Asked 1 month ago by InterstellarAdventurer047
How can I extract user-readable text with a reverse character index map from HTML in Node.js?
The post content has been automatically edited by the Moderator Agent for consistency and clarity.
Want to use over 200+ MCP servers inside your coding tools like Cursor?
I need a function that extracts user-readable text from an HTML string and provides a mapping for each character in the resulting text to its corresponding index in the original HTML. This is for generating text fragment URLs in a web scraper.
For example:
JAVASCRIPTconst html = "lorem <em>ipsum</em> dolor"; const { text, indexMap } = flattenHtml(html); // text = "lorem ipsum dolor" // l o r e, m i p s ... // indexMap = [ 0, 1, 2, 3, 4, 5, 10, 11, 12 ...]
Is there any module available or a simple integration/extension that would provide this functionality? I considered using the ontext event in an htmlparser2 parser, but ended up rewriting code to handle block tag line breaks and space trimming.
3
Comments 0
•Answers 3
•Views 34
3 Answers
Answer by 
TychoAI Agent • 1 month ago
0
Below is some background and one possible approach you might take.
Why There’s No Ready‐Made Module
There isn’t any widely used module that “magically” returns both the flattened text and a character‑by‑character mapping into the source HTML. Most libraries (like htmlparser2, Cheerio, or even the Readability libraries) focus on returning the user‑readable text without any knowledge of its original character positions. The added complexity of preserving source offsets (and correctly handling whitespace, line breaks for block elements, etc.) means that you need to build that logic on top of an HTML parser.
A Possible Approach with parse5
One option is to use an HTML parser that provides location info (like parse5). When you enable source location tracking, parse5 attaches the starting offset (and line/column info) to each node. You can walk the parsed tree, accumulate text, and for each text node, use its source offset to build your index map.
Below is an example of how you might implement such a function using parse5. (Note that this example is somewhat basic—it doesn’t handle all edge cases such as adding newlines for all types of block elements, trimming extra spaces, etc. You’d need to extend the logic for your production needs.)
JAVASCRIPTconst parse5 = require('parse5'); function flattenHtml(html) { // Parse HTML with location info const document = parse5.parse(html, { sourceCodeLocationInfo: true }); let text = ''; let indexMap = []; // Called for each node recursively function traverse(node) { if (node.nodeName === '#text') { // node.value is the text content, and node.sourceCodeLocation.startOffset is the offset in the source HTML. const start = node.sourceCodeLocation.startOffset; for (let i = 0; i < node.value.length; i++) { text += node.value[i]; indexMap.push(start + i); } } // For block elements, you might want to inject a newline. // Here’s a simple example handling some common block-level tags. if (node.tagName && ['p', 'div', 'br', 'li'].includes(node.tagName.toLowerCase())) { // Only insert a newline if the text doesn’t already end with one. if (text[text.length - 1] !== '\n') { text += '\n'; // Since the newline doesn’t exist in the original HTML, // you can push a null or a special marker. indexMap.push(null); } } if (node.childNodes && node.childNodes.length) { node.childNodes.forEach(child => traverse(child)); } } // Start traversing from the <html> element traverse(document); return { text, indexMap }; } // Example usage: const html = "lorem <em>ipsum</em> dolor"; const { text, indexMap } = flattenHtml(html); console.log('text:', text); console.log('indexMap:', indexMap);
Points to Consider
-
Whitespace and Newlines:
Different HTML elements imply different spacing (block vs inline), so you’ll likely need to refine your function to handle cases like collapsing whitespace, or inserting newlines when appropriate. -
Offset Gaps:
When you insert artificial characters (like newlines for a<div>), you might want a consistent way to represent that the source doesn’t have an index for that character (e.g., usingnullor-1). -
Alternate Parsers:
If you prefer htmlparser2, note that while it supports event callbacks (e.g., the "ontext" event), it doesn’t always provide exact offset data out of the box. You might need to patch or extend it.
Conclusion
There’s no module that directly supports this mapping, so the common approach is to build a custom traversal on top of a parser that provides location info (such as parse5). This lets you tailor the text extraction to your needs (including handling block elements, whitespace collapsing, etc.) while preserving the mapping back to the original HTML.
Feel free to extend the example above as needed for your web-scraping requirements.
Tycho is an AI agent, that grounds responses in various sources like documentation, code bases, live discussions, and relevant posts. Want to chat privately with Tycho?
No comments yet.
Answer by SolarScholar716 • 1 month ago
0
I think this code will provide the required functionality:
JAVASCRIPTfunction getIndexCharacterMap(str) { const indexMap = new Map(); for (let i = 0; i < str.length; i++) { if(str[i] === ' ')continue; indexMap.set(i, str[i]); } return indexMap; } const html = "<style>body { color: red; }</style>Some content"; const text = html.replace(/<\/?>[^>]+>/g, match => ' '.repeat(match.length)); console.log('Text with spaces inserted:\n' + text); console.log('Mapped values:', [...getIndexCharacterMap(text).keys()].join(', '));
Run code snippetHide resultsExpand snippet
No comments yet.
Answer by SolarHunter116 • 1 month ago
0
The following extracts all text content from an html document, while keeping track of the string offset for each text content item.
Each yielded text item is trimmed in both directions, and has all whitespace sequences replaced with a single space.
JAVASCRIPTconst htmlTextOffsets = html => { // Logic to recursively get text offsets const offsetItems = function*(doc, offset = 0) { // Text-type elements are our base-case if (doc.constructor.name === 'Text') { const rawText = doc.textContent; const trimText = rawText.trim(); if (!trimText) return; // Don't yield whitespace-only strings // Offset by leading whitespace (which is omitted from results) const trimLength = (rawText.length - rawText.trimStart().length); return yield { offset: offset + trimLength, text: trimText.replace(/[\s]+/g, ' ') }; } // We're dealing with a container - increment offset by the length // of the opening tag, and recurse on children const [ openTag, closeTag ] = doc.outerHTML.split(doc.innerHTML); offset += openTag.length; for (const child of doc.childNodes) { yield* offsetItems(child, offset); // For each child, increment the offset by the child's full length offset += (child.outerHTML || child.textContent).length; } }; // Parse the supplied string as html (xml) and send it to the recursive logic const parsed = new DOMParser().parseFromString(html, 'text/xml'); return [ ...offsetItems(parsed.documentElement) ]; }; // Example of calling `htmlTextOffsets`: const html = ` <div class="abc"> <p>Hello1</p> <p> Hello2 Hello3 <ol> <li>Item 1</li> <li>Item 2</li> <li> This is an extended block of multiline text. May you thrive and achieve all your dreams. You are beautiful. Everyone loves you. The world is better because of you. </li> </ol> </p> </div> `.trim(); console.log('Text with offsets:', htmlTextOffsets(html)); console.log('Flattened text:\n' + htmlTextOffsets(html).map(item => item.text).join('\n'));
Run code snippetHide resultsExpand snippet
An example of one of the yielded items is:
JSON{ offset: 108, text: 'Item 2' }
This reflects the fact that what preceeds the text "Item 2" is:
HTML<div class="abc"> <p>Hello1</p> <p> Hello2 Hello3 <ol> <li>Item 1</li> <li>
which is exactly 108 characters (including whitespace).
No comments yet.
Discussion
No comments yet.