What is the right way to get a web page into a model's context?

The pipeline has four stages, and stripping tags is only the second one.

1. Fetch what the reader sees. A large share of the web renders its content with scripts. A plain fetch gets you a shell. So either the page is server-rendered, or you need a real browser. This is where most home-made pipelines silently return nothing useful.

2. Extract the main content. This is the hard part and it has a name — boilerplate removal. The established approach is the readability algorithm used by browser reading modes, and there are good open implementations. Do not write your own heuristics; this problem has been worked on for fifteen years.

3. Convert to markdown, keeping the structure that carries meaning — headings, lists, tables, code blocks, link targets. Structure survives chunking; a wall of prose does not.

4. Chunk on structure, not on character count. Split at headings, keep tables and code blocks whole, and carry the heading path into each chunk so a fragment still knows what section it came from.

Stage 4 is where most retrieval quality is won or lost, and it is the stage everybody skips.

One thing worth deciding early, because it changes the whole design: are you retrieving or are you reading?

Reading means one page, now, into context. Then you want the full cleaned markdown and chunking is irrelevant. Cheap and easy.

Retrieving means a corpus you search over later. Then chunking, embedding, storage and the freshness question all appear, and it is a much bigger commitment.

A lot of teams build the second when they needed the first. If your agent looks at a page because a user mentioned it, you do not need a vector store — you need a fetch and a clean-up. That distinction has saved me more work than any tool choice.

What justifies paying somebody: the long tail of sites that fight you.

On a well-behaved page, the open-source stack above takes an afternoon and works. The trouble is that at any scale you meet:

• Pages needing JavaScript, which means running browsers, which means infrastructure.
• Bot protection, rate limits and geo-restrictions.
• Paywalls and consent walls that produce a page that parses fine and contains nothing.
• PDFs, which are a separate project entirely.
• Sites that change and quietly break your extraction.

So the honest framing is that you are not buying an HTML-to-markdown converter — that is a library. You are buying somebody else operating a browser fleet and absorbing the breakage.

If you fetch a handful of known sites, build it. If you fetch arbitrary URLs a user supplies, buy it or expect to run infrastructure.

Two practical points that get skipped and later become incidents.

Fetching a URL a user supplied is a server-side request forgery risk. Somebody will pass an internal address or a cloud metadata endpoint. Allow-list schemes, block private ranges, resolve before you connect and check what you resolved to.

Page content is untrusted input that reaches a model. Text on a page can be written to instruct your agent, and it will. Keep fetched content clearly separated from your instructions and never let it grant the agent anything it did not already have.