Roza

What justifies paying somebody: the long tail of sites that fight you.

On a well-behaved page, the open-source stack above takes an afternoon and works. The trouble is that at any scale you meet:

• Pages needing JavaScript, which means running browsers, which means infrastructure.
• Bot protection, rate limits and geo-restrictions.
• Paywalls and consent walls that produce a page that parses fine and contains nothing.
• PDFs, which are a separate project entirely.
• Sites that change and quietly break your extraction.

So the honest framing is that you are not buying an HTML-to-markdown converter — that is a library. You are buying somebody else operating a browser fleet and absorbing the breakage.

If you fetch a handful of known sites, build it. If you fetch arbitrary URLs a user supplies, buy it or expect to run infrastructure.

One thing to insist on when evaluating any of these: ask to see the failure rate on your own data, in both directions.

Vendors quote detection rates on public benchmarks. What you need is how often it blocks something legitimate in your domain, because that number decides whether your users can work.

Run it in shadow mode first — logging its verdicts without acting on them — for a couple of weeks. That gives you both numbers for free and it is the only honest way to choose a threshold.

The pipeline has four stages, and stripping tags is only the second one.

1. Fetch what the reader sees. A large share of the web renders its content with scripts. A plain fetch gets you a shell. So either the page is server-rendered, or you need a real browser. This is where most home-made pipelines silently return nothing useful.

2. Extract the main content. This is the hard part and it has a name — boilerplate removal. The established approach is the readability algorithm used by browser reading modes, and there are good open implementations. Do not write your own heuristics; this problem has been worked on for fifteen years.

3. Convert to markdown, keeping the structure that carries meaning — headings, lists, tables, code blocks, link targets. Structure survives chunking; a wall of prose does not.

4. Chunk on structure, not on character count. Split at headings, keep tables and code blocks whole, and carry the heading path into each chunk so a fragment still knows what section it came from.

Stage 4 is where most retrieval quality is won or lost, and it is the stage everybody skips.

On the bridging products specifically: the value is real and narrow.

What they can genuinely do is export from one product's account and import into another, and keep a canonical copy synchronised so a change lands in several places. That is fiddly work and worth paying for if you switch often.

What they cannot do is make the behaviour transfer. The same instructions produce noticeably different results across models and harnesses — an instruction that was necessary to stop one assistant doing something may be irrelevant or actively harmful with another. So a large accumulated instruction file is partly a record of one tool's quirks.

Which is an argument for periodically pruning it rather than carrying it forever. Half of mine turned out to be workarounds for a model that no longer exists.