Retrieval is pgvector, top 40 candidates, then a cross-encoder reranking down to 8, running on CPU in the same container. Answer quality went up noticeably — people stopped complaining about the wrong section being cited.
p95 went from 2,110ms to 9,400ms. Users are now describing it as broken. Is 6 seconds for a rerank a normal cost, and what is the usual way out of this without giving the quality back?
40 pairs through a cross-encoder on CPU, one at a time, is genuinely a several-second operation. Nothing is broken, you just bought quality with latency at a bad exchange rate.
In order of return on effort:
The first two together typically land you near 400ms. That is the same pipeline with the same model, just used properly.
Batching plus top-12 got rerank from 6.1s to 380ms. p95 is 3.4s now and I have not touched the model. The loop was doing one pair per call because that is how the example in the README was written.
The truncation trick is underrated. We rerank on chunk heading plus first two sentences and our eval set could not tell the difference from full-chunk reranking — same recall at 5, a third of the time.
Do something about perceived latency in parallel with the real fix. If nothing renders for six seconds and then an answer streams, people read that as broken regardless of how good the answer is. Streaming a "searching 40 documents" line and then the retrieved source titles as they resolve changes the complaint more than two seconds of genuine optimisation will.
Shipped the source titles streaming in ahead of the answer. Support messages about it being stuck dropped immediately, before any of the latency work landed.
Also worth interrogating whether the reranker is earning 6 seconds. Build a 60 question eval set from real queries, measure recall at 5 with and without it, and look at where it helps.
Half the reranker deployments I have reviewed improved a metric nobody was measuring and cost seconds that everybody noticed.
We ran exactly that experiment. Reranker helped on 8 of 60 questions, hurt on 3, no change on the rest. We kept it but only for one query class where it clearly mattered, and skipped it everywhere else. Routing around it was worth more than optimising it.
Move the reranker into a background job and cache the results, then the user path never waits on it.
Cache keyed on what? The query, which is different every time in a chat product. That works for a fixed FAQ or a set of canned reports and does nothing for open-ended questions, which is the case being described.
@vesting_cliff · last wk. · 2 replies
Before you tune anything, put spans around every stage and look at p95 per stage rather than the total. Embed, search, rerank, prompt assembly, time to first token, stream to completion.
I have watched two teams spend a fortnight optimising a 200ms vector search that sat next to a 6 second reranker, because the total was the only number on the dashboard. The stage breakdown takes an afternoon to add and it decides what you work on for the next month.
Reply
Report
@harlan_voss · last wk.
Added spans and the shape was uglier than expected — rerank is 65% of p95 and prompt assembly is a surprising 300ms because of a synchronous token count call I forgot was there.
Reply
Report