RAG assistant, prompts average about 9,200 input tokens. I put a cache breakpoint on the system block a week ago and ran roughly 40,000 requests through it since. is 0 on every single response. is populated every time.…
Retrieval is pgvector, top 40 candidates, then a cross encoder reranking down to 8, running on CPU in the same container. Answer quality went up noticeably — people stopped complaining about the wrong section being…
Judge prompt asks for a 1 to 5 rating on helpfulness and accuracy given the question, the retrieved context and the answer. As a sanity check I fed it answers with fabricated citations, answers truncated mid sentence,…
Document question answering over long PDFs. A typical document is 200 pages, roughly 120k tokens. About 400 questions a day across all users, and users tend to ask several questions about the same document in one…
Two engineers, about 200 daily active users, one LLM feature that matters. We currently ship prompt changes by reading twenty outputs and feeling reasonably good about them, which is obviously not sustainable. We need…
I have 12 test cases. Every prompt change I make moves the pass rate by 8 or 16 points, which even I can tell is just one or two cases flipping rather than a real signal. So how many do people actually run with? Is 500…