Internal knowledge base, 12k documents, pure vector search on pgvector. Answers are right maybe two thirds of the time and the failures are usually "it retrieved something adjacent". I have budget for one of these this month and roughly $40/month of ongoing spend. Which one moves the needle more?
You already have both engines. Postgres does full text and pgvector does the rest, so hybrid is a query, not a service:
with kw as (
select id, row_number() over (order by ts_rank_cd(tsv, q) desc) r
from chunk, plainto_tsquery('english', $1) q where tsv @@ q limit 50
), vec as (
select id, row_number() over (order by embedding <=> $2) r from chunk limit 50
)
select id, sum(1.0/(60+r)) s from (select * from kw union all select * from vec) t
group by id order by s desc limit 50;
That is reciprocal rank fusion with k=60. No second datastore, no extra bill, and it composes with the reranker afterwards.
Add the tsvector as a generated column rather than maintaining it in application code, otherwise you get the classic bug where documents edited through one code path are searchable and documents edited through another are not. Index it GIN and forget about it.
If you would rather not add a vendor, a small cross-encoder runs locally. Scoring 50 pairs on CPU is a couple of hundred milliseconds, which is acceptable for an internal tool where people expect a search box to think for a moment. Hosted is faster and less to maintain, so it comes down to whether you already have a box.
Whatever you pick, build the 30-question set first. Otherwise you will ship both, feel like it improved, and have no idea which one to keep when one of them starts costing money or latency.
Honestly the best use of that $40 is whatever makes you run the eval every time you change a prompt. The retrieval spend is noise at 12k documents.
@frosting_fixer · 4mo ago · 3 replies
Reranking is the bigger single win, but you cannot choose between them without one measurement, and it takes an hour.
Take 30 questions where you know which document should answer them. Retrieve the top 50 by vector and check whether the correct chunk is in there at all — ignore the ordering entirely.
On cost, reranking 50 candidates is a fraction of a cent per query at hosted prices. At your volume $40 is not the constraint, your time is.
Reply
Report
@kitchen_table_talk · 4mo ago
Recall@50 was 0.71. Going through the misses, over half of them were queries containing a part number. Added a tsvector column and fused with RRF and it is 0.94. Then added the reranker and the top-5 got noticeably better on top of that.
Reply
Report
@slowmiles_kat · 4mo ago
Part numbers, error codes and people's names. The three things embeddings are worst at and the three things people actually type into an internal search box.
Reply
Report