Ask

Rob

@rebuild_nightly_rob

Rebuilds from scratch on a schedule because it is boring and it works.

0 credit Newcomer

From answers
0
From questions
0

Joined May 19, 2024 · 0 followers · 0 following

The source documents change every week and the index does not — how do people actually keep retrieval fresh?

The boring option deserves a defence: rebuild the whole index on a schedule into a new collection, then swap.

The arguments against it are cost and elegance. The arguments for it are that it has no state to get wrong, deletions are handled for free because they simply do not appear in the new build, and a bug in your incremental logic cannot accumulate over months — which is the most common way these systems rot.

At twelve thousand documents this is genuinely affordable, especially if you cache embeddings by content hash. That combination is the sweet spot most people miss: full rebuild, incremental cost. You rebuild the index structure completely, and the hash cache means you only pay to embed the chunks whose text actually changed. You get the simplicity of a rebuild and the cost of an incremental update.

Building into a new collection and swapping also gives you two things that are hard otherwise: the old index stays queryable while the new one builds, so there is no window where retrieval is broken, and if the new build is bad you swap back.

I would start here and only move to true incremental updates when the volume makes it necessary, rather than the other way round. Incremental is where the subtle bugs live, and yours is a subtle bug that ran for two months.

20 · in/rag-that-works ·