Expanding on deletion, because it has a failure mode that survives even a correct implementation.
If your index and your source can ever disagree, you will eventually retrieve something that no longer exists — a document that was deleted while your job was halfway through, a rename that looked like a delete plus an add, a chunk that failed to delete because of an error nobody looked at. And a stale chunk is worse than a missing one, because the model will use it confidently.
Two defences:
Filter at query time, not only at index time. Store the document id on the chunk and check it against the current set of live documents before you put anything in the prompt. It is cheap and it means the worst case is a missing answer rather than a wrong one.
Make the model cite, and make the citation resolvable. If every claim carries a document id and a section, a stale citation becomes visible — someone clicks it and finds nothing, or finds text that does not match. Without citations, your exact bug is undetectable by anyone who does not already know the answer, which is the situation you are in now.
One more thing worth doing this week regardless of the redesign: run a reconciliation once. Count documents in the source, count distinct document ids in the index, and list the ones in the index that no longer exist. That number tells you how bad the current drift is, and it is usually much larger than people expect.