unrelated paragraphs score 0.86 cosine with text-embedding-3-small, is my index broken embeddings

Nothing is broken. Cosine similarity from these models lives in a narrow band — anything in the same language and roughly the same domain lands somewhere around 0.75 to 0.90, because a large part of the vector encodes "this is English business prose" and not the topic.

The absolute number is meaningless. Only the ordering within a single query means anything, and even that is weak. So:

• Never threshold on a global constant. It will either fire on everything or nothing, and which one changes when you switch models.
• If you need an abstain signal, use the shape of the results — the gap between rank 1 and rank 10 — or calibrate against a labelled set and recompute when anything changes.
• Better: let a reranker produce the score you threshold on. Its scores actually spread out.

Also make sure you know which direction you are measuring. In pgvector <=> is cosine distance, so smaller is better, and <#> is negative inner product. Half the "my similarity scores are backwards" posts are someone ordering by the wrong operator and getting a beautifully sorted list of the least relevant documents.

One thing that can genuinely narrow the band further: if you shortened the embedding dimensions to save space, you compressed away some of the separation along with the storage. Fine trade for speed and disk, but do not then be surprised that everything looks similar.

Practical version of all of the above: retrieve 50 by vector, rerank, threshold on the reranker score. You get a number that behaves like you expected cosine to behave.

Cheap diagnostic: take 20 pairs you know are related and 20 you know are not, plot the two score distributions. If they overlap heavily on your own content, the model is not the problem and neither is your index — the content is too uniform, and you need keyword search in the mix.