Worth checking one thing before you rewrite anything: is the key you are writing to the key the component is reading from?
A surprising share of "the rollback did not work" is that the optimistic write went to one key and the component is subscribed to another, a list key versus a detail key, or a key with a filter object that is constructed inline and therefore not identical between renders. The write succeeds, nothing complains, and the component never sees it. Then the eventual refetch corrects the screen and it looks like a timing bug.
How to check quickly: open the devtools for your query library and watch the cache while you toggle. You will see whether the optimistic write is landing on the entry the component uses. Thirty seconds and it rules out the entire category.
The related trap is a key built from an object literal. If any part of it is created fresh each render, or contains something non-serialisable, you can end up with more cache entries than you think - and your optimistic write goes to one while the screen reads another.
If your toggle affects both a list and a detail view, you need to handle both, and that is the case where invalidating on settled saves you from writing two rollbacks that have to agree.