It is a stage and day four is the worst of it. Soy typically needs one to two weeks to cure and there is a well known slump in the first week where cold throw drops before it comes back. Lids on, somewhere stable, smell them again at day 10 and day 14.
One thing to check though: 82C is fine for adding fragrance but two minutes of stirring is on the short side. Stir gently for a full two minutes minimum, longer for thick oils, and scrape the sides. Poorly bound fragrance smells great early and then disappears.