The second group are people who did know it was you and went to the bank anyway. That is a communication failure, and it is fixable.
The reasons people skip the seller, roughly in order:
They could not find how to contact you, or they did and got no reply for two days. A dispute is what people do when they think emailing will not work.
They do not believe the refund will happen. Enough companies make it hard that the bank feels like the reliable route. Your policy has to be visible before they need it, not discoverable in a support email.
They forgot they subscribed. A charge months later on something they stopped using looks like a mistake.
So, concretely:
Put the policy where people look, the checkout page, the receipt, and the footer. Not only in a terms page.
Make the contact route obvious in the receipt. The receipt is the document people find when they are looking at the charge.
Warn before recurring charges. A short email a few days before a renewal is the single most effective anti-dispute measure for subscriptions, and people rarely resent it.
Reply fast. An acknowledgement within an hour, even if the refund takes a day, prevents nearly all of the escalations in this group.