You are missing a feature rather than a setting, and it removes the trade-off entirely: direct monitoring.
Almost every audio interface can send the incoming signal straight to your headphones in hardware, before it ever reaches the computer. There is no round trip, so there is no delay - it is effectively instant regardless of your buffer size.
It is usually a knob labelled something like mix or blend, or a button labelled direct or monitor. The knob blends between what is coming into the interface and what is coming back from the computer.
How to use it:
Turn the blend toward input so you hear yourself directly.
Mute the track in the software or turn off its input monitoring, or you will hear yourself twice: once instantly and once late, which is worse than either.
Raise the buffer as high as you like. Once you are not monitoring through the computer, buffer size stops mattering for playing. This is the part people find counterintuitive.
The one thing you lose is hearing effects on your voice while you sing, since the direct signal is dry. For most people that is a fair trade, and there are ways around it if reverb helps your performance.