To understand it you need to know what the clutch and the synchromesh are each doing.
The clutch disconnects the engine from the gearbox input shaft, so that shaft is free to change speed.
The synchromesh is a small friction cone inside the gearbox that brings the gear you are selecting and the shaft to matching speeds before the teeth engage. It is what makes a modern gearbox shift smoothly instead of grinding.
When you shift without the clutch, the engine is still driving the input shaft. The synchromesh now has to overcome the engine as well as match speeds, which is far more than it was designed for. It can often do it — which is why the shift sometimes goes through — and it wears rapidly while doing so.
Synchromesh wear is gradual and invisible until the day a gear starts crunching, at which point the repair is a gearbox rebuild.