Mini PC, and it is not close once you said transcoding.
A used office machine with an 8th gen or newer Intel chip, or a new N100 box, gives you QuickSync. That handles four or five simultaneous 1080p transcodes without the fan even noticing. The Pi 5 has no usable hardware encoder for this job, so every transcode is software, and one 1080p software transcode will eat the whole board and still stutter.
Money wise: a small office desktop with 8GB and a modest SSD goes for 90 to 130 depending on how patient you are, which leaves room for a drive. A Pi 5 plus NVMe hat plus SSD plus cooler is roughly the same money for less machine.
The Pi wins if and only if everything direct plays forever, which stops being true the first time someone watches from a phone on a train.