The genuinely serious problem with saving your own seed potatoes is not size at all — it is disease.
Potatoes accumulate viruses through the season, carried mainly by aphids, and those viruses pass into the tubers. The plant may look fine and still be carrying several. Replant those tubers and the next generation starts already infected, and the crop declines noticeably over a few years.
This is why certified seed potatoes exist and why they are grown in specific regions — often cooler and windier places where aphid pressure is low — and inspected. It is not a marketing arrangement; it is the actual reason the crop stays productive.
So for the fictional case, with no aphids and a closed system, saving tubers indefinitely is fine. In a garden, refreshing your seed stock every few years is the thing that matters, and it matters far more than which size you replant.