Worth adding the contrast that makes the point stick: this reasoning would apply to a crop grown from true seed.
Save seed from your smallest tomatoes each year and you really are selecting, because those seeds carry a genetic sample from that specific plant. Do it for a decade and you will have smaller tomatoes.
That is why the same intuition is correct for most of the garden and wrong for potatoes, garlic and anything else propagated vegetatively. The question is always: am I planting a seed or a piece of the parent?