On why infinity specifically fails, your intuition is worth taking seriously because it is half right, and the half that is wrong is easy to see.
You divided ten by smaller and smaller positive numbers and the result grew without bound. True.
Now do it from the other side. Divide ten by numbers approaching zero from below, minus one, minus a tenth, minus a hundredth. The results are minus ten, minus a hundred, minus a thousand. It grows without bound in the negative direction.
So approaching zero from one side heads to positive infinity and from the other to negative infinity. There is no single value it is heading towards, which means no consistent answer exists even in the limiting sense.
That is the specific reason infinity is rejected: not that it is too big, but that the two sides disagree.
This is also why the distinction between undefined and infinite matters. Something like one divided by zero squared does head to positive infinity from both sides, and mathematicians are comfortable saying the limit is infinite there - while still saying the expression itself is undefined, because infinity is not a number you can do arithmetic with.