The argument is not that infinity number of balls were removed, and it is not about 'infinity - infinity' tricks.
The argument is that each particular ball was removed at specific moment. You can provide a formula that takes the number of the ball and gives the exact time it is removed. Each particular ball is inserted once and then removed once before noon, thus no ball can remain inside the box at 12:00.