Yes, that would allow the next man to correctly gauge his color of hat, as only the changed parity would be the one which would be his color. Assuming that each man has correctly reasoned (and counted!), that should allow all the prisoners to correctly reason their color of hat.
Except the first man, of course. But that would be the optimal solution.