Analogies are only suggestive, to guide thinking.
By achieving 1/5 of his objective we mean travel 1/5 of the distance from start to finish, not 1/5 of the remaining distance.
We can take away the determinism by saying the man had walked to the city a large number of times in the past and on average it took him 5 days to arrive.
If a day were 8 hours of walking, and on different days the man was well rested or not, or the weather was conducive or not to travel, then each time he made the trip his arrival time would be different from 40 hours although 40 hours was his average result.
Then it is valid to assert that, on average, a day's travel achieved 1/5 of the total distance. You get what I mean.
And then the equivalence of that statement with the statement that on average it takes him 1/(1/5) = 5 days is clear.
However, the first argument is a rigorous proof, not that the infinite series isn't, and is simpler.
Six trials, six equally likely results.
On average, one result each.