Agreed.
Imagine instead that the dad traveled back to the Roman Empire and died back there without changing the world in any way.
Consider the attached image which I just generated from an astoundingly powerful computer that observes space-time anomalies.
The blue line represents the timeline of a man who does not time travel. His life appears linear from this perspective. The red line represents the life of the father. Although it doesn't appear linear, it actually is. Time is a jumbled mess of a thing. I tried to make the image as clear as possible.
If the man traveled to the year 325 (arbitrary), everything he has done so far is in his past, even if it is in the future of the rest of earth. He didn't begin existing in the year 325. Even the direction of the time travel doesn't matter. If he went into the future 5000 years, he would still have existed linearly (is that a word) relative to his own observation.
I love Doctor Who, but I hate Matt Smith. A good example of this linearity is observing the timelines of the eleventh doctor and River Song. I wouldn't use the media to build any theories, but this is a great example of how linearity works.