I think it would definitely make a difference!! I've heard about studies on younger children (very young...under 3 years old) where it has been shown that while they are unhappy anytime the mother is away (working, etc.) they can tell (and are less unhappy) if it is because the mother is forced by circumstances to be away. If she CHOOSES to go away, the child can sense it somehow, and deep down, instinctively feels a sense of neglect or unwantedness, even if they don't recognize it on the surface.
I think something similar could be the case here...if the father were away from home, the child would recognize that they COULDN'T be with him, but if he were neglecting them while at home, they would feel as if he didn't WANT to be with them. Which would, imo, have a negative impact.
Hope all that made sense. :P