There are two small clues that the old gentleman saw very little of young Harry.

We read: "in occasional visits with his father and mother at Norland, had so far gained on the affections of his uncle, by such attractions as are by no means unusual in children of two or three years old"

First of all, the visits were only occasional, not frequent. Second of all, the mention of 2 or 3 years old makes me wonder if Harry was ever brought to Norland as an infant at all? A crying baby that cannot interact or talk would not gain the old gentleman's affection nearly so well as a cute toddler might.