In Chapter 12, Elinor & Marianne are discussing Willoughby's giving Marianne a horse. When Elinor raises concerns about the propriety of accepting this gift, Marianne answers (in part) with the following:
I should hold myself guilty of greater impropriety in accepting a horse from my brother than from Willoughby. Of John I know very little, though we have lived together for years; but of Willoughby my judgment has long been formed. [emphasis mine]
Assuming that Marianne is not exaggerating, then, it appears that John Dashwood did spend a good bit of time w/ his father's second family. This, imo, makes his treatment of them even worse. He must have always been cold-hearted, because I cannot imagine Mrs Dashwood not being very kind & loving toward him.