I think that Old Jolyon missed his son all along, but due to society's standards, he had to shun him in order to protect June. Since June was so young when Young Jolyon left her mother (Frances?), Old Jolyon felt an obligation to look after June.

There is no doubt in my mind that Old Jolyon turns to Young Jolyon because he is lonely. But I think that his justification is partly due to the fact that June doesn't need his protection any longer. Now he can focus on his two other grandchildren who could use his support.