It is introspective bits from Young Jolyon's point of view that I love most.

And I think you pretty much nailed the reasons, though two and three were probably more important than one.I think that just being witnessed talking to his son and grandchildren wether they met by accident or design, would be enough to get the rumours to fly.

Also, from the narrative point of view, the location both allows them to be seen by James and his 'female folk', and makes a parallel between the situation of the animals in the cages and Irene in hers.