Although we have seen some other clues, it appears the rumor mill started up in full force based on Swithin's dream, followed by Irene's seeming confirmation "I don't care if I never get home!" Are we supposed to assume that Swithin's dream mirrors what is going on between Irene and Bosinney? Or is the reality something different, and the gossip builds what happens into something dreadfully false? Hmmmm...

On a side note:
This chapter, with the relating of the "facts" from Forsyte to Forsyte, reminds me so much of Norman Rockwell's painting called The Gossips. I've pasted a link to the painting below if anyone wants to take a look. You may have to go back to the main page (http://www.normanrockwellvt.com) and search for it.