I think I misunderstood the sequence. I thought that after the girls left, they left!

"...their visitor had bowed and was gone, the Miss Musgroves were gone too, suddenly resolving to walk to the end of the village with the sportsmen: the room was cleared, and Anne might finish her breakfast as she could."

I missed the line that you pointed out that said that after they sent the guys off, they "returned and finished their visit at the Cottage."

What does that mean? I don't get it. Wasn't Anne in the same Cottage? Wasn't she amicable with the Musgrove sisters? Why wasn't she there? Did Louisa and Henrietta meet with Mary only? Was it outside in the garden, perhaps? Or had Anne gone out walking, taking in the autumnal air?