Anne doesn't talk to him either - except as required for decency. In fact, even though the parents of the hurt child came to dinner, she stayed at the house. That could easily be interpreted by him as a desire on her part for them not to interact. He hadn't met Mary before, he wouldn't have any understanding of her character, so I don't think he would have had reason to consider it anything but a personal slight.