Several points occur to me, in reading your answer (thank you for it, by the way -- very interesting!).

If the groom were Mrs. Smith's, Willoughby might be afraid that his loyalty was to HER; or would he be counting on his future inheritance of the property making the servants know on which side their bread was going to be buttered?

If caring for one's servants is too be similar to the care one has for one's children, I feel pity for Willoughby's household employees.