Even after this short acquaintance, Elinor is able to perceive that he's not as truly ill-natured as he tries to act, and that the source of his annoyance and resultant bad temper is his wife.

He would not need to be embarrassed by gaffes of various kinds if he had married Lady Middleton, nor would she annoy him with silly chatter. She is also good-looking, which seems to have been his main attraction to Charlotte, and would have had as much of a dowry as Charlotte had. She would, indeed, be a hostess for him. She is a hostess for Sir John's gatherings now, all the time. Why would she refuse to do the same if she were married to Mr. Palmer instead?

And really, if there are 8 sitting down at the table and there is a cook and other servants, could it make any difference to sit another couple down with them? None whatsoever. It's not like Lady Middleton (or Charlotte, for that matter) would have to be the one preparing their meal, or that they were lacking in chairs for them to sit on or plates on which to put their food.

The only reason she refused to have them is because she is 'standing on ceremony' that they 'owe' her a dinner invitation--nothing to do with the size of the party or anything like it. Keeping score in this way is really not to the advantage of a sociable man like Sir John.