LOL!

I agree that it probably began with some inkling that either the Barton Park people were encouraging or promoting a Brandon/Marianne match or that Marianne had hinted at this to him.

What is very telling, to me, is not only that Willoughby is criticizing the colonel, but the kinds of things for which he is being criticized:

• "Whom every body speaks well of, and nobody cares about; whom all are delighted to see, and nobody remembers to talk to." He is critcized because he has everyone's respect and good opinion, and because he is not the type of person to push his way into a discussion?
• "He is highly esteemed by all the family at the Park, and I never see him myself without taking pains to converse with him."

  "That he is patronized by you [Elinor] ," replied Willoughby, "is certainly in his favour; but as for the esteem of the others, it is a reproach in itself. Who would submit to the indignity of being approved by such women as Lady Middleton and Mrs. Jennings...?" He's criticizing Colonel Brandon because he has the respect and approval of two ladies who are at the centre of all the social life in the area?

• [Elinor]"He has seen a great deal of the world; has been abroad; has read, and has a thinking mind. I have found him capable of giving me much information on various subjects, and he has always answered my inquiries with the readiness of good-breeding and good nature."

  "That is to say," cried Marianne contemptuously, "he has told you that in the East Indies the climate is hot, and the mosquitoes are troublesome."

  "He would have told me so, I doubt not, had I made any such inquiries; but they happened to be points on which I had been previously informed."

  "Perhaps," said Willoughby, "his observations may have extended to the existence of nabobs, gold mohrs, and palanquins." He is being critcized for being worldly, well-travelled, and for having something to say for himself beyond the subjects of music, dancing and poetry?

• [Willoughby]"I consider him, on the contrary, as a very respectable man, who has every body's good word and nobody's notice; who has more money than he can spend, more time than he knows how to employ, and two new coats every year...I have three unanswerable reasons for disliking Colonel Brandon: he has threatened me with rain when I wanted it to be fine; he has found fault with the hanging of my curricle, and I cannot persuade him to buy my brown mare."
  He is being criticized for being respectable, unobtrusive, wealthy and not a spendthrift?

Taken as a whole, this whole conversation is more than unfair to Colonel Brandon. Since it does not appear that the colonel has done anything to Marianne and Willoughby or said anything to them beyond the most commonplace social interaction, it really makes you stop to think why Willoughby would be so eager to criticize a man for the faults of being wealthy, polite, respectable, gentlemanly, intelligent, well-travelled, worldly, and not inclined to spend money on things he doesn't need?

As for Marianne, to me it seems that she is doing one of two things here: she is either trying to let Willoughby know in no uncertain terms that whatever anyone else may think about matching her up with Brandon, she has no interest in him at all, and also trying to ingratiate herself to Willoughby or win his further approval by agreeing with him.

This comment of hers is particularly unfair, though: "He has neither genius, taste, nor spirit. That his understanding has no brilliancy, his feelings no ardour, and his voice no expression."

Beyond the quality of the tone of his voice, how exactly did she determine the rest of this to be true? Has she had any kind of conversation or interaction with him that would have allowed him to display any of the qualities she listed?