All these are good reasons for Willoughby to jealous of the Colonel. But, I think that he also expresses dislike because of Marianne's attitude. We are told in Chapter 10 that W conforms his opinions to hers.

It was only necessary to mention any favourite amusement to engage her to talk. She could not be silent when such points were introduced, and she had neither shyness nor reserve in their discussion. They speedily discovered that their enjoyment of dancing and music was mutual, and that it arose from a general conformity of judgment in all that related to either. Encouraged by this to a further examination of his opinions, she proceeded to question him on the subject of books; her favourite authors were brought forward and dwelt upon with so rapturous a delight, that any young man of five-and-twenty must have been insensible indeed, not to become an immediate convert to the excellence of such works, however disregarded before. Their taste was strikingly alike. The same books, the same passages were idolized by each -- or, if any difference appeared, any objection arose, it lasted no longer than till the force of her arguments and the brightness of her eyes could be displayed. He acquiesced in all her decisions, caught all her enthusiasm, and long before his visit concluded, they conversed with the familiarity of a long-established acquaintance.

Hearing that Marianne holds disdain for Colonel Brandon's age, lack of liveliness, etc., encourages his natural jealousy.