I think you're onto something with the idea that it's more than Brandon's comparative wealth that affects Willoughby's attitude. It's certainly enough to inspire envy, true, but Willoughby is, for all his external beauty, a man with a very ugly personality that all his charm is hard put to disguise. We can see it already with the bad effect he's having on Marianne, for one thing. There's some commentary elsewhere on the board about how people are perhaps less scandalized by Brandon's supposedly having a "natural daughter" (even though they are mistaken in this) because he makes the effort to take care of her, showing himself a responsible and decent man with a good heart. This, of course, will be another mark against Willoughby later on.

Willoughby's snarking at Brandon always reminds me of Othello, when Iago talks about the reasons he dislikes Cassio and means to do away with him. Among what some would consider very down to earth reasons, Iago has this interesting comment:

"He hath a daily beauty in his life
That makes me ugly . . ."