My impression is that Willoughby thinks all that wealth sadly wasted on a dull old fellow who doesn't know how to enjoy life. Whereas he himself would put it to excellent "use".
He has even tried to sell some horse he's no longer any use for himself and resents Brandon's refusal to spend money, especially since he could very well afford to do so.
Most annoyingly immature young brat actually. - Admittedly very handsome and charming though.