...says that Mr. Palmer was certainly a Tory, which would have made Willoughby a 'Foxite Whig' if he was in the opposition.

The note in that edition also states that rivalries between the parties at that time were such that people from opposing parties would refuse to see each other socially.

I don't know much at all about the politics in this era. However, I was reading two books about other people who lived in that time (late 1700s)--The Duchess of Devonshire and also Mary Robinson (George IV's mistress Perdita). Both of these ladies moved in the same circle as Charles James Fox, whom I assume is the same person as 'Foxite' indicates. I recall in the Duchess of Devonshire book there was a lot in there about her interest in politics and how the Devonshire Circle--a fashionable crowd who were into gambling and other vices--were all very much on Fox's side politically.

This little throwaway line about Willoughby being in the opposition might actually reveal quite a bit about his character and how he spends time in London.