"There is no persuading you to change your mind, Brandon. I know of old....once you are determined on anything."ch13

Sir John's words seem to me to suggest that they have been friends for a long time, and that Colonel Brandon's character is well known to him. We witness here that Brandon can and does say no to the entreaties of Sir John and Mrs Jennings, and he has probably had to do so on numerous occasions. There also seems to be a hint that the friendship is a strong one, for despite the Colonel's determined and sombre nature, and despite Sir John's constantly lively exuberance they have remained friends. We are left wondering how the friendship was formed in the first place, it surely must be based on more than being neighbours.
I think Sir John lets his disappointment about missing a days entertainment affect his behaviour towards his friend. Mrs Jennings is perhaps more reprehensible as she seems to care only for gossip.
Willoughby's assessment is perhaps the cruelest, "He was afraid of catching cold", and betrays some real dislike.