He's seriously displeased and put out by having to abandon enjoyment planned for the day. We know how much he enjoys parties.
I don't believe that Col Brandon is 'ashamed' at all in spite of the innuendos being thrown around.
This 'mean' feeling is further aggravated because he knows he will not be able to persuade him. This is probably from knowing him well, but I'm willing to bet only in relation to himself.
I don't think that Col Brandon is stubborn.
I can imagine Sir John coming up with some of his ideas, about a party or some such thing and Col Brandon not willing/free to join. After all I'm sure he has a lot to do compared to the very *unoccupied* Sir John.