I can't see his bitterness taking on that kind of vindictive spin (being glad she isn't having fun). I don't see that in him at all. He seemed truly shocked at her physical alterations, and my sense is that he felt more of a haunting sadness at her decline.

As angry as he made me at the Musgroves', I never have the sense that he's trying to deliberatly, vindictively rub her face in her losses (Ha! See what a brilliant success I am, and what a dried-up old maid you are?) His misstep in speaking his mind when asked of Anne's appearance seems to me to truly come from a shocked, saddened heart.

However, I do agree that he is subconsciously relieved/pleased that she isn't courting...even if it does surprise him.