I think that at first, surely, at least prior to his enquiry at the dance, he must be enjoying the fact that he is considered quite the catch and as you say rubbing it in. Early in the dinner party he is definitely a bit hostile, pointing his comments specifically to Anne.

And the fact that she is so passive, timid and seemingly shrinking from all contact from him must reinforce his opinion that he well shot of her. Then during the dance, I think he must feel a bit of pity for her thinking that she was once so lively and loved to dance herself.

But to someone of Wentworth's intelligence, his ego can only be stroked so long before he must get tired of it. At some point I think he begins to try to act more politely and more of a friend to Anne, like Darcy perhaps in his first half hour of meeting with Elizabeth at Pemberley, in order to show her that he had no hard feelings.

Still during the walk, I cannot deny that Louisa seems to score real points with him. She shows her taste by understanding how much The Admiral and Mrs. Croft are in love and by showing him that Anne is preferred by the family to Mary. She shows herself to be resolute. And she is an eager audience for his half-serious (I've aways thought, wholly ridiculous) nut speech.