But, like you, I have a lot more sympathy with Wentworth at this point in the story than seems to be the norm. Let's assume that although we do not really hear about how he is feeling, he is not indifferent to the presence of the woman who broke his heart. Even if he thinks her power with him is gone I think we can see that it is still a difficult situation.

In terms of that early morning visit to the cottage, what I see is someone who can only assume that Anne is keen to avoid his company and who goes to some lengths to deal with what is potentially a difficult meeting in a tactful way. He cannot say to Charles 'look, I think it is better if I keep away from Miss Elliot, and certainly do not want to take her by surprise'. So instead he suggests that it will inconvenience Mary and make things difficult because of the boy. When this is brushed aside, he insists on word being sent ahead to the cottage to warn Anne, though of course he cannot say that openly. When they meet, neither is able to do or say anything much to each other, but she has the woman's privilege of keeping quiet. He has to keep up the sociable front he has shown so far, but cannot really single her out and engage her in conversation when she cannot even look at him. It would actually be cruel to do so. So he keeps it short, is charming all round, polite if distant to her and leaves as quickly as possible.