Anne kept up with Naval lists/news and would have known that Captain Wentworth had earned enough prize money to come back and renew his addresses. The fact that he didn't, was a signal that he was no longer interested in having her for a wife and apparently didn't want to have anything to do with her. In the intervening years between that time and his appearance in her neighborhood, there was always the chance that he might have changed his mind. When he first meets Anne and gives only a bow and turns his attention to Mary without uttering a word to Anne, this is a signal that he has not changed his mind. If he had changed his mind, something along the lines of, "I hope you have been well," or a smile would have signaled that at the very least, he was no longer holding a grudge.

Since Anne was the person who broke the engagement, she really did not have much choice but to follow his lead on the determination of the relationship. I, and this is my personal opinion, have never felt that the Captain was trying to warn Anne of his visit to the Cottage. Having refused to have breakfast at the Cottage and yet accepting one at the House, I think he felt it incumbent upon him to make a stop and pay his respect to Mary, while having the excuse of going hunting to make it a quick one. If he had been so solicitous of Anne's feelings he would not have made the comment about her looks to the ladies knowing between the three of them it would likely get back to Anne. He knows Anne has a sweet, gentle nature and that a comment like that would likely be 'inflicting a wound' but obviously takes none of this into consideration. And, of course, that is exactly what happens---Mary quite cheerfully relays his comment to Anne.

He proceeds to flirt with both the Musgrove girls, a particular no-no of the times. Don't get me wrong, I like the Captain and despise the way he was treated by Anne's family and Lady Russell but his behavior on his return to the neighborhood is not worthy of our hero if that is what the author's intention is for him to be. So, as you see, I agree with you and others who believe Captain Wentworth's behavior as regards Anne, was not quite the thing. :)