I think you have not quite got this bit aright,Rae. Anne Elliot does reply to wentworth. He has begun the conversation and note he does not say the normal "Good morning,How do you do ?" commonplace pleasantries....He blurts out that he has only come to see the Miss Musgroves. Again he is setting the tone. Anne actually answers that outburst quite well considering,IMO :

The surprise of finding himself almost alone with Anne Elliot, deprived his manners of their usual composure: he started, and could only say, "I thought the Miss Musgroves had been here: Mrs. Musgrove told me I should find them here," before he walked to the window to recollect himself, and feel how he ought to behave.

"They are up stairs with my sister: they will be down in a few moments, I dare say," had been Anne's reply, in all the confusion that was natural; and if the child had not called her to come and do something for him, she would have been out of the room the next moment, and released Captain Wentworth as well as herself.

He continued at the window; and after calmly and politely saying, "I hope the little boy is better," was silent.

I agree they are both rattled : but he again sets the tone,telling Anne he had only come to see Louisa and Henrietta and walks off to look out of the window,presumably now having his back to her. Anne says no more, but you know I consider its hard to address an unfriendly back. His body language is adding to his exclusion of Anne, indicating he really does not want to enter into conversation with her... I think by now she could have easily discerned that.