Neglected may be too strong a word. CB "had a pleasing face and a melancholy air, just as he ought to have, and drew back from conversation." Also, they all seem surprised that CB comes with Harville to visit them, "he came, bringing his friend also, which was more than had been expected, it having been agreed that Captain Benwick had all the appearance of being oppressed by the presence of so many strangers. He ventured among them again, however, though his spirits certainly did not seem fit for the mirth of the party in general." (Chap 11)

So here's a guy that draws back from conversation & seems oppressed by so many strangers, yet Anne is still left to cope with him on her own. I don't mean to imply that this was intentionally done, just that Anne was sort of overlooked in Lyme as well as everywhere else.