Anne's looks have gone downhill and she's still not married and living under the dictatorship of her family. I think Frederick sees all these as coming directly from her unwillingness to marry him when she could have had her own house and be loved by him all this time. I actually don't see this as being unjust on his part at all. It seems extremely logical if you're looking at life through his eyes. And he doesn't know she's listening to him and Louisa doesn't understand what he's talking about. He never directly refers to Anne - so he's basically keeping her secret safe within him while talking out his opinion at the same time.