I too was struck by the injustice of this remark. However, when I thought about it for a bit I realized that between Fanny's shyness and Sir Thomas' rather formal air with the children, he really doesn't know her at all.

Austen says, when Fanny first comes to live with the family, that he really doesn't even know his own daughters that well because he doesn't inspire them to express themselves honestly in his presence. This effect would be even more pronounced with Fanny, who is naturally shy and feels herself to be so much beneath the rest of the Mansfield family--as indeed, Mrs. Norris has ensured that she should. (Darn Sir Thomas for assigning her the task of making sure Fanny knows she isn't a "Miss Bertram"!)