Have a look at chapter 8 and how his opinion of Harriet changes.

"He began speaking of Harriet, and speaking of her with more voluntary praise than Emma had ever heard before." Harriet is "a pretty little creature" and he is so much "inclined to think very well of her disposition" that he says she has the potential to "turn out a valuable woman."

He assured Robert Martin, of whom he thinks very highly, that "he could not do better" than marry Harriet.

But then he finds that Harriet refused his friend, and wow: "She is not a sensible girl, nor a girl of any information. She has been taught nothing useful, and is too young and too simple to have acquired any thing herself. At her age she can have no experience, and with her little wit, is not very likely ever to have any that can avail her. ... as to a rational companion or useful helpmate, [Mr. Martin] could not do worse."

When Mr. Knightley is irritated by her, Harriet's whole personality is reducable to one word: ignorance.