People can be considered handsome or beautiful because they just are drop dead gorgeous, but sometimes handsomeness is a consensus opinion. If someone has reasonably pleasing features, & decent height & posture their handsomeness can be a result of other considerations. A single vicar, because of his spiritual, social & emotional cache, can be considered handsome with regular features, & a good bearing, which could be entirely overlooked in someone in a different class. Moreover, opinion tends to enlarge & reinforce itself when it enters the local rumor mill.

That said, I agree that Mr. Knightley is the least lkely person to be influenced by extraneous factors so if he says Elton is handsome, he's handsome, but the community at large may be more so affected & may have inflated Mr. Elton's appearance.