I think we agree then, that the original not-handsome-enough-to-dance-with comment laid the foundation, but had little first-hand effect at the time of Darcy's proposal. It was a building block, well-hidden by the time Elizabeth is in Hunsford by the subsequent events and gossip. Had Elizabeth never heard anything more against Darcy, she would have thought him an unpleasant companion, but no worse.