If he knew well Mrs. Smith (or at least he knew about her from the Middletons), he would know that she "was unfortunately too infirm to mix with the world, and never stirred from home", so the probability that Willoughby's character were exposed through her was *very* dim, as it actually happens in the novel: W. tells the Dashwoods that Mrs. Smith sent him on business to London, and they never hear anything against him.