I think Mr. Bennet is poking fun at his wife and his daughters. In reply to Mrs. Bennet saying "you flatter me... When a woman has five grown-up daughters she ought to give over thinking of her own beauty." He replies:

"In such cases a woman has not often much beauty to think of."
That reply isn't complimentary to his wife or his daughters, all of whom he likened to his wife.

I was just thinking about the character of Mr. Bennet, and the different portrayals of him in the various adaptations. I prefer Benjamin Whitrow's Mr. Bennet in P&P2 - who is more quick and not so lazy - however reading JA's description of Mr. Bennet:
"Mr. Bennet was so odd a mixture of quick parts, sarcastic humour, reserve, and caprice,..."(Ch. 1)
I had to look up the word "caprice". One definition in Merriam Webster is:
"a disposition to do things impulsively"
and then synonyms to that word (as an act, not a disposition) are: whim, vagary, crotchet. And then the word caprice is further elaborated on as: caprice stresses lack of apparent motivation and suggests willfulness