"It is a proof of your own attachment to Hertfordshire. anything beyond the very neighbourhood of Longbourn, I suppose, would appear far."
As he spoke there was a sort of smile which Elizabeth fancied she understood; he must be supposing her to be thinking of Jane and Netherfield, "I do not mean to say that a woman may not be settled too near her family. The far and the near must be relative, and depend on many varying circumstances. Where there is fortune to make the expence of travelling unimportant, distance becomes no evil."
Mr. Darcy drew his chair a little towards her, and said, "You cannot have a right to such very strong local attachment. You cannot have been always at Longbourn." Chapter 32
Darcy is smiling, I think, because he is imaging Elizabeth at Pemberley. He probably interprets her words as that she would not mind being a fair distance from her own family, say in Derbyshire. That is why he pulls nearer to her to get her true opinion.