I thought that if two people were engaged & the man called off the engagement it tainted the woman's reputation. The woman was able to call it off without as much trouble, but it would still be known. I thought this was why Edward didn't break off with Lucy Steele, but she's able to break off with him later.

I know this case is different, but I guess since outside parties like the Harvilles believe CW & Louisa have an understanding, CW feels obligated to her. Also, I think he's worried Louisa may truly have feelings for him. It wouldn't be honorable for him to raise her expectations & then let her down. Good thing he finally makes that visit to his brother!