I think if he had had a different wife, he would have been a decent person. He does have guilt over his broken promise to his father, so he spends all his time trying to justify himself because he simply can't let himself see his wife in anything other than the very best light. It would completely rock his world if he realized just what she and her mother actually were. So he is continually misinterpreting everything they do and making it out to be for very decent, sensible reasons. They manipulate him so easily because he too has a disposition to be selfish, but knows better, so he covers it up and goes along with them. And his whole life is making excuses and trying to bury his guilt.