I thought the "education to render them callous" referred to a kind of mental or moral training - e.g: you should train yourself always to be spiritually ready for adversity, focus on the really important things so hardship won't take you by surprise, etc., etc. - the kind of advice that was especially popular with later Victorian children's authors. Obviously, Mr. Primrose is saying that his own family hasn't had this kind of training (perhaps because nobody thought it would be needed?)