I don't think that things changed in places like Downton that easily. So Tom is just in a silly position - and IMO his initial eloping with a lady of the house was enough of a stretch. Then, they've had to stretch plausibility a lot to get him a role where he could stay at D Abbey when it was more likely that he would leave. But since they've done these scenes of him wandering around shirtless, I suppose they have retained him because he appeals to viewers. IF the Granthams were a family who visited London, he might find some kind of role there, where he could be a visitor or resident at their house, and work at some job which wasn't too low or too high status.. and say leave little Sybil with her mother's family, but from what I've seen of the show, the Crawley family are "home based" for the most part and dont go to London much.