I think he felt as though he had no real right to remain there with them unless he had been expressly invited. He offers to go, thinking it is the socially correct thing to di, but he was "gratifying the first wish of his heart" to stay there with them. It's funny that Mrs. Jennings invites him to stay for the sake of "his love", Elinor, but in the end the effect accomplished is the same for the colonel.