in CH 43, want to go away from Cleveland?

"Colonel Brandon himself, with a much greater exertion, began to talk of going likewise. "

One would have thought his anxiety would keep him there. Was it impolite to stay there because he came with Mr. Palmer, who was no longer there?