]No, it isn't really polite, but it is the sort of thing that gets said when one is caught off guard by someone looking very upset, especially someone we care about.

I agree that, in general, we all have outbursts of feeling and then regret the way the phrase came out. However, if we read the text:

"'Good God! what is the matter?' cried he, with more feeling than politeness; then recollecting himself, 'I will not detain you a minute, but let me, or let the servant, go after Mr. and Mrs. Gardiner.'"