While he's speaking with both of them, I think it was Mrs. Allen who said "I was afraid you had left Bath" - that would have been entirely too forward and inappropriate for Catherine to say!

As for his response, I think you're right that he's amusing himself, and hoping to amuse Catherine perhaps a bit at Mrs. Allen's expense, by playing with words in response to the use of "afraid". As we've seen from his introductory conversation with Catherine, he's all about poking fun at the delicacies of polite society, and I think this is more of the same. Mrs. Allen's comment invokes an extreme of emotion - fear - which is really a bit of overkill for the situation, and I see Mr. Tilney smirking inwardly as he follows up on it.