that he was having breakfast with the Thorpes was because he was actually staying with them.

From Chapter 7:
This critique, the justness of which was unfortunately lost on poor Catherine, brought them to the door of Mrs. Thorpe’s lodgings, and the feelings of the discerning and unprejudiced reader of Camilla gave way to the feelings of the dutiful and affectionate son, as they met Mrs. Thorpe, who had descried them from above, in the passage. “Ah, Mother! How do you do?” said he, giving her a hearty shake of the hand. “Where did you get that quiz of a hat? It makes you look like an old witch. Here is Morland and I come to stay a few days with you, so you must look out for a couple of good beds somewhere near.”

So it seems that James is the guest of the Thorpes, and thus his time is at their disposal, rather than that of the Allens and Catherine. The Allens never extended an invitation to James.