We learn in this section, that he has at least one sister, possibly two, and a cousin who is close enough that Mrs. Jennings wonders if the letter that upsets him at breakfast is from her. Whitwell is owned by "a" brother-in-law of Colonel Brandon's--does that mean he could have more than one?
That brother-in-law is abroad, and Colonel Brandon's sister is in Avignon--the assumption would easily be that these are the owners of Whitwell, if it weren't for the indefinite article 'a' used with brother-in-law.
The fact that his sister is in Avignon is, according to Deirdre LeFaye in Jane Austen: The World of Her Novels, something that Jane Austen overlooked in updating the novel for publication. Brandon's sister might have been living in the south of France when the novel was originally written in 1795, but it was far less likely anyone from England would have been living there in 1811.