I was able to find a few instances of a similar claim being made by sources other than the History Wardrobe ladies--although I certainly don't dispute your resources, for I know they are extensive, detailed and accurate!

However, with other references to this around, I also don't think the History Wardrobe ladies were making that up.

Here, FWIW:

There was from an article that appeared in The Jane Austen Journal Persuasions
quoted here, where the author of the article states that the flannel waistcoat, while unfashionable, implies "danger, endurance and courage, a kind of military uniform and aligns him with the bravest military men of his time."

This online resource talks about uniforms issued to British Cavalry units serving in the War of 1812. Wrong contintent, but right approximate time period and also the British Army. Among the standard issue for them was a flannel waistcoat--so perhaps such a thing is a remnant of Brandon's military experience?

I also found references of men being advised to wear them to avoid taking a chill. If Brandon had been in the habit of doing this, perhaps it speaks more of a wish to avoid illness (since he has no one to look after him, really) than actually being infirm himself.

Here's a passage from The Adventures of Ferdinand Count Fathom (1753)
by Tobias Smollett

...the company was surprised by the entrance of such a figure as had never appeared before in that place. This was no other than a person habited in the exact uniform of an English jockey. His leathern cap, cut bob, fustian frock, flannel waistcoat, buff breeches, hunting-boots and whip, were sufficient of themselves to furnish out a phenomenon for the admiration of all Paris. But these peculiarities were rendered still more conspicuous by the behaviour of the man who owned them. When he crossed the threshold of the outward door, he produced such a sound from the smack of his whip, as equalled the explosion of an ordinary cohorn; and then broke forth into the halloo of a foxhunter