]Mr. B didn't say no to his wife when she sent Jane on horse back under the rain, and he let lizzy ramble in mud in her way to Netherfield, I think Darcy must think of all of that especially that he admits that he doesn't like to see his sister doing this.

... we know this as readers. He (Darcy) can't know this for sure. Mr. B's behavior at Netherfield was his only evidence, which I still find too weak to base his assertions on. What else could Mr. B have done under those circumstances?