to kill a man in a duel and get away with it? I mean if they were both good shots the"Winner" would have to leave the country after the death of his opponent. Willoughby had no means to support himself abroad and Miss Gray who was probably not to be trifled with can't be expected to follow him in exile after a debacle involving another woman.
Brandon had several reasons to remain in England. Marianne, Eliza and his estate. Besides he'd been in the army and likely knew what it was to shoot another man. Not something to wish for and also an obstacle between himself and any hope of a future with Marianne.