I think both Col Brandon and Willoughby had reasonable explanations for their sudden departure, an urgent message and being sent away by an elder relative respectively. I think what was found wanting was their manners. I am not sure Marianne had any ill suspicion of the colonel until Willoughby suggested:

“There are some people who cannot bear a party of pleasure. Brandon is one of them. He was afraid of catching cold, I dare say, and invented this trick for getting out of it. I would lay fifty guineas the letter was of his own writing.” (Ch. 13)

It is a very silly suggestion IMHO and it seems to me Willoughby says it because Col Brandon would not acquiesce to the demands of the party. He is polite but resolute. I think the reason for Elinor’s and Mrs. Dashwood’s ill suspicions of Willoughby was that he was acting unlike himself:

Elinor's uneasiness was at least equal to her mother's. She thought of what had just passed with anxiety and distrust. Willoughby's behaviour in taking leave of them, his embarrassment, and affectation of cheerfulness, and, above all, his unwillingness to accept her mother's invitation, a backwardness so unlike a lover -- so unlike himself, greatly disturbed her. (Ch. 15)