Here's my take on poor Mr Rushworth:

He's not the sharpest knife in the box, and quite likely he's aware of it, if only subconsciously. I think he may often suspect that he is missing the point of conversation and sometimes wonder if he is somehow being got the better of by sharper witted people.

I agree with Angela L that Rushworth is reluctant to show his jealousy - he's probably reluctant even to admit it clearly to himself.

I reckon Edmund's denseness in overlooking Mary's plainly stated unsuitability far surpasses any stupidity on Mr Rushworth's part regarding Maria, because what is going on between Maria and Henry is so very hard to pin down and is easily explained away . Maria's coolness towards him can be seen as commendable observance of the proprieties. Look at this exchange between Mrs Rushworth and Mrs Norris (ch.12):

"Yes, ma’am, indeed,” replied the other (Mrs R), with a stately simper, “there will be some satisfaction in looking on now, and I think it was rather a pity they should have been obliged to part. Young folks in their situation should be excused complying with the common forms. I wonder my son did not propose it.”
“I dare say he did, ma’am. Mr. Rushworth is never remiss. But dear Maria has such a strict sense of propriety, so much of that true delicacy which one seldom meets with nowadays, Mrs. Rushworth—that wish of avoiding particularity!

Henry is very careful to flirt with Julia as well as Maria - "... his manners being to each so animated and agreeable as to lose no ground with either, and just stopping short of the consistence, the steadiness, the solicitude, and the warmth which might excite general notice."

I wouldn't call Mr Rushworth particularly gutless at this stage. Even if he were sure his fears about Maria are justified there's not a lot he can do about it. If he confronts Mr Crawford about it Henry would probably say "Just being pleasant to the ladies. Nothing to worry about old chap." Rushworth certainly doesn't have grounds for challenging Henry to a duel, even supposing he wanted to go down that route.

If he confronts Maria she would very likely turn on the charm and make him feel like an idiot for suspecting her. If he offers to release her from their engagement he risks losing her.

He cannot end the engagement himself without very clear evidence of immoral behaviour. A woman could withdraw from an engagement, though it might harm her reputation. For a man to do so would be seen as dishonourable.

I don't read Rushworth's 'complacency' when he is taken in by Henry taking pains to soothe away his resentment as meaning that his suspicions are done away with completely. I think they are still there but soothed further by Julia taking the seat on the box with Henry.

I believe that he is reluctant to risk a break up. He is smitten by Maria and doesn't want to lose her.
The private pain and humiliation of having your fiancee flirting or worse with someone else is bad enough; the public humiliation of an actual break up and losing her to another man would be even worse.