The word of Bingley and the word of Wickham may both be "just one man's word," but they are not equal here. Whatever Bingley knows is secondhand, and we don't even know exactly what he knows. Wickham is speaking from his own experience. That doesn't, of course, make him truthful (and we don't yet know whether he is), but it is a difference between Wickham's firsthand word and Bingley's secondhand word in this case. To put it in legal terms, Wickham could testify in court to much of what he told Lizzy, but Bingley couldn't, because it would be hearsay.

I actually think that Lizzy does a good job in analyzing Caroline's warning here. Lizzy may be biased against Darcy, but she unerringly puts her finger on the source Caroline's bias, and she rightly ignores the warning.

In addition to saying that he has no right to an opinion, Wickham says that he won't shame Darcy over his infamous conduct, but he is telling the story to a stranger at a party, and for all he knows, to the worst gossip in the county. That should have caused Lizzy to wonder, even if it is not definitive. (It's possible that Wickham just got carried away telling his sad story to a pretty girl.)