I agree; Col Brandon did not know about Willoughby until he got to London. But Willoughby must have know who the Col. was when he was introduced at Barton. It seems likely that Eliza had mentioned the name of her ward to him. That explains why W. disliked him instantly and talked about him in such a insulting manner.
I think that S&S3 implied that Brandon was suspicious about him. I could only think that perhaps W. may have had a reputation in Devonshire already that Brandon may have heard. But it was W. who was so critical. Now I get it!