Yes, people do sometimes fall in love at first sight. Other times they don't. JA presents a number of couples in her novels that haven't fallen in love at first sight, and I think we all know of examples in our own lives of people who have had long and happy marriages after a rather slower start.
Secondly, in most potential relationships, even where people fall in love at first sight, there is a negotiating phrase around "does she/he return my feelings?". In Romeo & Juliet Shakespeare gets around this by having Romeo overhear Juliet expressing her feelings for him in what she thinks is privacy. Since Colonel Brandon has no such encouragement, it would be risky for him to go too fast.
It would also be dangerous to court Marianne openly before he is sure of *his* regard for her. Willoughby's behaviour to Marianne convinces his acquaintances that he is engaged, and when Willoughby's engagement to Miss Grey is found out he is ruined in the opinion of his acquaintances:
Well, said I, all I can say is, that if it is true, he has used a young lady of my acquaintance abominably ill, and I wish with all my soul his wife may plague his heart out. And so I shall always say, my dear, you may depend on it. I have no notion of men's going on in this way: and if ever I meet him again, I will give him such a dressing as he has not had this many a day. (chpt 30).
Of course Willoughby is thoroughly dishonourable. In _Per_ Captain Wentworth finds himself nearly caught by his sense of honour when he realises he has been paying too many attentions to a girl he realises he doesn't truly care for. Colonel Brandon, if he moved too fast, ran such a risk, and I think unlike Willoughby he would feel himself caught by honour.
And Marianne was set against Colonel Brandon, not on the basis of his manners, but on the basis of his age and her romantic opinions:
To the former her raillery was probably, as far as it regarded only himself, perfectly indifferent; but to the latter it was at first incomprehensible; and when its object was understood, she hardly knew whether most to laugh at its absurdity, or censure its impertinence, for she considered it as an unfeeling reflection on the colonel's advanced years, and on his forlorn condition as an old bachelor.
...
But thirty-five has nothing to do with matrimony." (chpt 8).
It strikes me as rather cruel to an orphaned child to propose to rename the child something else, and break her last links to her mother. Colonel Brandon may have felt the same way.