I think that Brandon is a very private, honorable man who is very self consious, it is blatantly obvious to him (however wrongly) that Marianne could never love an infirm, stuck in his ways old man. He can't see his own worth be it monetary or otherwise. All he sees is her high spirits and does not wish to kill them, as he has witnessed in his past the disasterous results of this.
I believe it was not love at first sight, simply memories she rekindled of a first love which is never forgotton.