I agree that the lovers were having a bit of light-hearted fun and he certainly didn't realize that he was in love with Emma until Frank Churchill was regraded as a possible match for her. In chap 5, Mr Knigtley also said --

"She always declares she will never marry, which, of course, means just nothing at all. But I have no idea that she has yet ever seen a man she cared for. It would not be a bad thing for her to be very much in love with a proper object. I should like to see Emma in love, and in some doubt of a return; it would do her good. But there is nobody hereabouts to attach her; and she goes so seldom from home".

It would have been impossible for him to have said these words e.g. "nobody hereabouts to attach her" if he had realized that he himself was in love with her. He knew her declaration of never marrying "means just nothing", so it was not because he took it seriously that prevented him from asking her, or acting more like a lover and courting her, if he had felt himself to be in love with her.