I can't claim to have figured Frank out either, but I have a couple of ideas.

He might imagine that he had made enough 'blunders' to have given himself away. These 'blunders' might have seemed glaringly obvious to him, but were not in fact obvious enough for Emma to have figured out the truth. As we know, Emma was actually interpreting his behaviour so as to make it fit with her own imaginings, but Frank does not know this and thinks that she is observing him objectively.

Also, I think Frank could have been quite clear in his own mind that Emma was not really becoming attached to him. He would sense a great difference between the feeling of his interactions with her and his interactions with Jane - all sorts of non-verbal details of body language and eye contact would be absent from his interactions with Emma. This could well lead him to suppose she was not serious about him. Luckily he was right about this, though he could easily have been wrong - people often mask their true emotions and may be quite unaware that they are doing so.
So if Frank believes Emma is not seriously attracted to him and yet she still cheerfully engages in their banter and flirtation, he might interpret this as evidence that she is aware of what is really going on but is continuing to 'play the game' for amusement or other reasons of her own.

This is just a theory, and I cannot support it with evidence from the text, though clues may be buried in there somewhere. As to what he would have done if he felt that Emma was getting seriously involved, I think he would have probably backed off. He is a bumptious and rather thoughtless young man, but I don't get the sense that he is deliberately cruel. He would also be risking things getting to a point where he would be under pressure to propose to her or be labelled callous and heartless by his family and hers. This sort of complication is something I think he would wish to avoid.