Tom isn't a ladies man. He probalby finds dancing parties dull and does not particualarly want to exert himself on the dance floor. He prefers outdoors sports and pursuits.

Hes not all that flirtatious so hes not pushed about making himself agreeable to the young ladies present. And he just seees Fanny as the "poor relation" who should be very glad to be invited to the dance at all..
and Fanny IS very grateful to be allowed this litlte indulgence and is not annoyed that she's not dancing all the time