I should think that his "affected indifference" is more to do with a childish desire to be seen as superior. He likes girls, but wants to be seen as despising them unless tehy are obviously very rich or dazzlingly beautiful. He gives them a few stares, notes that they are quite pretty but not very rich, from their clothes etc and then makes out he's quite indifferent and does nto take any interest in such girls...by walking off.