It's a small detail, but in ch.30, the Omniscient Narrator calls him *the* younger son of the Earl of __________, and so does Elizabeth in ch.33. If there were more than one, IMO they would say *a* younger son of the Earl, or *one of* the younger sons.

Another semi-piece of evidence is that Caroline Bingley is referred to as the younger sister, and there are only two of them (but "younger" in her case is confusing anyway, so I don't count it as full evidence!)

Also, the characters in JA's novels don't necessarily have huge families - the Earl's sisters only had one or two each.