]why didn't she say something to the point of, "As I am, however slightly, acquainted with Mr Darcy, I should feel wrong to go without being invited." In fact, why didn't the Gardeners consider this in the first place?

According to the novel,

Elizabeth was distressed. She felt that she had no business at Pemberley, and was obliged to assume a disinclination for seeing it. She must own that she was tired of great houses; after going over so many, she really had no pleasure in fine carpets or satin curtains.

Mrs. Gardiner abused her stupidity. "If it were merely a fine house richly furnished," said she, "I should not care about it myself; but the grounds are delightful. They have some of the finest woods in the country."

Elizabeth said no more—but her mind could not acquiesce. The possibility of meeting Mr. Darcy, while viewing the place, instantly occurred. It would be dreadful! She blushed at the very idea; and thought it would be better to speak openly to her aunt than to run such a risk. But against this there were objections; and she finally resolved that it could be the last resource, if her private enquiries as to the absence of the family were unfavourably answered.

So, she will bring it up with her aunt, if she finds out that the family is there. I am not sure why she doesn't talk to her aunt right away. Maybe because her Aunt wants to go so much, she would feel bad being a "wet blanket," so to speak.

Accordingly, when she retired at night, she asked the chambermaid whether Pemberley were not a very fine place, what was the name of its proprietor, and, with no little alarm, whether the family were down for the summer. A most welcome negative followed the last question—and her alarms being now removed, she was at leisure to feel a great deal of curiosity to see the house herself; and when the subject was revived the next morning, and she was again applied to, could readily answer, and with a proper air of indifference, that she had not really any dislike to the scheme.