I'm not sure we can assume the groom was with them when they went to Allenham. Obviously Willoughby's own groom traveled with him to Allenham or possibly he was assigned one for his use once he got to Allenham though the first possibility is the more likely one. We know that he went to Barton Park, to Allenham, back to Barton Park and if he changed for dinner, he may have gone back to Allenham and then back to Barton Park. Would the groom have been on every trip, especially the one with Marianne? It was only a mile and a half away. And even if the groom was there, as you point out, Elinor's indication that it was against propriety for Marianne to have gone without a companion would mean it would be thought of as having gone alone. And you're right, servants sometimes come of as invisible in JA's works, at least to us, but I imagine her original readers would have imagined them as part of the scenery without even thinking about it. Here is some information from the The Domestic Servant in Eighteenth-Century England (Jean Hecht) that may answer some of your questions. I will try and water it down as much as possible.

The theory of the relationship of master and servant current in the eighteenth century was in some ways inconsistent. In part the product of post-medieval society with its flourishing individualism, it acknowledged the relationship to be basically contractual. In part an inheritance from the Middle Ages, when status was fixed and involved recognized rights and duties, it conceived of the relationship as essentially a family one.

...Thus, despite the contractual foundation of the relationship, the dominion exercised by the master was regarded as almost unlimited, the bounds of the servant's obligation as all but undefined. The servant was looked upon as having temporarily relinquished his freedom.

..The harshness of this state was presumed to be greatly mitigated by the emotional and sentimental tie binding master and servant; for it was supposed that what was in the first instance a contract would develop into a truly family bond, characterized by mutual devotion. Indeed, the servant was thought of as becoming an integral part of the family, as the contemporary practice of referring to mans's domestics as his 'family' clearly indicates....the authority he exercised over his servants was viewed at the equivalent in scope and character to that which he exercised over his children. [The master was thus supposed to see to his domestics moral and spiritual needs through guidance as well as take care of their physical needs.]

Moreover, according to this concept, the master might expect fidelity and attachment from his domestics no less than from the other members of his family. They were supposed to guard his secrets, defend his good name against calumny and hostile criticism, and in general make his interests their own.

While there were master and mistress/servant relationships that met this theory the text continues, No such harmony as characterized the relationship in theory existed in fact. The chorus of complaint raised against domestics throughout the period makes it abundantly clear that eighteenth century was no golden age of service.

Reasons for discord: 1) An increase in the middle classes also increased the demand for servants multiplying opportunities for employment giving the servant more independence. This improved the servant's bargaining power. 2) Expansion of commerce and industry raised more and more people up the class levels and the impulse spread to raise oneself up reaching into the servant class. 3) Gratuities, more liberal now, from visitors and tradesmen freed servants from complete reliance on the master making him less mindful of being an excellent domestic. Insubordination ensued with greater frequency and servants began to refuse to submit to physical correction. There were suits.

Hand in hand with insubordination went the pursuit of self-interest: servants tended to exploit their places to the full...purloining of provisions, the padding of tradesmen's bills to increase the commissions, the neglect and ill-treatment of guests who failed to give generously. .however, the greatest of the attendant evils was the stifling of fidelity and devotion in the servant. It seems that as the relationship became more about the contract and less about an attachment, there was a rise in a multitude of petty annoyances from which few employers were entirely immune. Not the least harassing of these was the propensity of servants to retail their masters' business. The greatest inconvenience, however, was viewed as the high turnover of domestics by this time.