I do not think Elinor was in any physical danger. She would be hesitant to make a scene, but if she was actually frightened there are servants in the house, including whoever opened the door for Willoughby, who would have come in at her first call.

However, the social strictures at the time left her almost as bound as physical violence. She partially decides to hear him because her doing so will make this awkward situation easier, and quicken his departure. Her curiosity, and her wish of treating him with some degree of the acquaintanceship they shared in Barton were not the only things that make her sit down and let him tell his tale in person.