Willoughby left Eliza "ignorant of his address". Although he promised to return to her, he evidently did not wish to be found by her.

I cannot imagine that if Willoughby was trying to avoid being found by Eliza, that he would knowingly go to the very part of England where her guardian was most likely to be found. I also don't know why Eliza would share that information--Brandon's name--with Willoughby.