I had always assumed it meant there weren't many people out and about in the streets and shops to look at. (And "little expectation of company" would refer to the Inn itself.)

Another thought now occurs to me -- perhaps because Lyme itself is not on a main road/ thoroughfare, you would not have people stopping by on their way to somewhere else. This would further reduce the number of people around.