In Ecclesiastical law, Sunday observance is a vespera ad vesperam, but the Sunday Observance Act 1780 was not a holy law. Bishop Porteus, its author, was attempting to suppress "disputing societies" in the lower classes, (who were preaching atheism, as well as unionism and labour reform), so a secular act was needed. In his preamble, he mentions particularly "places within the cities of London or Westminster or in the neighbourhood thereof have of late frequently been open for entertainment or amusement on the evening of the Lords day" - and if I am correct in interpreting "vesperam" as meaning "the church service conducted at sunset on Sunday", then these meetings would not be prohibited by Ecclesiastical law.

Not that Bishop Porteus foresaw the effect that his legislation would have on making the towns of England (and Canada, and Australia) renowned by Sunday travellers from other countries, for their dullness and emptiness. He was quoted as claiming, in defence of his act that "It imposes no other restraint than this (which is surely not a great hardship), that no one shall either pay or be paid for talking blasphemy or profaneness in a public room on the Lords day."
Well, the kindest thing I can suppose is that Bishop Porteous was myopic -- this act was far reaching and in our era, contributed to the hardships of the observant working class as well as the godless. For example, while the act does not prevent a family that worked sixteen hours a day six days a week, from dressing meat to eat on Sunday lunch, if they get paid their wages Saturday night, it effectively prevents them from buying meat, and if they rely
on their baker for the use of an oven, they would have practical difficulties legally obtaining a hot Sunday roast before 1822.

Still, I am not supposing that Mr Elliot in his impoverished youth would have suffered much from the manifold restrictions of this act, even though he happened to reside in the area that the Bishop particularly mentioned when explaining the necessity for the act. He wouldn't have to. Statutes like this, banning circuses, fairs, and all sorts of fun, had been enacted since medieval times, and the fun still went on. For example, the Sunday Observance Act of 1625 already prohibited people assembling outside their own parish for "any sport or pastime whatsoever, or within the parish for any bear-baiting, bull-baiting, interludes,common plays or unlawful exercises or past-times" -- but people still recorded attending such events, or occasionally were brought before a magistrate, charged with holding them. People ignored the laws, forgot them or found ways around them. In the case of this act, they were helped out by a Good-time Charlie in the Aristocracy.
In 1633, Charles I restored the right of men to practice archery, women to decorate churches with rushes, and all to dance, leap, vault, play May-games,and drink Whitsun-ales (although he had resistance from the clergy, and 800 priests lost their livings by refusing to post this proclamation on their church doors).

Even without overt help from the Regent (who could not defy parliament in this way, anyway), people (and particularly the denizens of the cities of London and Westminster) found ways around the Sunday Observance Act of 1780. Concert halls could give free Sunday concerts (free to get in, but you would have to pay for a seat). Subscribers could attend concerts on a Sunday. Private assemblies were exempted from the act, and private clubs. Benefit concerts were within the law also (gold coin donation gratefully accepted). Debates, dancing halls and picture galleries also opened for "free" and charged for incidental expenses. Barrows could not be described as 'houses' nor their locations as 'places', so Sunday markets continued to trade. For retail shops, Sunday trading had been specifically prohibited in an earlier Sunday Observance Act, but the fine was fixed at five shillings and the volume of trade made it worthwhile to pay it, when they must. Cricket, football and Tennis were (according to the responses of their governing bodies to a 1964 committee investigating the repeal of the act) never affected by the act, although technically, the hire of a court, or paying to attend a game, were prohibited by it.
Circuses were covered by this act, but had been covered by acts like it since the age of Constantine, so Sunday had become the traditional day to dismantle their equipment and move on, before this particular act came into force.

What made the 1780 act more effective than its predecessors, was a "sudden and almost universal adoption by county and borough benches of a policy of restriction and regulation" in the licensing of ale, victualling and coffee houses, inns, taverns and dram shops. These had proliferated out of control (or rather, with the assistance of J.P.s who got five shillings fee for every licence issued, and without opposition from the farmers, landlords and clergy whose incomes were influenced by the amount of malted corn sold) during the first half of the eighteenth century, and were colourfully and famously described by Smollet ("dead drunk for two-pence,...straw for nothing") in his History of England in 1802. By the late Georgian however, the various reforming Societies for the Prevention of Vice , including the ones lead by Henry Zouch and William Wilberforce, were determined to shut down these dens of iniquity and put a stop to the misery they created, change the national drink from ale to tea, and they sought the assistance of the local magistrates in their quest.
Sidney and Beatrice Webb, in their history of Local Government in England, noted that there was a lull in "crime and rioting, licentious living,disorderly conduct and brutal amusements" between 1790 and 1810, as a consequence of this regime of strict suppression and removal of the licence of any drinking place that could be named a disorderly house under the terms of the Sunday Observance Act 1790, and through refusing to licence new applications.

Of course there was a huge loophole -- a person who wanted a drink, had only to remove himself from the inn in his local hundredth to the inn in the next in order to legally enjoy one. Westminster and London in particular seemed to have hordes of bona-fide travellers on a Sunday.

So Sunday travelling could imply a lot more than an irreligious character - it could imply a man that can't endure an alcohol free day; a libertine devoted to frivolity; a preference for low and seditious company; dabbling in radical politics; supporting democracy and revolt for the lower classes; conspiring with the agents of her enemies against England; and also, in Mr Elliot's case, demonstrating a willingness to break the spirit of the law even when he is ostensibly earning his living from enforcing it in the letter.

References:
Report of the Departmental Committee on the Law of Sunday ObservanceCmnd2528 chaired by Rt.Hon. Lard Crathorne, T.D. (London: Her Majesty's Stationary Office, 1964)pp.2-5, 11,16,17,21,22,24-8,35,40,41,64

S&B Webb, The History of Liquor Licensing in England Principally from 1700 to 1830,(GB:Frank Cass,1963) ch.3 in particular.

T.Smollett,The History of England(Philadelphia:Levis & Weaver, 1810)p.499 pt.XXXVI

Sunday Observance Act 1780, as it currently stands(yes, still not repealed), not as it was in our era,unfortunately.
(UrL:http://www.opsi.gov.uk/RevisedStatutes/Acts/apgb/1780/capgb_17800049_en_1)
Online excerpt on 'Sunday' from the 1911 Encyclopaedia Britannica
(URL:http://www.1911encyclopedia.org/Sunday)