Yes, ESTA is required for citizens of Visa Waiver countries (eg British citizens) to enter the US without a visa, even through a land border from Mexico or Canada.
According to the US Embassy (UK) FAQ on the Visa Waiver Program and ESTA
If you enter by land there is no requirement that you be in possession
of a round trip or onward ticket.
See US Embassy & Consulates in the United Kingdom - Visa Waiver Program and ESTA FAQs
However, I would add that I'd expect them to ask you to demonstrate intent to depart within 90 days, whether that's proving that you live in Canada or Mexico (as a legal resident, not as a visitor/tourist) or can show your future travel plans.
For further context - I am a legal resident of Mexico and citizen of a visa waiver country. I travel into the US (on a B1/B2 visa, not ESTA/VWP) often, and still am occasionally asked to prove that I'm a legal resident of Mexico, to justify the fact that I have not returned to my country of citizenship. I imagine if I entered on the VWP rather than a visa I'd be asked this far more often.