‘The proposed solution, so characteristic of the modern state, is regulation. In place of internal mission comes external monitoring’.

These arresting sentences occur at the start of a recent piece by the FT’s Martin Wolf attacking the government’s proposals to open up the UK university sector to market forces. In this case, the problem which regulation, in the shape of an Office for Students (OfS), is meant to be an answer to is ensuring quality of provision, and to do this it has a battery of nuclear-powered weapons at its disposal – including, extraordinarily (this is a directly government-appointed body, after all), the ability, by order, to revoke any university’s historic right, granted by Royal Charter or Act of Parliament, to award degrees.

In Innovation and Entrepreneurship, Peter Drucker observed that in time all ‘institutions, systems, policies outlive themselves, as do products, processes and services’, and turn into a new problem to be solved. This is surely the case with regulation. One reason is what has been called ‘the tyranny of misapplied doctrine’ – the indiscriminate application of faddish nostrums (privatisation, lean, digital are three that instantly come to mind) to any institution or organisation thought to need reform, whether the remedy is appropriate or not.

Wolf does a good job of showing why turning the UK’s highly regarded university sector into a competitive market-driven one is a thoroughly bad idea, and I won’t rehearse it here. But it’s worth pausing to consider where ‘regulation’ is heading and the issues it is leaving behind.

Recall that the idea behind the formula ‘privatisation plus regulation’ was to put essentially commercial activities such as telecoms, energy generation and the railways out of reach of political meddling so that markets and competition could work their impartial magic unhindered. In spheres that couldn’t be privatised, regulation was intended to extend the positive functions of the state in a predictable, rule-based rather than arbitrary fashion.

Good luck with that. Relocated in the private sector, energy policy is a complete political and practical shambles. The railways are never out of controversy. ‘Big Bang’ and light-touch regulation were heavily implicated in the Great Crash of 2008 – regulators never even twigged that a crisis was brewing. In the public sector, by contrast, ever-tightening regulation has signally failed to prevent tragedies and scandals such as Mid-Staffs, Baby P and countless others. The truth is that not only has regulation not improved public services; it has actually prevented improvement. To put it another way, regulation fails the bottom-line test. It costs more than the sum of its benefits.

Fallacies live on both sides of the privatisation-regulation equation. Privatisation and markets aren’t always the right or only answer, as in universities (and education generally), health and other social fields. Even when they are a conceivable answer, regulation often doesn’t help, mostly, as John Seddon explains in The Whitehall Effect, because ‘regulators bring with them their own theories of management and control’. These are as ideologically based as privatisation itself, reflecting a fixed belief for example in efficiencies of scale and command and control and incentives to get the work done. Thus regulation arbitrarily decrees that public services should be target-driven, outsourced where possible and ‘digital by default’; in the private sector all that matters is that there should be competition on price.

Regulation-by-decree largely explains how we have ended up with a command version of the market economy which is more flexible than that of Soviet Russia, but only in degree. So we may have a market economy, and therefore ‘choice’. But ‘choice’ in the regulatory sense has a special and limited meaning. It doesn’t mean getting a service that you actually want or meets your individual needs; it means getting a service from providers using methods that the regulator approves.

Those methods rely heavily on compliance, budgeting and performance management activities which add little or no value, and the emphasis on unit cost and scale efficiencies effectively lock services into the industrial-age paradigm of low-cost, low-quality mass production, as (shamefully) in care services. If you ever wondered how in an age when ‘adtech’ boasts of its ability to micro-target ads according to everything from your age to weight to your gastronomic or sexual tastes, your long-time banking, financial services, retailing, energy and public service providers are all indistinguishable from each other in their mediocrity and inability to tell you from Adam, here is your answer. By extension, this is also the reason why all our High Streets resemble each other – for all these organisations it is the regulator who is king, not the customer, and regulators care about economic competition, not distinctiveness or quality.

There is a role for regulation – but it categorically does not include decreeing method, which on the contrary is exhibit No 1 in the museum of regulatory howlers. For a start, there is no so such thing as ‘best practice’, which is contingent on purpose and measures; and setting the means in stone is Soviet-style management that stops innovation dead in its tracks, which is why we have fragmented, lowest-common denominator public (and often private) services, hopeless banks and dismal High Streets.

Regulation – where needed – should be about holding organisations to account against their ‘internal mission’ (Wolf’s words) or purpose. Seddon proposes that regulators should be entitled to ask one and a half questions: What are your measures and how do they show that you are improving against your purpose? After 1000 years, British universities have developed a fairly good idea of their internal mission, and have powerful traditions of defending it, which is one reason they rate highly in international comparisons. That mission has already been compromised by government pushing of market forces: UK (like US) universities now have more, and more highly paid, managers and administrators than lecturers, and compete through marketing and ‘credentialling’ as much as by teaching and research. Using the screen of a regulator to justify and enforce what threatens to be a full-scale government takeover of the university sector is another large step in the Sovietisation of UK society, and the exact opposite of the advancement of the public interest that regulation was supposed to achieve.