Sometimes an overnight revelation takes 20 years.

When I started writing a weekly management column for The Observer in 1993, I didn’t have an overall ‘theory’ about management. I knew it was important, that it was about people, and I sensed it was more craft than science. I knew it was about more than shareholder value, and suspected that, as in other fields, short cuts would turn out to be the reverse: organic growth was likely to take us to a better place than growth by acquisition and financial engineering. I wondered if I would run out of subjects.

There were also a number of puzzles, often to do with the glaring gap between reality and the rhetoric. Big companies were clearly essential to developed economies, but despite the comforting prose of their annual reports, why did they have to be such dispiriting places to work? Could tobacco companies and hamburger chains really achieve model citizenship through programmes of corporate social responsibility while their products were indirectly imposing huge costs on the rest of society? What about giant retailers whose ‘efficiencies’ (ie low prices) came at the expense of suppliers (sometimes whole industries) and low-paid employees?

More generally, if managers were the hard-nosed pragmatists and management the empirically-based discipline that convention says it is, why were they doing so many things that were at best ineffective and at worst harmful, even in their own terms? Research repeatedly said that acquisitions mostly destroyed value, but that didn’t stop M&A hitting record levels year by year. Companies that aggressively pursued shareholder value didn’t seem to do better than those that had a purpose other than maximising investor returns, at least in the long term. Intrinsic motivation was much praised when it applied to nurses and carers, the sense of vocation used to keep wages at subsistence level, so how come CEOs needed extravagant extrinsic incentives to persuade them to deliver a good day’s work? Especially since no one could find a link between CEO pay and company performance (not, it should be said, for lack of trying). Companies have spent most of the last two decades putting their supposed ‘greatest asset’ out of work, and the financial crisis and its aftermath revealed for all to see just what the financial sector really thought of the customer who is meant to be king. A major disappointment was the New Labour government that came to power in 1997, which instead of making the UK a role model for enlightened public-sector management simply grafted on to public services the industrialised practices that were turning customers off in droves in the private sector.

By the mid-Noughties it was hard not to believe that there was as much wrong with present-day management as right. Writing a weekly column was an extraordinary compressed education. On the one hand the work brought contact with leading management thinkers and experimenters, and on the other with readers who brought the fads and theories back to a touchstone: never mind the PR, here’s what it was really like to be managed in modern Britain. It was this combination that counselled caution in the face of the triumphalism that accompanied the fall of communism and later what was optimistically billed as ‘the Great Moderation’. Sceptism was of course vindicated in spades by the spectacular implosion of the financial system in 2008: we were right, management didn’t do what it said on the tin, and now, knitting together what I had sensed before, I thought I could see why, although I struggled to express it.

But although I had most of the pieces, the final epiphany only came later. It arrived in three parts. One was at a conference in Brussels last February, put on by an enterprising Czech-based NGO, the Frank Bold Society, on The Purpose of the Corporation. The briefing included a memo which set out the legal position in black and white: across jurisdictions, as a matter of law, shareholders don’t own the corporation, and directors’ fiduciary duty is to the company with which they have a contract. So in brief, shareholder capitalism, and the whole theory of corporate governance that has evolved to sustain it, including the assumptions about human nature and behaviour that it is supposed to control, is based on a myth.

The second ‘aha’ moment was at a Vanguard conference on health, some of the profound findings of which I wrote about here. One of them was that the thinking that would make the difference between a manageable and unmanageable NHS was not inherently difficult: it was just different. So different, in fact, that the existing management worldview couldn’t be modified to incorporate it – change could only come if that worldview was replaced. That helped to explain why initial resistance to the ideas was so strong.

The third element was an invitation to a workshop put on by the alumni of the Open University’s Systems Thinking in Practice course. The aim of the event was to give support and sustenance to systems thinkers who, for the reasons outlined above, could easily find themselves isolated and discouraged at work. I had expected to be interested and stimulated by the occasion, but it turned out to be rather more than that. Slightly unwillingly I found myself participating in an exercise designed to draw the lessons from a situation where systems thinking had helped in the past and consider how to apply them again in the future.

Bingo! Suddenly, reflecting on my trajectory, I could see what had been staring me in the face all along. It’s a system, stupid. The management apparatus that has been developed in business school and university economics and finance departments to further shareholder value and control is all of a piece, from governance, through the measures and techniques used, right down to performance management on the shop or call-centre floor. If the organising principle of shareholder primacy can’t be justified, it’s not an accident that so much of management designed around it is ‘wrong’ – the surprise would be if any of it were right.

Here’s the reason why management gets inexorably more complicated and regulation more intrusive, as ever tighter rules are drafted to deal with its increasingly harmful side-effects, whether for individuals, society or the planet. Management has become its opposite. It doesn’t solve problems, it creates them. And the companies it animates are a Frankenstein’s monster.

Everything you know about management is wrong. Literally.