# The Data Was There All Along *Part 2 of 3: The signals we missed, and what they reveal about how organisations really change* ![[agile-market-that-vanished-data.png]] --- After the initial shock of discovering that the Agile coaching market had essentially collapsed, I did what any self-respecting practitioner should have done years earlier. I went back to the data. Not new data but the data I'd already had. The annual State of Agile, DevOps and Kanban reports I'd been casually tracking for nearly a decade. The industry surveys I'd skimmed and filed. The market signals I'd noticed but never interrogated. What I found was uncomfortable. ## Hiding in plain sight The decline didn't arrive suddenly. It had been signalling for years, buried beneath optimistic headlines and upward-trending certification numbers. Ninety-eight percent of coaches reported "measurable impact". However, when you looked at what was actually being measured, the primary metric was mindset and culture shift, not business value. The measures that mattered most (employee wellbeing and leadership behaviour) accounted for just 5% combined. Over 11% of coaches weren't measured on any performance metric at all. The role was fragmenting. Coaching was increasingly bundled into other positions: Product Owner, Delivery Manager, Team Lead. Later editions of the State of Agile reports showed the Agile Coach role steadily declining in visibility, dropping from featured prominence to near-absence over the span of a few years. Additionally, organisational barriers were named, repeatedly, and repeatedly ignored: legacy command-and-control structures, functional silos, and inadequate leadership engagement. The reports kept documenting the same obstacles year after year. Organisations were hiring coaches but weren't prepared to ***be*** coached. And perhaps most telling: Agile maturity stopped being measured altogether. Coaches were increasingly assessed against general business OKRs rather than agile-specific outcomes. The very thing that made the role distinct was being quietly erased from the performance conversation. The data was always there. The narrative around it was optimistic. And most of us (myself included) chose the narrative. ## The guilt of hindsight I won't pretend this realisation was purely intellectual. There was a period where the dominant feeling was something closer to shame. Had I been wilfully blind? Had I chosen comfort over critical thinking, the very thing I'd spent years coaching others to avoid? The honest answer is: partially, yes. I was inside the system, benefiting from it, and that proximity made it harder to see clearly. This is not an unusual dynamic. Alvesson's research on functional stupidity describes exactly this pattern: the way organisations, and the people within them, develop a learned incapacity for critical reflection when that reflection threatens the status quo. I was my own case study. ## The Jedi parallel Star Wars fans like me might recognise this story. Guides who rise during turmoil, valued for a deeper understanding most people don't possess. They become institutionalised, absorbed into bureaucratic structures that dull their effectiveness. The Jedi became functionaries of the Senate; coaches became functionaries of scaled frameworks. Both were ultimately discarded by the systems they'd helped sustain. The pattern (wisdom that becomes institutionalised, then commodified, then discarded) is older than either Star Wars or Agile. It's an archetype. This wasn't just a market correction. It was the predictable outcome of a system that values novelty over depth, and that disposes of its guides once it believes it no longer needs guidance. ## What the pattern reveals Step back from Agile specifically and a broader dynamic comes into focus. Organisations adopt new ways of working with enthusiasm and investment. They hire specialists to support the transition. They measure early outputs (ceremonies held, teams formed, velocity tracked) and mistake these for transformation. The deeper shifts (in leadership behaviour, decision-making quality, psychological safety, learning culture) are harder to measure and easier to ignore. Over time, the gap between visible progress and actual capability grows. The specialists notice, but raising concerns feels risky when the organisation has already declared success. So the concerns go unvoiced. The specialists themselves become part of the performance: facilitating rituals that look like agility without challenging the structures that prevent it. When the organisation eventually confronts the gap between investment and outcomes, it blames the methodology, the specialists, or both. Rarely does it examine its own readiness to change. This is the pattern that killed the Agile coaching market. And it's the same pattern currently playing out with AI transformation, digital programmes, and every other change initiative that treats adoption as the goal rather than the starting point. ## The question that changed everything Somewhere in the middle of this reckoning, the question shifted. It stopped being "What happened to Agile coaching?" and became something more useful: *Why do organisations consistently fail to capture the value of their own transformation investments?* That question led somewhere entirely different. It led to McKinsey's research showing 43% average value loss across transformations. It led to Kegan's work on the invisible psychological labour that consumes people during change. It led to a diagnostic model that could quantify what dashboards can't show. But that's a different article. What matters here is the pivot: from looking backward at a collapsing market to looking forward at an unsolved problem. The Agile coaching crisis didn't end my career. It clarified it. The skills I'd built over two decades (reading human systems, surfacing what's hidden, helping people navigate uncertainty) turned out to be exactly what was needed. Just not in the container the market had provided. The container had to change. The work didn't. --- *Part 3: "What Comes Next": where the real demand lives, and why the future belongs to practitioners who can bridge psychology, coaching, and organisational systems.* *Andrew Kidd is the founder of Daring Futures and an MSc Business Psychology researcher at the Open University.* ## References Alvesson, M. and Spicer, A. (2012) 'A stupidity-based theory of organizations', *Journal of Management Studies*, 49(7), pp. 1194-1220. Digital.ai (2015-2023) *Annual state of agile reports, 10th-17th editions*. Available at: https://digital.ai/resource/state-of-agile-report/ (Accessed: 3 April 2026). Kegan, R. and Lahey, L.L. (2009) *Immunity to change: how to overcome it and unlock the potential in yourself and your organization*. Boston, MA: Harvard Business Press. McKinsey and Company (2021) *Losing from day one: why even successful transformations fall short*. New York: McKinsey and Company. Available at: https://www.mckinsey.com/capabilities/people-and-organizational-performance/our-insights/losing-from-day-one-why-even-successful-transformations-fall-short (Accessed: 3 April 2026). Scrum Alliance and Business Agility Institute (2022) *State of agile coaching report 2022*. Indianapolis, IN: Scrum Alliance. Available at: https://resources.scrumalliance.org/Article/state-agile-coaching-report (Accessed: 3 April 2026).