The History of Agile Software Development: From Toyota to the Manifesto

The History of Agile Software Development: From Toyota to the Manifesto

How Software Development’s Most Successful Revolution Became Its Most Abused Brand

Agile is the most successful methodology in the history of software development. It’s also, in many of its current implementations, barely recognizable as what its founders intended. Understanding how that happened requires knowing where it came from.

The Problem Agile Was Solving

Before Agile, the dominant model was Waterfall: gather all requirements, design the full system, build it, test it, deploy it. The logic was impeccable. The results were frequently catastrophic.

The Standish Group began tracking software project outcomes in 1994. Their CHAOS reports found that roughly a third of software projects were cancelled outright, more than half were significantly over budget and behind schedule, and fewer than 17% were delivered on time, on budget, and with the originally specified features. The “specified features” criterion turned out to be the most revealing problem: by the time software was delivered, the requirements had changed enough that what was built wasn’t quite what was needed.

The Waterfall model had a fundamental assumption baked into it: that requirements could be fully and accurately specified before development began. In practice, this assumption was almost always wrong. Requirements changed. Technology constraints emerged. Users discovered what they actually wanted when they saw working software. The model treated these realities as failures of process rather than inherent properties of complex work.

Several approaches were already attacking this problem before 2001.

Iterative and incremental development had roots going back to the 1950s — NASA’s Mercury capsule project used iterative development in the early 1960s. Barry Boehm’s Spiral Model in the 1980s introduced risk-driven, iterative planning. These approaches recognized that learning happens during development and that plans should accommodate new information.

Rapid Application Development (RAD), introduced by James Martin in 1991, compressed development cycles and emphasized continuous customer involvement. RAD practitioners were already doing many things that would become Agile best practices.

The Toyota Production System contributed ideas that would eventually become Lean software development: eliminating waste, just-in-time production, and the concept that workers closest to the problem have the most relevant knowledge.

What Happened in Snowbird in February 2001

Seventeen software developers gathered at a ski resort in Utah. They called themselves “organizational anarchists” — they had various methodologies (Extreme Programming, Scrum, Crystal, DSDM, Pragmatic Programming) but shared a frustration with heavyweight, document-driven development processes.

The group included Kent Beck (XP creator), Jeff Sutherland and Ken Schwaber (Scrum), Alistair Cockburn (Crystal), and others who had been thinking seriously about lightweight development approaches. They weren’t theorists presenting academic research — they were practitioners describing what had worked for them.

The result was The Agile Manifesto: four value statements and twelve principles that could fit on a single page. That brevity was deliberate. The authors wanted something that couldn’t be bureaucratized into a methodology without losing its meaning.

Whether that aspiration succeeded is debatable.

The Rise of the Methodologies

The Manifesto was a values document, not a process document. The frameworks that translated those values into practice had already been developing in parallel and continued to evolve afterward.

Scrum had been developing since 1993 (Jeff Sutherland) and 1995 (Ken Schwaber presented it at OOPSLA). By the early 2000s it was the most widely adopted Agile framework, primarily because it was prescriptive enough to implement and lightweight enough to adopt without a full organizational transformation.

Extreme Programming (XP), Kent Beck’s creation, was arguably the most technically rigorous Agile methodology — Test-Driven Development, pair programming, continuous integration, collective code ownership. XP had significant influence on engineering practices even in teams that didn’t formally adopt XP as a framework.

Kanban’s adaptation to software began in the mid-2000s, most notably through David Anderson’s work. Its origins in Toyota’s manufacturing system gave it a different character from the other Agile frameworks — less prescriptive in ceremony, more focused on flow and waste elimination.

Large-scale frameworks emerged as Agile adoption grew into enterprises that needed to coordinate many teams. The Scaled Agile Framework (SAFe), Large-Scale Scrum (LeSS), and Disciplined Agile all emerged in the 2010s to address this problem, with varying degrees of success and various criticisms about whether they actually preserved Agile values at scale.

The Success and Its Costs

Agile adoption spread remarkably quickly. By 2010, Agile had moved from a software development approach to a general organizational philosophy. Marketing teams did “Agile marketing.” HR departments did “Agile HR.” Agile principles appeared in management consulting proposals to companies that had nothing to do with software.

This success created its own problems.

The certification industry grew rapidly around Scrum and other Agile frameworks. Two-day certifications proliferated. “Certified Scrum Master” became a common LinkedIn credential for people who had attended a training and passed a multiple-choice exam. The certifications signaled exposure to vocabulary rather than practical competence.

Cargo cult Agile emerged at scale: organizations adopted the ceremonies and terminology of Agile while maintaining the governance, incentives, and culture of traditional management. Daily standups happened in front of Jira boards, sprint reviews were conducted, retrospectives were held — and nothing changed about how decisions were made, how work was prioritized, or how teams were empowered. The research organization VersionOne (later Digital.ai) found in annual surveys that “organizational culture” was consistently the top barrier to Agile adoption, even among organizations that considered themselves Agile.

“Agile” became a brand applied to anything iterative, anything customer-focused, anything collaborative. This dilution made the term almost meaningless in some contexts.

Where Agile Stands Today

The core ideas of the Agile Manifesto have held up well. Short feedback loops are better than long ones. Working software is a better progress measure than documents. Customer collaboration produces better outcomes than requirements handoffs. Adapting to change is more realistic than following a fixed plan.

The implementations have become more varied and more contested. Scrum remains dominant. Remote work, which accelerated dramatically in 2020, has forced adaptation of practices designed for co-located teams. AI-assisted development is changing what “the work” looks like in ways that current frameworks haven’t fully addressed.

The critics of Agile — and there are thoughtful ones — point to real problems: Agile teams can become reactive rather than strategic, sprinting from priority to priority without architectural vision; “responding to change” can become an excuse for poor upfront thinking; the human cost of endless iteration without stable foundations is real.

These are fair criticisms of implementations, not of the values. The answer isn’t to return to Waterfall — the problems Waterfall failed to solve haven’t become easier. The answer is continued reflection on what the values actually require in any given context.

That’s the part the Manifesto got right that most implementations miss: it called for continuous improvement, not adoption of a permanent methodology. The history of Agile is a story of practitioners trying to solve a hard problem. The next chapter is the same story.