Agile Versus Waterfall: Part One

The agile crowd always bashes waterfall. It's silly. It's dumb. It's guaranteed to fail. But waterfall is responsible for most of the software available today. There must have been something good going on there. What was it?

In my opinion, it was the illusion of control. Managers, often non-technical, had to manage teams of very technical developers. It was their job to report on the progress of the teams, but they couldn't. The technical work was so, well... technical, and so heavily based in the developer's heads, that it was very difficult to evaluate or understand progress.

The result is fairly easy to understand. A manager felt a strong need to understand what was going to happen, so he or she tasked their teams to do a great deal of analysis before the project started. This didn't work, so like violence and XML, they reasoned if it's not working, add more! This became what is often referred today as Big Design Up Front (BDUF).

So developers were tasked with doing very detailed studies and analysis when they knew the least about the product, and the least about the solution. And the customer they were trying to interview knew the least about the solution as well. It seemed that no matter how well the teams did their Big Design Up Front (BDUF), the customer was never happy. They either changed their minds or wanted a completely different product? The solution? Add more design!

Waterfall projects tended to last a long time... years in some cases. Of course, the bigger the project, the more work had to be done up front when everyone was in their optimum state of ignorance. However, the one thing that waterfall had going for it was the long time frames. It gave the teams a relative period of quiet, uninterrupted work. They could make mistakes, recover from them, and move on towards success. And this happened often enough that it took hold.

Unfortunately, it more frequently failed. That quiet period could also be used to hide problems. Issues ranging from developer apathy to incompetence weren't surfaced very often. If the team misunderstood the requirements, or missed a key point, no-one noticed until the end of the waterfall product cycle.

The first advantage of agile over waterfall is the short iteration. Developers focus on completing tasks on a weekly or bi-weekly rhythm. This short cadence has many benefits, the two that spring to mind at the moment are the surfacing of issues, and the ability to change directions.

If you have four teams, and three of them finish their short iterations very well on a regular basis, and one can't complete half of their work on most iterations, you know which team has problems. We don't know what the problem is, but we know where to start looking. A waterfall project can hide the problem teams, or the less than competent developer, until about the 75% mark, when teams start trying to integrate their work. At this point it's often too late to recover.

The other benefit is the ability to show work to the customer after an iteration is done and get feedback. The problem with getting feedback is that it might change what you're doing. The problem with not getting feedback is your project could fail, the customer might withhold payment, and the company could go broke. I think I'd prefer to change what I'm doing from time to time.

The world we live in is extremely dynamic and your customers may not be very good at describing their problems (assuming they understand them at all!) If you don't get feedback as you go, the product is at risk. Your success as a team is at risk. The only way to manage or minimize that risk is to show the customer your work as often as possible. The iteration provides a great stopping point for those demos. In reality most teams don't demo for a customer after each iteration, but they could. And some do.

Another agile technique for minimizing risk is the continuous integration system. After every code push, the entire project is compiled and every automated test is run. The alternative is waiting until "later" to merge everyone's code, and that's both painful, and risky. What do you do when people have been in the same projects, changing the same code? Frequent integration catches those problems quickly, minimizing the impact of the collision.

Which is better? I'd have trouble picking waterfall for most projects. Even a project for a government entity that had very stable requirements can still benefit from the frequent iterations as a risk management tool.

I'll post more on this topic next week... for now, what are your thoughts?