DevOps Zone is brought to you in partnership with:

Jez Humble is a principal consultant with ThoughtWorks and co-author of Continuous Delivery (Addison Wesley, 2010). He has been fascinated by computers and electronics since getting his first ZX Spectrum aged 11, and spent several years hacking on Acorn machines in 6502 and ARM assembler and BASIC until he was old enough to get a proper job. He got into IT in 2000, just in time for the dot com bust. Since then he has worked as a developer, system administrator, trainer, consultant, manager, and speaker. Jez has posted 13 posts at DZone. You can read more from them at their website. View Full User Profile

Jez Humble: Why Software Development Methodologies Suck

08.03.2012
| 40462 views |
  • submit to reddit

There’s a lot of dogma in the religious wars around software development practices and methodologies. Are phase-gate methodologies effective at managing the risk of software development, or just risk management kabuki? Does TDD really make for higher quality software? Is pair programming a superior replacement for code review or just a way to inflate consulting rates? I’m going to argue that while scientific evidence to decide these claims is lacking, there are two general principles which can help us choose good practices while at the same time improving the value of the software we deliver: reduce cycle time and increase feedback.

Michael Feathers makes the following observation:

I think that, in the end, we just have to accept that developer skill is a far more significant variable than language choice or methodological nuances1. Frankly, I think we all know that, but we seem to suffer from the delusion that they are the primary knobs to tweak. Maybe it’s an extension of the deeply held view that from an economic viewpoint, it would be ideal if people were interchangeable.


The problem is, how do we get skilled developers? Since the concept of individual productivity in IT has never been satisfactorily defined, this is a particularly hard problem to solve. Lines of code – still a popular measure – suffers from the devastating flaw that a line of code is a liability, not an asset as is often thought. Measuring number of hours worked encourages heroic behavior – but experience shows that the “heroes” are usually the same people that cause projects to become late through taking unacceptable risks early on, and working long hours makes people stupid and leads to poor quality software. There is still no generally accepted set of professional standards or chartering system for IT professionals, and recruiting good people is very much an art rather than a science.

Psychologists have at least addressed the problem of why it is so difficult to acquire and measure skill in IT. As Daniel Kahneman says in Thinking Fast and Slow, there are “two basic conditions for acquiring a skill: an environment that is sufficiently regular to be predictable; [and] an opportunity to learn these regularities through prolonged practice.”

But traditional software projects are the opposite of a regular, predictable environment. The only good measure of success of a project – did the end result create the expected value over its lifetime? – is so distant from the critical decisions that caused that success or failure that it’s rare for anybody from the original team even to be present to get the feedback. It’s practically impossible to determine which of those decisions led to success or failure (in artificial intelligence, this is known as the credit-assignment problem).

These factors make it very hard for IT professionals to acquire the skills that lead to successful products and services. Instead, developers acquire the skills that allow them to most efficiently reach the goals they are incentivized by – usually declaring their work “dev complete” as rapidly as possible irrespective of whether the functionality is integrated and production-ready – and similar problems arise in other functional areas too.

The fact that software projects are complex systems rather than regular environments leads to another problem – the extreme difficulty of gathering data on which techniques, practices, and methodologies are actually effective, and the near impossibility of generalizing this data outside the context in which it was gathered.

In his excellent book The Leprechauns of Software Engineering Laurent Bossavit executes a devastating attack on software development folklore such as the “cost of change” (or “cost of defects”) “curve”, the claim that the variance in developer productivity is an order of magnitude, the idea of the cone of certainty, and many other cornerstones of methodological lore in software development. He shows that these theories – and many others – depend on very small sets of data that are gathered either from informal experiments run on computer science students, or projects which cannot possibly have been effectively controlled. The organization of the studies that form the basis of these claims is often methodologically unsound, the data poorly analyzed, and – most egregiously – the findings generalized well beyond their domain of applicability2.

As a result, it’s not possible to take seriously any of the general claims as to whether agile development practices are better than waterfall ones, or vice-versa. The intuitions of “thought leaders” are also a poor guide. As Kahneman says, “The confidence that people have in their intuitions is not a reliable guide to their validity… when evaluating expert intuition you should always consider whether there was an adequate opportunity to learn the cues, even in a regular environment.” As Ben Butler-Cole points out in his companion post, “why software development methodologies rock”, the very act of introducing a new methodology can generate some of the results the adopters of the methodology intend to bring about.

You might think that puts us in an impossible position when it comes to deciding how to run teams. But consider why software development is not a regular environment, and why it is so hard to run experiments, to acquire skills, and to measure which practices and decisions lead to success, and which to failure. The root cause in all these cases – the reason the environment is not regular – is that the feedback loop between making a change and understanding the result of that change is too long. The word “change” here should be understood very generally to mean change in requirements, change in methodology, change in development practices, change in business plan, or code or configuration change.

There are many benefits to reducing cycle time – it’s one of the most important principles that emerges when we apply Lean Thinking to software development. Short cycle times are certainly essential for creating great products: as Bret Victor says in his mind-blowing video Inventing on Principle, “so much of creation is discovery, and you can’t discover anything if you can’t see what you’re doing.”

But for me this is the clincher: It’s virtually impossible for us to practice continuous improvement, to learn how to get better as teams or as individuals, and to acquire the skills that enable the successful creation of great products and services – unless we focus on getting that feedback loop as short as possible so we can actually detect correlations, and discern cause and effect.

In fact, the benefits of having a short cycle time from idea to feedback are so important that they should form one of the most important criteria for your business model. If you have to decide between creating your product as a user-installed package or software-as-a-service, this consideration should push you strongly in the direction of software-as-a-service (I speak from experience here). If you’re building a system which involves hardware, work out how you can get prototypes out as quickly as possible, and how you can modularize both the hardware and the software so you can update them fast and independently. 3D printing is likely to make a huge impact in this area since it allows for the application of software development practices to the evolution of hardware systems. Working in cross-functional teams is more or less a requirement if you want to achieve a sufficiently short cycle time.

Software methodologies – even the “hire a bunch of awesome people and let them self-organize” methodology – suck because they so often lead to cargo-cult behaviour: we’re doing stand-ups, we have a prioritized backlog, we’re even practicing continuous integration for goodness’ sake – why is the stuff we make still shitty and late? Because you forgot the most important thing: building an organization which learns and adapts as fast as possible.

1 Although as Laurent Bossavit points out (private communication) “A developer’s skill is in part the method he/she knows and his/her reasons for preferring one language over another.”

2 I am not suggesting that we give up on running experiments to learn more about what works and what doesn’t in software development, and the contexts in which such claims are valid – quite the contrary, I’m saying we’re not trying nearly hard enough.

Published at DZone with permission of its author, Jez Humble. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Jilles Van Gurp replied on Sat, 2012/08/04 - 7:18am

I totally agree with everything you say here. I've been a quite vocal critic of especially the role of scrum in our industry and you articulate very well a few of my favorite arguments.

I actually believe methodologies are just a means to an end, which is producing good software and creating value of some sort (money, vanity, honor and glory, etc.) for somebody. A lot of the things that methodologies like kanban or scrum try to achieve are actually things that you could argue are good using logical reasoning. For example short cycles, which are indeed very essential, have a few benefits that once you understand them make them more or less a no brainer.

The key advantage of short cycles is small change sets. Small change sets are easier to understand, test, and integrate. Easy here means cost. The testing benefit follows from the fact that in order to test a particular change you have to excercise the code. The more change you have the more combinations of things to test you have, i.e. more work. This amount of work not scale linearly with the amount of change but exponential (elementary mathematic combinatory explosion of inputs). This means testing larger change sets requires more effort than testing smaller change sets. This is the sort of intuition that both sounds logical and actually has some meaningful empirical validation behind it (and forgive me for not producing the references here).

Similarly lines of code may not be a great measurement tool for productivity but a bunch of things have been empirically correlated to them. For example, productivity in lines of code is more or less a constant across different languages for individuals but the productivity between a good and an excellent engineer can be magnitudes when measured in lines of code. That means using languages that are less verbose, can be a good idea because it makes every body more productive ultimately. Also cost models tend to factor in lines of code in some way.

Why is this relevant? Well as you said, software engineering is very much a process of discovery and iterative refinement. Having short feedback cycles means that you validate core assumptions more frequently and that you have a smaller risk of spending effort on redundant (i.e. non value adding) things. So there are some strong economical arguments for keeping cycles short, keeping change sets smal, and using sensible technologies that allow you to achieve things with less effort. That's in a nutshell the economic reasoning behind lean development.

The ultimate short cycle is a single commit. Continuos integration therefore makes sense from a cost perspective since the test effort for a single commit is cheaper than for multiple commits. You can make the same arguments for continuous deployment.

 There's a distinction between the what (e.g. short cycles, TDD, CI, etc.) and the how (scrum, kanban). Mostly these things are orthogonal and mostly the how part is not that interesting once you figure out the what part.

In my view collaborating on complex projects gets progressively more difficult as you add more people to a team. Developing stuff alone, you don't need any form of process. With two or three, if you can communicate, you don't need to be pretending to do retrospectives and standups. It's called the two pizza rule (you should be able to feed the entire team with that) and there are companies that apply that.

Process is usually brought in to fix situations where teams are bigger or where there are multiple teams. The real kicker here is that things like scrum are typically aimed at small teams. Small teams are easy and if you have few strong people they can be productive no matter what process they use. The real problem is building big stuff with lots of people. That's where the real challenges are in terms of how to develop software. I don't think many consultants have a good answer here. The better ones tend to take on a coaching role instead of a dogmatic role which means leaning on their experience to guide and gently nudge people in the right direction.  

Nicolas Seyvet replied on Sun, 2012/08/05 - 4:43am

I agree with both of you.  My problem: how to explain this to a management team supervising many groups of developers? For example, how to convince them that scaling up a project from 24 to 50 teams is counter productive? Politics play a big role in this too...

Jilles Van Gurp replied on Sun, 2012/08/05 - 5:55am

24 to 50 teams sounds like madness indeed. Doesn't anybody read the mythical man month? I've seen large teams like that getting in sort of a collective failure mode. Basically, at some point you start bleeding talent (i.e. the people that keep things going walk away) and at that point you have basically a couple of hundred people that aren't producing much value. Inevitably the shit will hit the fan and you don't want to be there when that happens. Layoffs are usually involved.

In my experience, you can't really explain your way out of such a situation. It takes a lot of energy, it pisses people off, and it ultimately gets you nowhere. Instead just lead by example. In the end, what matters is results and if you are consistently getting results, people will notice and appreciate what you are doing. You only need a handful of people to form a really strong team and once you start producing stuff, you are very hard to ignore.

The key thing is building alliances. Find people across the different team that are interested in getting stuff done and work with those people exclusively. You pretty much want to ignore everything else because it is not likely to be productive. That's a way of building a meta team that eventually just becomes the team. 

 

Mike Dzone replied on Mon, 2012/08/06 - 7:02am

All this "methodology" stuff is pretty much crap and everyone knows it but for some reason will not admit it and not work toward something else.  Every "methodology" shop I've been in have operated in one of two ways.  Either the team took 1 part of the methodology (say scrum) and stuck to that religiously and threw out the rest, or the team was super dedicated to every part of the methodology for about 1 week then everyone gave it up.

 What you have said about getting good, quality people is really the key to everything. For a while I worked on a small team of highly skilled great people and things "self organized" great without any "methodology"  throw crappy people into the mix, and that is when things fall apart. 

 People are not interchangable and just because a person has skills with Java  does not mean they are experts on any Java project.  Companies need to realize, like you stated in your article, code is a liablity, people are the asset.  As long as we stick to thinking the opposite, we will be in trouble.  

 

Stephen Wilson replied on Wed, 2012/08/08 - 3:10pm

The arguments about methodologies (and tools, and languages, and training) will continue and will remain largely unresolved while we continue to misunderstand the nature of software.  It is completely unlike the stuff of other branches of engineering -- so much so that calling it "engineering" is probably misleading.

I suggest a radical change of perspective. 

Most practitioners and observers see the creative aspect to writing software; clever programs seem to involve more inspiration than perspiration; and we have all known the odd truly gifted developer.  They're intuitive, idiosyncratic, different. 

Then consider the fact that software does not obey the laws of physics.  The very physicality of orthodox materials leads to predictability and reliability, but not so with software -- where badly written code, when compiled and looked at like a fabric, is indistinguishable from the good stuff.  This is why process is so crucial to software: we can only gauge the quality of a program by knowing its history. 

The mixture of virtuality and virtuosity leads me to conclude that coding is more art than science. Literally. Programming is actually like playwriting. Maybe this different model could lead to better understanding of the software craft and how to manage it, than the engineering paradigm.

More at http://lockstep.com.au/blog/2011/02/23/programming-is-like-playwriti

Dean Schulze replied on Sun, 2012/08/12 - 1:13pm

You've gone too far when you say that code is a liability, not an asset.  If code is a liability rather than an asset then get rid of all of it.  Of course, code is both an asset and liability.

A better way to think about this is that code is to a software project what weight is to an airplane.  You need to mimimize it and be sure that what is there is optimum, but you can't eliminate it because it is essential.  The heaviest components on an airplane are its engines.  They are essential, but they are also as light as possible.

Dean Schulze replied on Sun, 2012/08/12 - 1:29pm

Great article with some good links.

I agree with you about the need for good experiments.  The software industry needs its own version of the Underwriters Laboratories.  The UL is an independent testing lab that started out assessing fire and electrical safety.  Many local electrical codes are based on the work of the UL.  We need this kind of independent experimentation and assessment in software development.

An ad-hoc experiment of the type that we need more of is described here:

http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-solvers.html

I think there's something to learn from this example, but meaningful experiments would not use geniuses (Norvig) or someone whose trying to show that what he is selling will work (Jeffries).

Daniel Rocha replied on Thu, 2013/05/02 - 5:11am

in my opnion, we fail at following methodologies because our incapacity of understands its core concepts and issues of our own work environment. the scrum works very well when you understand the way of doing it. it's our fault.

Daybreaker Gijane replied on Thu, 2013/06/20 - 5:40am in response to: Daniel Rocha

Ah the old 'methodology is sound, you're just not doing it right' argument gets wheeled out again. Shame

John Quinn replied on Fri, 2013/07/26 - 12:49pm


Outstanding essay! I've been having a lot of the same thoughts but would never have been able to express them so well.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.