DevOps Zone is brought to you in partnership with:

Allan Kelly has held just about every job in the software world, from sys admin to development manager. Today he provides training and coaching to teams in the use of Agile and Lean techniques. He is the author of "Changing Software Development: Learning to become Agile" (2008) and "Business Patterns for Software Developers" (2012) and a frequent conference speaker. Allan is a DZone MVB and is not an employee of DZone and has posted 77 posts at DZone. You can read more from them at their website. View Full User Profile

Is manual testing sinful?

02.17.2014
| 6278 views |
  • submit to reddit
One of the asides I made in “Programmers without TDD will be unemployable” which caused a bit of outrage in the testing community was my comment “Manual testing is a sin.” While I have been unfair to many testers, and somewhat simplistic, I still stand by the statement. Let me expand on why I still stand by the comment and why I am wrong.

It is all a question of context. Look at the full line the quote appeared in:
“Unit testing will be overwhelmingly automated. Manual testing is a sin. Manual unit testing doubly so.”

In the context of Unit Testing, as one who has undertaken formal and informal manual unit testing I believe unit testing can and should be fully automated. Informal unit testing is too ad hoc, prone to error, time consuming and difficult to replicate. Formal manual unit testing suffers all these problems and is hideously expensive.

In the context of unit testing I stand by “Manual testing is a sin.” I will also go further.

The vast majority of testing I see performed by professional testers - i.e. not unit testing - is manual, the vast majority of this testing could be automated. If you automate the testing the initial costs might go up - I say might because a good natural language test script is expensive to write itself - but the cost of executing the test will fall. More importantly the time it takes to run the test will fall dramatically.

The combination of low cost and fast turn execution means the tests become a very very effective feedback loop.

Failure to cease this amazing opportunity is something I consider sinful. It may be expensive but it is necessary. Even seizing 20% of this testing would be massively beneficial.

What I do not consider sinful is Exploratory Testing which almost by definition cannot be automated and therefore needs to be a manual process. Exploratory testing is about looking for the unusual, the unexpected, the thing you didn’t think of before, the thing you only think of because you now see it.

Automated exploratory testing might even be a contradiction in terms therefore manual exploratory testing is not sinful.

But, and this is a pretty big BUT: if the initial quality is low (e.g. unit testing is absent or poor) and the testing (e.g. system, integration, acceptance testing) that comes before exploratory testing is ineffective then exploratory testing will be effective at finding defects (because the earlier rounds didn’t find them) but it will be ineffective as exploratory testing. The time and effort will be spent on exploratory testing is really system testing, and because real exploratory testing itself will not be possible when there are many defects in the way to see what is really happening.

Unfortunately the subtly of this logic is lost on most organizations who just label everything that happens after writing actual production code “testing” and is made worse when testing is separated from the developers and, more importantly, those who understand what the system should do (the business, the BAs, the product managers, the users, etc.)

Worse still, the easy availability of cheap offshore testing capability means that rather than address the root cause of defects (low initial quality, system ineffective testing, misunderstanding of exploratory testing) means the whole thing can be hidden away cheaply.

There are probably some other examples of testing which cannot be automated and are not sinful. But I repeatedly find individuals and organizations who too ready to jump to “it cannot be automated” after very limited, or no, effort is spent on trying to automate it.

Similarly the fact that a professional tester cannot automate the testing of a piece of software system does not mean it cannot be automated. Automated testing - at least the setup - involves programming skills which many testers simply do not have, indeed it is wrong to assume they would have them.

Finally, there are a class of systems which as far as I know defy automated testing. Or rather defy automation at the moment. They defy automation because the creators of these systems have little or no interest in allowing automation. This set includes: SAP, Microsoft Sharepoint, most CRM systems such as Microsoft Dynamics, Murex (although I believe efforts are afoot here) and some other large packages.

I find it incredible that companies like SAP, Oracle and Microsoft can sell systems which cannot be effectively tested. But actually this is part of the sales-pitch. Because these systems are marketed and sold as “no programming required” to offer automated test tools would expose the lie. If you are involved with selecting one of these systems ask: how will we test it?

The inability to effectively test these systems is their Achilles heal, it is one of the reasons I don’t like these systems and it makes me wonder about whether organizations are right to use these systems.
Published at DZone with permission of Allan Kelly, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Andre Bogus replied on Mon, 2014/02/17 - 2:21am

Your claim that all manual testing can easily be automated away for massive benefits is challenged by the fact that software can and will change during development, and automated tests (UI tests doubly so) tend to become brittle over time and will need to be maintained along the code.

Now you must add the cost of maintaining the tests to the balance. Also a good tester will bring her intelligence into the deal, whereas an automated test is intentionally dumb. The former can tell the difference between a bug and a desirable change, while the latter cannot. On the other hand, automated tests will only find errors they have been programmed to spot, while humans have an uncanny ability to find the spot that looks just a bit off.

Some tests will be made only a very limited number of times. It makes little sense to invest in automation then. For other tests, you will lack the tools required to automate them easily, so automation would require creating or otherwise obtaining the needed tools first (though this will probably be easier as time passes)-

Finally, automated tests can lull you into a false sense of security - I have seen systems fail in production because a bug was not caught by a test (that was changed because "that couldn't have been a bug) and no one thought to at least manually run through a few usage scenarios which would have found the error quite easily.

Conclusion: 

  • Automated testing: good
  • Manual testing: good
  • Mindful combination of both: better

Peter Verhas replied on Mon, 2014/02/17 - 11:43am

Generally test automation is worth if the

<ul><li>cost of manual execution of the manual test during the lifetime of the software plus

<li>the cost induced by the different quality resulted by the lack of automated testing for the manually tested feature</ul>

(manual testing cost) is more than the cost of test automation (automated test cost) including the development of the automated tests as well as the cost of the execution.

In my experience automated test cost was always significantly lower than manual testing cost. Even in projects where the first estimated underestimated the cost of manual testing and the automation was started only after a few iteration of manual tests.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.