DevOps Zone is brought to you in partnership with:

I am an experienced software development manager, project manager and CTO focused on hard problems in software development and maintenance, software quality and security. For the last 15 years I have been managing teams building electronic trading platforms for stock exchanges and investment banks around the world. My special interest is how small teams can be most effective at building real software: high-quality, secure systems at the extreme limits of reliability, performance, and adaptability. Jim is a DZone MVB and is not an employee of DZone and has posted 101 posts at DZone. You can read more from them at their website. View Full User Profile

Static Analysis isn’t Development Testing

  • submit to reddit
I constantly get emails from Static Analysis vendors telling me why I need to buy their technology. Recently I’ve been receiving emails explaining how my team can use static analysis tools to do impressive things like “test millions of complex lines of codes [sic] in minutes”.

Hold on now, pardner. Running static analysis tools against your code to enforce good coding practices and check for common coding mistakes and violations of security coding rules is a darn good idea. But this isn’t testing, and it’s not a substitute for testing.

Static Analysis tools like Coverity, Klocwork Insight or Klocwork Solo for Java Developers,Findbugs,HP Fortify,IBM’s Software Analyzer, Microsoft’s FXCop and Veracode’s on-demand static analysis checking are a step up from what compilers and IDEs check for you (although IDEs like IntelliJ IDEA include a good set of built-in static analysis code checkers). These tools should be part of your development and continuous integration environment.

They are fast and efficient at finding common coding mistakes and inconsistencies in large code bases. The kinds of mistakes that are easy to miss and hard to test for, and that are usually easy to fix. This makes code reviews more efficient and effective, because it lets reviewers focus on higher-level problems rather than low-level correctness. And they can identify areas in the code that need to be refactored and simplified, and areas of the code that may need more testing or deeper review – especially if the tool finds problems in code that is already “working”.

But static analysis isn’t the same as, or as good as, testing

Static analysis tools can’t test the code to validate that it does what it is supposed to do. That the features that the customer requested are all there and that the business logic is correct (unit testing and functional testing and acceptance testing). They can’t check that the interfaces of a module and its behavior is correct (component-level testing), or that the larger pieces of the system work correctly together and with other systems (system testing and integration testing). Or that the system will hold up under load (performance and stress testing and soak testing), or whether it is vulnerable to security attacks (pen testing), or that it people can actually understand how to use it (usability testing). Static analysis tools, even the best ones, are limited to finding “a narrow band of code-related defects”.

I appreciate that static analysis tool vendors want to find new ways to sell their solutions. Maybe this is why Coverity is now calling its static analysis toolset the “Industry’s First Development Testing Platform”. Their message seems to be that developers don’t test their code and won’t fix bugs found by testers “because of organizational silos and conflicting priorities”. So static analysis has to fill this perceived gap.

By improving integration with development tools like Eclipse and Visual Studio and making the static analysis checks run faster, Coverity has made it easier for developers to find and fix more coding bugs earlier – that’s good. But better packaging and workflow and faster performance doesn’t change what the tool actually does.

Development Testing or Developer Testing

Running a static analysis tool isn’t “Development Testing”. Development Testing or Developer Testing is what developers do to test their own code to see if it works: walking through their code in a debugger as it executes and making sure that it does what they expect, profiling it to make sure that performance doesn’t suck, writing automated unit tests and component tests to make sure that the code does what it is actually supposed to do.

Confusing static analysis tools with other automated testing tools is a bad idea. It misrepresents what static analysis tools can do. It could lead people to believe that static analysis is a substitute for testing. Bill Pugh, an expert in static analysis and the father of FindBugs, makes it clear that while static analysis tools definitely work, “testing is far more valuable than static analysis” at finding bugs. And security experts like Jeremiah Grossman at White Hat Security debunk the idea that static analysis is enough to check for security bugs or other kinds of bugs in a system.

Static analysis tools are getting better and easier to use, they’re getting faster and more accurate and they throw off less false positives than they did even a couple of years ago. Everyone should use them if they can find a tool that works well for them and that they can afford. If you’re in a Java shop for example, there’s no excuse not to be using Findbugs at least – it’s free and it works. My team uses multiple different static analysis tools at different stages of development and continuous integration, because different tools find different kinds of problems, and some tools take longer to run. These tools find real bugs and coding problems. But there’s too much hype in this market. In the land of software testing, static analysis tools don’t “reign king”.

Published at DZone with permission of Jim Bird, author and DZone MVB.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)


Tom Gardner replied on Sat, 2012/01/14 - 3:39pm

Static analysis tools can’t test the code to validate that it does what it is supposed to do.
Of course, neither can testing (especially unit testing) validate that the code does what it is supposed to!

But I'm sure you knew that!

dieter von holten replied on Sun, 2012/01/15 - 11:17am

i am a fan of static code-analysis and i see a big gap between what is achieved in research and what is available as practical tools to the programmer in the trenches. static analysis can point you to resource leaks, concurrency problems, value-out-of-range problems, usage anomalies, architectural problems much better and easier than any dynamic test could do. in some areas (embedded sw) its possible to prove absence of certain errors.

writing and running dynamic tests before all complaints of the static analysers are silent is a waste of time.

Fabien Bergeret replied on Mon, 2012/01/16 - 7:44am

I totally agree that static code analysis shouldn't prevent you from doing Unit Testing and Functional Testing.

However, there are some very interesting tools that can indeed find bugs in your code just by analyzing the source code. Findbugs, for instance, can actually find bugs. 90% of the alerts are false positive, but it sometimes finds errors that would have taken hours of testing otherwise.

Andrey Karpov replied on Wed, 2012/01/18 - 3:25am

What do you think of on-the-fly mode, in the analyzer PVS-Studio?

Paul Russel replied on Sun, 2012/06/10 - 8:07am

I very much agree that static analysis does not replace all of the different types of testing. That's just crazy. I would argue that static analysis is a type of testing. From my point of view, quality is a part of testing and I think static analysis falls into that category. Maybe I'm biased because I'm one of the no-good-dirty-vendors (HP), but I think static analysis is a form of testing that can complement other types of development testing.

BTW, really enjoy the blog.

Arthur Hicken replied on Wed, 2012/08/08 - 4:41pm

I heartily agree that static analysis isn't testing. Disclaimer - I work for Parasoft where we've been producting static analysis tools for almost 20 years.

Over the last couple of years somehow the market has started to get the perception that the primary function of static analysis is to find bugs. The real truth is that static analysis is very well suited to finding weaknesses in an application, and not nearly as well suited to finding bugs, especially at the last moment.

Frequently the information you get from a tool is about how you've done something poorly and it would be better another way. Late in the game developers tend to ignore such advice that is too late to be helpful and most often write it off as a false-positive, when in fact it's more of a "too-late to be helpful" message. The later in the process you use static analysis, the more likely it is to produce messages that aren't helpful.

To get the most benefit, you have to decide early on what kinds of weaknesses you would prefer to avoid, and put static analysis in place to help catch offenders as the code is being written - either as part of a regular daily build, continuous integration, or on the desktop as the code is written in the IDE.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.