Agile Zone is brought to you in partnership with:

Meera has posted 70 posts at DZone. You can read more from them at their website. View Full User Profile

Is Code Coverage Important?

10.20.2008
| 22552 views |
  • submit to reddit

How much code coverage is enough? Do you consider having 100% code coverage an uphill task? Based on whom you talk to, the reply you get varies considerably. I have worked with Managers who would not compromise on anything less than 100% code coverage.  On the other hand, I have had the chance to work with clients who had about 30% or even less code coverage and relied heavily on their testers and QA, and had successful releases month after month.

In my opinion, code coverage data gives us an important insight on how effective our tests are, what parts of our source code are thoroughly executed. You can also look at the code coverage report and find out specific areas of code which are not exercised by our tests.

If you are following TDD, than I assume you must and will be having 100% coverage. The test early principle helps you in detecting and fixing bugs early in the life cycle of your project.

However, if you are working with legacy code, it is a herculean task to achieve 100% code coverage. I suggest that you start by taking baby steps. When I started working with one of my clients, they had 10% code coverage. The technical lead decided that it was humanely impossible to achieve 100% coverage with their several hundred thousands of lines of code. So, we decided to run metrics and code coverage on changes the developer made on a daily basis. We were able to fail the build when the coverage or quality went down for just those files which were checked in by the developer. We were able to easily convince the developers to maintain the quality and also write tests for the specific functionality they were implementing.

I remember reading an interview with Rod Jonhson here at Javalobby where Rod says:

Sugrue: When you talk about quality,  I presume as followers of best practice that you do a lot of test-driven development and a lot of unit testing. But, do you worry about coverage statistics. What is your opinion on code coverage with unit tests?

Johnson: We do. Pretty much all development in Spring projects is test-driven. And I think that something has been extremely helpful in terms of getting the good quality that we have achieved. We do worry about coverage, so we definitely track coverage across all of our projects.

We are not kind of the Taliban of test coverage, in that we don't have people writing pointless code so they can get from 97% to 99%, but we do really care about coverage and we like to try to ensure that the coverage gradually increases across all of our projects.

Actually, I have had some interesting discussions with Peter Cooper Ellis. Peter is really surprised at just how few bugs that we have in Spring. And I think, that is an example of how our embrace of test-driven development definitely gives us an advantage versus many traditional software vendors.
 

 If you want your project to be as successful as Spring, than you know what % of coverage you need. Let's all strive to "Aim for the highest".

Code coverage reports should be used on a daily basis to:

    * see what percentage of tests actually test our code
    * communicate with your testers and the QA team about the code bases which have very low test coverage
    * improve the coverage by writing additional tests as more functionality is added.

 What is the % you strive to achieve in your project? Is there a specific person designated to look at these reports? Do you break the build when the coverage drops? Do you have 100% coverage?

Share your thoughts and code coverage tips and tricks?

Published at DZone with permission of its author, Meera Subbarao.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

James Sugrue replied on Mon, 2008/10/20 - 1:55am

In my experience, the code coverage balance is a delicate one. Code in the UI layer,  or generated code, can prove difficult when trying to achieve a high percentage (unless you just have a policy to not include that type of code).

Another important point for me is that code coverage metrics/goals are not enforced on the developer by managers etc, but that it is a natural by-product of TDD. All developers would like high code coverage, but not at the cost of developing other features.
It's a terrible thing when tests are just written to achieve a good code coverage metric.

James

Rasmus Grouleff replied on Mon, 2008/10/20 - 3:10am

You can achieve 100% coverage without writing a single assert, so coverage as a measure of code quality isn't really all that good, unless you factor in some sort of measure of how good the tests are along with different kinds of metrics such as cyclomatic complexity, lines of code per method and so on.

As James points out it also matters what kind of code you're working on. It's not all parts of the software that are equally important and it perhaps isn't all that important if you don't cover the last 10 or 20%, as long as the tests that has been written are of high quality.

Thomas Mueller replied on Mon, 2008/10/20 - 3:22am

TDD is great, but code coverage is just one aspect: it helps finding dead and untested code. But high coverage doesn't mean good software. If all you want is good code coverage, write randomized tests (fuzz tests). Randomized tests help a lot, but the problem is: they don't usually test correctness. Assertions improves code quality a lot, but don't improve coverage at all. So I think that most tests should be feature driven, not coverage driven.

UI tests are hard to automate, but you could measure the coverage of manual tests.

James Sugrue replied on Mon, 2008/10/20 - 3:26am in response to: Thomas Mueller

Very true Thomas - but then I wonder if you're measuring the coverage of manual tests are you then just getting metrics for the sake of it. Ideally, TDD should be automated.

Nicholas Pellow replied on Mon, 2008/10/20 - 4:55am

Maybe a better question would be: "How much testing is enough?"

Rather than focus on % of Code Covered, I find it better to focus on Test Quality. A Code Coverage report can certainly help do this.

I would rather have a project with low code coverage, and high quality tests, than one with high coverage and low quality tests. Since coverage is a negative metric, the uncovered code in a coverage report actually holds more information than the covered lines. ie if a statement is uncovered, you can be certain it is not tested. The same is not true for a covered line.

 

 

Rob Grzywinski replied on Mon, 2008/10/20 - 6:48am

As long as we remember that 100% (or 200% or ...) code coverage tells you absolutely nothing about the number of defects that you have.

 Canonical example:

  •  Input:  rob;  Output:  true
  • Input:  bob;  Output:  true
  • Input:  rod;  Output:  true

 Code (which is 300% covered by the above tests):

    public boolean login(final String username) {
return true;
}

 

 Whoops.

 

http://www.realityinteractive.com/rgrzywinski/archives/000062.html

Meera Subbarao replied on Mon, 2008/10/20 - 7:15am in response to: Rob Grzywinski

There isn't any doubt at all about the quality of tests we should be writing. Say we have 30% coverage with really good quality tests, how do you release your product knowing that the remaining 70% of your code isn't tested at all?

Meera Subbarao replied on Mon, 2008/10/20 - 7:23am in response to: Rob Grzywinski

[quote=rgrzywinski]

 Canonical example:

  •  Input:  rob;  Output:  true
  • Input:  bob;  Output:  true
  • Input:  rod;  Output:  true

 Code (which is 300% covered by the above tests):

    public boolean login(final String username) {
return true;
}

 

[/quote]

These tests looks like were written by developers who consider tests as second class citizens. So, there isn't any point in arguing with developers who write such tests. 

Meera Subbarao

Jean-Francois P... replied on Mon, 2008/10/20 - 8:52am in response to: Meera Subbarao

The fact is that, when you start to impose some minimum coverage ratio, then developers will little by little slip to this kind of tests, althougn they may be usually good tests writers.

 That's the problem with "enforcing" thresholds (that I personnally find stupid, because you can't always expect to have only first-class developers in your team).

Occasionally, in my company, I perform a review of the code of automatic tests themselves in order ot ensure that they do test something; I have bad surprises sometimes. I do that particularly for projects that claim high code-coverage;-) Note that we don't use TDD (with TDD the situation would be different of course).

The real problem is that imposing thresholds like that may shift the focus of developers (even the best) to the wrong locations. I prefer to responsibilize Project Leaders and their developers.

I find tools like Clover interesting (I didn't use them yet but I have seen the generated reports) because they correlate coverage with code complexity thus show where the real risks are. Indeed, having specific unit tests providing 100% coverage of Java Beans methods is generally a total waste of time. Whereas complex methods (with business rules)should call for extra testing efforts.

 

Josh Marotti replied on Mon, 2008/10/20 - 9:07am

I could write 1 unit test with lots of reflection that will give me 100% code coverage, yet will not give me any value as a test.  Code coverage is a nice thing to see, especially with an IDE plugin to help you write unit tests (like Eclemma for Eclipse), but I believe there should never be a line in the sand as to how much you should cover...

We should also consider the type of tests run by junit.  Integration tests (junit tests that depend on outside sources, such as other layers of code, db's, etc...) tend to cover a lot but not test very much, whereas true unit tests (outside sources mocked out) tend to cover very little, but have tremendous value.  Unit tests are something to be run everytime you compile, whereas integration tests run daily or at specific intervals since they traditionally take so long to run...

I've seen people try to say 100% test coverage is necessary, but it really depends on the application.  Reporting apps, for example, do not need 100% coverage.  In fact, I've seen a VERY well tested reporting app only have 40% coverage.  CRUD apps, on the other hand, tend to need more coverage... and yet I've seen a CRUD app with over 90% test coverage that was a buggy mess.

Coverage only shows us one small aspect.  Comparing coverage with the amount of tests, and the types of tests, along with other good metrics can show you the quality of the code to a degree.

And, lastly, remember that unit tests and coverage only prove one point:   The code does what the coder intended... which doesn't necessarily mean that the code is CORRECT...

Josh Marotti replied on Mon, 2008/10/20 - 9:10am in response to: Meera Subbarao

what if only 30% of the code is actual code, and the other 70% is DAO calls to a DB?  Is testing DAO test (not unit tests, but integration tests) really necessary when the results are just delegated to the UI where UAT and testers will be seeing it?  I'd say it'd be a waste of time for a developer to write tests on something like that...

Meera Subbarao replied on Mon, 2008/10/20 - 9:23am

I remember reading an article by Andrew Glover at his Disco Blog which summarizes this topic very well. To quote his conclusion:

coverage reports are hip in ascertaining what’s not tested (which, in this case, would practically force me to execute the 2nd condition in the if) but don’t depend too heavily on them telling you what’s tested. Otherwise you could end up short-circuiting yourself into a false sense of security…or is that coverage?

Alex Ruiz replied on Mon, 2008/10/20 - 9:38am

I don't agree with "If you are following TDD, than I assume you must and will be having 100% coverage." IMHO, TDD does not mean 100% code coverage. We can apply TDD 100% of the time and still have less than 100% code coverage. Examples that I have seen: private constructors in utility classes and generated code (implementations of "equals" and "hashCode"). In addition, TDD-ers that do not believe in mock objects, may encounter situations that cannot be reproduced even with stubs (e.g. testing handling an exception thrown by an external dependency under very specific conditions.)

I use code coverage tools as a reference only. I don't have a specific percentage in mind. What I do is look at the least tested code, and depending on its complexity or how critical that code is, I decide to add a test or not. In another words, I don't aim for a specific number blindly. I use coverage only to analyze where tests are needed.

Cheers,

-Alex

 

 

 

Andrew Newdigate replied on Mon, 2008/10/20 - 9:44am

My team uses cobertura to enforce a minimum coverage of 80% on our unit tests, although for most of our code the actual percentage is higher. We have two exceptions to this rule:

  • Bean / model classes. We don't have explicit unit cases for classes consisting of getters and setters
  • DAO classes: mocking out Hibernate and/or JDBC API's is painful and a waste of time. Making changes to the DAOs later on in the project is especially painful when this is done.

In general, bean classes are implicitely tested through the unit tests of the classes which manipulate the beans, and DAO's are tested via System and Integration tests.

In principal, I agree that you shouldn't have to enforce minimum coverage, but I find that if you don't have some type of automatic enforcement, the level of coverage quickly drops to unacceptable levels, and as a team we would never be able to justify the time taken to improve coverage rates if we found them to be too low - it's much, much easier to maintain a high level of coverage up front.

That said, it's very easy to write tests that obtain high coverage, but don't assert anything. In my experience, the only way to ensure the quality, as opposed to the coverage, of tests is regular code reviews.

Rob Grzywinski replied on Mon, 2008/10/20 - 1:35pm

Actaully Ms. Subbarao, the point is that 100% code coverage with only positive tests (regardless of their simplity, complexity or quality) tells you absolutely nothing about your software quality.  This isn't a question of "good vs. bad" tests.  It's a question of "positive vs. negative testing".  100% code coverage with negative tests tells you nothing.  100% code coverage with positive tests tells you nothing.  So what combination of positive and negative tests together with code coverage is sufficient?

 

Nicholas Pellow replied on Mon, 2008/10/20 - 7:15pm in response to: Josh Marotti

Coverage only shows us one small aspect.  Comparing coverage with the amount of tests, and the types of tests, along with other good metrics can show you the quality of the code to a degree.

We have attempted to make this possible to some degree with Clover's new per-test coverage feature. For each line of covered code, you can view the tests (click in the bottle green marker in the margin on this example) that covered that line. This makes it easy to check, that the test case is actually testing.

 

per-test pop-up 

 

Alternatively, you can get a sense of a Test's quality by viewing the list of classes each test case exercised. Generally speaking, a test which exercises only a few classes (ideally just one) will be a better test than one which covers many. A test case that hits many parts of the code will be more brittle, requiring more maintenance.

For example, compare StringUtilTest.testReplaceWithNull to CyclomaticComplexityTest.testNastyComplicatedMethod

 

These features are designed to make jumping from Source Code Coverage to a Test Case and vice-versa as easy as possible - giving you a better sense of "test quality" of your project.

Jens Schauder replied on Tue, 2008/10/21 - 1:08pm in response to: Meera Subbarao

[quote=meera]Say we have 30% coverage with really good quality tests, how do you release your product knowing that the remaining 70% of your code isn't tested at all?[/quote]

This statement is conveying some fundamental misconception. There seems to be the assumption that 100% code coverage means anything. But it doesn't.

 1. It is not at all clear what kind of coverage are you talking about? Statement, branch, method coverage? Or one of the dozens other alternatives?

 2. Assuming we agreed on one kind of coverage metric. 100% of it does NOT mean that the piece of code is completely covered in the sense that it can't contain any bugs anymore. So if you are not comfortable in releasing code with a coverage of 30% why would you agree on 90% or 100% these are just arbitrary points on a line ranging from 0 to infinity.

 What you really need is a understanding if the quality of the code base is sufficient. Code coverage can't give you that. Measures like mean time between failure might be usefull.

 I wrote about my opinion on code coverage in my blog.

Tim Lavers replied on Wed, 2008/10/22 - 2:41pm in response to: Rasmus Grouleff

Good point to distinguish coverage from effectiveness. A nice situation to be in is where everyone on the team is committed to writing good tests, which are reviewed or pair-programmed. Code coverage tools can assist in finding untested branches of complex statements and untested methods.

Meera Subbarao replied on Thu, 2008/10/23 - 11:03am

We had an hour long discussion at my work place also yesterday. Each one had a different opinion about code coverage. There isn't any one size fits all for this. As I said, we need to use the coverage report as a guide for writing additional tests. Of course, we need to use other tools as well to know what class or package has a high complexity level, how to refactor, or write additional tests so that we cover all permutations and combinations before we release our product to the outside world.

Jeroen Wenting replied on Wed, 2008/10/29 - 1:50am

What coverage is reported also seems to depend on the coverage tool used.
I've seen coverage reports that listed 0% coverage for some interfaces that weren't used directly in Java code (only through Spring configuration).
The classes implemented by those interfaces were reported as covered something like 75%, the coverage tool reported import statements and Javadoc as "uncovered code"...

Can't remember what version of what tool it was, but it's clear that even with every executable line of code covered by tests you can still encounter reports of incomplete code coverage.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.