DevelopsenseLogo

100% Coverage is Possible

In testing, what does “100% coverage” mean? 100% of what, specifically?

Some people might say that “100% coverage” could refer to lines of code, or branches within the code, or the conditions associated with the branches. That’s fine, but saying “100% of the lines (or branches, or conditions) in the program were executed” doesn’t tell us anything about whether those lines were good or bad, useful or useless.

“100% code coverage” doesn’t tell us anything about what the programmers intended, what the user desired, or what the tester observed. It says nothing about the tester’s engagement with the testing; whether the tester was asleep or awake. It ignores the oracles that the tester applied;how the tester recognized—or failed to recognize—bugs and other problems that were encountered during the testing. It suggests that some machinery processed something; nothing more.

Here’s a potentially helpful way to think about this:

“X coverage is how thoroughly we have examined the product with respect to some model of X”.

So: risk coverage is how thoroughly we have examined the product with respect to some model of risk; requirements coverage is how how thoroughly we have examined the product with respect to some model of requirements; code coverage is how thoroughly we have examined the product with respect to some model of code.

To claim 100% coverage is essentially the same as saying “We’ve looked for bugs everywhere!” For a skilled tester, any “100%” claim about coverage should prompt critical thinking: “How much” compared to what? 100% of what, specifically? Some model of X—which one? Whose model? How well does the “model of X” model reality? What does the model of X leave out of the universe of possible ways of thinking about X? And what non-X things should we also be considering when we’re testing?

Here’s just one example: code coverage is usually described in terms of the code that we’ve written, or that we have available to evaluate. Yet every program we write interacts with some platform that might include third-party libraries, browsers, plug-ins, operating systems, file systems, firmware. Our code might interact with our own libraries that we haven’t instrumented this time. So “code coverage” refers to some code in the system, but not all the code in the system.

Once I did a test (or was it 10,000 tests?) wherein I used an automated check to run through all 10,000 possible settings of a particular variable. That was 100% coverage of that variable being used in a particular moment in the execution of the system, on that day.

But it was not 100% of all the possible sequences of those settings, nor 100% of the possible subsequent paths through the product. It wasn’t 100% of the possible variations in pacing, or system load, or times of day when the system could be used. That test wasn’t representative of all of the possible stakeholders who might be using that variable, nor how they might use it.

What would “100% requirements coverage” mean? Would it mean that every statement in the requirements document was covered by a test? If you think so, it might be worthwhile to consider all the models that are in play.

The requirements document is a model of the product’s requirements. It refers to ideas that have been explicitly expressed by some people, but not by all of the people who might have requirements for the product. The requirements document models what those people thought they wanted at a certain point, but not necessarily what they want now.
The requirements document doesn’t account for all of the ideas or ideas that people had that may have been tacit, or implicit, or latent.

You can subject “statement”, “covered”, and “test” to the same kind of treatment. A statement is a model of what someone is thinking at a given point in time; our notion of what “covered” means is governed our models of coverage; our notion of “a test” is conditioned by our models of testing. It’s models all the way down.

Things in testing keep reminding me of a passage from Jerry Weinberg’s work:

“One of the lessons to be learned … is that the sheer number of tests performed is of little significance in itself. Too often, the series of tests simply proves how good the computer is at doing the same things with different numbers. As in many instances, we are probably misled here by our experiences with people, whose inherent reliability on repetitive work is at best variable. With a computer program, however, the greater problem is to prove adaptability, something which is not trivial in human functions either. Consequently we must be sure that each test does some work not done by previous tests. To do this, we must struggle to develop a suspicious nature as well as a lively imagination.“

Computer Programming Fundamentals, Leeds & Weinberg, 1961

Testing is an open investigation. 100% coverage of a particular factor may be possible—but that requires a model so constrained that we leave out practically everything else that might be important.

Test coverage, like quality, is not something that yields very well to quantitative measurements, except when we’re talking of very narrow and specific conditions. But we can discuss coverage, and ask questions about whether it’s what we want, whether we’re happy with it, or whether we want more.

Further reading:

Got You Covered http://developsense.com/articles/2008-09-GotYouCovered.pdf
Cover or Discover http://developsense.com/articles/2008-10-CoverOrDiscover.pdf
A Map by Any Other Name http://developsense.com/articles/2008-11-AMapByAnyOtherName.pdf
What Counts http://www.developsense.com/articles/2007-11-WhatCounts.pdf

Subscribe to the Newsletter

13 replies to “100% Coverage is Possible”

  4. Hi Michael, good article, a couple of observations:
    you say “..saying “100% of the lines (or branches, or conditions) in the program were executed” doesn’t tell us anything about whether those lines were good or bad, useful or useless. It doesn’t tell us anything about what the programmers intended, what the user desired, or what the tester observed.”

    If you wrote those checks doing Test Driven Development I could argue that the statement above is not correct, it is actually the opposite. In TDD (BDD) the check describes the behaviour that includes the intentions of the user, the developer and the tester. BDD also makes sure we don’t gold plate (hence we eliminate the useless), we introduce the useful, remove the bad and adds the good.

    Michael replies: I observe that in order to provide meaning for the statement “100% of the lines (or branches, or conditions) in the program were executed”, you had to say a good deal more than that. In addition, I’m not aware of anything intrinsic to BDD that limits gold plating. Time constraints might do that; resource constraints might do that. A check cannot encode intentions. If that were possible, then it would be possible for a program to reliably encode intentions. If that were so, we would have no need for checking, testing, testers, or programmers. There is no set of checks that I’m aware of that would be capable of stipulating that ‘these checks shall pass and no other checks shall pass’. For further reference, see http://www.satisfice.com/blog/archives/638.

    It seems to me that you’re giving credit to BDD for things that normal people are normally capable of managing by an enormous variety of means, particularly the conversation part. I seem to remember having conversations a long time before there was BDD.

    On a different note, I talk about code coverage only when I talk about unit tests, in fact experience taught me that measuring coverage against checks at a higher level is pretty much useless.

    My point is that talking about code coverage at a lower level is similarly unhelpful. I could say that talking about what has not been covered might be a good deal more interesting, but that’s not at all a new idea; Brian Marick was saying that roughly 15 years ago.

    In general, coverage doesn’t tell me that the code is tested, not at all. Once the unit tests pass I have some information, only after performing exploratory testing and all other testing I reckon will be necessary I then can say that I am happy.

    Okay… but isn’t that the point of my post—and that the point that you were trying to argue against in your first paragraph above?

    But at that point the checks that I have written will serve their MOST IMPORTANT function that it to fail when in the future I break something and tell me straight away. That’s what checks are for.

    Let’s be careful here: the checks that you have written will serve their most important function when you break something in a way that those checks detect. That’s a good thing, and quite useful. Nonetheless, as testers and developers, it behooves us to be alert to the possibility that the product will have problems that the checks will not detect. Therefore, we test.

    Reply

  6. I understand what you are saying. Great article.

    Michael replies: Based on what you write below, I’m not convinced that you do understand what I’m saying.

    However, we practitioners struggle with this myth everyday with our business partners. I’ve had experienced architects that stared blankly at me when I tried to explain that 100% is not only possible, you might as well burn Benjamins in your futility to attain. No, if you want to present this topic, encourage the phase to be stricken from the lexicon and replaced with Product Risk-based Testing.

    What specifically is going on here? Was it that you read the title, but not the rest of the post? Let’s read this part again: “100% coverage of a particular factor may be possible—but that requires a model so constrained that we leave out practically everything else that might be important.” (I think you meant to say that “100% coverage is not only not possible…”, by the way.)

    Thank you for your suggestion, and not to put too fine a point on it, but I do want to present this topic, and that being the case, I will say whatever I damned well please. I don’t have control over “the lexicon”, except in the Rapid Software Testing namespace. If I did have control over “the lexicon”, I wouldn’t replace discussions about coverage with “product risk-based testing”, since those are different topics. Meanwhile, if you want to say something different from what I said, feel free to say it on your own blog.

    All the best,

    —Michael B.

    Reply

  7. Agreed but all contractors say 100 percent as a requirement. How do we get away from this?

    Michael replies: My answer to the “100% is a requirement” statement would be the blog post that you’ve just been reading, along with the “further reading” link I provided at the end of it. If pressed, I would respond that I don’t know how to achieve 100% test coverage. James Bach doesn’t know how to do it. Jerry Weinberg doesn’t know how to do it. Cem Kaner doesn’t know how to do it. Other than what I’ve said above, to my knowledge, no one knows how to do it. So, it would be up to the client to explain how they think I could do it.

    Also some measure is required otherwise we wouldn’t know about the depth of coverage. Any straight measures available?

    I don’t know what you mean by a “straight” measure. Can you explain what you mean by that?

    Reply

  10. Hi Michael,

    yes, yes, yes! Also my knowledge is that the coverage is just an indicator of what the code development has done to check that they done the right things. There is and will be IMHO always a need to get other indicators to get an idea of the software quality.

    Well done article(s)!

    Reply

Leave a Comment