Testing Deep and Shallow (1): Coverage

Many years ago, I went on a quest.

“Coverage” seemed to be an important word in testing, but it began to occur to me that I had been thinking about it in a vague, hand-wavey kind of way. I sensed that I was not alone in that.

I wanted to know what people meant by coverage. I wanted to know what I meant by coverage.

In the Rapid Software Testing class material, James Bach had been describing coverage as “the proportion of the product that has been tested”. That didn’t make sense to me.

Could we think of a product in terms of proportions? A product can be a lot of things to a lot of people. We could consider proportions of a product in terms of the number of bytes on a hard drive, but that wouldn’t be very helpful or illuminating. A product is a set of files and modules that contain code that instantiate objects and data and functions; could we take a proportion of that?

A product has interactions with hardware and software, some created by us, and some created by other people. How could we get proportions of the underlying system? A product provides (presumably valuable) functions and features to people. How would we make those things proportional? A product has interfaces, whereby people and programs can interact with it, feed it data, probe its internals, produce output. Can we render those proportionally?

A software product is not a static, tangible thing; it’s a set of relationships. What would 100% of a product, a set of relationships look like? That’s an important question, because unless we know what 100% looks like, the idea of “proportion” doesn’t carry much water.

So, as we do, James and I argued about it.

I went to the testing books. If they referred to coverage at all, most of them begged the question of what coverage is. The books that did describe coverage talked about it in terms of code coverage—lines of code, branches, paths, conditions… Testing Computer Software, for instance, cited Boris Beizer as saying that “testing to the level of ‘complete’ coverage will find, at best, half the bugs”. Huh? How could that make sense?

I eventually found a copy, in India, of Beizer’s Software Testing Techniques, which contained this intriuging hint in the index: “any metric of completeness with respect to a test selection criterion”. While the book talked about code coverage, it also talked about paths in terms of functional flows through the program.

James argued that “any metric of completeness with respect to a test selection criterion” wasn’t very helpful either. “Test selection criteria” are always based on some model of the product, he said.

A model is a an idea, activity, or object (such as an idea in your mind, a diagram, a list of words, a spreadsheet, a person, a toy, an equation, a demonstration, or a program…) that represents—literally, re-presents—something complex in terms of something simpler. By understanding something about the simpler thing, a good model can give us leverage on understanding the more complex thing.

There are as many ways to model a software product as there are ways to represent it, or its parts, or the things to which it relates. For instance: we can model a product by representing its components, in a diagram. We can model a product by describing it in a requirements document—which is itself a model of the requirements for the product. We can represent the information stored by a product by way of a database schema.

We can model a product in terms of its interfaces—APIs and command lines and GUIs and network protocols and printer ports. We can represent people’s interactions with a product by means of flowcharts, user stories, tutorials, or task lists. And of course, we are always modeling a product tacitly, with sets of ideas in our heads. We can represent those ideas in any number of ways.

The code is not the product. The product is that set of relationships between software, hardware, people, and their needs and desires, individually and in social groups. The code for the product is itself a model of the product. Code coverage is one way to describe how we’ve covered the product with testing.

And somewhere, in all of that back-and-forth discussion between James and me, a light began to dawn.

In the Rapid Software Testing namespace, when we’re talking about coverage generally,

Coverage is how thoroughly we have examined the product with respect to some model related to the product.

When we’re speaking about some kind of coverage, we’re referring the thoroughness of testing with respect to a specific way of modeling the product.

• Functional coverage is how thoroughly we have examined the product with respect to some model of the functions in the product.
• Requirements coverage is how thoroughly we have examined the product with respect to some model of the requirements.
• Performance coverage is how thoroughly we have examined the product with respect to some model of performance.
• Risk coverage is how thoroughly we have examined the product with respect to some model of risk.

Therefore: Code coverage is how thoroughly we have examined the product with respect to some model of the code.

It should be plain to see that code coverage is not the same as risk coverage; that covering the code doesn’t cover all of the possible risks that might beset a product. It should be equally clear that risk coverage (how thoroughly we have examined the product with respect to some model of risk) doesn’t necessarily cover all the code, either.

Which brings us to the next exciting installment: what we mean by deep and shallow testing.

Related: Talking About Coverage