If you don’t read the forums on the Software Testing Club, I’d recommend that you consider it. In my view, the STC is one of the more thoughtful venues for conversation about testing. (I’d recommend subscribing to the Software Testing mailing list, too.)
A correspondent recently posted a request for help in recommending an automation approach. I answered something like what follows:
Need to get a code coverage of at least 70% and as it stands, Selenium is probably only covering about 20% as it just tests basic user flows.
I’m going to try to help in a different way, by encouraging you to question some assumptions.
What does “100% code coverage” mean to you? 100% of the code your programmers write? Or 100% of the code that is invoked by the application? Remember, much of the code that your customers will use is in the platform—the code on which your application depends and that depends on the browser(s) and the operating system(s) and the application framework(s) and the plugin(s) and the version(s) of each and the…)
But more importantly, what does it mean to cover the code? To execute each line? Each branch? Each branch, with each condition that might drive code down that branch? Have we covered the code if we run it and it doesn’t crash? If it doesn’t cause a problem that we observe right away? If it doesn’t cause a problem that some automaton can observe? If there’s a problem that our test automation hasn’t been programmed to detect, how are we going to find out about it?
Let’s use an analogy. Suppose that you worked for a police force, and you wanted to monitor what was going on in the city by setting up closed-circuit TV cameras. (This is being done in many cities in the United Kingdom these days. What would it mean to cover everything with CCTV cameras? Would you cover the entire city with cameras? Every street? If you did, would you be able to see inside buildings? Down every alleyway? Inside every rubbish bin? How expensive would this become? How long would it take to install the equipment?
What sorts of problems would you use the cameras to detect? Murder? Assault? Drunk and disorderly conduct? Embezzlement? Pickpockets? Littering? Parking violations?
So suppose you sorted out these problems and you went to the effort of setting up a bunch of cameras. How would the cameras know the difference between bad behaviour and behaviour that wasn’t so bad? They wouldn’t; they’d need human monitors. How many? Would the humans get bored or inattentive watching a bunch of non-events? Would it be more helpful to have some of those humans outside, on the street, actively helping to keep the peace and stopping trouble before it happens?
Now, these questions may sound ridiculous, and to some degree they are. But here’s an important question: at what point would they not become ridiculous? Clearly we couldn’t obtain 100% monitoring of what was going on in the city, so maybe we’d settle for 70%. But 70% of what?
How do we make sure that crime isn’t bursting out all over the place? What kinds of problems aren’t observable by cameras? Are there senses other than vision that humans might bring to bear in identifying problems? Might we find out about certain kinds of problems earlier if we could smell smoke or sewage? Might we be able to identify criminals if we could listen in on their conversations?
But maybe all this monitoring isn’t as helpful as it might be. A city with little crime isn’t the result of total vigilance by the police; it’s the result of various things—many of them little things, many of them social conventions—such that there’s little need to monitor or investigate. When people are well-fed and well-housed and well-educated, when they all have valuable jobs to do, when they’re interdependent and feel responsible to one another, the crime rate goes down. When issues are debated and decided in public, based on consensus, people recognize the value in the rules and conventions, and generally feel less inclined to try to get around them. People tend not to cheat when they feel they’re getting a fair deal. Security comes with freedom, responsibility, visibility, and trust. That’s not infallibly true, but recognizing it can save a lot of effort.
Similarly, in a software development project, some practices are going to make it possible to reduce the emphasis on automation coverage at the GUI level. If your programmers are unit testing their code thoroughly, certain kinds of high-level code coverage aren’t going to be so necessary; in fact, they’ll likely be better tested than if they’re tested via the GUI, since risks and questions about the code can be targeted specifically. If testers are performing tests (interactively or getting help from automation) at some level below the GUI, there’s code coverage happening there too. If testers are operating the product as regular users do, the testers are obtaining some of the kind of coverage that users need. If testers are operating the product according to extreme, unusual, harsh, complex, challenging scenarios, then they’re getting even more of the coverage that users will need.
The thing that usually gets left out of these discussions is that coverage is multi-dimensional. For testers, the issue isn’t just code coverage (statement coverage, or branch coverage, or branch-condition coverage, or…). The issue is test coverage, testing for all kinds of dimensions of value, and code coverage is a manifestation of that. Neither test automation nor code coverage tools will tell you anything about whether a product follows the user’s workflow, or whether an error message was informative and clear, or whether the product has a useful logging system. Neither test automation nor code coverage tools will report on missing functionality, nor will they point out that your product doesn’t support some relevant standard.
It’s not that these tools are unhelpful; on the contrary, they can be very helpful. It’s just that they’re not the be-all and end-all of testing. Test automation (which we define as any use of tools to support testing) makes many impossible things possible, and makes many hard things easy. One of most useful things that code coverage does do: it points us to places where we haven’t run the code at all, and therefore haven’t tested. That might be very interesting.
I’ve written a number of articles on the distinction between code coverage and test coverage. Two of them are here:
http://www.developsense.com/articles/2008-10-GotYouCovered.pdf
http://www.developsense.com/articles/2008-11-CoverOrDiscover.pdf
In addition, Brian Marick (a long time ago, in tester years) wrote an article on how to misuse code coverage tools. That’s here: http://www.exampler.com/testing-com/writings/coverage.pdf.