(This post is adapted from my recent article on LinkedIn.)
Out there in the world, there is a persistent notion that “preventing problems early in the software development process will lead to higher-quality products than testing later will”. That isn’t true.
It’s untrue, but not for the reason that might first occur to most people. The issue is not that addressing problems early on is a bad idea. That’s usually a really good idea.
The issue is the statement is incoherent. Testing on its own, whether done early or late, will not lead to higher quality products at all.
Problem prevention, product improvements, and testing are different pursuits within development work. These activities are related, but testing can neither prevent problems nor improve the product. Something else, beyond testing, must happen.
Coming from me—a teacher and an advocate for skilled testing—that might seem crazy, but it’s true: testing doesn’t improve the product.
Investigative journalism is an information-gathering activity. Investigative journalists reveal problems in companies and governments and groups, problems that affect society.
Awareness of those problems may lead to public concern or outcry. The news reports on their own, though, don’t change anything. Change happens when boards, regulators, politicians, legal systems, leaders, workers, or social groups take action. (You could say that the news reports change perception, or change awareness, and that’s a fine thing; but being aware of a problem doesn’t address the problem on its own.)
Testing, too, is an investigative information-gathering activity. That information can be used to recognize problems in the product (“bugs”), or to identify aspects of the product that do not represent errors but that nonetheless could be improved (“enhancement requests”). Gathering information plays a role in making things better, but it doesn’t make things better intrinsically and automatically.
Consider: weighing yourself doesn’t cause you to lose weight. Blood tests don’t make you healthier. Standardized tests in schools don’t make kids smarter, and certainly don’t improve the quality of education.
What testing can do is to improve our understanding and awareness of whatever might be in front of us. Testing—the process of evaluating a product by learning about it through experiencing, exploring, and experimenting—helps our teams and our clients to become aware of problems. On becoming aware of them, our teams and clients might decide to address them.
To put it another way: testing is questioning a product in order to evaluate it. Neither the questions nor the answers make the product better. People acting on the answers can make the product better.
Similarly, in daily life, a particular reading on a bathroom scale might prompt us to eat more carefully, or to get more exercise, whereupon we might become more fit. A blood test might prompt a doctor to prescribe anti-malarial drugs, and if we take them as prescribed, we’re likely to control the malaria. Those standardized school tests might suggest changes to the curriculum, or to funding for education, or to teacher training. But until someone takes action, the test only improves awareness of the situation, not the situation itself.
In software development, improvement doesn’t happen unlesss someone addresses the problems that testing helps us to discover. Of course, if the problems aren’t discovered, improvement is much less likely to happen—and that’s why testing is so important. Testing helps us to understand the product we’ve got, so we can decide whether it’s the product we want. Where improvement is necessary, testing can reveal the need for improvement.
Some people believe that testing requires us to operate a product, thinking in terms of the product as a built piece of software. That’s a very important kind of testing, but it’s only one kind of testing activity, referring to one kind of product.
It can be helpful to consider a more expansive notion of a product as something that someone has produced. This means that testing can be applied to units or components or mockups or prototypes of an application.
And although we might typically call it review, we can a kind of testing to things people have written, or sketched, or said about a software product that does not yet exist. In these cases, the product is the artifact or the ideas that is represents. In this kind of test, experimentation consists of thought experiments; exploration applies to the product and to the space, or context, in which it is situated; experiencing the product applies to the process of analysis, and to experiences that we could imagine.
The outcome of the test-as-thought-experiment is the evaluation and learning that happens through these activities. That learning can be applied to correcting errors in the design and the development of the product—but once again, it’s the work that happens in response to testing, not the testing itself, that improves the product.
Just as testing doesn’t improve products, testing doesn’t prevent problems either. As testers, we have an abiding faith that there are already problems in anything that we’ve been asked to test. That is, the problems in the product are there before we encounter them. We must believe that problems have not been prevented. Indeed, our belief that problems have not been successfully prevented is a key motivating idea for testing work.
So what good is testing if it can’t prevent problems? Testing can help us to become aware of real problems that are really there. That’s good. That might even be great, because with that awareness, people can make changes to prevent those unprevented problems from going any further, and that’s good too.
It’s a great idea for people who design and build products to try to prevent problems early in development. To the degree that the attempt can be successful, the builders are more likely to develop a high-quality product. Nonetheless, problems can elude even a highly disciplined development process. There are at least two ways to find out if that has happened.
One way is to test the product all the way through its development, from intention to realization. Test the understanding of what the customer really wants, by engaging with a range of customers and learning about what they do. Test the initially fuzzy and ever-sharper vision of the designer, through review and discussion and what-if questions.
Test the code at its smallest units and at every stage of integration, through more review, pairing, static analysis tools, and automated output checking. Check the outputs of the build process for bad or missing components and incorrect settings. These forms of testing are usually not terribly deep. That’s a a good thing, because deep testing may take time, effort, and preparation that can be disruptive to developers. Without deep testing, though, bugs can elude the developers.
So, in parallel to the developers’ testing, assign some people to focus on and to perform deep testing. Deep testing is targeted towards rare, hidden, subtle, intermittent, emergent bugs that can get past the speedy, shallow, non-disruptive testing that developers—quite reasonably—prefer most of the time.
If your problem-prevention and problem-mitigation strategies have been successful, if you’ve been testing all along, and if you’ve built testability into the product, you’ll have a better understanding of it. You’ll also be less likely to encounter shallow problems late in the game. If you don’t have to investigate and report those problems, deep testing can be relatively quicker and easier.
If your problem-prevention and problem-mitigation strategies have been unsuccessful, deep testing is one way to find out. The problems that you discover can be addressed; the builders can make improvements to the product; and problems for the business and for the customer can be prevented before the product ships.
The other way to find out if a problem has eluded your problem prevention processes is to release the product to your otherwise unsuspecting customers, and take the chance that the problems will be both infrequent and insignificant enough that your customers won’t suffer much.
Here are some potential objections:
If software testing does not reveal the need for improvement then the improvement will not happen.
That’s not true. Although testing shines light on things that can be improved, improvement can happen without testing.
Testing can happen without improvement, too. For instance…
- I perform a test. I find a bug in a feature. The program manager says, “I disagree that that’s a bug. We’re not doing anything in response to that report.”
- I perform a test. I find a bug in a feature. The program manager says “I agree that that’s a bug. However, we don’t have time to fix it before we ship. We’ll fix it in the next cycle.”
- I test. I find a bug. The program manager agrees that it’s a bug. The developer tries to fix it, but makes a mistake and the fix is ineffective.
- I test. I find a bug. The program manager agrees, the developer fixes that bug, but along the way introduces new bugs, each of which is worse than the first.
In each case above, 1) Has the product been tested? (Yes.) 2) Has the product been improved? (No.)
Saying that testing doesn’t improve the product diminishes the perceived value of testing.
Saying that testing does improve the product isn’t true, and miscalibrates the role of the tester relative to the people who design, build, and manage the product.
Let’s be straight about this: we play a role in product improvement, and that’s fine and valuable and honourable. Being truthful and appropriately humble about the extents and limits of what testing can actually do diminishes none of its value. We don’t design or develop or improve the product, but we give insight to the people who do.
The value argument in favour of testing is easy to make. As I pointed out above, investigative journalists don’t run governments and don’t set public policy. Would you want them to? Probably not; making policy is appropriately the role of policy-makers.
On the other hand, would you want to live in a society without investigative journalism? Now: would you want to live a world of products that had been released without sufficiently deep testing?
When there’s the risk of loss, harm, bad feelings, or diminished value for people, it’s a good idea to be aware of problems before it’s too late, and that’s where testing helps. Testing on its own neither prevents problems nor improves the product. But testing does make it possible to anticipate problems that need to be prevented, and testing shines light on the places where the product might need to be improved.