Over the last 35 years in the software business, I’ve heard the expression “You can’t test quality into a product.” To support that statement, I’ve seen this quote — or a part of it — repeated from time to time:
Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product. As Harold F. Dodge said, “You can not inspect quality into a product.”
— W. Edwards Deming, Out of the Crisis
The entire quote is from Deming, but people often attribute Dodge’s quote to Deming himself. Nothing wrong with a little misattribution; getting things right would only distract from a good bromide.
Dodge’s statement is typically uttered by someone who wants to dismiss the role that serious testing plays in software development. The statement “inspection does not improve the quality” is true with respect to quality, but it’s obvious and pretty banal. Examining a patient does not improve her health. Interviewing a government official does not improve his ethics. Looking over the books of a corporation doesn’t improve its business practices.
And if testing is the focus of the conversation, the statement is incoherent — since testing is not simply inspection. Testing is more like diagnosis, or investigative journalism, or scientific experiments. These things help us find out truths about the world, but they do not change the world unless someone does something with what the diagnosis, report, or experiment reveals.
Of course you can’t test quality into a product. You can’t diagnose health into a person either; you can’t report ethics into a government; you can’t audit sound business practices into a corporation; and you can’t fly safety into a commercial airliner. All these statements are true and obvious; detecting a cancerous tumour would not inevitably cause the patient to get better.
What’s weird is that some people say “you can’t test quality into a product” to dismiss testing. As they do so, they dismiss why anyone might choose to observe the state of something from a critical perspective, and perform experiments to find deeper truths about what it is, what it does, and how it might entail risk.
So let’s be clear: we don’t test (operate, observe, evaluate) to put quality into something. No even tries to do that, because testing doesn’t put quality in. More testing on its own doesn’t make the product better. Designing, building, and fixing — redesigning and rebuilding — makes the product better. Yet that doesn’t mean that inspection — or testing — is irrelevant, or unnecessary. On the contrary.
We test to find out the actual status of the product, empirically. We test to find vulnerabilities that represent threats to its value. We test to get experience with the built product. We perform experiments to find deeply hidden, rare, subtle, intermittent, emergent, and elusive problems that matter.
In particular, we test to challenge beliefs about the product that might be reckless or risky — or even relatively safe, according to some — but nonetheless false. We test to find out where someone might need to change the product to improve its quality in the eyes of some person(s) who matter.
You can’t test quality into a product; and you can’t critique sound reasoning into a dismissive argument. Similarly, we can’t inspect (or test) quality into a product. But we can test to bring problems out into the light where people can see them — so that the clients of our testing can address quality problems for people we all serve.