Quality Engineering Is Not Testing

There are subtle but serious problems in the testing business right now, and if we don’t attend to them they will lead to bigger problems. The core problem I’m attending to today is confusion between testing and quality engineering.

Engineering

Let me start by saying I believe that quality engineering is a good thing. Or rather, I believe that engineering is a good thing. As Billy Vaughan Koen said in his wonderful book, Discussion of the Method, engineering is the universal problem solving method. Specifically, he said, the engineering method is:

the use of heuristics to cause the best change in a poorly understood situation within the available resources.

Koen, Billy Vaughan, Discussion of the Method, Oxford University Press, 2003, p. 28

This is a very broad definition, but we often see “engineering” used in a broad sense. Koen provides examples: “the reigning prince engineered the divorce of his daughter”; “doctors have engineered a better bacterial host”. In that sense, we humans are all engineers, applying imperfect methods to solving problems as best we can.

(Why imperfect methods? Do you know of a perfect method for accomplishing anything in every context? We use imperfect methods because those are the only methods that are available.)

When we are engineering, we’re engineering while aspiring to provide quality — value to some person. That is: anyone who is doing engineering work is doing quality engineering, even if the value is only to one’s self. So “quality engineering” is a pleonasm — an expression that uses more words than strictly necessary to convey meaning, like “vegetarian broccoli”, “solid ice cube”, or “imperfect heuristics”.

Words are important, because they reflect and influence our ideas. Muddy terms lead to muddy thinking. This is not a new problem. Orwell wrote about in the 1940s, in Politics and the English Language.

(The English language) becomes ugly and inaccurate because our thoughts are foolish, but the slovenliness of our language makes it easier for us to have foolish thoughts.

Confucius wrote about it in Analects in the 400s BCE.

“If names be not correct, language is not in accordance with the truth of things.”

Analects, Book XIII, Chatper 3

Quality and Testing Aren’t the Same

Some people in the software development community — mostly testers, so it seems — are using the words “quality” and “testing” carelessly and interchangeably. Here’s an example, picked from LinkedIn after 15 seconds of searching on “testing quality”:

“Quality isn’t just about finding bugs it’s about building confidence.”

Here’s another from a video blog:

“SmartBear’s recent research found that seven out of ten respondents are concerned about quality keeping up with the speed of AI-driven code creation.”

“Quality ceases to be a function and becomes the mechanism through which enterprises sustain velocity without compromising stability or compliance.”

Now, you might think, “Well… quality, testing; tomayto, tomahto. Same diff, right?”

No. NO. Please pay attention.

Quality can’t cease to be a function, because it never was a function. Quality is not something that keeps up (or doesn’t) with code generation. It’s not that quality “isn’t just about finding bugs”; quality is not about finding bugs at all, nor is it something that builds confidence.

Quality is value to some person(s). It’s not a property of a product; not a function, nor a task. Quality is a set of relationships between a product or system, its context and setting, people, and their needs and desires. Those relationships are constantly in flux. They change over time. Today’s cutting-edge product is tomorrow’s shovelware.

Quality for one person is not the same as quality for another. The developer’s perspective on the quality of a software product (“for instance, “is it easy to maintain?”) is different from some user’s perspective (“does it solve my problem for me?”). But users’ notions of quality are different too. Even dimensions of quality like “usability” differ from one person and context to the next. A novice user might see usability in terms of ease of learning (like a new driver in a car with automatic transmission); an expert user might see usability in terms of the capacity to control the system (like the driver of a truck with 18-gear transmissions).

Testing is not quality. Testing the process of evaluating a product by learning about it through experiencing, exploring and experimenting. Testing is one of the ways in which we come to know about the quality of a product.

There’s an important difference between the state of something, and how we come to know about the state of something. This is completely obvious in domains other than software development and testing. Health is one thing; diagnosis is another. Flavour is one thing; tasting is another. Government integrity is different from auditing, and from investigative journalism. The ability to drive is one thing; driver examination is something different.

All of the things I’ve mentioned are important. We want health, but without diagnosis, we don’t know about deep problems in the state of our health. We like our food to be delicious, but until we taste it, we can’t be sure about its flavour. We want ethical governments, but without auditing and journalism, we are likely to be unaware of deep problems in the state of our governments. We want good drivers, but without good driving exams, we don’t find out early enough which drivers shouldn’t be on the road. (And if you want to counter with “But look at all the bad drivers out there!”, I’ll reply, “Then I guess you’ll agree that the exams aren’t halfway stringent enough.”)

Quality Strategy is Not Test Strategy

Quality is not the same thing as testing. Yet some people refer to a quality strategy as a test strategy as though they were the same. As I’ve said before, and as should be obvious, they’re not. A quality strategy refers to the ways we design, develop, and manage projects to develop products that have value to people. That is: a quality strategy is a set of ideas focused on achieving quality goals. Applying a good quality strategy helps us to do good work, and to develop good feelings about the bits and pieces and precursors to the product. A good quality strategy is focused on achieving success.

A test strategy refers to testing — a set of ideas for examining, exploring, experiencing, and experimenting with the product. The primary — yet often unspoken — goal of testing is to discover situations and aspects of the product where quality goals haven’t been achieved. Applying a good test strategy includes challenging and testing what we’re designing, building and, managing every step of the way. That is: a good test strategy is focused on finding trouble.

As a complete product emerges from the development process, we may feel pretty good about it. Yet unless we’ve been testing all the way along, and until we can engage with the final product, all of what we know about it is theory. Some problems are subtle, rare, intermittent, condition-dependent, platform-depended. And some problems are emergent — not present in any particular precursor or component in isolation, but manifesting in the relationships between them or in what the product present to humans who are using it. Testing helps us to learn about the actual, built product — and in particular, problems that the quality strategy hasn’t addressed successfully.

The quality strategy and the test strategy can and should influence each other. But when we confuse them — which happens when squeamish people try to avoid the T word — I can guarantee that important test strategy and testing work won’t get done; won’t even be considered. Without that work, even highly disciplined teams risk being unpleasantly surprised by problems that hurt their customers.

For Testing, Quality is a Distraction

If what I’ve said hasn’t blown your mind already, this might: effective testing is not a search for quality. Testing is a search for truth and trouble — the true status of the product, and problems that threaten its value. Looking for quality is a distraction from that goal.

That may sound crazy, but consider: when your attention is directed towards something (like quality), it is inevitably directed away from other things. In our Rapid Software Testing classes, we make this explicit: when your job is to find needles in the haystack, they’ll be harder to find when you’re focused on the hay.

Testing is Not Quality Engineering

Today’s problem is a new version of an old problem — the “quality assurance” label — that I wrote about in 2010, and then again in 2024. It’s old, vinegared wine in the same old dusty bottles, but with a new label: quality engineering. That’s how some people in and around testing want to label it. Why? I believe there are several reasons.

• Some testers don’t want to be thought of as doing “just testing”, by which, so it seems, they mean “just executing formally scripted, procedurally structured test cases”. Or something. But that is and always has been an impoverished view of what testing is, and what testers do — as though developers doing “just developing” were typing and compiling code.
  
  For instance, in a recent LinkedIn post, someone suggested that quality engineers, or “QA” or even more weirdly, “owners” ask questions like “What could go wrong?” “What assumptions are we making?” ‘What happens when real users do something unexpected?” — and implied that testers don’t. Aren’t those questions that testers are asking all the time, throughout the project? What kind of tester doesn’t ask those questions? If those questions are not at the forefront of a tester’s mind and work process, they’re not testing.
  
• Some testers want different (and, in their minds, perhaps more glamourous) work than testing: fixing product code, managing build pipelines, writing requirement documents, or designing the product. And that’s fine; any person can contribute to these kinds of work, and these kinds of work are very valuable.
  
  But while someone is doing fixing, building, managing, writing, or design, they’re not doing testing; time and focus on testing work is being displaced.
• Some testers perceive that the title “tester” has a poor reputation. I have some sympathy for that. Through the history of software development, testing and the search for errors has been sidelined at best. Jerry Weinberg recounted that people were dismayed by his chapter on errors in Computer Programming Fundamentals in 1961!
  
  For far too long, testing has been reduced to rote repetition of test cases, leaving out the essence of testing: exploration, discovery, investigation, and learning. The Agile movement, focused on developers, marginalized testing and testers; neither the Manifesto for Agile Software Development nor the Principles that followed mentioned testing at all. And the ISTQB reduced testing to a glossary, and qualification of testers to a multiple choice test where a 62.5% grade was sufficient to pass. (Never mind evaluating a person’s complex set of technical and social skills; would you even evaluate a piece of software that way?)

So why not make testing sound better by rebranding it as “quality engineering”?

Here are four reasons to resist that temptation:

1. Quality engineering may be informed by testing, but “quality engineering” is not a description of what testers do. The term “quality engineering” is a description of is what the people actually designing, building and managing the product do.
2. The term “quality engineering” applied to testing is misrepresentation. Testing is about investigating, or assessing, or evaluating quality, not engineering it. Again, testing may inform quality engineering, just as auditing informs a government or a business of what might need to change.
3. The term “quality engineering” applied to testing is a form of usurpation. If testers are quality engineers, are the people who are actually building the product and running the project not quality engineers? Isn’t it presumptuous — even insulting — to insinuate that we testers are quality engineers and other people aren’t?
4. Rebranding something to make it sound better is not the same as making it better. Will calling testing “quality engineering” improve the state of testing? Or is it simply the latest attempt to avoid saying the T-word?

“I Still Don’t Like Saying ‘Testing’!”

Okay, so you’d prefer grander words? How about some titles or job descriptions that hew closer to the truth of what we do:

• Risk Investigator (credit to Sam Connelly for this)
• Quality Assessment
• System Analysis
• Quality Awareness
• Risk Analyst
• Product Investigation
• Toolsmith (or Testing Toolsmith)
• Applied Critical Thinking
• Problem Investigation
• Risk Research
• Problem Detection
• Product Exploration
• Software Diagnosis
• Forensic Software Investigation
• Troublehunting

I bet some of those sound awkward. I like “risk investigator”, myself. But in the end, I prefer testing, and tester.

As testers, we help people to see things as they are. We make informed decisions about quality possible by enacting critical thinking about software. We do that by getting experience with the software, exploring it, and performing experiments on it. Let’s not make the claim that we engineer quality, because we don’t. We can, however, be enormously helpful to the people who do.

Related reading:

Blog Post: Testing Is Not Quality; Quality Is Not Testing

Blog Post: Quality Assurance and Testing

Blog Post: Testers, Get Out of the Quality Assurance Business

Blog Post: Everyone is NOT Responsible for Quality (James Bach, Satisfice)

RST Supplementary Document: Why Testers? https://www.satisfice.com/download/why-testers