Quality: Not Merely The Absence Of Bugs

“Quality is value to some person.” —Jerry Weinberg

In the agile-testing mailing list, Steven Gordon says “The reality is that meeting the actual needs of the market beats quality (which is why we observe so many low quality systems surviving in the wild). Get over it. Just focus on how to attain the most quality we can while still delivering fast enough to discover, evolve and implement the right requirements faster than our competitors.” Ron Jeffries disagrees strongly, and responds in this blog post.

I think Steven is incorrect, because meeting the needs of the market doesn’t “beat quality”. A product that doesn’t meet the needs of the market (or at least of its own customers) is by definition not a quality product. Steven errs, in my view, by suggesting that “low quality systems” survive. Systems survive when, as buggy as they might be, they supply some value to someone. Otherwise, those systems would die. What Steven means, I think, is that these products fail in some dimensions of quality, for some people, while supplying at least some value for some other people.

Yet I think Ron is incorrect too when he claims that there must not be a trade-off between speed and quality, for the same reason; speed is also a dimension of quality, value to some person. So if I’ve offended either Steven or Ron, I hasten to point out that I’m probably offending the other equally.

I think it’s a mistake to suggest that quality is merely the absence of bugs, as both appear to suggest. Here’s why:

  • “Some person” is a variable; there are many “some persons” in every project community. The client and the end user are examples of “some person”, but so are the programmers, the managers, the testers, the documenters, the support people, etc., etc.
  • Value to a given person is multivariate (that is, for each person there is a collection of variables, several things that that person might value in varying degrees).
  • Capability and functionality are important dimensions of value to some person(s).
  • Rapid iteration and time to delivery are dimensions of value to some person(s).
  • Security, reliability, usability, scalability, performance, compatibility, maintainability and many other -ilities are also dimensions of quality, some of which may be of paramount importance and some of which are of lower importance to some person(s).
  • The absence of bugs is one (and only one) dimension of value to some person(s), if it’s even that. It’s more accurate, I contend, to think of bugs as things that threaten or limit value. For this reason…
  • We get severely mixed up when we describe “quality” solely in terms of the absence of bugs.
  • The absence of bugs might matter less, much less, than the presence of other things that are valuable. Despite the protestations of some “quality assurance” people who are neither managers nor business people, it might not be insane to value features over fixes. Questionable, I would argue, but among other things, wouldn’t the judgment depend on the severity of the problem and the risk of the fix?
  • The absence of bugs is completely irrelevant if the software doesn’t provide value to some person(s). A bug-free program that nobody cares about is a lousy program.

These differing views of value mean that there will be differing views on the notion of working software (also known as valuable software). How do we handle these different views when they compete? By responding to change with customer collaboration which happens by interactions between individuals. That’s what “agile” is supposed to mean. It’s not just about shorter morning meetings with no chairs; it’s about human approaches to solving human problems all day long. Well… it used to be, maybe, for a while. Maybe not any more.

So I disagree with Ron when he suggests that there isn’t a trade-off between time to delivery and quality. That’s because time to delivery isn’t distinct from quality either; it’s another dimension of quality. And there’s always a trade-off between all of the dimensions of quality, depending on what people value, who and what informs the ultimate decisions, and who has the power to make those decisions.

I do agree strongly with Ron, though, when he suggests that the presence of problems in the code is a serious threat to the many of the other dimensions of quality, and that reducing those problems as early as possible tends to be a good investment of time. In his blog post, he has articulated numerous ways in which those problems threaten the ability of the programmers to do valuable work. Test-driven development and unit tests can be powerful ways of avoiding these problems. Collaboration and technical review—pair programming, walkthroughs, inspections, knowledge crunching sessions (as Eric Evans calls them in Domain Driven Design)—not only help to prevent problems, but also afford opportunities for people to learn and exchange knowledge about the product.

I’m not in the business of telling programmers how to do their work, but as a tester I can say that problems in the code threaten the quality of our work too. Specifically, they constrain the testers’ ability to provide value in the form of knowledge about the product. Testers are often asked, “Why didn’t you find that bug?” One plausible answer is “because we were so busy finding other bugs.” Well-programmed code, already tested to some degree by the programmers themselves, is enormously important for the testers. Bugs block our ability to observe certain parts of the program, they add uncertainty and noise to our observations of the system, and they cause us to spend time in investigation and reporting of the problems. This represents opportunity cost; bug investigation and reporting compromises our capacity to investigate and cover the rest of the test space, which in turn gives bugs more time and more places in which to hide out.

So a well-tested program is easier to explore more quickly. Having a hard time persuading your manager or your (cough, cough) Scrum team? This presentation on finding bugs vs. coverage sets out the problem from the point of view of the testers. As usual, the business decisions are for the those who manage the project. It’s up to us—the programmers, the testers, and the other developers on the project—to present the technical risks in the context of the business risks. It’s up to all of us to collaborate on balancing them.

6 replies to “Quality: Not Merely The Absence Of Bugs”

  1. I’m struggling with your following 2 reasons of why quality is more than the absence of bugs:
    # Capability and functionality are important dimensions of value to some person(s).
    # Rapid iteration and time to delivery are dimensions of value to some person(s).

    Together with Jerry Weinberg’s quoted definition of quality (“Quality is value to some person.”), it suggests that these are dimensions of quality.

    I agree that Quality is Value to some person, but can Quality and Value be interchanged? Does all that is of value to that person falls under the flag of quality?

    I doubt it, but that is more a feeling. Intuitively I think that if a feature is missing, e.g. a car without a radio, the value for some person is less, but not the quality.
    Similarly if a person orders a car, but that car is delivered 1 month later, the quality of the car is not changed (at least not by the sole fact of later delivery).

    Just my thoughts…

  2. >I agree that Quality is Value to >some person, but can Quality and >Value be interchanged? Does all >that is of value to that person >falls under the flag of quality?

    My opinion is that people should talk more about what they value and less about what they have determined is quality. Quality seems to be used in a judgmental way. It is like people use it to mask what they value and instead make a blanket statement about what is right.

    Let's take your car scenario as an example. I am unsure what you mean by "quality". I know that I am assuming that you mean the strength of the frame, the aesthetics, the safety are what you consider "quality", but I think it would me more safe for you to say that you value those things and then talk about at what level of strength and safety you require a car to be for you to value it.

    Some people won't value a radio in a car so their judgment of the quality won't decrease when the car doesn't come with it. Others don't value strength in the parts, or the safety of the vehicle so much as they value gas mileage or the ability to drive down very narrow roads. A "well-made" American car in the crowded and narrow streets of Europe might not be "quality" in the opinion of smart car drivers.

    In short I think the word "quality" is a shortcut. If you know what I value in a car and I say that a car is "quality" then you know what I mean. Otherwise, you can only assume what I mean by the word.

  3. Michael, when Ron is saying “quality” here, I think he’s mostly referring to internal quality–the quality of the code being clean, flexible, easily extended, and easily modified. It’s different from the absence of discernible bugs. They are different sorts of value to different (though overlapping) groups of people.

    It is internal quality that enables us to go faster. You could have that, but still have bugs.

    Lack of visible bugs can be extraordinarily expensive if achieved without internal quality. I’ve seen companies try to achieve this by patching every duplicate path through the code rather than by removing the duplication.

    The two types of quality are somewhat orthogonal.

    I’ve weighed in on the topic from the perspective of “what to do about it” at

    – George

  4. To Arjan:

    I was going to respond to you, but Joe did an excellent job of it.

    In short I think the word “quality” is a shortcut. If you know what I value in a car and I say that a car is “quality” then you know what I mean. Otherwise, you can only assume what I mean by the word. … so you have to question that. See how that’s reflected in George’s response.

    Michael, when Ron is saying “quality” here, I think he’s mostly referring to internal quality–the quality of the code being clean, flexible, easily extended, and easily modified.

    Those are some of the "problems in the code" that I was referring to in the original blog post. (Note that I referred to "problems in the code" far more often than "bugs".) I agree with you, George, that these are all important aspects of quality to the programmer. (I wrap most of them up under the rubric of "maintainability", the extent to which the code is readable, coherent, easily adaptable, easily fixable, well-understood—plus the business that I was referring to about few bugs.)

    As George correctly notes, the dimension of "few bugs" and "elegant code" might be somewhat orthogonal; and that which the programmer values might be orthogonal to that which the client wants, what the testers want, what the end-user wants. The agile-testing thread that prompted all this was one that started with someone saying "the people I work with don't care about quality!" as if they had no values. My point is that they do have values, but that something in the company culture—the mindset, the reward system, the management focus—is allowing people to take rationally but radically divergent views about what constitutes quality. That can get sorted out when (and only when) we sort out what we value in common.

    There's been a recent meme shift; some people on the agile-testing list have suddenly been writing differently about value, noticing that it's at the core of what we need to think about in service organizations and in teams. I find this encouraging.


Leave a Comment