Why Do Some Testers Find The Critical Problems?

Today, someone on Twitter pointed to an interesting blog post by Alan Page of Microsoft. He says:

“How do testers determine if a bug is a bug anyone would care about vs. a bug that directly impacts quality (or the customers perception of quality)? (or something in between?) Of course, testers should report anything that may annoy a user, but learning to differentiate between an ‘it could be better’ bug and a ‘oh-my-gosh-fix-this’ bug is a skill that some testers seem to learn slowly. … “So what is it that makes some testers zero in on critical issues, while others get lost in the weeds?”

I believe I have some answers to this. My answers are based on roughly 20 years of observation and experience in consulting, training, and working with other testers. The forms of interaction have included in-class training; online coaching via video, voice, and text; face-to-face conversation in workplaces, conferences, and workshops; direct collaboration with other working testers in mass-market commercial software, financial services, retail services, specialized mathematical applications, and several other domains.

My first answer is that testing, for a long time and in many places, has been myopically focused on functional correctness, rather than on value to people. Cem Kaner discusses this issue in his talk Software Testing as a Social Science, and later variations on it. This problem in testing is a subset of a larger problem in computer science and software engineering. Introductory texts often observe that a computer program is “a set of instructions for a computer”. Kaner’s definition of a computer program as “a communication among several humans and computers, distributed over distance and time, that contains instructions that can be executed a computer” goes some distance towards addressing the problem; his explication that “the point of the program is to provide value to the stakeholders” goes further still. When the definition of programming is reduced to producing “a set of instructions for a computer”, it misses the point—value to people—and when testing is reduced to the checking of those instructions, the “testing” will miss the same point. I’ve suggested in recent talks that testing is “the investigation of systems composed of people, computer programs, related products and services.” Successful testers avoid a fascination with functional correctness, and focus on ways in which people might obtain value from a program—or have their value unfulfilled or threatened.

This first answer gives rise to my second: that when testing is focused on functional correctness, it becomes a confirmatory, verification-oriented task, rather than an exploratory, discovery-oriented set of processes. This is not a new problem. It’s old enough that Glenford Myers tried (more or less unsuccessfully, it seems) to argue against it in The Art of Software Testing in 1979. Myers’ point was the testing should be premised on trying to expose the program’s failures, rather than on trying to confirm that it works. Psychological research before and since Myers’ book (in particular Klayman and Ha’s paper on confirmation bias) shows that the positive test heuristic biases people towards choosing tests that demonstrate fit with a working hypothesis (showing THAT it works), rather than tests that drive towards final rule discovery (showing how it works, and more important, how it might fail). Worse yet, I’ve heard numerous reports of development and test managers urging testers to “make sure the tests pass”. The trouble with passing tests is that they don’t expose threats to value. Every function in the program code might be checked and found correct, but the product might be unusable. As in Alan’s example, the phone might make calls perfectly, but unless we model the way people actually use the product—talking for more than three minutes at a time, say—we will miss important problems. Every function might work perfectly, but we might fail to observe missing functionality. Every function might work perfectly, but we might miss terrible compatibility problems. Functional correctness is a very important thing in computer software, but it’s not the only thing. (See the “Quality Criteria” section of the Heuristic Test Strategy Model for suggestions.) Testers “who zero in critical issues” avoid the confirmation trap.

My third answer (related to the first two) is that when testing is focused on confirming functional correctness, a lot of other information gets left lying on the table. Testing becomes a search for finding errors, rather than on finding issues. That is, testers become oriented towards reporting bugs, and less oriented towards the discovery of issues—things that aren’t bugs, necessarily, but that threaten the value of testing and of the project generally. I’ve written recently about issues here. Successful testers recognize issues that represent obstacles to their missions and strategies, and work around them or seek help.

My fourth answer is that many (in my unscientific sample, most) testers are poorly versed in the skills of test framing. This is understandable, at least in part because test framing itself wasn’t known by that name as recently as a year ago as I write. Test framing is the set of logical connections that structure and inform a test. It involves the capacity to follow and express a line of perhaps informal yet reasonably structured logic that directly links the testing mission to the tests and their results. In my experience, most testers are unable to trace this logical line quickly and expertly. There are many roots for this problem. The earlier answers above provide part of the explanation; the mission of value to the customer is overwhelmed by the mission of proving functional correctness. In situations where the process of test design is separated from test execution (as in environments that take a highly scripted approach to testing), the steps to perform the test and observe the results are typically listed explicitly, but the motivation for performing the test is often left out. In situations where test execution, observation of outcomes, and reporting of test results is heavily delegated to automation, motivation is even further disconnected from the mission. In such environments, focus is directed towards getting the automation to follow a script, rather using than automation to assist in probing for problems. In such environments, focus is often on the quantity of tests or the quantity of bug reports, rather than on the quality, the value, of the information revealed by testing. Testers who find problems successfully can link tests, test activities, and test results to the mission. They’re far more concerned about the quality of the information they provide than the quantity.

My fifth answer is that in many organizations there is insufficient diversity of tester skills, mindsets, and approaches for finding the great diversity of problems that might lurk in the product. This problem starts in various ways. In some organizations, testers are drawn exclusively from the business. In others, testers are required to have programming skills before they can be considered for the job. And then things get left out. Testers who need training or experience in the business domain don’t get it, and are kept separated from the business people (that’s a classic example of an issue). Testers aren’t given training in software design, programming, or related skills. They’re not given training in testing, problem reporting and bug advocacy, design of experiments. They’re not given training or education in anthropology, critical thinking, systems thinking, or philosophy and other disciplines that inform excellent testing. Successful testers tend to take on diversified skills, knowledge, and tactics, and when those skills are lacking, they collaborate with people who have them.

Note that I’m not suggesting here that anyone become a Donald Knuth-level programmer, a Pierre Bourdieu-league anthropologist, a Ross Ashby-class systems thinker, a Wittgenstein-grade philosopher. I am suggesting that testers be given sufficient training and opportunity to learn to program to the level of Brian Marick’s Everyday Scripting with Ruby, and that they be given classes, experience, and challenges in observation, the business domain, systems thinking and critical thinking. I am suggesting that people who are testing computer software do need some exposure to core ideas about logic (if we see this, can we justifiably infer that?), about ontology (what are our systems of knowledge about the way things work—especially related to computer programs and to testing), and about epistemology (how do we know what we know?).

I’ve been told by people involved in the design of testing standards that “you can’t expect regular testers to learn epistemology, for goodness’ sake”. Well, I’m saying that we can and that we must at least provide opportunities for learning, to the degree that testers can frame their mission, their ideas about risk, their testing, and their evaluation of the product in the ways that their clients value. Moreover, I’ve worked with testing organizations that have done that, and the results have been impressive. Sometimes I hear people saying “what if we train our testers and they leave?” As one wag on Twitter replied (I wish I knew who), “What if you don’t train them and they stay?”

In our classes, James Bach and I have the experience of inspriring testers to become interested in and excited by these topics. We find that it’s not hard to do that. We remain concerned about the capacity of some organizations to sustain that enthusiasm, often because some middle managers’ misconceptions about the practice and value of testing can squash both enthusiasm and value in a hurry. Testers, to be successful, must be given the freedom and responsibility to explore and to contribute what they’ve learned back to their team and to the rest of the organization.

So, what would we advise?

Read this set of ideas as a system, rather than as a linear list:

  • The purpose of testing is to identify threats to the value of the program. Functional errors are only one kind of threat to the value of the program.
  • Take on expansive ideas about what might constitute—or threaten—the quality of the product.
  • Dynamically manage your focus to exercise the product and test those ideas about value.
  • In hiring, staffing, and training, focus on the mindset and the skill set of the individual tester as a member of a highly diversified team.
  • As an individual tester, develop and diversify your skills and your strategies.
  • Immediately identify report issues that threaten the value of the testing effort and of the project generally. Solve the ones you can; raise team and management awareness of the costs and risks of issues, in order to get attention and help.
  • Learn to frame your testing and to compose, edit, narrate and justify a compelling testing story.
  • Don’t try to control or restrain testers; grant them the freedom—along with the responsibility to discover what they will. Given that… they will.

14 replies to “Why Do Some Testers Find The Critical Problems?”

  1. […] This post was mentioned on Twitter by Michael Bolton, AlanPage, Adam Knight, ronpih, Pete Walen and others. Pete Walen said: Interesting post @michaelbolton Had a similar quandry yesterday @ day-job #testing #qa #agile . […]

  2. My advise is the “rule of three”, for instance: do test case identification in three approaches – functions/requirements, risks/importance/mission?, applications… depending on the setting. It will get you going in the direction of the above.

    You have elaborated briliantly on my preffered interview question: “Are you OK on working on a range of things?” Some specialization is good, but can’t be taken for granted over time, as I tried to convey through Software testing is a skill of many skills (—jesper-ottosen.aspx#comments)

    Now I must go and “bring back to the team and the rest of the organization”

  3. You wrote:
    I’ve suggested in recent talks that testing is “the investigation of systems composed of people, computer programs, related products and services.” Successful testers avoid a fascination with functional correctness, and focus on ways in which people might obtain value from a program—or have their value unfulfilled or threatened.

    This also is exactly why i hardly ever use the term “software tester” to describe what i do. Most of the time I prefer to stick to “tester”. I test more than just software, or a program. (which in turn is also a small suggestion for your advice: maybe it’s better to refer to a “system” instead of a “program”).

    Michael replies: I often add “software” to tester when the context warrants it. I often refer to “systems” instead of “programs”, too—thanks for the reminder.

  4. Great post , as always !

    On your point about
    “As an individual tester, develop and diversify your skills and your strategies”

    Diversification of skills in terms of acquiring (and implementing) knowledge about “how” the product’s is used by customers.
    This crucial information has helped me finding several critical problems which without the customer context would have been average functional failures.

    In my company a lot of senior testers are continuously pushing testers to have an active learning relationship with “customer advocates” .
    On lines of “what would this failure mean for xyz customer” ,”would they worry about it ? ” etc .
    The good thing is that these “customer advocates” are from across all departments – requirements,marketing,product managers,senior developers or even actual customers in some cases.
    And the best things is , these communications (the effective ones) do not involve any pointing to documents, just quick chat “gyan” 🙂

  5. “Finding” I don’t know about, probably that’s about developing skill in context-driven, exploratory testing. But once found, the “sorting” into fix-now versus fix-later is a keen interest of mine.

    From what I’ve seen, and as others have pointed out, some folks think about what they observe (the bugs they find) in the larger context (a la your Kaner quote above).

    When a tester (programmer, business analyst, etc.) comes to me with a “bug” and proposes fixing it (on the spot usually, dropping everything else) I say:

    “Tell me about the bad things that will happen because of this bug”.

    At first they talk about this exception, that error message, the other database inconsistency, and so on. So I ask them to tell me about what will happen in the *real world*. One team I worked with was responsible for a medical record application, I would ask them, “How many patients will die because of this bug?” They were always shocked! “None”, they would answer. “Good! At least we’ve established an upper limit on the severity”.

    Michael replies: An important notion, to be sure. But I’d counsel caution for testers in terms of understating the severity of the bug. The issue here is that what we see is the observation of a symptom, not the cause. When I see an exception, an error message, or a database inconsistency, I might know that it’s happening, but I don’t know why. Moreover, it’s risky to extrapolate the consequences. In a medical record application, the database inconsistency might mean that the patient’s home phone number and business phone number are switched—or it might mean that some’s person’s record gets associated with someone else’s medication.

    Further questions were along the lines of: “What will the customer *do* if they run into this bug?”, “How will they work around it?”, “How many hours, days, weeks will it take to clean up the database, if the problem occurs?”. One of my favorites questions is, “How long has it been doing this, and what has the effect been so far?”. Often the answer is – a long time, and not much.

    That can be valuable, but it can also be misleading. 2007: “How long have we been lending money to people who normally wouldn’t qualify for them, and what have we seen as the consequences of that?” “A long time, and not much.”

    Sorting these correctly frees up time and mental energy to focus on things that matter most.

    I agree, to a point—after the cause, rather than the symptom, has been determined. I’d suggest having a look at my blog post, A Letter to the Programmer—especially the Feynman quote within—and The Black Swan: The Impact of the Highly Improbable.

    The insight folks start to get is that our software isn’t just “a set of instructions for a computer”, but actually does something in the real world. The corollary good that comes of this insight is that they get better at seeing the value of the “good things” the software does, along with perspective on the “bad things” it does. It always does both.

    I agree that awareness of what the program does in its context, and that value that it provides to people, is a good thing.

  6. Thanks for the response.

    I am looking for some more prescriptive test cases. Since we are talking in generalities, wouldnt you agree that the best tester is the one that creates the requirement?

    Michael replies: I don’t know what you mean by “more prescriptive test cases”. Can you give me an example?

    I wouldn’t agree that the best tester is the one that creates the requirement. I don’t understand the context that you’re apparently proposing, in which the tester would create the requirement. Doesn’t the customer—the person for whom the software is being created, or the product owner—create the requirement? Do you mean that the best tester is the one who writes a line in a requirements document?

    Thank you for the thought provoking article.

    You’re welcome.

  7. My short answer to the original question is:
    The testers that know what is important.

    These are also the testers that can make the best test strategies, test design, and test execution, with the right deviations to find most of the important stuff about the product.

    But who knows what is important?
    The traditional answer is: “not needed for testers, this is already covered in the approved requirements”.
    The best, and latest, answer is this blog post, which is brilliant.
    From Page’s blog, you can add domain knowledge (understanding what is important in the area of the product’s usage.)
    Other important additions could be knowledge about the product, and the technology that is used.
    And the ability to find more sources of information that point to what is important about the product.

    Sounds difficult? Yeah, it is. There are so many things that are important about software, so prepare for a long and fun learning journey.

  8. Thanks for the post Michael. I found it very interesting. Bolding the final sentence in each paragraph is a clever idea. It puts a nice emphasis on the conclusion of each paragraph and makes it easy to re-read the key points of the article.

    Michael replies: Thank you for saying so. Note that providing a concise summary of key points is an important testing skill, whether you’re reporting bugs or issues.

    Your first and second answers got me thinking about why some testers might have a fascination with testing for functional correctness. One of my conclusions was this: Testing for functional correctness is easy. It involves tests you can often see with your eyes and easily identify, making test ideas come naturally. Additionally, by physically looking at what is being tested, a tester can see what he/she has tested, therefore giving the illusion of coverage. I think inexperienced testers that are unsure of what testing would be most effective would naturally gravitate to easier testing tasks, which would include functional correctness testing.

    Yes, I think you’re right about that—and not just with respect to junior testers, but also (alas) more senior managers.

    Finally, I really think you’re onto something with your definition of “Issues”. I have observed testers, including myself, who will overlook problems or assume they will be dealt with without needing to raise an issue because the problem is not a software bug. With this new (or at least new to me) definition, I think I will be more aware of issues and help other testers become aware as well.

    Yes. When an issue remains buried below the level of awareness, you can usually expect to hear someone ask eventually, “Why didn’t you find that bug?” When you answer that a buried issue blocked you or slowed you down, you can expect to hear, “Why didn’t you raise that issue?” Good question.

  9. Nice article, emphasizing on what testers should actually focus on, than merely on checking system correctness. I do agree majority of testers do not think beyond the requirements that is provided by the customer and try to set the evaluation benchmark against the criteria. However they spend little or no time thinking in exploring the business cause or the business objective of testing a particular system.

    The other important point that struck me was that lack of domain experience to perform a business driven testing rather than a requirement driven testing. I do agree that testers need to be trained and mentored to think beyond just delivering a product based on the requirement, but also determining the value that the system can bring to the business and help in enhancing the customer delivery excellence.

  10. Hi,

    Great Post! Testing with awareness, competency, thorough understanding of feature and skill is required for finding critical bugs. Testing for functional verification is definitely short-sighted. If we are just after validating the product against spec, at the end of the project often late we realize that ensuring quality is not that easy.

    “The intersection of Technology and Leadership”

  11. I love this: “When programming is reduced to producing “a set of instructions for a computer”, it misses the point—value to people—and when testing is reduced to the checking of those instructions, it too will miss the same point.”

    This post really resonates with me. I think that continuing to educate yourself is important for testers who want to grow and become more well rounded. However, companies that encourage and promote tester education seem to be few and far between. I hope this changes as testing becomes more integrated with development processes and testers demonstrate an ability and willingness to learn – how to write some code, how to handle clients, how to reason out logic problems and all the other various skills we need to aquire as we work to become better testers.

    Michael requires: It’s sad that companies seem to choose to invest so little in training for everyone, not just testers. But let’s emphasize “continuing to educate yourself is important for testers who want to grow and become more well rounded”. In other words, self-education is entirely up to ourselves. Thanks for reminding us.


Leave a Comment