DevelopsenseLogo

Problems with Problems

People sometimes seem to struggle with a concept that’s central to testing, the concept of “oracle”. In the three-day Rapid Software Testing class, we define an oracle as

a principle or mechanism means by which we recognize a problem.

Sometimes I like to emphasize that oracles are fallible and context-dependent. When that’s so, I say that an oracle is

a heuristic principle or mechanism means by which we recognize a problem.

(Updated in 2019 to reflect a change that we made some time ago. –MAB)

That means that an oracle can work but might fail in helping us to recognize a problem. A heuristic is something that helps us learn; it’s a fallible method for solving a problem or making a decision. So an oracle, as a special kind of heuristic, helps us to make a particular decision in answer to the question, “Is there a problem here?” If the answer is Yes, it’s because an oracle is enabling us to recognize a problem.

For some time, I’ve been surprised at how tricky a concept this seems to be for some people. Possibly, I thought, it was because people simply weren’t used to thinking about oracles; for many it’s a new word and a new concept. Maybe the problem was with “heuristic”, or “principle”, or “mechanism”. Recently, though, I began to wonder: perhaps the difficulty comes from the fact that people aren’t used to thinking deeply about problems. I tested that idea by asking some people, and I found that they struggled in describing how they thought of problems. So what is a problem, anyway?

Here are two ways of thinking about problems that I’ve found useful.

1) A problem is “a difference between what is perceived and what is desired.” (Dewey, J. (1933), How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process) I first learned this definition of “problem” in a session led by Don Gray at the Amplifying Your Effecctiveness Conference a few years ago, and I believe it shows up in several places in Jerry Weinberg‘s work.

2) A problem is “an undesirable situation that is significant to and maybe solvable by some agent, though probably with some difficulty.” (G.F Smith, “Towards a Heuristic Theory of Problem Structuring”, quoted in Weick, Karl E. Sensemaking in Organizations. Sage Publications, Inc, 1995.)

Both definitions, I think, point to the same thing: problems are based on desires, perceptions, and situations. But both definitions have a minor problem for me, which is that they do not make explicit something I think to be very important: desires, perceptions, and situations are centred around people and context. Problems are subject to the Relative Rule:

For any abstract X, X is X to some person, at some time.

People often talk of problems (bugs, defects, issues, and so forth) as though they were attributes of a product or of a situation. But problems aren’t attributes as such. They’re relationships between some person(s), the product, and the system that includes them, and the situation that encompasses it all. A problem is a problem for some person, at some time. Differences, perceptions, and desires are like that too; they’re relative. If I were to expand out Dewey’s notion of “problem”, it would look like this:

A problem is a difference (according to some person) between what is perceived (by some person) and what is desired (by some person) (all at some point in time).

There are several implications here.

  • The problem might be in the difference, or in the perception, or in the desire.
  • Different people will have different notions of differences, different perceptions, different desires.
  • A problem may also be influenced by timing; something that’s a problem today might not be a problem tomorrow, and something that isn’t a problem today might be a catastrophe tomorrow.
  • Since, as testers, we’re in the business of finding potential problems, we must be alert to the enormous variety of people who may be affected by the product and the project.
  • Our mission as testers typically includes the task of identifying possible problems. As such, we should resist the temptation to dismiss something as “not a problem”, because…
  • Something that we might dismiss as a trivial problem for some people might be a serious problem for other people and…
  • Something that we might see as a serious problem in our perception might appear as a trivial problem for other people so…
  • We’re obliged as testers to identify a possible problem in terms of its most serious consequences for people who might matter to the product owner. However…
  • We’re not in the business of deciding whether something is a problem or not, so for a given problem, our testing clients or the project owners decide on its ultimate significance, and on their response to it.

So the bottom line is that problems are slippery. A problem is not a thing in a product or in the world, but a relationship between some situation and some person.

Smith’s emphasis on an ability to solve a problem may matter too. People often accept things that they perceive as beyond anyone’s ability to solve. For example, I can’t do anything about a meteor hitting my computer in the next few minutes, so I’m not going to treat the possibility that it might happen as a problem. Yet again, I should be careful to suppress any of my own prejudices that nothing can be done with respect to a given problem. That’s a decision for those who build and those who own the product, since they may have information and more resources of which I am unaware.

One of the fundamental questions of testing is “Is there a problem here?” Oracles are media, in the McLuhan sense. Media are tools, extensions of ourselves that enhance, enable, accelerate, or intensify our capacity to do things. McLuhan pointed out, though, that media are agnostic about what they extend. If our concept of “problem” is limited, our oracles will extend and accelerate our ability to recognize a limited set of problems. In other words, with too narrow an idea of what a problem might be, oracles may only help us to recognize too narrow a set of possible problems.

So: one key to understanding and applying oracles skilfully is to recognize the richness and diversity of what we might mean by problem. What we’re testing is not simply source code or a collection of compiled object code files. We’re testing products or services that are systems, sets of things in meaningful relationship to each other. Those systems are related to other systems. Products don’t stand on their own. Every product is part of a system that includes people who build it, people who support it, people who maintain it, people who buy it, people who use it directly, and people who are affected by it. That’s an incomplete list of people who might experience problems related to the product or system. Try asking these questions:

  • What are the elements of system that I’m testing?
  • Who might have a relationship to that system, or to its elements?
  • Who else might have a relationship to it?
  • What desires might they have that are related to the system?
  • What aspects of the system might provide value to those people by fulfilling their desires? What might threaten that value by dashing their desires?
  • How might people perceive the system? What might they see, hear, smell, touch, taste, or feel as they use it or interact with it?
  • What might influence, magnify, sharpen, or distort their perceptions?
  • What differences might they experience between their perceptions and their desires?
  • Could someone, or something—some change in the situation—help them to solve or otherwise deal with such differences?
  • What might change in the system over time? How might people’s perceptions, desires, or notions of differences change over time?
  • What ideas, tools, or conversations might help me to recognize perceptions, desires, and differences?

Reflecting on these questions periodically, even briefly, may help you to expand your notion of what a problem is. That in turn may extend your ability to use oracles to recognize potential problems in the system you’re testing.

(What’s an oracle? An oracle is a heuristic for addressing the question “Is there a problem here?” But wait… that raises the question, “What’s a heuristic?” Find out here.)

This blog post was inspired and sharpened by conversations with Anne-Marie Charrett, Peter Walen, and James Bach. Thanks to them.

Subscribe to the Newsletter

13 replies to “Problems with Problems”

  3. Hi Michael

    Thank you for helping to clarify this important subject. I admit I was one of the people who struggle with the concept of Oracles and heuristics and thanks to your skills as a teacher you managed eventually to get the concept over to me.

    One thing I think could be missing from the questions is the importance of our own self justification and beliefs which could give a false answer to the problem. This is especially true if we are experiencing cognitive dissonance and unknowingly using confirmation bias to resolve the problem in a way in which is incorrect but in our own minds we would like it to be correct.

    Maybe a question that dis-confirms our own views, beliefs about the problem would be useful. There are far more ways to prove a thought/theory/belief is incorrect than it is correct but our natural instinct is to only confirm what we already believe.

    Reply

  4. A lot of projects seem (as it remains implicit) to have the following definition of a problem: “A problem is a difference between what has been produced and what was documented.” So no relative rule, no perception and no desires.

    Michael replies: I’ve seen that too. And this implicitly means that quality is based on a match between a document and the product’s apparent behaviour, rather than a broader notion of value to some person.

    One reason for this is in that “a project consists of a temporary endeavor undertaken to create a unique product, service or result. [wikipedia]” If the project part (Do I reach the agreed upon targets on time?) becomes too important compared to the product, the definition of ‘problem’ will become equally narrow. It will be focused on attaining the project goal as documented and agreed upon at the expense of focus on the goal of the actual product or service. (I do understand why that shift of focus happens, btw. If only because the consequences of not attaining your project targets are more close-by than not attaining the product goals.)

    The same thing happens with the definition of ‘oracle’. In TMap and ISTQB testing consist of comparing the design documents with the product. So a problem (i.e. a defect) is a difference between the actual test result and the result as predicted by the design documents. So yet again, no relative rule, no perception and no desires.

    When you are used to this approach, it’s a big change to do what you propose: look beyond the project, the documentation, etc. and approach the product from the systems it will be become a part of.

    I agree; it might be a big change. To me, much depends on what kind of tester you want to be, and what kind of organization you want to work for.

    Reply

  5. I like Gause and Weinberg’s thought process for problems from “Are your lights on?”:
    – What is the problem?
    – Who has it?
    – Who should fix it?

    In testing, oracles help u recognize problems, test-framing appears to me to be all over on describing the problem, but the latter two questions are probably more interesting for any business owner. Of course, we can help with the second one as well, thinking through meaningful scenarios for our tests, but in the end it will be the product owner to make the decision who is fixing the problem.

    Reply

  6. Interesting article as always!

    Markus,
    I’ve got this book on order so without having read it…

    Who has the problem has an impact on “What is the problem?” so these two questions can only be answered together, not in isolation, imo. I had experiences in the past where the problem statement (bug) was a problem to one person, whereas I saw the problem in a different area. Same bug, only different perception of what’s important.

    Who should fix it is not always the product owner. I can think of a situation from my experience where the product owner has no say on who’s fixing the problem. For example, the product owner sits in a business unit and IT is a separate department with different teams within that. The product owner may say “IT, fix it”, but who that’s going to be may be completely out of her control.

    Interesting questions, looking forward to reading the book now.

    Reply

  7. This is really a nice & interesting article. I think this will help testers to think about the bugs or the problems in different dimensions. And also this writing will help to think about the problem fixing.

    Reply

  12. I do think there should be some work to identify the significance. Most business owners are not as thoughtful on problems and their impact. Providing them the boundaries of the problem and the impact helps them to be better decisions. I’ve got business owners that identify everything as critical, but when I present it in relation to another problem or a new feature they want, they can better rank the problem. I do think there is huge value in identifying how many widgets are impacted (assuming you are impacting widgets; obviously lives have a bigger impact if lives are the impact). Frequency should be measured across individual users and all users. If one user has 100 problems a day, and all other users have 0 vs if all users have 1 problem and the are 100 users. The frequency is the same, but I can make a better decision on the problem.

    Michael replies: Certainly contextualizing a problem in terms of their significance can be important, both in terms of the problem on its own and in relation to other problems.

    I’d caution against relying too much on counting either frequency or number of users, although those may be factors. For many problems, these things will be a projection or a guess. Frequency is often a guess; impact is often a guess; how many people it will impact is a guess; and each of these may vary from situation to situation. So I’d counsel against measurement, and suggest assessment, which helps to shine light appropriately on our fundamental uncertainty about things.

    Moreover, most people — even testers — have a hard time projection downstream consequences of things that might look like minor problems from our perspective. A more important thing, to me, is constructing stories about problems. Ultimately, it seems to me, problems are mostly about how we feel about things, and stories tend to land with people. The counts to which you refer may be very helpful in illustrating those stories, as long as we remember they’re illustrations, and not stories in themselves.

    I’d also like to advise caution on the tester making a better decision about the problem. It’s the tester’s job to shine light on problems; to present a reasoned basis for why we think they’re problems; and to provide our perceptions on the significance of them. It’s not our job to decide whether something is a problem or not; nor to decide on its significance; nor to decide on what to do about them. Managers make those decisions. As a tester, I want to be helpful, but I don’t run the project.

    Your comment prompted me to think about this stuff and articulate it, so thank you for that.

    Reply

Leave a Comment