DevelopsenseLogo

Oracles: The Brainstream Media

In the past, James Bach and I have defined “oracle” as “a principle or mechanism by which we recognize a problem,” and we’ve focused on principles rooted in ideas about consistency and inconsistency. At its foundation, applying an oracle and recognizing a problem always involves some principle that links an observation of a product to someone’s desires for it. That is: we observe a problem when we see some kind of undesirable relationship between related things: inconsistency with things someone wants, consistency with things someone doesn’t want.

Most traditional ideas about oracles focus on mechanisms—typically in the form of a reference document, or a tool that provides “the correct answer”. (There are enormous problems with the notion of an oracle as the source of the correct answer, but I’ll save that for a later post.) Documents and tools are media—things that lie between ourselves and some person who matters. Media, as McLuhan points out, extend, accelerate, enhance, enable, or intensify some human capability. Oracles are media that extend, accelerate, enhance, enable, or intensify our ability to recognize a problem—presumably a problem for some person who matters.

In a conversation in 2009, Cem Kaner remarked to me that a tester is himself or herself a test instrument. As I’ve reflected on that, I’ve come to realize that the role of the tester is that of a medium—one that extends, accelerates, enhances, enables, or intensifies our clients’ abilities to understand their products and the problems in them.

At least since 2006, James has been talking about the concept of the “live oracle”. A live oracle may be the product owner, a programmer, a domain expert, a business analyst, a designer, another tester—any person who can alert us to the presence of a problem.

It’s the nature of media that they are indirect; mediate (as an adjective), rather than immediate. That is, they’re in between the observer of the problem and the person to whom the problem matters. But naturally, sometimes our recognition of a problem can be more immediate; that is, we recognize a problem based on a principle, without a specific mechanism or instrument or person to mediate it.

So: sometimes as we’re testing, we recognize a problem by a direct feeling, or through a principle that we’re aware of. Sometimes our recognition of a problem comes from a document or tool; sometimes from another person. That may entail embodied mental processes, rational principles, or mechanisms, but all that’s kind a mouthful. So lately, James and I have started to say that an oracle is simply “a way to recognize a problem”. Here’s a high-level, work-in-progress list of ways a tester might recognize a problem:

Feelings like confusion, surprise, frustration, annoyance, impatience, or amusement.
People, such as a person whose opinion matters, or the opinion of that person; or disagreements among people who matter.
Artifacts, including reference documents with useful information; sets of known good or known bad example outputs; comparable products or features or algorithms.
Mechanisms, such as processes or tools by which output can be checked, or that help a tester to identify patterns.
Principlesb)—undesirable inconsistencies between the product and something related to it; or undesirable consistency with things we don’t like.

There are two things that we encourage Rapid Software Testing students to do with this list:

1) What examples can you provide as specific instances of each of these categories? (The FEW HICCUPPS heuristics are examples of principles)
2) What are the factors or dimensions that might cause us to use or prefer one oracle over another?

I’ll try to provide some answers over the coming days.

Subscribe to the Newsletter

7 replies to “Oracles: The Brainstream Media”

  1. Hi
    I am very confused about the oracle now,I always have the opposite view with the developer about a question.Whenever I come up my opinion,he would say,the customer would not act that function in my way or the function is too small,there is no need to modify,I don’t know how to solve this problem,We are a small team and our product manager let us solve this problems ourselves since we have been familiar with our product.

    Michael replies: Hi, Nicole; thanks for writing.

    Here’s something that we say in the Rapid Software Testing class: When a programmer says, “No user would ever do that,” what he really means is “No user that I’ve thought of, and that I like, would ever do that on purpose (in a way that I’ve imagined).” So in cases where programmers do say “No user would ever do that,” it’s important to ask “What kind of user haven’t you thought of? Have you considered customers that you don’t like (such as hackers, malicious people, or poorly-trained people)? Have you considered that a customer you do like might do something by accident?”

    As testers, I argue that it’s not our job to make decisions about quality. Another thing that we say in the Rapid Testing class is that oracles are not perfect, and testers are not judges. There’s no reason to believe that the tester’s opinion should automatically prevail in decisions about bugs and quality. Testers may bring product knowledge, technical knowledge, domain knowledge, awareness of customer needs, systems thinking, critical thinking, and all kinds of other information to the table. But programmers, designers, technical support people, product managers, may do that too. Why should the tester’s point of view prevail?

    (I’ll tell you one important reason why the tester’s point of view might prevail: because, for a given problem, the tester can tell a compelling, credible story about what the problem is, why it’s a problem, and why the problem matters. This is what Cem Kaner calls “bug advocacy“, and it’s closely associated with a testing skill that James Bach and I call test framing, and with the three-part testing story and how we tell it. Please follow the links I’ve provided here.)

    In fact, neither the tester nor the programmer is really the decision-maker when it comes to decisions about bugs (which are decisions about quality). As a tester, as soon as I find myself in disagreement with a programmer, I try not to take too long in the argument. As best I can, I present the best description of the problem, why I think it’s a problem, and what the risk is. If the programmer disagrees, I recognize that it’s neither my decision nor his. I reply, “Okay, we disagree. Let’s ask the person that we’re both serving here: the product owner.”

    The ultimate decision-maker is the person who is ultimately responsible for the quality the product: the product manager. It seems as though your product manager has decided that product management isn’t his job. Without more information, I can’t tell the reason for this. Perhaps he (she?) is afraid of responsibility or blame for problems in the field. Maybe he believes that because he’s newer to the product, he’s not as qualified as you are to make decisions. Maybe he’s okay with delegating the decision to you or to the programmer. Maybe he simply wants to avoid conflict. But (perhaps unfortunately for him), he’s either going to have to start making decisions himself, or to identify one person who is ultimately responsible for product decisions. Otherwise, you and the programmer may be stalemated, or you may be blamed for problems that are the responsibility of the person charged with making product decisions.

    Reply

  3. Hi Michael
    Thanks for your reply
    I think I have made lots of mistakes since I always want to let the developer accept my opinions,I will think carefully about myself to found out the shortcomings.

    Reply

  4. Hi Michael,

    I have been thinking over the list and the first thing that I can think of is “motivation”. A tester might recognise a problem if there is enough motivation isn’t it? What do you think? or is it a Pre-requisite to the list?

    Thanks,
    Sharath

    Michael replies: Motivation is orthogonal to the principles or mechanisms that would help you to recognize a problem. Blindfold two people such that they can’t see inconsistencies. Will the more motivated one find more problems?

    Reply

Leave a Comment