In which the conversation about heuristics and oracles continues…
“So what’s the difference,” I asked my tester friend Tony, “between an oracle and a heuristic?”
“Hmm. Well, I’ve read the Rapid Testing stuff, and you and James keep saying an oracle is a principle or mechanism by which we recognize a problem.“
“Yes,” I said. “That’s what we call an oracle. What’s the difference between that and a heuristic?”
“An oracle helps us recognize a problem, but it’s not a method for solving a problem, or for making a decision.” He suddenly paused.
“Wait,” he said. “There’s that question you say testers should always be asking—Is there a problem here? An oracle does help us make a decision: it helps us to decide whether there’s a problem in the product we’re testing. And oracles can fail, too. So an oracle’s not different from a heuristic; an oracle is a heuristic. They’re the same.”
“Okay,” I said. “But that’s like saying ‘an iPhone isn’t different from a smartphone; an iPhone is a smartphone. They’re the same.'”
“But? But what? What’s the problem with that? Aren’t all iPhones smartphones?”
“Well, I’d say so,” I replied. “But let me ask you: are all smartphones iPhones?”
He paused for a second. “Oooh. Oracles are heuristic, but not all heuristics are oracles. An oracle is a heuristic, but it’s a specific kind of heuristic. Okay, let me see if I’ve got this: tossing a coin is a heuristic for making a decision. A heuristic approach for making a decision, I mean. You’d use the Coin Toss heuristic in some contexts—random decisions, or unimportant decisions, or… or intractable decisions, or decisions that you want to be fair. The approach can fail. It might not be a fair coin. Or it might be a high-stakes decision that shouldn’t be left to chance. So the Coin Toss heuristic might work, it can fail.”
“Right,” I said. “Tossing a coin is a heuristic approach for making a decision.”
“But it’s not an oracle,” Tony said, “because tossing a coin doesn’t help us to recognize a problem. So tossing a coin is a heuristic, but it’s not an oracle.”
“All right. What does an oracle do for us?”
Tony started confidently. “An oracle is something that gives us the right answer, so that we can compare it to the result the product gives us. If there’s a difference between the oracle’s answer and the product’s result, there’s a problem. If the product’s answer is the same as the oracle’s answer, then there’s no problem.”
“Are you sure about that?” I asked. “Is a specification an oracle?”
“Yes. The specification tells us how the product is supposed to behave.”
“And how reliable are the specifications where you work?”
Tony paused, and then he grinned. “Okay. They suck, to be honest with you,” he said. “They’re ambiguous. They’re unclear. They’re incomplete; they usually miss a bunch of requirements. They contradict each other, sometimes on the same page. So we have to talk about them a lot to clear them up—and then when we sort things out, the job of updating the written spec usually gets left for last, if it ever happens at all.”
“Still,” I said, “if you see an inconsistency between the spec and the product, you at least suspect a problem, don’t you?”
“Well, yeah. When the spec and the product disagree, there’s usually a problem somewhere—either with the product, or with the spec. Or both. When we’re not sure, the program manager is usually the one who clears things up. Sometimes the programmers fix the product. Sometimes the the product turns out to be right, and it’s the spec that’s wrong—but then we know at least the BA’s ought to fix the spec, even if they don’t get around to it right away.”
“So if you use a specification as an oracle, it’s somewhat reliable, but it’s not guaranteed to be right. What does that sound like?”
He paused again. “It’s a heuristic. An oracle is a special kind of heuristic. An oracle is a heuristic principle or mechanism by which we recognize a problem.“
“That’s the way I like to say it these days, yes,” I replied. “For one thing, having the word ‘heuristic’ in the definition of ‘oracle’ seems to help people recognize that there’s some kind of distinction to be made between heuristics and oracles. But for another, I think it’s important to emphasize that oracles help us to learn things. And that, since they’re heuristics, oracles are fallible and context-dependent. No oracle comes with a guarantee that it’s giving you the right answer. An oracle can only point you to a possible problem.“
Tony’s brow furrowed again.
Hi Michael,
Very interesting blog post.
Doug Hoffman considers heuristic test oracles as one type of test oracles. So, although I am convinced with the discussion in this post, I am somewhat confused w.r.t. whether there are any deterministic oracles at all.
Can you please throw some light on what you think about Doug’s description of types of oracles: True/Stochastic/Heuristic/Sampling/Consistent?
Thanks for this Michael.
I have struggled with this the definitions and understanding of each) for some time. This is helping, a lot. Look forward to the next part.
Hi Michael,
As far as I understand the definitions, we apply fallible methods to decide (answer) the “is there a problem here?” question. Fallible, because even if the method suggests the existence of problem, in certain cases it might turn out that the ‘problem’ is no problem.
Therefore I wonder why the ‘potential’ word is included into the definition sometimes:
“heuristic principle or mechanism by which we recognize a potential problem”
potential problem ~ it might be a problem, it might be not
But the ‘heuristic’ adjective already includes the fact of fallibility per def., so I can’t feel the need for ‘potential’.
Or could we say oracle is simply “any principle or mechanism by which we recognize a potential problem”? (no ‘heuristic’ included)
Regards,
Zoltan
Just extend my
“Fallible, because even if the method suggests the existence of problem, in certain cases it might turn out that the ‘problem’ is no problem.”
statement with “vica versa”. Of course, sometimes the applied oracles do not reveal any problem however there is.
[…] Blog: All Oracles Are Heuristic Written by: Michael Bolton […]
Dear Michael
We toss a dice for whom of the boys get to take the bath first (1-3-5 for the oldest, 2-4-6 for the youngest). Today the oldest said:
“How can a dice decide, if it is not human”
Michael: Help me, please.
Michael replies: Your son has raised an important question, which I might address in further dialogs, but for now I’ll address it here.
It’s common for people to say things like “the spelling checker found the mistake” or “the calculator did the math” or “the dice made the decision”. It’s a common (and erroneous) linguistic shortcut to say things like that. Yet none of those statements is accurate, and your son has identified why: people find mistakes; people do math; people make decisions. As Pradeep Soundararajan put it once, “The test doesn’t find the bug. The tester finds the bug, and the test plays a role in finding the bug.” The tools we use are media, as Marshall McLuhan put it. They extend or enhance or enable or accelerate or intensify our ability to do things, but they don’t do the things.
Also – will it make a difference if I told you it’s a red dice with numbers on….
Jesper (with a grin on his face)
I’ve always wondered what the real definition of oracle is. If you look it up in the dictionary, you have to bend the definition to meet the testing context. I don’t think you’ve cleared it up totally with this conversation, but its helped some, thanks.
Michael replies: I’m glad it helped some. Be careful of the real definition of oracle and the dictionary. Consider instead someone’s definition of oracle, or a dictionary.
[…] (If you want to understand in very simple language above statement then please read Michael Bolton’s post on “All Oracles are heuristic”) […]
[…] (If you want to understand in very simple language above statement then please read Michael Bolton’s post on “All Oracles are heuristic”) […]
[…] using heuristics ; emphasizing the use of lightweight, flexible tools; framing testing ; applying oracles ; identifying coverage and the gaps in it; and reporting on all of those things. The premises of […]
[…] All Oracles are Heuristic by Michael Bolton […]
[…] oracles, which are heuristic principles or mechanisms by which we recognize a problem. More on that here. We can use oracles to recognize problems and createwhich are heuristic principles or mechanisms […]
[…] Bolton draws an analogy to clarify: all iPhones are smartphones, but not all smartphones are iPhones. In the same way, […]