Heuristics for Understanding Heuristics

This conversation is fictitious, but it’s also representative of several chats that I’ve had with testers over the last few weeks.

Tony, a tester friend, approached me recently, and told me that he was having trouble understanding heuristics and oracles. I have a heuristic approach for solving the problem of people not understanding a word:

Give ’em a definition.

So, I told him:

A heuristic is a fallible method for solving a problem or making a decision.

After I tried the “Give ’em a definition” heuristic, I tested to see if Tony seemed to understand. His eyes were a little glazed over. I applied a heuristic for making the decision, did he get it?

When someone’s eyes glaze over, they don’t get it.

Heuristics aren’t guaranteed to work. For example, sometimes the general “Give ’em a definition” heuristic solves the problem of people not understanding something, and sometimes it doesn’t. In the latter case, I apply another heuristic:

Give ’em an explanation.

So I told him:

“When you know how to solve a problem, you might follow a rule. When you’re not so sure about how to solve the problem, following a rule won’t help you. Not knowing how to solve a problem means not knowing which rule to apply, or whether there’s a rule at all. When you’re in uncertain conditions, or dealing with imperfect or incomplete information, you apply heuristics—methods that might work, or that might fail.

“As an adjective, ‘heuristic’ means ‘serving to discover’ or ‘helping to learn’. When Archimedes realized that things that sink displace their volume of water, and things that float displace their mass, he ran naked through the streets of Athens yelling, ‘Eureka!’ or ‘I’ve discovered it!’ ‘Eureka’ and ‘heuristic’ come from the same root word in Greek.

Tony was listening thoughtfully, but his brow was still furrowed. So I applied another teaching heuristic:

Give ’em something to compare.

I said, “Here’s one way of understanding heuristics: compare ‘heuristic’ with ‘algorithm’. An algorithm is a method for solving a problem that’s guaranteed to have a right answer. So an algorithm is like a rule that you follow; a heuristic is like a rule of thumb that you apply. Rules of thumb usually work, but not always.”

Sometimes providing a comparable idea solves the problem of understanding something, and sometimes it doesn’t. Tony nodded, but still looked a little puzzled. I wasn’t sure I had solved the problem, so I applied a new heuristic:

Point ’em to a book.

I suggested that he read George Polya’s book How to Solve It. “In that book, Polya presents a set of ideas and questions you can ask yourself that can help you to solve math problems.”

“Wait… I thought you always solved math problems with algorithms,” Tony said.

“That’s when you know how to solve the problem. When you don’t, Polya’s suggestions—heuristics—can get you started. They don’t always work, but they tend to be pretty powerful, and when one doesn’t work, you try another one. You never know which questions or ideas will help you solve the problem most quickly. So you practice this cycle: apply a heuristic, and if you’re still stuck, try another one. After a while, you develop judgement and skill, which is what you need to apply heuristics well. Polya talks about that a lot. He also emphasizes just how much heuristics are fallible and context-dependent.”

Mind you, neither Tony nor I had a copy of Polya’s book right handy, and Tony wanted to understand “heuristics” better now. The “point ’em to a book” heuristic had failed this time, even though it might have worked in a different context. So I tried yet another heuristic to solve the problem:

Point ’em to another book.

I suggested that he read Gut Feelings by Gerd Gigerenzer. “In that book, Gigerenzer emphasizes that heuristics tend to be fast and frugal (that is, quick and inexpensive). That’s important, he says: humans need heuristics because they’re typically dealing with bounded rationality.”

Uh-oh. Tony’s eyes had glazed over again at the mention of “bounded rationality”. So I applied a heuristic:

Even when it’s a deep concept, a fast and frugal explanation might do.

After all, Polya says that a heuristic isn’t intended to be perfect. Instead, heuristics are provisional and context-dependent. So in order to provide a quick understanding of “bounded rationality”, I said, “In a nutshell, bounded rationality is a situation when you have incomplete knowledge, imperfect understanding, and limited time.”

He grinned, and said, “What, like when you’re testing? Like most of the time in life?”

“Yes. Billy Vaughan Koen, in another book, Discussion of the Method, says that the engineering method is ‘to cause the best change in a poorly understood situation within the available resources.'”

“So he’s saying that engineers apply heuristics?” Tony asked. “I guess that makes sense, since engineers solve problems in ways that usually work, but sometimes there are failures.”

He seemed to be getting it. But I wanted to test that, so I applied a heuristic for making the decision, “Does he get it?”

Ask the student to provide an example.

So I said, “I think you might have it. But can you provide me with an example of a heuristic?”

He said, “Okay. I think so.” He paused. “Here’s a heuristic for solving the problem of opening a door: ‘Pull on the handle; push on the plate.’ That’s what you do when you get to a door, right? It’s a heuristic that usually works. Well… it might fail. It could be one of those annoying doors that have handles on both sides, where you have to push the handle or pull the handle to open the door. It might be one of those doors that opens both ways, like the doors for restaurant kitchens, so there’s no handle. The door might not even have a handle or a plate; it might have a knob. In that case, you apply another heuristic: ‘Turn the knob’. That’s a solution for the problem of opening a door that doesn’t have a handle or a plate. But that heuristic might fail too. The door might be locked, even though the knob turns. It might be one of those fancy doors that have dead-bolt locks and knobs that don’t turn. It might not have a knob at all; it might have one of those old-fashioned latches. So none of those heuristics guarantees a solution, but each one might help to solve the problem of getting through the door.”

“Great! I think you’ve got it.”

“To be precise about it,” he said, “you can’t be sure, so you’re applying heuristics that help you to make the decision that I get it.”

I laughed. “Right. So what’s the difference,” I asked, “between an oracle and a heuristic?”

He paused.

(to be continued…)