Oracles from the Inside Out, Part 2: Experience, Mental Models, and Feelings

In the first post in this series, I introduced some of the factors in recognizing a problem when one occurs during testing. Let’s walk through some examples of recognizing and relating problems.

Imagine that I’m a tester at Adobe, a few years back, testing a version of Acrobat. As it happens, my laptop’s screen is smaller than a desktop monitor, but at 1920 by 960 pixels, the resolution is quite fine (in more ways than one). On my display, the standard Windows fonts are tiny and my eyes aren’t so good, so I use Windows’ Large Fonts setting. When large fonts are active, Adobe Acrobat displays the text in its File / Properties / Additional Metadata dialog like this:

My process of recognizing and reporting a problem typically starts with a feeling that I experience. I might be confused by some behaviour that I don’t understand; or I might be frustrated that I can’t get something done. I might be annoyed by a feature that’s obscure, hard to find, or hard to use. I’m impatient waiting for the system to finish what it’s doing so that I can move on to the next thing. Maybe I’ve seen this problem before, and I’m mildly disgusted that it has returned, or I’m worried that the programmers misunderstood a bug report or misimplemented a fix somehow. When I experience feelings like these, I begin to suspect that I’m seeing a problem.

When I see the Advanced File Properties dialog, I’m surprised and to some degree amused by the cut-off text in its labels, which seems inappropriate and somewhat silly because it’s in conflict with the way I think text should look. Formally, that’s called a schema; in this case, it’s an internal, private, tacit mental model of how things should be. My models provide framing for why I believe something is a problem, and the intensity of my feelings provides clues about the seriousness of the problem. My models have been shaped by experiences of one kind or another, and in turn my models shape my perceptions and my experiences.

Sometimes I might encode some of my mental models, actions, and observations in a set of automated checks, and at some point one might be returning a red result. You could say that my recognition of a problem starts not with a feeling, but with my observation of the red bar, or even with my encoding of the check. I would reply that the observation alone doesn’t have any real meaning or significance for me—that is, I don’t interpret the observation as a problem until the feeling arrives, which it does pretty much immediately anyway: I’m surprised by the red bar and curious about what has produced it.

An experience of a problem—represented by the upper left quadrant of the diagram—is private to me, internal and tacit. Inside, I say something like “Euuugh!” or “Oh!” or “Damn!” or “Ha ha!”. I have the feeling that there’s a problem, but I haven’t yet made it explicit, not to anyone else, and maybe not even to myself. Feelings don’t come with return addresses, and we usually don’t have to justify or explain what we feel or why we feel them. When I’m testing, though, I’m acting as an agent for other people, so when I get a feeling of a problem, there’s more work to do. That’s the subject of the next post in this series.