Not-So-Great Expectations

In my teaching and consulting work, I often ask people how they recognize a problem, and they often offer “inconsistency with expectations” as one way to do it. I agree, but I also think we should be careful to think things through. A product that fulfills our expectations may not be satisfying, and a product that violates our expectations may be terrific.

Several years ago, I bought a new computer for the kids. The discount retailer offered only Windows Vista as the operating system. I had already heard plenty about Vista, and I considered going elsewhere. However, I figured that it might be a good thing to have one computer with Vista in the house for testing purposes—and for the kids, it would Build Character.

In this house, “Daddy” also means “technical support”, so I frequently found myself troubleshooting and configuring the kids’ system. I was a Windows XP user at the time. As usual, Microsoft had run its User Interface Scrambling and Feature Hiding tool just before Vista’s general release, so I couldn’t find the controls and settings I wanted to find. Moreover, every time I tried to something, Vista would interrupt me, pop up a dialog, and ask if I had really asked to do that something. If Vista had been an employee, I would have fired it on the first day. “Yes, I did want to do that. Why do you keep asking me? Yes, I am who I say I am. Why do you keep asking me that? Don’t you think if I weren’t who I said I was, I would simply lie to you and answer Yes?” I expected Vista to suck. And it sucked in all the ways I had expected, and more. But it was consistent with those expectations.

More recently, I purchased a Nest thermostat. My expectations weren’t met; they were exceeded. I was expecting to have to fetch a screwdriver to install it; nope, the unit comes with a screwdriver (both slot and Philips head). I was expecting to have to do the fiddly wrap-wires-around-a-terminal thing. When I opened the package, the thermostat didn’t have screw terminals, as I had expected; it had clearly labeled, spring-loaded, easy-to-use wire clips. I was expecting that I’d have to patch and paint where the big old thermostat had been; the Nest is small and round. No problem; the Nest comes with a backing plate that’s slightly larger than my old unit’s backing plate. I was expecting to have to connect to my computer or do some other elaborate setup stuff. Nope; the unit automatically found our wireless router, asked me for the access key, and connected itself to the Web. So the product wasn’t what I expected, but that wasn’t a problem.

Expectations typically refer to the future, but many things that we might call “expectations” in casual speech are not clear in advance. They’re subconscious, tacit, or constructed on the fly. The set of expectations that we could have about something is intractably large, and it would be cognitively difficult and practically expensive to make them explicit and to note them—and for many problems, it wouldn’t be necessary to do that, either. Here’s an example. Look closely, and ask “what’s wrong with this picture?”

Outlook Truncates the Fields

In my version of Microsoft Outlook 2010, I use the search function to find an email message that has a particular string in it. The search returns no results for the current folder, so Outlook offers to search all mail items. In the search result, the fields for the sender, the subject of the message, and the folder all have the last characters truncated.

Neat bug, huh? Now, I could say that I recognized a problem because the behavior was inconsistent with my expectations, and you might say the same. Yet I doubt that we would ever have bothered to express this expectation: “I expect all data fields to display all characters, even the last one”. It might be more accurate to say that, when I spotted the bug, I constructed the expectation in the moment, in response to an observation of a violation of some pattern or consistency heuristic.

Our expectations, desires and requirements for a product aren’t static. We develop them as we develop the product, as we become familiar with it, and as we develop our relationship with it. When people say that something is a bug because it’s inconsistent with their expectations, I think they might be confusing expectations with desires. That is, when people say “that’s not what I expected,” they really mean “that’s not what I want“. Inconsistency with expectations isn’t a big deal at all when people are surprised or delighted by the product. In some cases, inconsistency with expectations might cause temporary discomfort or disorientation, but after a bit of experience and familiarity, what we have might turn out to be better than what we expected. Moreover, someone might have a set of expectations fulfilled, and upon observation and reflection they might realize that what they expected was not what they wanted after all. As oracles—means by which we identify a problem during testing—an inconsistency with history or expectations might be superseded by a consistency with purpose.

Every moment that we’re testing, we have the capacity to recognize a new risk, a new problem, or a buried assumption, even without an explicit expectation. Humans can do this at the drop of a hat, but machines cannot. This is why it’s so important to have humans testing, interacting with the product, observing and experimenting at all levels and with all of the product’s interfaces. By all means use tools to help you, but beware The Narcotic Comfort of the Green Bar! Operate and observe the product directly, and re-evaluate your checks and their outputs from time to time. Don’t just ask, “Is this what we expect?” Ask also, “Is this (still) what we want?”

8 replies to “Not-So-Great Expectations”

  1. Nice thoughts Michael. I always enjoy when people define Quality as ‘meeting expectations’ and then ask what ‘exceeding expectation’ is? The expectations and desires of the users keep on changing and it is always good to ask questions to clarify.

    Thanks for yet another great post!

    Michael replies: Thank you for the kind words. I’m glad you find it helpful.

  2. Neat explanation.

    And that is true for testers. Their levels of expectation! I’ve seen people accept horrid UI/functionality or even errors. And when prompted why you get the answer that that’s how software xyz (usually Windows/Word) works too. So by inference they think that makes it all right.

    Michael replies: People often prefer the familiar to the comfortable, which avoids one kind of pain while ramping up another.

    One of the big moments in my testing career was my 1st Mac. Like your Nest experience my expectations were surpassed and immediately led me to adopt a higher expectation level in anything I test. Which should be a good thing but very few (especially developers) people agree.

    I have to remind myself to be careful there. I think most people do their best with respect to the expectations they believe they should fulfill. But people have different beliefs about which expectations matter. Programmers may respond to expectations of a large number of features; of fast delivery; of optimized performance. What’s more likely, I think, is that we don’t necessarily see eye-to-eye on the relative importance of certain expectations. Conversations need to happen, and people need to speak clearly and articulately for many quality criteria, in my view. Good examples can help.

    Cheers Oliver

    Thanks, as always, for the comment.

  3. Hi Michael,
    At my company, we’re “guilty” of using a “parameter-expected results-observed results” paradigm when writing scripted tests to verify customer requirements. I have found that this has on occasion revealed misinterpretations of requirements, in that what we said we expected was inconsistent with what the customer expected. In your examples above, would you say that there was a problem with your expectations, perhaps in that the heuristics you used to develop them failed? Presuming that Microsoft didn’t set out to make Vista suck, isn’t it inconsistent to expect it to?

    You don’t say whether the thermostat box indicated that a screwdriver was included. Could the inconsistency between your expectation and the product be considered a problem of a lack on information on the packaging that would have made you expect one? It seems like there’s a Schrodinger-esque aspect to labeling something as a problem.

    – Dan


Leave a Comment