Why Exploratory? Isn’t It All Just Testing?

The post “Exploratory Testing and Review” continues to prompt comments whose responses, I think, are worthy of their own posts. Thank you to Parthi, who provides some thoughtful comments and questions.

I always wondered and in attempted to see the difference between the Exploratory testing that you are talking about and the testing that I am doing. Unlike the rest of the commenter’s, this post made this question all the more valid and haunting.

From what you have written, as long as there is a loop between the test design and execution, its exploratory testing? And the shorter the loop, exploratory nature goes up?

Yes, that’s right. A completely linear process would be entirely scripted, with no exploratory element to it. The existence of a loop suggests that the testing is to some degree exploratory. This suggests (to me, at least) a link to one of the points of Jerry Weinberg’s Perfect Software and Other Illusions About Testing. Testing, he suggests, is gathering information with the intention of informing a decision, and he also says that if you’re not going to use that information, you might as well not test. I’ll go a little further and suggest that if you “test” with no intention of using the information in any way, you might be doing something, but you’re not really testing.

As we’ve said before, some people seem to have interpreted the fact that there’s a distinction between exploratory testing and scripted testing as meaning that you can only be doing one or the other. That’s a misconception. It’s like saying that there are only two kinds of liquid water: hot or cold. Yet there are varying gradations of water: almost freezing, extremely cold, chilly, cool, room temperature, tepid, warm, hot, scalding, boiling. To stretch the metaphor, a test is it’s being done by a machine (that is, a check) is like ice. It’s frozen and it’s not going anywhere. An investigation of a piece of software done by a tester with no purpose other than to assuage his curiosity is like steam; it’s invisible and vaporous. But testing in most cases is to some extent scripted and to some extent exploratory. No matter how exploratory, a test is to some degree informed by a mission that typically came from someone else, at some point in the past; that is, the test is to some degree scripted. No matter how scripted, a test is to some degree informed by decisions and actions that come from the individual tester in the moment—otherwise the tester would freeze and stop working, just like a machine, as soon as he or she was unable to perform some step specified in the script. That is, all testing is to some degree exploratory.

In addition to the existence of loops, there other elements too. Very generally,

• the extent to which the tester has freedom to make his or her own choices about which step to take next, which tests to perform, which tools to use, which oracles to apply, and which coverage to obtain (more freedom means more exploratory and less scripted; more control means less exploratory and more scripted);
• the extent to which the tester is held responsible for the choices being made and the quality of his or her work. More responsibility on the tester means more exploratory and less scripted; more responsibility on some other agency means less exploratory and more scripted.
• the extent to which all available information (including the most recent information) informs the design and execution of the next test. The broader the scope of the information that informs the test, the more exploratory; the narrower the scope of information that informs the test , the more scripted.
• the extent to which the mission—the search for information—is open-ended and new information is welcomed. The more new information will be embraced, the more exploratory the mission; the more new information will be ignored or rejected, the less exploratory the mission.
• again, very generally, the length of the loops that include designing, performing, and interpreting an activity and learning from it, and then feeding that information back into the next cycle of design, performance, interpretation, and learning. I’m not talking here so much about timing and sequences of actions so much as the cognitive engagement. Timing is a factor; that’s one reason one reason that we now favour “parallel” over “simultaneous”. But more importantly, the more difficult it is to unsnarl the tangle of your interactions and your ideas, the more exploratory a process you’re in. The more rapidly you are able to shift from one heavy focus (say on executing the test) to another heavy focus (pondering the implications of what you’ve just seen) to another (running a thought experiment in your head) to yet another (revising your design), very generally, the more exploratory the process. Another way to put it: the more organic the process, the more exploratory it is; the more linear the process, the more scripted it is.

Is this what you are saying? If yes, there is hardly any difference in what I do at my work and what you preach and this is true with most of my team (am talking about 600+ testers in my organization) and we simply call this Testing.

I’d smilingly suggest that you can “simply” call it whatever you like. The more important issue is whether you want to simply call it something, or whether you want to achieve a deeper understanding of it. The risk associated with “simply” calling it something is that you’ll end up doing it simply, and that may fail to serve your clients when they are producing and using very complex products and services and systems. Which is, these days, mostly what’s happening.

For example, is there really a difference between what I’m talking about and what are your 600+ testers doing? Can you describe what they’re doing? How would you describe it? How would you frame their actions in terms of risk, cost, value, skill, diversity, heuristics, oracles, coverage, procedures, context, quality criteria, product elements, recording, reporting? Is all that stuff “simply” testing? For any one of those elements of testing, where are your testers in control of their own process, and when are they being controlled? Are all 600+ at equivalent stages of development and experience? Are they all simply testing simply, or are some testing in more complex ways?

Watch out for the magic words “simply” or “just”. Those are magic words. They cast a spell, blinding and deafening people to complexity. Yet the blindness and deafness don’t make the complexity go away. Even though these words have all the weight of snowflakes, their cumulative effect is to cover up complexity like a heavy snowfall covers up a garden.

May be these posts should be titled “Testing” than “Exploratory Testing”?

There is already good number of groups/people taking advantage of the (confused state of the larger) testing community (like certification boards). Why to add fuel to this instead of simplifying things?

There’s a set of important answers to that, in my view.

• Testing is a complex cognitive activity comprising many other complex cognitive activities. If we want to understand testing and learn how to do it well, we need to confront and embrace that complexity, instead of trying to simplify it away.
• If we want our clients to understand the value, the costs, the extents, and the limitations of the services we can provide for them, we need to be able to explain what we’re doing, how we’re doing it, and why we’re doing it. That’s important so that both we and they can collaborate in making better informed choices about the information that we’re all seeking and the ways we go about obtaining that information.
• One way to “simplify” matters is to pretend that testing is “simply” the preparation and then following of a script, or that exploratory testing is “simply” fooling around with the computer. If you’re upset at all about the certification boards that trivialize testing (as I am), it’s important to articulate and demonstrate the fact that testing is not at all a simple activity, or that comprehension of it can be assessed with any validity via a 40-question multiple choice test. Such a claim, in my opinion, is false, and charging money for such a test while making such a claim is, in my opinion, morally equivalent to theft. The whole scheme is founded in the premise that testing a tester is “simply” a matter of putting the tester through 40 checks. If we really wanted to evaluate and qualify a tester, we’d use an exploratory process: interviews, auditions, field testing, long sequence tests, compatibility tests, and so on. And we wouldn’t weed people out on the basis of them failing to take a bogus exam, any more than we’d reject a program for not being run against a set of automated checks that were irrelevant to what the program was actually supposed to do.
• Just as software development is done in many contexts, so testing is done in many contexts. As we say in the Rapid Testing class, in excellent testing, your context informs your choices and vice versa. And in excellent testing, both your context and your choices evolve over time. I would argue that a heavily scripted process is more resistant to this evolution. That might be a good thing for certain purposes and certain contexts, and a not-at-all good thing for other purposes and other contexts.

Many people say, for example, that to test medical devices, you must do scripted testing. There is indeed much in medical device testing that must be checked. Problems of a certain class yield very nicely to scripted tests (checks), such that a scripted approach is warranted. The trouble comes with the implicit suggestion that if you must do scripted testing, you must not do exploratory testing. Yet if we agree that problems in a product don’t follow scripts; if we agree that there will be problems in requirements as well as in code; if we agree that we can’t recognize incompleteness or ambiguity in advance of encountering their consequences; if we agree that although we can address the unexpected we can’t eliminate it; and if we agree that people’s lives may be at stake: isn’t it the case that we must do exploratory testing in addition to any scripted testing that we might or might not do?

The answer is, to my mind, certainly Yes. So, to what extent, from moment to moment, are we emphasising one approach or the other? That’s not a question that we can answer by saying that we’re “just” testing.

Thanks again, Parthi, for prompting this post.