A Few Observations on Structure in Testing

On Twitter, Johan Jonasson reported today that he was about to attend a presentation called “Structured Testing vs Exploratory Testing”. This led to a few observations and comments that I’d like to collect here.

Over the years, it’s been common for people in our community to mention exploratory testing, only to have someone reply, “Oh, so that’s like unstructured testing, right?” That’s a little like someone refer to a cadenza or a musical solo as “unstructured music”; to improv theatre as “unstructured theatre”; to hiking without a map or a precise schedule as “unstructured walking”; to all forms of education outside of a school as “unstructured learning”. When someone says “Exploratory testing is unstructured,” I immediately hear, “I have a very limited view of what ‘structure’ means.”

Typically, by “structured testing”, such people appear to mean “scripted testing”. To me, a script is a set of detailed, ordered set of steps of specific actions and specific observations. Some people call that a procedure, but I think a procedure is characterized by a set of activities that might be but are not necessarily highly specific, and that might be but are not necessarily strictly ordered. So what is structure?

Structure, to me, is any aspect of something (like an object or system) that is required to maintain the thing as it is. You can change some element of a system, add something to it or remove something from it. If the system explodes, or collapses, or changes such that we would consider it a different system, the element or the change was structural. If the system retains its identity, the element or the change wasn’t structural. Someone once described structure to me as “that which remains”.

A system, in James Bach’s useful definition, is a set of things in meaningful relationship. As such, structure is the specific set of things and factors and relationships that make the system a system. Note that the observer is central here: you may see a system where I see only a mess. That is, you see structure, but I don’t. When a system changes, you may call it a different system, where I might see the original system, but modified. In this case, I’m seeing structure that you don’t see or acknowledge as such. We see these things differently because systems, structures, elements, changes, meaningfulness, and relationships are all subject to the Relative Rule: “For any abstract X, X is X to some person.”

One of the principles of general systems thinking is that the extent to which you are able to observe and control a system is limited by the Law of Requisite Variety, coined by W. Ross Ashby: “only variety can destroy variety.” Jurgen Appelo recently produced a marvelous blog post on the subject, which you can read here.

Variety, says Ashby, is the number of distinct states of a system. In order to observe variety, “The observer and his powers of discrimination may have to be specified if the variety is to be well defined.” The observer is limited by the number of states that he (or she or it) can see. This also implies that however many states there are to a system, another system that controls it has to be able to observe that many states and have at least one state in which it can exert control. Wow, that’s pretty heady. What does it mean? The practical upshot is that if you intend to control something, you have to be able to observe it and to change something about it, and that means that you need to have more states available to you than the object has. Or, as Karl Weick put it in Sensemaking in Organizations, “if you want to understand something complicated, you have to complicate yourself.”

This pattern repeats in lots of places. Glenford Myers suggested that testing a program requires more creativity than writing one; more things can go wrong than go right in writing a program. Machines can do a lot of the work of keeping an aircraft aloft, but managing those machines and the work that they do requires more intelligent and more adaptable systems—people. The Law of Requisite Variety also suggests that we can never understand or control ourselves completely, people are insufficient to understand people completely.

So: some people might see exploratory testing as unstructured. I suspect they’re thinking of a particular kind of structure: that scripted procedure that I mentioned above. For me, that suggests an impoverished view of both exploratory testing and of structure. Exploratory testing has tons of structures.

Over the years, many colleagues have been collecting and cataloguing the structures of exploratory testing. James and Jon Bach have led the way, but there have been lots of other contributions from people like Cem Kaner, Mike Kelly, Elisabeth Hendrickson, Michael Hunter, Karen Johnston, Jonathan Kohl, and me (if there’s someone else who should appear in this list, remind me). I’ve collected some of these in this list of structures of exploratory testing. For those who are just joining us, and for the old hands who might need a refresher, you can find my list of exploratory testing structures here.

Programs had structures long before “structured programming” came along. Programming that isn’t structured programing is not “unstructured programming.” The structure in “structured programming” is a particular kind of structure. To many people, there is only one kind of testing structure: scripted procedures. I will guarantee that an incapacity to see alternative kinds of structure will limit your testing. And to paraphrase Weick, if you want to understand testing, you must test yourself.