A Testopsy: Learning from Performance

What’s the difference between Rapid Software Testing (RST) and other forms of testing? In RST, the process model is not the centre of testing; neither is formal documentation; nor are tools. All of those things play a role in testing, of course, but they’re not at the centre.

In RST, the centre of testing is the skill set and the mindset of the individual tester, and heuristics that testers apply.

A heuristic is a fallible means of solving a problem. That is, a heuristic might work, or it might fail. A heuristic will fail when it is applied to the wrong kind of problem; or when it is applied with insufficient judgement, wisdom, skill, or care; or when some context factor or another derails it. All of the models that we apply to the product and to the test space are heuristic. All test techniques are heuristic. All of the ways in which we could apply tools are heuristic. All the ways we have of deciding that there’s a problem (that is, all of our oracles) are heuristic. And this doesn’t apply only to testing; everything in software development, and in the broader field of engineering itself, is heuristic.

So, in order to get good at testing, we must learn about heuristics that we can apply powerfully in our work. We must also consider how our heuristics can fail, too. One of the better ways to do that is to review and evaluate our work periodically in a very detailed way. In Rapid Software Testing, we call that a testopsy.

Earlier this year, James Bach and I did a testopsy on a session of testing that we had performed together about six months earlier, in preparation for the Rapid Software Testing Applied class that we teach. By examining our performance, we able to notice and name heuristics and patterns that help us to think about testing, to describe it, and to understand how testing can go right—and sometimes not so right.

Here are just a few things we learned—or learned more deeply—from that session and the testopsy we performed:

• When we’re doing pair testing, a lot of tacit knowledge emerges into the explicit. Each person’s performance is visible to the other, raising observations and questions about things that have not been shared up to that point. Through that, knowledge gets shared, discussed, and refined.
• Products often give us lists of their own features in odd places, in interesting ways, that afford some efficiency for identifying coverage ideas.
• There’s a phenomenon that happens in testing that we’re calling a “bug cascade“—periods where we are stressed or even overwhelmed by overlapping and competing investigations of complex and confusing behaviour.
• During a bug cascade, we often recognize that we don’t know enough about the product to perform good analysis and troubleshooting.
• Bugs get noticed and then lost, or missed altogether, during a bug cascade…
• …but having a video and reviewing it can help us to recover what we’ve lost.
• Analyzing the product (which had been our original misison for the session) can be severely disrupted by a cascade of bugs.
• We coined a term, “mutually disruptive processes”, to describe one of the consequence of the bug cascade—which, when you’re working alone, is self-disruptive.
• We coined another term, “the money booth effect”, to account for the collapse of productivity that is the consequence of mutually- or self-disruptive processes.
• It is a good idea to be forgiving of ourselves for these problems. Although we can try to manage them to some degree, they are intrinsic to the process of learning and testing a product.

There’s lots more to the testopsy, which you can see here.

Why is this all important? Because in order to do something well, we must understand it, and testing is often terribly misunderstood—by managers, by developers, and, sadly, by testers themselves. By doing deep study of our work from time to time, we can begin the process of framing it, describing it, dicussing it, and developing expertise in it.

Want to learn how to observe, analyze, and investigate software? Want to learn how to talk more clearly about testing with your clients and colleagues? Rapid Software Testing Explored, presented by me and set up for the daytime in North America and evenings in Europe and the UK, November 9-12. James Bach will be teaching Rapid Software Testing Managed November 17-20, and a flight of Rapid Software Testing Explored from December 8-11. There are also classes of Rapid Software Testing Applied coming up. See the full schedule, with links to register here.