Testing Deep and Shallow (3): Determination

After almost a year of the blog lying fallow, it’s time to continue the series on Testing Deep and Shallow that begins here. (Shallow testing (not an insult!) is testing that has a chance of finding every easy bug. Deep testing maximizes the chance of finding every elusive bug that matters.)

Premise 6 of Rapid Software Testing is about cost and value.

“We commit to performing credible, cost-effective testing, and we will inform our clients of anything that threatens that commitment. Rapid Testing seeks the fastest, least expensive testing that completely fulfills the mission of testing. We should not suggest million dollar testing when ten dollar testing will do the job.”

That is: we want to do shallow testing where it’s appropriate, and deep testing where it’s necessary.

What does it take to get there? What do we need for testing that maximizes the chance of finding elusive, important problems?

In this post, I’ll talk about the first thing we need for deep testing: determination.

Some product problems are near the surface; easy to find. Some can be detected easily by simple tests and automated checks done by developers as they build the product. Others can be noticed by pretty much anyone during early encounters with the product.

By contrast, some problems are subtle, rare, intermittent, latent, condition-dependent, or emergent. Problems like that can be elusive — and important. The important bugs aren’t always right there on the surface.

Finding such problems may take deep testing, but that doesn’t happen by accident. Deep testing has to happen intentionally and deliberately. We won’t simply blunder into deep testing; we have to want it.

The determination to do deep testing is motivated by risk. The possibility of bugs and problems about the product that will affect people means risk to the business.

Problems:

• Everyone else on the project is envisioning success, and focused on getting the product built. Some of these people might be considering the possibility of trouble, but with very few exceptions, only testers are focused on it.

• Deep testing might be seen as highly formalized testing.

• Traditional approaches to testing (for instance, highly formalized procedure test scripts) make testing dull and destroy people’s initiative and determination to investigate the product.

• Testing leads to the discovery of problems that seem to slow the project down.

• All this leads to testing being socially challenging in two ways. That is, we’re challenging the product and our beliefs about it in a social context; and doing that is sometimes difficult for people to accept.

Heuristic workarounds:

• Make sure that there is at least one person on the team who takes on the role of Responsible Tester, as we call it in the Rapid Software Testing Namespace. The responsible tester is the tester who bears personal responsibility for testing a particular thing in a particular way for a particular project. The responsible tester is someone who does testing work, sees to it that it gets done, and where needed, recruits supporting testers to help.

• Keep the whole team — designers, developers, managers, and testers — alert to risk by revealing product problems, by telling stories about problems that others have had, and by developing and narrating stories about the consequences of product problems.

• Shallow bugs interrrupt deep testing. Discovering errors requires recording and reporting work, and then revisiting and retesting after fixing. For testers, all this can sometimes feel unnecessary when those errors are easy to find and seem easily avoidable. That, in turn, can reduce testers’ motivation.
  
  Meanwhile, developers find that deep testing requires a change in mindset and approach, and interrupts their progress in building the product. How many developers do you know who say, “I love spending lots of time on testing work!”? When people aren’t motivated to do something, they probably won’t do it very well or very thoroughly. Therefore: give developers time, resources, and responsibility to check for errors close to the surface of their work, so that shallow bugs don’t interrupt deep testing. At the same time, give testers mission to do the deep testing that would otherwise disrupt the developers’ flow.
  
  No one wants those tricky bugs to be there, but finding them can be exciting and feels valuable. The mission to find them can increase testers’ motivation.

• Deep testing is not necessarily formally proceduralized testing. To the degree that testing might be formalized, the formality might be more about careful design and performance of interesting experiments, rather than adherence to rote, prodecural, heavily scripted, confirmatory test cases. Test cases may play a role in testing, but testing is not test cases, and test cases are not testing.

• The essence of testing is evaluating the product by learning about it; experiencing the product; exploring it; and experimenting with it. Recognize that testing depth is not necessarily about formality, but it is about thoroughness of the experiences, the exploration, and experimentation.

• Eliminate everything that makes testing work dull, boring, drab, and awful. Avoid turning human beings into test robots. Give responsible testers and supporting testers freedom and responsibility to explore, discover, investigate, and describe the product — and avoid overstructuring reporting of the work when formality isn’t absolutely necessary.

• Help people to remember that discovering problems deliberately before release tends to be better than finding problems by surprise in production. Yes, the discovery of a bug is usually bad news on one level, but revealing a bug now gives us a chance to address the problem before it’s too late. Praise and reward people for engaging in that discovery. Testing need not slow the project down, but prudence might require going more slowly to go faster.

Deep testing requires rich models; we’ll talk about those next. Stay tuned!