Evaluating Test Cases, Checks, and Tools

For testers who are being asked to focus on test cases and testing tools, remember this: a test case never finds a bug. The tester finds a bug, and the test case may play a role in finding the bug. (Credit to Pradeep Soundararajan for putting this so succinctly, all those years ago.)

Similarly, an automated check never finds a bug. The tester finds a bug, and the check may play a role in finding the bug.

A testing tool never finds a bug. The tester finds a bug, and the tool may play a role in finding the bug.

If you suspect that managers are putting too much emphasis on test cases, or automated checks, or testing tools—artifacts—, try this:

Start a list.

Whenever you find a bug, make a quick note about the bug and how you found it. Next to that, put a score on the value of the artifact. Write another quick note to describe and explain why you gave the the artifact a particular score.

Score 3 when you notice that an artifact was essential in finding the bug; there’s no way you could have found the bug without the artifact.

Score 2 if the artifact was significant in finding the bug; you could have found the bug, but the artifact was reasonably helpful.

Score 1 if the artifact helped, but not very much.

Score 0 if the artifact played no role either way.

Score -1 whenever you notice the artifact costing you some small amount of time, or distracting you somewhat.

Score -2 whenever the artifact when you notice the artifact costing you significant time or disruption from the task of finding problems that matter.

Score -3 whenever you notice that the artifact is actively preventing you from finding problems—when your attention has been completely diverted from the product, learning about it, and discovering possible problems in it, and has been directed towards the care and feeding of the artifact.

Notice that you don’t need to find a bug to offer a score. Pause your work periodically to evaluate your status and take a note. If you haven’t found a bug in the last little while, note that. In any case, every now and then, identify how long you’ve been on a particular thread of investigation using a test case, or a set of checks, or a tool. Evaluate your interaction with the artifact.

Periodically review the list with your manager and your team. The current total score might be interesting; if it’s high, that might suggest that your tools or test cases or other artifacts are helping you. If it’s low or negative, that might suggest that the tools or test cases or other artifacts are getting in your way.

Don’t take too long on the aggregate score; practically no time at all. It’s far more important to go through the list in detail. The more extreme numbers might be the most interesting. You might want to pay the greatest or earliest attention to the things that score the lowest and highest first, but maybe not. You might prefer to go through the list in order.

In any case, as soon as you begin your review of a particular item, throw away the score, because the score doesn’t really mean anything. It’s arbitrary. You could call it data, but it’s probably not valid data, and it’s almost certainly not reliable data. If people start using the data to control the decisions, eventually the data will be used to control you. Throw the score away.

What matters is your experience, and what you and the rest of the team can learn from it. Turn your attention to your notes and your experience. Then start having a real conversation with your manager and team about the bug, about the artifact or tool, and about your testing. If the artifact was helpful, identify how it helped, and how it might help next time, and how it could fool you if you became over-reliant on it. If the artifact wasn’t helpful, consider how it interfered with your testing, how you might improve or adjust it or whether you should put it to bed for a while or throw it away.

Learn from every discovery. Learn from every bug.

Related reading:
https://www.satisfice.com/blog/archives/487091