Our principal job as testers is to discover the actual status of the product. Remember: everyone else on the project is focused on Success, Preventing Problems, Building Quality In, Adding Value, and all that. Those are all good things, and there’s nothing wrong with them. But there is a catch.
The optimistic focus required to build the product can direct people’s attention away from problems that threaten its quality and value — problems that slipped by everyone’s best efforts to prevent them. As testers, we experience, explore, and experiment with the product to discover its true status, with a special focus on finding bugs and risk. That makes us special; no one else on the project has the mission of looking for trouble — and the potential for trouble — as their *primary* focus.
A bug, in the Rapid Software Testing namespace, is anything about the product that threatens its value; so one of the most valuable services we offer to the project is the bug report.
So what should a bug report contain? Let’s remember that we’re testing to find problems for people. “People” starts with P, E, O… Problem, Example, Oracle.
Problem: Provide a description of the problem you perceive. What Bad Thing happens? What Good Thing fails to happen? A problem is something that might bring loss, harm, diminished value, or bad feelings. Be specific and clear about the problem and its significance to people who matter.
Example: What did you observe about the product itself? Describe context, data, state, steps, logs, behaviour or evidence that would allow other people to see what you saw. Do a favour to your reader, though: make your description efficient — as detailed as necessary, and as concise you reasonably can.
Oracle: The means by which you recognized the problem. That’s generally about an observable, describable inconsistency between the product and some specification, example, tool, preference, or principle, in the view of someone who matters. Bear in mind that some oracles are stronger and more authoritative; others are weaker, and merely suggestive of a problem. The strength of the oracle often — but not always — sheds light on the importance of the problem.
Update: Melissa Fisher has pointed out the risk of misunderstanding what I mean by “product”, so let me be clear: “product”, in this context, refers to anything that someone has produced. The problems we’re reporting on, as testers, could certainly be in a product that consists of running code; but they could also be in other things that someone has produced: a specification; a prototype; a plan; a requirements document; an idea; a service; a system; a drawing on a whiteboard…
“About”, in this context, refers not only to things intrinsic to a product or inside of it, but also to the other products, systems, people, and the sets of relationships netween those things.
Here’s a detailed version of that quick summary: The Rapid Software Testing Guide to Making Good Bug Reports
Confused by the idea of an oracle? This series might help.
1 reply to “Problem, Example, Oracle: A Quick Checklist for Bug Reports”