When Management Asks “Why Didn’t You Find That Bug?”

A tester asks…

How do we handle production bugs? When management asks “Did you test this?”. how do I respond?

When management asks “Why didn’t you find that bug?”, the first step is to accept in your own mind responsibility for looking for bugs, but not a commitment to finding every bug. The latter is a great aspiration, but an unreasonable commitment, and management shouldn’t be holding you to it.

Remember: as testers, we don’t put the bugs in! Most bugs are hidden—and empirically, any bug that no one found — until now — was hidden deeply enough that even the people who put them in didn’t find it.

The next step is to take the position that the question could be reasonable; management might be asking for an explanation, rather than an excuse or a defense. Of course, the tone of the question and the manner of the person asking might imply an accusation, but for the moment put that aside.

Take a moment to come up with a sober and honest evaluation of your own work. When you reflect on a production problem, was it something that you believe that you should have noticed? Or did you act reasonably, work diligently, and test thoroughly? Could the answer to both questions be “Yes”?

A big part of that evaluation is asking what you actually did. Were you doing things that were productive — like investigating and reporting other bugs — but at the same time disruptive to the task of finding this bug? Were you doing important things, like covering parts of the product that entailed risk, but that turned out to have relatively few bugs? Were you setting up tools that helped to make the rest of your testing more efficient, and that led to the discovery of important problems?

Or were you doing things that were less productive? For instance…

• Were you spending more time than you needed to on investigating and reporting bugs? Maybe mipping — a quick conversation with the developer or a manager — might have been a better use of your time.
• Were you covering relatively unimportant areas of the product? Perhaps revisiting the testing mission with the clients more often could address that, such that your testing was better focused on problems that would matter to them.
• Were you creating highly formal, procedurally structured test scripts that you didn’t need, or that were later invalidated by changes to the product? Maybe a set of risk lists and checklists would have been a better choice.
• Were you creating automated checks that simply confirmed that things were okay in a shallow way (arguably work that the developers should be doing)? Did that take time away from learning the product more deeply and searching for more subtle bugs? Were you spending a lot of time wrestling with interpretation and debugging and maintenance of the checks?
• If you were doing unproductive things, were you doing them by your own choices, or were those unproductive things mandated by management?

So: when answering the question, keep those possibilities in mind. If indeed you did miss something that you feel, in your professional judgement, that you should have noticed — if you simply missed something or made a mistake — then cop to that, and indicate that you intend to learn from the experience.

But maybe you were acting reasonably, and you were doing other things that mattered, and you can justify your work. Maybe this is a learning opportunity for everyone:

• Maybe we need a more testable product.
• Maybe we need better access to information, or to the developers. Maybe we need help from more people, with different kinds of expertise.
• Maybe the developers need more time to do checking of their own work. Maybe each developer needs a buddy.
• Maybe if the product weren’t so danged buggy, we’d spend less time on investigating and reporting bugs, which might have given us more time to find this bug.

There’s a cold, hard fact about life and about testing: no one can do everything, and no one can do everything perfectly. Considering that, both you and management must accept the fact that you simply can’t promise to find every bug — and that we can learn from the ones we hadn’t been aware of until now.

Here’s an older, related post: “Why Didn’t We Catch This In QA?”