Testers ask: “I’m often given a product to test, but not enough time to test it. How am I supposed to sign off on the release when I haven’t tested enough?”
My reply goes like this:
If you’re a tester, it seems profoundly odd to me that you are responsible for signing off on a release. The decision to release a product is a business decision, not a technical one. The decision is, of course, informed by technical considerations, so it’s entirely reasonable for you to provide a report on those considerations.
It would be foolish for for someone in a business role to ignore your report. But it would be just as foolish for a someone in a business role to abdicate the business decision to technical people. We serve the business; we don’t run it, and technical people often aren’t privy to things in the business’ domain.
The idea that testers can either promote or prevent a release can be tested easily. Try refusing to sign off until you’re entirely happy with what you know about the product, and with the extent of what you know. You’ll get a fairly quick result.
Perhaps managers will go along with you. You’ll test the product until you are satisfied that you have covered everything important, that you have discovered all the significant problems, that those problems have been fixed, and that the discovery of any more problems that matter would be a big surprise to you. Your client (that is, the person to whom you report) will give you all the time and all the resources you need until the product meets your standards. Yes, that seems unlikely to me, too.
More likely, at some point you will be overruled. Management will decide that your concerns are not important enough to block the release. Then you will be able to say, “So, the decision to approve or prevent the release is not really my decision? Cool.” Then, perhaps silently, “I’m glad that’s your job after all. I’m happy not being a fake product owner.”
Without a product owner’s authority (and title, and salary), I’m not willing to sign—or even make—a go/no-go decision. That decision is not my responsibility, but the responsibility of people with the authority to make it. If they ask for my recommendation, I’ll provide a list of problems that I know about, and any reasons to believe that there might be more problems lurking. Then I will politely recommend that they weigh these against the needs and objectives of everyone else involved—development, operations, support, sales, marketing, finance, and so forth.
So what if someone asks you to sign something? I am willing to sign an account of what I know about the status of the product, and what I’ve done to learn about it. That account can be fairly concise, but in expanded form, it will look like this:
I typically start with “I’ve tested the product, and I’ve found these problems in it.” I then provide a list of a few problems that I believe to be most important to inform management’s decisions. For the rest, I’ll provide a summary, including patterns or classes of general problems, or pointers to problem reports. My special focus is on the problems; as the newspaper reporters will tell you, “if it bleeds it leads”.
I’ll welcome requests for more information. If there’s any doubt about the matter, I emphasize that the decision to ship or not rests with the person responsible for making the release decision.
(Some people claim “the whole team decides when to release and when not to”. That’s true when the whole team agrees, or when disagreements are tractable. When they’re not, in every business situation I’ve been in, there is a single person who is ultimately responsible for the release decision.)
If I haven’t found any problems—which is rare—I won’t sign anything claiming that there are no problems in the product. I’m willing to assert that I’m not aware of any problems. I cannot responsibly say that there are no problems, or that I’m capable of finding all problems. To say that there are no problems is only an inference; to say that I’m not aware of any problems is a fact.
Whether I’ve found problems or not, I’m willing to sign a statement like this: “I have covered these conditions to this degree, and I have not covered these other conditions.” The conditions include specific product factors and quality criteria, like those found in the Heuristic Test Strategy model, or others that are specific to the project’s context.
My statement about coverage gives warrant to my statement that there are problems (or that I’m not aware of them), and identifies why management should be appropriately trusting and appropriately skeptical about my evaluation. For an informed release decision, management needs to know about things I haven’t covered, and my perception of the risk associated with not covering them.
Happy news about the product might be worth mentioning, but it takes second place to reporting the problems and risks. I want to make sure that any concerns I have are prominent and not buried in my report.
I’m also willing to sign a statement saying “Here are some of the things that helped me, and here are some of the things that didn’t help; things that slowed my testing down, made it more difficult, reduced the breadth and depth of coverage I was able to obtain.”
Whether I sign such a statement or not, I want to make sure I’ve been heard. I also want to offer some ideas that address the obstacles, and note that with management help, maybe we can reduce or remove some of them so that I can provide more timely, more valuable coverage of the product. When I can do that, I can find deeper, or more subtle, or more intermittent, and possibly more dangerous bugs.
Of course, I don’t run the project. There may be business considerations that prevent management from helping me to address the obstacles. If I’ve been heard, I’ll play the hand I’ve been dealt; I’ll do my best to address any problems I’ve got, using any resources I can bring to bear. It’s my job to make management aware of any risks associated with not dealing with the obstacles—on paper or in a quick email, if I’m worried about accountability. After that, decisions on how to manage the project belong with management.
In other words: I’m prepared to sign off on a three-part testing story. As a tester, I’m prepared to accept responsibility for my story about the quality of the product, but the product does not have to adhere to my quality standards.
I’ll sign off on my report, but not on a shipping decision. The business doesn’t need my permission to ship.