Testers ask: “I’m often given a product to test, but not enough time to test it. How am I supposed to sign off on the release when I haven’t tested enough?”
My reply goes like this:
If you’re a tester, it seems profoundly odd to me that you are responsible for signing off on a release. The decision to release a product is a business decision, not a technical one. The decision is, of course, informed by technical considerations, so it’s entirely reasonable for you to provide a report on those considerations.
It would be foolish for for someone in a business role to ignore your report. But it would be just as foolish for a someone in a business role to abdicate the business decision to technical people. We serve the business; we don’t run it, and technical people often aren’t privy to things in the business’ domain.
The idea that testers can either promote or prevent a release can be tested easily. Try refusing to sign off until you’re entirely happy with what you know about the product, and with the extent of what you know. You’ll get a fairly quick result.
Perhaps managers will go along with you. You’ll test the product until you are satisfied that you have covered everything important, that you have discovered all the significant problems, that those problems have been fixed, and that the discovery of any more problems that matter would be a big surprise to you. Your client (that is, the person to whom you report) will give you all the time and all the resources you need until the product meets your standards. Yes, that seems unlikely to me, too.
More likely, at some point you will be overruled. Management will decide that your concerns are not important enough to block the release. Then you will be able to say, “So, the decision to approve or prevent the release is not really my decision? Cool.” Then, perhaps silently, “I’m glad that’s your job after all. I’m happy not being a fake product owner.”
Without a product owner’s authority (and title, and salary), I’m not willing to sign—or even make—a go/no-go decision. That decision is not my responsibility, but the responsibility of people with the authority to make it. If they ask for my recommendation, I’ll provide a list of problems that I know about, and any reasons to believe that there might be more problems lurking. Then I will politely recommend that they weigh these against the needs and objectives of everyone else involved—development, operations, support, sales, marketing, finance, and so forth.
So what if someone asks you to sign something? I am willing to sign an account of what I know about the status of the product, and what I’ve done to learn about it. That account can be fairly concise, but in expanded form, it will look like this:
I typically start with “I’ve tested the product, and I’ve found these problems in it.” I then provide a list of a few problems that I believe to be most important to inform management’s decisions. For the rest, I’ll provide a summary, including patterns or classes of general problems, or pointers to problem reports. My special focus is on the problems; as the newspaper reporters will tell you, “if it bleeds it leads”.
I’ll welcome requests for more information. If there’s any doubt about the matter, I emphasize that the decision to ship or not rests with the person responsible for making the release decision.
(Some people claim “the whole team decides when to release and when not to”. That’s true when the whole team agrees, or when disagreements are tractable. When they’re not, in every business situation I’ve been in, there is a single person who is ultimately responsible for the release decision.)
If I haven’t found any problems—which is rare—I won’t sign anything claiming that there are no problems in the product. I’m willing to assert that I’m not aware of any problems. I cannot responsibly say that there are no problems, or that I’m capable of finding all problems. To say that there are no problems is only an inference; to say that I’m not aware of any problems is a fact.
Whether I’ve found problems or not, I’m willing to sign a statement like this: “I have covered these conditions to this degree, and I have not covered these other conditions.” The conditions include specific product factors and quality criteria, like those found in the Heuristic Test Strategy model, or others that are specific to the project’s context.
My statement about coverage gives warrant to my statement that there are problems (or that I’m not aware of them), and identifies why management should be appropriately trusting and appropriately skeptical about my evaluation. For an informed release decision, management needs to know about things I haven’t covered, and my perception of the risk associated with not covering them.
Happy news about the product might be worth mentioning, but it takes second place to reporting the problems and risks. I want to make sure that any concerns I have are prominent and not buried in my report.
I’m also willing to sign a statement saying “Here are some of the things that helped me, and here are some of the things that didn’t help; things that slowed my testing down, made it more difficult, reduced the breadth and depth of coverage I was able to obtain.”
Whether I sign such a statement or not, I want to make sure I’ve been heard. I also want to offer some ideas that address the obstacles, and note that with management help, maybe we can reduce or remove some of them so that I can provide more timely, more valuable coverage of the product. When I can do that, I can find deeper, or more subtle, or more intermittent, and possibly more dangerous bugs.
Of course, I don’t run the project. There may be business considerations that prevent management from helping me to address the obstacles. If I’ve been heard, I’ll play the hand I’ve been dealt; I’ll do my best to address any problems I’ve got, using any resources I can bring to bear. It’s my job to make management aware of any risks associated with not dealing with the obstacles—on paper or in a quick email, if I’m worried about accountability. After that, decisions on how to manage the project belong with management.
In other words: I’m prepared to sign off on a three-part testing story. As a tester, I’m prepared to accept responsibility for my story about the quality of the product, but the product does not have to adhere to my quality standards.
I’ll sign off on my report, but not on a shipping decision. The business doesn’t need my permission to ship.
Early in my career, the company I worked for had a policy that when testing on a batch of code was completed, the tester in charge would sign off on it. Literally, signing a piece of paper that said, essentially, “testing on this has been completed.” Managers were under instruction not to release to production without that sign-off.
In this story, I was leading a small team of 4-5 people, and what we were doing was mostly mindless link-checking that really should have been automated. We’d get a page filled with links, check to make sure they all went where they were supposed to, and I’d sign the paper saying we checked all of them. Then one day, they sent us a batch that was about four times the size of the typical ones. And when we weren’t done checking it the next day like we usually were, the project manager came to my desk to complain. I explained that this was a significantly larger piece of work than usual, it’ll take a little longer.
So they started pressuring me to just go ahead and sign off on it, so they could release it on time. I dug my heels in: No, I’m not going to sign my name on a piece of paper that says I’ve fully tested something, when I haven’t fully tested it. If the schedule really is more important than the testing, then you can go ahead and release it without my signature on that piece of paper. They eventually backed down and gave us the extra couple of days we needed.
Michael replies: This is a really good story. Thank you, Scott, for telling it for us here. For me, the key point is that you wouldn’t assent to what amounts to lying. As soon as you’ve done that once, your integrity and reputation become vulnerable.
Some cautionary notes, though. Those who are not savvy about testing may make a systematic error when they hear (or read) “the testing has been completed” or “fully tested”. I would offer instead “I (or we) have performed (this) testing”, with an account of the testing that I’ve performed and that I haven’t, or with a reference to a report that provides the details. I want to emphasize that the testing didn’t just happen; I want to make it clear that people performed it, and that people are accepting responsibility for the testing work that was done. And although testing does stop, I also don’t want to suggest that testing is ever completed. I could assert that testing stopped based on specific criteria, like coverage or time.
Note also that while I can accept responsibility for the quality of the testing work, I cannot accept responsibility for the quality of the product.
[…] Signing Off – Michael Bolton – http://www.developsense.com/blog/2018/03/signing-off/ […]
You’ve summed up my approach wonderfully. I always encourage my testers to report facts and let others make the business decision. What we’ve tested in the time; and the key anomalies found.
These anomaly reports are our shop window or ‘raisin d’etre’. They describe the findings in factual terms & how the anomaly is triggered. Only the business and/or product owner can really put this into the context of the release schedule as a whole.
As a test lead or manager, I usually carry the top 2-3 in my mental basket, so that I can summarise them whenever asked about testing progress. Coupled with factual reporting on things like stability & progress against time has won me many friends and few enemies. Like this article, I too stress that the decision making rests beyond Testing alone and all that I can do is tell the testers side of the story.
I can be assertive, when needed though. In a recent role, we saw a problem with EURO to GBP currency exchanges (no other combinations) that would lose the client money. There I spelt it out in minute detail on multiple occasions to many people. As the Head of IT expressed it; “you were like a terrier with a rat.” Needless to say that one was fixed pre-release. But a business decision could still have been to go live with the system but with currency exchange switched off.
Michael replies: Nice story. Fascinating how losing the client money can garner a bit of attention, isn’t it?
[…] Then, let’s talk about signing off on a release, Michael Bolton explains that testers should never do that. The decision being a business decision, not a technical one. Read how to deal with that if you are asked to decide to ship or not: Signing off. […]
[…] Signing Off Written by: Michael Bolton […]