In response to my post, Testers: Get Out of the Quality Assurance Business, my colleague Adam White writes,
I want ask for your experience when you’ve your first 3 points for managers:
- Provide them with the information they need to make informed decisions, and then let them make the decisions.
- Remain fully aware that they’re making business decisions, not just technical ones.
- Know that the product doesn’t necessarily have to hew to your standard of quality.
In my experience I took this to an extreme. I got to the point where I wouldn’t give my opinion on whether or not we should ship the product because I clearly didn’t have all the information to make a business decision like this.
I don’t think that’s taking things to extreme. I think that’s professional and responsible behaviour for a tester.
Back in the 90s, I was a program manager for several important mass-market software products. As I said in the original post, I believe that I was pretty good at it. Several times, I tried to quit, and the company kept asking me to take on the role again. As a program manager, I would not have asked you for your opinion on a shipping decision. If you had merely told me, “You have to ship this product!” or “You can’t ship this product!” I would have thanked you for your opinion, and probed into why you thought that way. I would have also counselled you to frame your concerns in terms of threats to the value of the product, rather than telling me what I should or should not do. I likely would have explained the business reasons for my decisions; I liked to keep people on the team informed. But had you continued in telling me what decision I “should” be making, and had you done it stridently enough, I would have stopped thanking you, and I would have insisted to you (and, if you persisted, your manager) to stop telling me how to do my job.
On the other hand, I would have asked you constantly for solid technical information about the product. I would have expected it as your primary deliverable, I would have paid very close attention to it, and would have weighed it very seriously in my decisions.
This issue is related to what Cem Kaner calls “bug advocacy“. “Bug advocacy” is an easy label to misinterpret. The idea is not that you’re advocating in favour of bugs, of course, but I don’t believe that you’re advocating management to do something specific with respect to a particular bug, either. Instead, you’re “selling” bugs. You’re an advocate in the sense that you’re trying to show each bug as the most important bug it can be, presenting it in terms of its clearest manifestation, its richest story, its greatest possible impact, its maximum threat to the value of the product. Like a good salesman, you’re trying to sell the customer on all of the dimensions and features of the bug and trying to overcome all of the possible objections to the sale. A sale, in this case, results in a decision by management to address the bug. But (and this is a key point) like a good salesman, you’re not there to tell the customer that he must buy what you’re selling; you’re there to provide your your customers with information about the product that allows them to make the best choice for them. And (corresponding key point) a responsible customer doesn’t let a salesman make his decisions for him.
I started to get pushback from my boss and others in development that it’s my job to give this opinion.
“Please, Tester! I’m scared to commit! Will you please do my job for me? At least part of it?”
What do you think? Should testers give their opinion on whether or not to ship the product or should they take the approach of presenting “just the facts ma’am”?
I remember being in something like your position when I was at Quarterdeck in the late 1990s. The decision to ship a certain product lay with the product manager, which at the time was a marketing function. She was young, and ambitious, but (justifiably) nervous about whether to ship, so she asked the rest of us what she should do. I insisted that it was her decision; that, since she was the owner of the product, it was up to her to make the decision. The testers, also being asked to provide their opinion, also declined. Then she wanted to put it to a vote of the development team. We declined to vote; businesses aren’t democracies. Anticipating the way James Bach would answer the question (a couple of years before I met him), I answered more or less with “I’m in the technical domain. Making this kind of business decision is not a service that I offer”.
But it’s not that we were obstacles to the product, and it’s not that we were unhelpful. Here are the services that we did offer:
- We told her about what we considered the most serious problems in the product.
- We made back-of-the-envelope calculations of technical support burdens for those problems (and were specific about the uncertainty of those calculations).
- We contextualized those problems in terms of the corresponding benefits of the product.
- We answered any questions for which we had reliable information.
- We provided information about known features and limitations of competitive products.
- We offered rapid help in answering questions about the known unknowns.
Yet we insisted, respectfully, that the decision was hers to make. And in the end, she made it.
There are other potentially appropriate answers to the question, “Do you think we should ship this product?”
- “I don’t know,” is a very good one, since it’s inarguable; you don’t know, and they can’t tell you that you do.
- “Would anything I said change your decision?” is another, since if they’re sure of their decision, their answer would be the appropriate one: No. If the answer is Yes, then return to “I don’t know.”
- “What if I said Yes?” immediately followed by “What if I said No?” might kick-start some thinking about business business based alternatives.
- “Would I get a product owner’s position and salary if I gave you an answer?”, said with a smile and a wink might work in some rare, low-pressure cases although, typically, the product owner will have lost his sense of humour by the time you’re being asked for your ship/no-ship opinion.
As I said in the previous post, managers (and in particular, the product owners) are the brains of the project. Testers are sense organs, eyes for the project team. In dieting, car purchases, or romance, we know what happens when we let our eyes, instead of our brains, make decisions for us.
Michael, good analysis.
Having recently started “Are your lights on?” I’ve got a little more clarity on my approaches to problems that aren’t solely my problems… (I think I had certain clarity before – I just hadn’t been forced to actively think about it! Great exercise.)
I’ve recently had the opportunity to apply this and it goes along the lines:
Me: “I can express an opinion from my perspective… But you must look at the problem/question from your perspective as well.
It could be that when we weigh up the different aspects that your considerations out-weigh mine.”
It provides a stunning clarity to the problem – it forces everyone to take a step back and think about “Is my problem the same as your problem and which problem/question do we need to answer right now?”
Even though it is not part of the tester’s job, I think we can answer the question (but of course stress that it is our judgment, without knowing the full business picture.)
I would see this as a help to the manager, in the same way as I expect them to help us by reporting a bug they see, even though it isn’t part of their job description.
If we see the making of the software as a team effort, I think it’s good to get involved in each others areas, it increases understanding and collaboration.
It would still be the role’s responsibility, but that doesn’t mean people shouldn’t help in the best way they can.
And sometimes that can be saying “I wouldn’t release this product, in the long term, I think we would benefit from a couple of more weeks in the oven. But it’s your call.”
I think good management practice is to listen your people / team. That said I think everyone should have their say about the maturity of the release (not just “QA”) but the responsibility of saying “go / no go” should belong to the guy whose money is in the pot (or is responsible for the money).
Wow – a whole post in response to my question!! I wish I was home so we could have discussions in person too!
I should have clarified that in the situations where I’ve being asked for my opinion I was presented the facts that the test team’s findings and as well being the testing manager. This happened frequently.
I have had a lack of success with these.
“I don’t know,” is a very good one, since it’s inarguable; you don’t know, and they can’t tell you that you do.
“Would anything I said change your decision?” is another, since if they’re sure of their decision, their answer would be the appropriate one: No. If the answer is Yes, then return to “I don’t know.”
These answers often get interpreted as being obtuse, incompetent or just being a plain jack a**. I know because people have told me that is how it comes across. Perhaps the problem is my delivery and not the data. I’ve dug in on this before and discovered the times when it was delivery.
“Would I get a product owner’s position and salary if I gave you an answer?”, said with a smile and a wink might work in some rare, low-pressure cases although, typically, the product owner will have lost his sense of humour by the time you’re being asked for your ship/no-ship opinion.
While a humorous question to me..the last sentence is true and it only adds fuel to a usually already hot fire
This is one I’ve never tried
” “What if I said Yes?” immediately followed by “What if I said No?” might kick-start some thinking about business business based alternatives”
Thanks for the ideas!
Once again, excellent insight. Another way to look at this is, why would you want to assume risk / responsibility for something over which you have no ultimate control? I learned this principle at the expense of a former manager. This guy actually felt offended that as manager of so-called “QA,” he did not have the authority to put a halt on shipments. Going back to your last post, we were (and still are) mislabeled as QA when we are in fact testers. My poor manager did not understand this important distinction and chose to define his own role based on the “QA” title rather than the actual job he was hired to perform. He felt that if he was to indeed assure quality, he needed the power to keep releases from going into production if he couldn’t sanction them from a quality standpoint.
Needless to say, his constant opposition to the release schedule didn’t sit well with upper management. They got tired of explaining his job to him and he’s working elsewhere.
I’ve been asked the “should we ship” quite a few times, in fact the last time it was the project manager and the analyst asking my opinion. I let them know what the crazy, possibly business-stopping defects were and then said, “based on that information, you can make that decision.” Or something like that… I don’t make it a habit to quote myself.
In the end, if managers are expecting testers to make their decisions for them, then let’s switch positions.
An excellent analysis. And so familiar, been there sometimes. Reading this story triggered me to come up with some questions and thoughts which I wrote about on my blog: http://testconsultant.blogspot.com/2010/05/response-on-gono-go-to-ship.html . It triggered me to question myself and share this information with my colleagues also.
Thanks for this!
Regards
Jeroen
Hi, Jeroen… Thanks for the kind words, and good for you for continuing the discussion both at work and on your blog.
Interesting take. What came to my mind while reading through, is the fact that I hear: “I don’t understand that technical aspects from you” as a reply from a manager when I think I start to explain my concerns about the product.
So, there might be a disconnection between the business understanding and the technical understanding, since the managers responsible for taking the business decisions do not want to hear technical things.
On the other hand, the flaw could be located in the selling of the technical concerns about the product. When dived too deep into the material many managers just don’t want to be bothered with it.
Michael replies: They might not want to hear the technical details. I’m convinced, though, that they want to hear the implications of those details—that is, the way in which the technical concerns represent a threat to the value of the product. You might want to avoid burying the lead.
Third (applying Weinberg’s Rule of Three), there might be a misunderstanding of the context. Maybe managers are shifting the business decision to the technical staff.
So, in fact, looking at the root cause of the try to ship or don’t ship is worthwhile to do here. I’ll keep an eye out on it.
As always, thanks for writing, Markus.
[…] out the blog posts – most recently Testers: Get Out Of The Quality Assurance Business and When Testers Are Asked for a Ship/NoShip Decision. Both of these articles stand on their own, but if there’s enough interest, I might do a […]
Good article, clearly Michael had been in the trenches.
I think the title says it all, the tester is only asked for an opinion, not a decision.
Michael replies: I’m even leery of the “opinion” business. Goodness and badness isn’t for me to decide. What I can provide is some framing: IF you believe that this quality criterion is important; and IF we have an oracle that would point to a problem that threatens that quality criterion; and IF we believe that this test exercises the software in a way that would evinces that problem; then there’s a problem here. We tend to assume that we’re on the same page as the product owner, and we tend to compress a lot of those ideas, but that’s what going on. Since I’m human, my opinion tends to leak through my report. but I’m skeptical that it should.
The only contribution I’d like to make is regarding the advice that a possible response is “I don’t know”.
That is in fact a very good response, especially when followed immediately with “but I can help you try to find out.”
The tester should be giving a technical opinion only, along the following lines:
YES
1) Yes, all testing completed, all bugs closed.
2) Yes, all testing completed, some bugs not closed but not serious, mitigation are in place to handle that – shouldn’t be to much of a problem.
3) Yes, but not all testing completed, no outstanding bugs, the client may find more bugs.
4) Yes, but not all testing completed, some bugs not closed, mitigation in place, client may find more bugs.
NO
5) No, high risk bugs still open without mitigation.
6) No, not all testing completed, based on testing existing results highly suspect that more serious bugs may be discovered.
Hmmm. Are those testing reports? Or are they reports from a program manager? Again, I might not have the information to decide whether a problem is “serious” or not. And what does “all testing completed” mean? All testing that the client wants completed, perhaps—or at least, that’s the only notion of completeness that ultimately matters.
As you allured to, the ship/no ship decision by the programme manager is not based only on the technical point of view. However, if the tech view states – software is totally shot, well… 🙂
The astute reader will most likely point out that I’m contradicting myself since sometimes I say YES and sometimes I say NO when not all off the testing had been completed. It all comes down to a mix of knowing your software products, your colleagues, the history of the SDLC, your client and many other factors.
I have a set of heuristics that might help decide when a tester might choose to stop a given test or a test cycle. You can find that here. But ultimately, your client—the person to whom you are responsible, be that the product owner or someone beneath her in the management chain—is responsible for that decision too.
Thank you for writing.