Testers don’t prevent errors, and errors aren’t necessarily waste.
Testing, in and of itself, does not prevent bugs. Platform testing that reveals a compatibility bug provides a developer with information. That information prompts him to correct an error in the product, which prevents that already-existing error from reaching and bugging a customer.
Stress testing that reveals a bug in a function provides a developer with information. That information helps her to rewrite the code and remove an error, which prevents that already-existing error from turning into a bug in an integrated build.
Review (a form of testing) that reveals an error in a specification provides a product team with information. That information helps the team in rewriting the spec correctly, which prevents that already-existing error from turning into a bug in the code.
Transpection (which might happen as a part of something we now call prospective testing) reveals an error in a designer’s idea. The conversation helps the designer to change his idea to prevent the error from turning into a design flaw.
Automated regression checks reveal inconsistencies between some output and some automated oracle. That revelation prompts interactive retesting to evaluate whether the product, or the check, or our understanding of the relationship between them is incorrect or inadequte.
You see? In each case, there is an error, and nothing prevented it. Just as smoke detectors don’t prevent fires, testing on its own doesn’t prevent problems. Smoke detectors direct our attention to something that’s already creating smoke, so we can do something about it and prevent the situation from getting worse. Testing directs our attention to existing errors. Those errors will persist—presumably with consequences—unless someone makes some change that fixes them.
Some people say that errors, bugs, and problems are waste, but they are not in themselves wasteful unless no one learns from them and does something about them. On the other hand, every error that someone discovers represents an opportunity to take action that prevents the error from becoming a more serious problem.
As a tester, I’m fascinated by errors. I study errors: how people commit errors (bug stories; the history of engineering), why they make errors (fallible heuristics; cognitive biases), where we we might find errors (coverage), how we might recognize errors (oracles). From that perspective, I love errors. Every error that is discovered represents an opportunity to learn something. That learning can help people to change their products or themselves in order to prevent future errors — or to make changes to prevent present errors from going any further.
So, as a tester, I don’t prevent problems. I play a role in preventing problems by helping people to detect errors. That allows those people to prevent those errors from turning into problems that bug people.
Still squeamish about errors? Read Jerry Weinberg’s e-book, Errors: Bugs, Boo-boos, Blunders.
This post was lightly updated 2024-01-21.
Michael, I agree with that statement. It’s been a while since the discussion raised up on Twitter if testers can prevent bugs, and so on. So I like that approach, by dissecting a bug into its parts and define the relation of a tester to them.
A thought that came up while reading. I lately describe the job of a tester to raise the awareness about problems and risks.
You cannot prevent a problem when it’s already there. But a risk is something that has a probability of becoming a problem.
So you could say that testers are helping to prevent risks from turning into problems?
Michael replies: Yes; helping to.
Of course a tester herself cannot do anything against the fact of an uncovered risk, except raising the question, describing the risk and the potential impact on the product and the user. If someone is taking action upon that step does not lie within the tester’s hands.
Yes, exactly. Part of the goal here is to identify the extents and limits of what testing can do; to identify that testers serve the project but don’t run the project.
Nice post! thanks!
Anyway, I tend to say that “testers help to prevent errors” when they participate in the very beginning of a project asking questions like “what could happen if…”, and thanks to that, designers, analysts and developers think about those situations and their designs, analysis and software are prepared to these situations, before they commit the error.
I’d like to hear what you think. Thanks!
Michael replies: Yes; testers help to prevent errors. But we cannot say that the error has bee prevented unless something else has happened; unless someone has taken some preventive action.
I can almost agree that testing itself does not prevent problems, it just provides information.
However, I would disagree that ‘testers’ cannot prevent problems. As a tester I do a number of jobs, including pairing with a developer, which do directly prevent issues from occurring.
Michael replies: I love pairing with programmers. It’s the fastest way I know of getting feedback to the program, and it allows for a ton of learning for both of us.
When I’m pairing, it’s almost a certainty that the programmer will tend to be in the builder’s mindset. For that reason, I try to remain in the tester’s mindset. While we have very close social distance, I want to try to maintain maximum critical distance. The programmer is anticipating success, while I’m imagining potential failure. The programmer is thinking “how can I make this work”, while I’m thinking “how might this fail?”
Now, of course, to some degree, I might have a notion of risk, and offer a strategy to the programmer to avoid that risk. In that moment, though, I’ve popped into a programming role, and I have—even if only for a moment—have lost my testing focus. I hope the programmer, in that moment, adopts the testing role so that someone is critical of the idea. In any case, the programmer remains in charge of the programming. I might offer ideas, but he makes the decisions. So, I’d say what you say above just a little differently: “As a tester I do a number of jobs, including pairing with a developer, which is a really efficient way to help him and the team to prevent issues from occurring.”
Think of rallying. I’m in the co-driver’s seat, in close partnership with the driver. As collaborative as that might be, I want to be clear on our roles. My job is to help the driver by identifying potential problems and alerting the driver to them. That’s my specialty. The driver remains in the driver’s seat. Things tend to go worse if I relax my vigilance on reading the map, observing the landscape, and communicating with the driver. Things will go even worse if I try to grab the wheel or shift gears.
If we win the race, I want to make sure that the driver gets appropriate credit for the victory. I’m happy to say I helped.
Greetings Michael,
Why is the distinction important?
Regards,
Vernon
Michael replies: I sometimes see or hear people saying “I don’t want to focus on detecting problems; I want to focus on preventing problems.” Preventing problems is a terrific goal, and I like to help with that. As a tester, I have a specific contribution to make to the overarching goal of preventing problems: finding the problems that are already there. To do that, I must tune and polish my skepticism and my problem-detecting tools.
No matter what we might do to prevent problems, there will be problems. In the testing role, my contribution is to to explore and experiment with the product, and to help the team learn about the relationship between the product and the people it’s intended to serve. My emphasis is on finding hidden, deep, rare, subtle, surprising problems. That requires focus, practice, study, and skill. Focus on other parts of the task of preventing problems takes time, energy, and attention away from seeking and detecting problems that we haven’t yet managed to prevent.
Another goal here is to identify the extents and limits of what testing can do; to identify the fact that testers serve the project but don’t run the project. We’re not in the quality assurance business. We help the people who put the quality in. I believe it’s important for us to remain humble about that.
I think there’s a certain squeamishness surrounding talk about problems, such that the statement may really be amounting to “I don’t want to focus on detecting problems; I want to focus on avoiding controversy and bad feelings.” There are lots of ways to do that, of course. One powerful way is to focus on detecting problems so that people can deal with them, before they turn into bigger problems.
Jerry Weinberg’s answer to my second question here addresses the issue nicely. If we want to learn from mistakes, we must set ourselves to up find them and to learn from them.
I’d be interested in your take about the relationship between preventing problems and causing problems. Are there problems one could prevent but not cause? Are there problems one could cause but not prevent?
Michael, nice article. Short but sweet.
1) So, likening testers to smoke detectors. OK, I get it, but smoke detectors are algorithmic, and don’t have wisdom, only systematic knowledge, put there /by/ a wise human, at best. Maybe a better comparison would be likening testers to scientists? They observe, run tests, report back, but both have wisdom, unlike smoke detectors.
Michael replies: Sure; of course the work of a tester is more than algorithmic. I chose that simile to illustrate the fact that an agency that alerts us to an existing problem does not prevent that problem, but putting us on alert might help us to prevent a worse problem. In the same way, automated checks don’t prevent bugs; they prevent certain bugs from going undetected.
2) Quote: “Some people say that errors, bugs, and problems are waste”
I’ve been around a while, but never heard this stated before about bugs itself. I /have/ heard people say (or mean) that bug /fixing/ is a waste of time, in which I talk to them about it being a form of iterative learning. Their mindset usually comes from a place of believing that maintenance is a problem that should be avoided rather than an intentional part of your product investment mix.
3) On a similar note, I’d love to see a blog post on how to inform upper management about the pains of not focusing on maintenance as you go. So many “Agile” (big A, Agile) implementations are taking the “done, not perfect” approach (which is great if you are thinking of it in a ‘good enough’ testing sense) but not to the extreme we’ve started to see. That extreme results in mountains of technical debt that the same product management folks end up pulling their hair out about years later. However, they only see the criticality of it when it gathers in it’s mountain-like form. So, speaking outside of testing now, is that an “okay” practice for product management to do, even when they themselves feel the pain later and want to fix that stuff? (i.e. How do you get them to feel the pain earlier, in the small chunks that aren’t as visible, so we don’t have to get there first?).
I don’t know how to make someone feel pain (other than hitting them or insulting them, but I’m pretty sure that’s not what you meant). I can tell stories about other people’s experiences. If they’re in a class or a workshop I can give people an experiential exercise. I can point people to any number of books and blogs that refer to technical debt. In meetings, I can offer metaphors show charts and images to show the relationships between technical debt, cost, value, and risk (with emphasis on risk). Rob Sabourin likes to show this video and have people analyze it and discuss it. Nonetheless, sometimes people simply won’t understand consequences until they get dealt a painful experience—and maybe not even then.
Whether all this is “okay” is not for me to decide, and it’s not for you to decide either, in the sense that neither you nor I run the business. I can say that I don’t like being in situations like the one you describe, and I am concerned for people who are.
[…] Testers Don’t Prevent Problems Written by: Michael Bolton […]
Or we should say “testers detect errors to prevent problems getting worse”
Michael replies: I’d say it a little more specifically: “to help other people to prevent problems from getting worse.”
I think you confuse “testing” with “testers”.
Michael replies: I don’t think I do.
When testing, I don’t prevent errors.
I agree.
As a tester – I definitely do.
Are you changing the code yourself? Revising the requirements? Mandating a change in behaviour? Making decisions about the schedule? The budget?
I prevent errors when I’m asking “How will the feature we’re planning impact that other feature nearly forgotten feature?” Now we won’t create that specific error in our design.
All of a sudden there, “I” seems to have turned into “we”. Do you have control over the design? Or do you help the people who have control over the design?
I prevent errors when I push for a simpler architecture. Now everyone dealing with the product have less place to make mistakes.
What happens when you push for a simpler architecture, and people with authority decide against they perceive a risk in that idea? The decision to go complex was theirs, not yours. Subsequently, a problem arises associated with the more complex architecture. You didn’t put the problem in. Would you say you failed to prevent that subsequent problem? Or would you say that you tried, unsuccessfully, to help them prevent that problem?
I prevent errors by sharing my intended test ideas with the developer, who, being in the builder’s mindset (I really like this notion, thanks) will think of the happy path and now will take into consideration some error flows.
Did you write that code? Was that your code? No? Then you helped the developer to prevent problems in his code. And well done.
I prevent errors by opening defects that relate to usability and maintenance and following them to resolution – now the users operation teams will make fewer mistakes that will lead to errors.
You did not prevent the defects that you reported (there were there; you reported them), nor did you prevent the downstream problems (unless you revised the requirements, wrote the code yourself, or had managerial authority over the people who did the work). But you helped the people who fixed the problems that you reported. You helped to prevent a problem from getting worse.
I prevent errors when I sit with my product manager and teach him a bit more about how our software works – now the will know what requirements are problematic to pose.
You help that product manager to prevent problems in his work. Unless you’re actually doing his work, you’re not doing the preventing part either.
Testing, as it is a process of learning what I currently have in front of me, cannot prevent anything. It spots, reports, and then sometimes things are fixed. But as a tester I do more than testing – I utilize the knowledge I gained from my testing efforts to drive for a better product: error prevention is part of that (As is coming up with new feature ideas, tools & processes improvements and providing support to whomever needs it). It just happens that Since testing is a great practice of spotting errors, I’m just better at spotting them in advance than I am in thinking of new features.
The fact that sometimes things are fixed—and only sometimes—points to the fact that you, in the tester’s role do not prevent problems. You also seem to have a limited (albeit very common) notion of what it means to test. In the Rapid Software Testing namespace, to test does not simply mean to bang on keys, or to run automated checks. To test is to evaluate a product by learning about it through exploration and experimentation. That includes exploring and experimenting with ideas (via thought experiments), specifications (via review), or running products (via interactive testing). It includes modeling and conjecturing and resourcing and tooling and collaborating and studying and observing and lots of other things. All of these things help other people to prevent the mistakes that they might have made from getting worse. To say that we prevent problems is pretty much the same thing as claiming that we assure quality. We do neither of those, and there’s nothing wrong with that.
If you suddenly have the authority and the responsibility to change the code, or revise the specification, or shift the schedule to prevent a problem, you have for at least a moment stepped out of the tester role in into the builder’s role. Keeping clear on roles is important.
Let’s look at a slightly modified statement: “testers can help the business analysts / developers to prevent the errors that they do when injecting ambiguities / defects into specification requirements / code”.
Michael replies: That’s longer, but it’s not materially different from what I’ve been saying. Business analysts and developers do the prevention work, and testers help.
How may a tester do this? By providing them with feedback after performing specifications, architecture or code reviews, static or dynamic analysis, or testing, and thus enabling them to avoid repeating the same error in the future. This feedback should contain some “error root cause analysis” or at least hints about some possible causes of the business analyst / programmer’s mistakes.
Providing analysis of the problem might be of great benefit, but I don’t see any reason to say that the tester must or even should provide a cause.
Of course, this may be done only in a certain context, when the tester:
• has a developer / business analyst background, or
• knows about programming / requirements engineering, or
• has a solid architectural knowledge about the system being tested, or
• has been working since a long period of time with the same developers
A combination of the above factors provides best chances to perform efficiently this kind of “problem prevention”.
You’re not really confusing helping to prevent problems with preventing problems again, are you?
Two examples:
1. After a requirements document review, you find too many ambiguities related to the improper use of terms like “quick”, “flexible” or “enough” ( testability is critically affected ). You should warn the business analyst to avoid using these terms.
Yes; and the analyst must change the terms, or something about the way he uses those terms, for us to be able to say that a problem has been prevented.
2. Noticing a systematic computation error in case of a simple algorithm (add, multiply…) might suggest the use of a non-initialized variable involved in the computation. You should warn the programmer about this.
Yes; and the programmer must change the code, or something about the way she writes code, for us to be able to say that a problem has been prevented.
I have used frequently this approach in my tester career and considered this as one of the various informations that a tester should provide to the project team. In my opinion, the informations transmitted by a tester at bug reporting should be augmented also by product risk related informations and problem prevention efforts, in a proactive approach.
If you do this, you are probably being helpful.
[…] Blog: Testers Don’t Prevent Problems – Michael Bolton – http://www.developsense.com/blog/2016/05/testers-dont-prevent-problems/ […]
This post throws an interesting perspective. I agree that testing on its own is not a proactive approach to prevent errors. But if done the right way, it does teach the developers to keep an eye for potential bugs.
What about TDD. Can we say that this test – dev approach is proactive in preventing errors.
Michael replies: When someone uses information from testing and does something to change the product, we can safely say that that person has taken action to prevent a problem. That happens during TDD. The programmer is responsible for making the change; the programmer has prevented a problem.
[…] 1. Testers Don’t Prevent Problems […]
I agree with you, except for this statement:
“Providing analysis of the problem might be of great benefit, but I don’t see any reason to say that the tester must or even should provide a cause.”
You are in a big error here.
Michael replies: Thank you for pointing that out. Let’s examine your assertion and see if it holds.
If you look at the issue only from the tester’s viewpoint, you’re perhaps right.
Hmmm… you seem to be backpedalling pretty quickly, there.
But if you consider the team perspective on the issue, you’re absolutely wrong.
Careful there, Sunny Jim. 🙂
As a tester, I must be a team player. Here, there is no “should”, but a categorical “must”. And I must help the team in whatever possible context.
There is no “should”, but a categorical “must”? Where does that derive from? Stone tablets? A voice from the sky? The ISTQB syllabus?
Why to ignore, as a tester, the contexts of bug finding, reporting and retrospectives, why to ignore the benefit (for the team) of providing hints, possible causes, or even root-cause-analysis about the errors made by other members of the team?
I’m not ignoring the benefit of that. But notice how you’ve subtly changed “a cause” into “possible causes”; how you’ve changed “a cause” into “root cause analysis”; how you’ve changed “Providing analysis of the problem might be of great benefit, but I don’t see any reason to say that the tester must or even should provide a cause” into “ignoring the benefit”. In fact, you yourself acknowledge this, when you say “Of course, this may be done only in a certain context, when the tester…” Well, yes. And that, to me, rather takes the steam out of your objection that I am “in a big error here”, or “absolutely wrong”. If a tester is in that context; if a tester has a well-reasoned explanation; if a tester frames a guess about the as a guess, and so on, then the tester stands a good chance of being helpful. But if a tester doesn’t have the requisite knowledge or background or skills or mandate, then it might be a distraction, a waste of time, a pain in the ass to the rest of the team. It might also be a learning opportunity, a chance to develop those skills. So let’s drop the “absolute”, “big error” stuff here, okay?
This is one of the most direct ways of helping the team, as a tester. And by providing this information, the tester will have a major contribution in minimizing the “rework” quality costs.
I may only agree that this is not a simple task.
That would be a good idea, in my view.
As a tester, you must “steal with pride” various skills from other software engineering disciplines, as programmer, requirements engineer, or architect. By doing this, your potential for “providing hints, possible causes, or even root-cause-analysis about the errors made by other members of the team” will increase steadily. You should learn about Ishikawa diagrams, cause-effect analysis, 5-why and other quality management instruments, in order to be efficient, effective and repeatable in this difficult task. But, little by little, you will start to provide those “hints, possible causes, or even root-cause-analysis”.
Yes; and I would argue that it’s also a good idea to know when not to use these things.
And, by doing this, you will help the team.
I agree.
This is the reason that you could not see.
Please don’t mistake your misinterpretation and imprecision for something that I could not see. That won’t help the team.
I read your response, and I am far from convinced (thanks for the link about the roles though, it’s very interesting, and I suspect I’ll have to read it two or three more times to be process it).
Michael replies: I’m glad that you’re prepared to think about this.
It did help me spot the point of difference – the way I understand it, you define both “tester” and “prevent an error” in a far more restrictive manner than I do.
That’s almost certainly true, so we’ll try to clear that up.
First, I’ll sum up what I understood from your response, to make sure I understand you correctly:
Note to spectators: this is an excellent rhetorical trick and a really good thinking trick, too. Amit here is providing an insight into his interpretation of what I said. Those may be aligned, and may not be. If there are problems there, we’ll be able to detect them and sort them out. Many—perhaps most—arguments seem to be about conclusions when they’re really about premises. So, restating your interpretation of the premises is a powerful approach, like stepping through the code for a really hard-to-understand bug.
You claim that:
1) Preventing an error requires a generation of an artifact – a piece of code, a requirement, a document or a decision.
That’s close. I would say that preventing a problem requires something to change, and it requires a person responsible for making that change and authorized to make it to actually make it.
2) When I’m doing any of the above, I put aside my tester hat and take on the hat of the product manager(if I’m defining requirements) or of the developer (if I’m changing the code).
Yes; you take off one hat and put on another. You move from tester to builder; from tester to designer; from tester to manager. You adopt, however temporarily, those roles.
I don’t accept any of the two points above.
The first point is simply narrowing down the term “error prevention” to an absurd level – It is just as insisting that warning signs (such as”slippery when wet” or “winding road”) do not prevent accidents but only help prevent them.
They do not prevent accidents. They do help to prevent them. Do accidents happen even though the sign is there? Yes, they do. Are accidents prevented just because the sign is there? No, because the driver is free to ignore the sign (at his peril, of course). The sign helps some drivers, not others. The sign doesn’t fix the bend in the road and doesn’t change the weather. It reminds people of those conditions, but the sign doesn’t take the wheel. I’m sorry—and a little surprised—that you find this absurd. When I’m driving, I’m in charge of the car, and the sign isn’t. However, the sign prompts me to do two things: to think, and slow down. The sign helps me. The point is that both of my actions here are optional.
If you are walking in a dark room full of obstacles and I turn on the light – wouldn’t you say I prevented you from stumbling into things? Sure, you could prevent this by walking slowly with your hands reached, and even with the light on you might have been distracted enough to still hit something, but still – I did something, and the result is less errors.
Imagine that you’ve turned the light on, and I still stumble into an obstacle. Did you prevent me from doing that? Can I say that you failed me? You turned on the light; that helped. But isn’t my clumsiness my responsibility? My problem with your line of reasoning is that it explains success in one way, and it explains failure in a completely different way.
Or, if we take on another analogy – when I push a domino piece and start a chain reaction that breaks a window – did I break the window or merely “helped” it to break? The same goes for people – if my action triggers another action (that would not be triggered otherwise) of another person, and that action changes something in a manner that prevents an error – I have just prevented that error.
There’s a crucial distinction to be made here, I think: people are not dominoes. People have agency. The domino is not free to say “screw him; I’m responsible for my own actions”. A responsible adult receives your input as a tester, processes it, and makes a decision on what to do next. That person is authorized by the business to make that decision and (unless you that person yourself, or you have supervisory authority over that person), you are not so authorized. You do have power—substantial power, I expect—to describe, to explain, to persuade, to troubleshoot; to tell stories; to connect dots. Those are all good things. But to me, to say that you’re the person preventing the problem is to deny the agency of the person who makes the change. To say that you helped them is to provide yourself with appropriate credit for your role in the interaction. And good on you for that.
2) Not every time I specify requirements I take off my tester’s hat, and the same is true for some code changes one might do.
As above, I would suggest that you do take off those hats, however momentarily.
As a tester, I can’t really work with incomplete or flawed requirements, so I have a bunch of oracles I use to fill in the gaps when I’m testing.
Of course you can really work with incomplete or flawed requirements. I can prove that pretty easily, in two ways. 1) Neither you nor I have ever seen requirements that we know to be complete or perfect. Even if we’re really happy with them, we don’t know whether incompleteness or flaws are absent, or merely concealed. 2) You yourself say “you have a bunch of oracles that (you) use to fill in the gaps when (you’re) testing. You wouldn’t do that if you knew you had complete or perfect requirements.
If I share these oracles I don’t don the role of the product manager, I am doing what is expected of any tester – the same that is expected when I participate in a design meeting. I am expected to take part of this as a tester, not as a developer. Since I maintain my skills as a tester way more than I invest in my programming skills, such short intervention does not shake me out of my testing shoes (unlike when I’m creating an automated check, and then I am no longer functioning as a tester).
Sharing an oracle is something that anyone on the team can do. (I assume that you are using “oracle” in the Rapid Software Testing namespace: any means by which we recognize a problem when we encounter one during testing.) Everyone on the team is welcome to adopt a tester’s mindset; to recognize a problem. Anyone on the team is free to offer oracles. In the paragraph above, though, I don’t see a moment when you exit the testing role.
Unlike being responsible for quality, error prevention is something that is judged on a case by case basis – I do not need authority to enforce anything, since I might fail at preventing a certain error, but when things go my way and an error is prevented I have no problem taking full credit for that. Yes, even if someone else did the coding.
Again, to me, this is to explain success at preventing a problem in one way, but a failure to prevent a problem in a completely different way. “I told the programmer about a problem, and she fixed the bug. Yay me! I prevented a problem.” “I told the programmer about a problem, and she didn’t fix the bug. No skin off my back; she’s the one who could have prevented the problem, and she didn’t do it.”
I’d prefer to be humble, and in both cases say that I did my best to help others prevent a problem. If they acted successfully, I’m happy to have helped. If they didn’t act, or unsuccessfully, I’m also happy to have helped, though I do wonder if I could have been more helpful. And I’m willing and eager to examine that.
Hmm…
After reading your response I think that the issue I’m having is that it seems to me that you define “prevention” as “After taking action x, we are 100% certain that we won’t be seeing this error”, while I tend to interpret it in terms such as “after action X, we are seeing 30% less errors of that type”.
Michael replies: I’m defining “I am preventing” as “I am responsible for an action, and am taking that action to change the current situation, such that a certain problem does not happen”. Unless you have the authority to change something in the product, it seems to me that you cannot prevent any problem in particular from happening, with one important exception. Here is what you can prevent, as a tester: unawareness of a problem that exists. You can change the current situation by reporting your perception of an existing problem or risk. You did not prevent that problem from happening, and it will remain just as bad or get worse unless someone else takes action—presumably, although not necessarily, because you pointed out the problem. Still, if you hadn’t pointed it out, someone else might have noticed it.
Let’s follow some logic here: suppose you had not raised awareness of the problem, and nobody fixed it. Did you prevent the problem? I think you would say No. Suppose you had not raised awareness of the problem—someone else did&mash;and someone else fixed it. Did you prevent the problem? You had no part in it, so I would assume you would also say No.
In the next case, you noticed a smaller or earlier problem and someone else fixed it, preventing a bigger problem. At this point, I think you would claim that you prevented the problem. I would claim that you had a role in preventing the problem, but you did not.
This time you raise awareness of a similar small problem in the same way as before, and nobody does anything about it. Then the big problem happens. Did you prevent the problem? Did you fail to prevent the problem? Remember, the difference is in what others did or did not do; your actions were exactly the same in both cases. I don’t know what you would say to that. I know that I would say “you tried to help prevent the problem, but nobody acted on the information you provided.” And I think that would be entirely reasonable. But to say either you prevented or didn’t prevent the problem in this case misses the mark; neither is true.
Meanwhile, your example is very odd to me. Reducing the frequency or the count of observations of a problem is hardly preventing the problem, is it? Why 30%? Why not 90%? Or 10%? Or 31%?
I’m having quite a difficult time bending my mind around this definition, as it seems to me that in order to prevent an error, you have to be able to point at it and then an action will be taken to remedy that – but if you can point at the error, were fixing, not preventing.
Perhaps you’re having a hard time with that because I never said it.
When we alert people to an error, we are neither fixing nor preventing.
That’s right.
Most preventative actions I can think about are more like a warning sign than they are actually doing something – choosing a simple design over a complex one doesn’t mean we won’t be seeing any errors, only that we are likely to see less of them.
Am I missing here something?
Yes. You’re ignorning the relevance of agency. Do you choose the design? Do you make the decisions? Do you write the code, set the budget, hire and fire the programmers, tell them what and what not to do such that you can make it stick?
P.S.
The point you raise about people having agency is interesting, but I don’t think it’s relevant to this case – my measurement is simple: I can confidently say that without my action, the error would have been done.
I’m not sure you can be so confident; see above. How do you know that no one else would have noticed the problem you noticed a little while later?
If I try to apply your definition of “preventing a problem requires something to change”, I am changing the way people think – their awareness to potential problems, their priorities, the information they have about a problem. Do any of those count as “something”?
When you change the way people think, are you performing surgery? Drugging them, and then brainwashing them? When they change the code to fix a bug, is it because you are controlling their minds and their fingers? Or are they changing the way they think with your help? Are you changing the code, or are they changing the code with your help?
While it’s a bit tough to speak about being “authorized” to make those changes (I can’t control how’s anyone thinking), I would definitely say this is my responsibility as a tester to have an impact on the way others think and act.
You can have an impact. No one is saying that you can’t have an impact. You cannot control people. You can tell compelling stories, but as a tester you cannot coerce people into doing things. You can help them to change themselves or their products, but they are not your puppets. You work for them. You help them. Am I right?
I said:
As a tester, I must be a team player. Here, there is no “should”, but a categorical “must”. And I must help the team in whatever possible context.
Your reply was:
There is no “should”, but a categorical “must”? Where does that derive from? Stone tablets? A voice from the sky? The ISTQB syllabus?
It seems I hit a sensible point here. I do not understand your position. Perhaps you may explain to the testing community why a tester must NOT be a team player!?
Let’s trace this through. In any earlier comment, you said, “This feedback should contain some ‘error root cause analysis’ or at least hints about some possible causes of the business analyst / programmer’s mistakes.” To that, I responded that I didn’t see any reason why a test must or even should provide a cause. You responded that in say that, I was “in a big error”. You didn’t actually respond to that, except to assert that you must help the team in whatever possible context. In other words, you’re simply generalizing your earlier assertion.
Let me be clear: I agree that a tester must be a team player, but that does not mean that the tester must help the team in whatever context possible; that second “must” is the one that I’m disagreeing with. For instance must the tester help the team when the team has asked the tester not to help, thank you? Must the tester help the team when the tester does not feel competent and is not receiving appropriate supervision? Must the tester help the team when the team is trying to hide information or risk from management? If we could see the stone tablets, or hear the voice together, or analyze the syllabus, we could talk about what they’re saying.
I started working in IT in the 70’s, as a programmer. Real-time kernels, drivers, multitasking, industrial systems, stuff like that. Since ’95 I do also testing. Meantime, I have done some requirements engineering too, and learned a lot about software architectures. But, not depending on the role I performed, I always considered myself as a team-player and for me, “must help the team” was and will be one of the fundamental things in software.
There’s a big difference between “must help the team” and “must help the team in whatever possible context”. There’s also a difference between “not always be a team player” and “must never be a team player”, and it seems to me you’re confusing the two. Maybe not; feel free to clarify.
You said:
“Let me be clear: I agree that a tester must be a team player, but that does not mean that the tester must help the team in whatever context possible; that second “must” is the one that I’m disagreeing with. For instance must the tester help the team when the team has asked the tester not to help, thank you? Must the tester help the team when the tester does not feel competent and is not receiving appropriate supervision? Must the tester help the team when the team is trying to hide information or risk from management?”
All these contexts are not “friendly” to the testers, and they prove that some necessary preparatory steps were skipped:
1. the team has asked the tester not to help, thank you? – perhaps some training on feedback, communication, emotional intelligence should be done, before trying helping others?
Yes. Training on feedback, communication, etc. seems to me to be an awfully good idea in many contexts, including that one.
2. Must the tester help the team when the tester does not feel competent and is not receiving appropriate supervision? – perhaps some training on testing foundations or some missing testing skills should be acquired, before trying helping others?
Yes; this seems to me to be a good idea in such circumstances.
3. Must the tester help the team when the team is trying to hide information or risk from management? – perhaps the team has not made yet through the “forming, storming, norming, performing” steps?
Yes; or whatever model you choose for developing understanding of roles, responsibilities, and commitments.
Now, to be honest, I was talking about contexts where the testing activity is well understood, prepared and performed, where the tester is accepted as a real team member. In these contexts, helping the team is a must.
Welcome to my world. In even the highest-functioning organizations I’ve worked with, you only have to look a little to one side before you find that testing is not well understood, prepared, or performed. In many places, I see a good deal of shallow agreement about that stuff. This is an attempt to look at the issue more precisely; to understand the extents and limits of what a tester can reasonably claim within the testing role.
In the contexts you mentioned, not only helping is impossible, but also testing.
In those contexts, neither helping nor testing is impossible, but as you note and as I agree, some other things will have to shift if helping or testing can be done.
Now, to finish our nice debate, “Sunny Jim” will tell you only that he is in a very good company when saying that defect prevention is a testing activity.
Are you appealing to authority, or are you reasoning this out for yourself?
A short list of references:
Boris Beizer – “More than the act of testing, the act of designing tests is one of the best bug preventers known. The thinking that must be done to create a useful test can discover and eliminate bugs before they are coded – indeed, test-design thinking can discover and eliminate bugs at every stage in the creation of software, from conception to specification, to design, coding and the rest.”
I agree with Boris to the degree that the act of designing tests may help people to prevent bugs. Yet designing a test, on its own, does not prevent a bug. And unless you did the coding, you didn’t prevent the bug—even though you might have been enormously helpful to the person who did. I’m curious as to why you wouldn’t be entirely pleased with that.
James Bach – “Process evolution in a mad world”, published 1994; please check the “Prevention” chapter in the Appendix “Key questions for bug review”.
I don’t have access to that material, but it’s been 22 years since it was published. As we’ve thought and reflected deeply over the last 22 years, I’ve changed my thinking and James has changed his. Based on my current and very frequent interactions with him, and whatever that article actually says, I can tell you that James would not agree that testers prevent bugs. I can tell you that with great confidence because this post is profoundly influenced by a recent conversation between us.
Should I mention also Cem Kaner, Elisabeth Hendrickson, Ilene Burstein, Gojko Adzic?
Mention them if you like. I don’t think any of them have responded to the point raised in this post, though. Cem himself was the first person who planted in my head the idea that testers don’t assure quality, which is very similar indeed to the point I’m making here. So it would surprise me if he suddenly started claiming otherwise.
Have you noticed the testing community reaction to your article?
Yes. You can have a look on Twitter, if you like. Some people disagree, and disagree forcefully. Others agree. You’ll notice, however, that the ones who disagree most strenuously are not making clear the distinction between [helping others to prevent problems] and [preventing problems]. They don’t feel that’s significant. I see at least two interesting things about this. One: those who disagree have disagreed with on other things before on the same kinds of basis: they don’t understand the motivation for the distinction, or they understand it and declare it doesn’t matter. That’s okay for them, for now, or maybe forever. Second, it feels to me very similar to some of the reactions to this post; a kind of panic over the idea that testers don’t run the show.
Meanwhile, other people who disagreed at first have bothered to reason this through, and are now fine with it. Some will always disagree. Oh well.
So you see, I’m not the only one out there.
People who intend to vote for Donald Trump could also say “See, I’m not the only one out there.” No, I’m not equating Boris Beizer or anyone else to a Trump voter. What I am saying is that appealing to what others say doesn’t prove your point for me.
I’m trying to follow your argument, and I’m having some real trouble with how I understand your definition of prevention.
1. What is the difference between the action of providing information to any other action?
The answer to this is straightforward: providing information is neither necessary nor sufficient to change a situation. For example: I have said that several times in these comments (that is, I have provided information) and yet I have not been able to change your mind (which would prevent what I believe to be further disagreement on your part). I’ve provided information, yet I cannot change you.
2. What does it mean that a *certain* problem does not happen? can’t someone prevent errors unintentionally (for instance, by implementing a simple design, I would say that a developer is preventing problems even without having something specific in mind)?
I did not mean “certain” in the sense of “for sure”. I meant “certain” in the sense of “specific”, “particular”, or “given”; as when you have a conversation with a certain person.
3. What about reducing the risk of a problem to happen? does this fall under preventing or is it something else altogether? If I have 17% less chance of an error occurring(17 is just my favorite random number) due to some action – I’d say that this was a preventative action.
If you are taking the action that reduces the probability of a problem, you are undertaking a preventative action. The key is that you are taking an action. Now, if by reporting an error that you’ve seen or a risk that you perceive, it seems to me there’s a far greater than 17% chance that that person will be aware of the consequent problem or risk; it seems near certain to me that they will become aware of the problem and will take action. And that is an attempt, on your part, to help someone who could prevent the existing problem from remaining or from leading to a worse problem. But here’s the thing: no one who changes the situation is being controlled by you.
4.You mention agency as an important barrier – if i follow you correctly, the only one preventing errors is the last agent in the chain.
Not necessarily the last. People who prevent problems are those who take necessary and sufficient action to change the situation. The fact that you provided information is neither necessary nor sufficient to do that. As I’ve said before, what you have prevented is oblivion, unawareness of the problem. That’s a good thing. Yay. Enjoy it. Take credit for it.
Why? In such chain of agency, every agent is able to perform only by relying on the other agents. A construction worker won’t be able to build much without the crane operator to do some heavy lifting (which, in turn, needs someone down below to tie things to the crane), and a Edmund Hillary wouldn’t have gone anywhere without the Sherpa aiding him.
Do you understand the difference between doing something and helping someone to do something? Your programmers do not rely on you to change the product. They DO rely on you to address their oblivion, but that’s a different thing, in my view, because they have agency to change the code or to ignore you. They are not your puppets.
Also, you have raised a valid question about how can I be certain that without my action an error would have been introduced – In some (rare) cases, I’m the only one around with the relevant piece of knowledge, in other cases, I get a bit late and get a response of “we didn’t think about it”. In other cases I can’t be certain – maybe someone else would have prevented the error in a later phase.
That’s right. So what was your part in all this? It seems to me you have not addressed the question of logic that I raised in my last reply to you.
The notion of agency is completely irrelevant when we look at the results – if on average you stub your foot 17 times when crossing a dark room, but only 1 time every 500 times you cross a well lit room, then even though the light is only helping you avoid hitting random stuff in the room, the fact is that light==> no pain in toes. Yes, you might have prevented the pain by wearing shoes, or sweeping the room – there’s more than one way to prevent an error.
sta
Yes, there is more than one way for me to prevent that error. But unless you have control over the room (such that you remove the obstacle yourself), or unless you take control over me (such that I become an automaton under your control), you’re not preventing me from stubbing my toe. If I respond to your cry of “watch out!”, I (not you) am the one changing my actions, in which case you helped me by prompting me to take that action. Moreover, you did not prevent my original error of me attempting to walk through the dark room; nor did you prevent the earlier error of leaving the lights out or putting an obstacle on the floor.
Nope, still not seeing the difference between shoving stuff aside so that you have a straight path to tread and giving you a warning about obstacles in the room.
Michael replies: If you can’t see the difference between those two things—between effecting an actual physical change on the environment on the one hand and a warning on the other—I afraid can’t help you. I urge you to think about the possible differences between them. Then come back, and we’ll talk about it.
In the first case, I changed the room so you are less likely to stub your toe. In the second, I changed you so you are less likely to stub your toe.
Let us assume here that I am responding as a developer. I would say this: YOU. DID. NOT. CHANGE. ME. You gave me information. Unless I’m open to it; unless I decide to pay attention to you; unless I do something that I choose to do, nothing different happens. The suggestion that you changed me just because you told me something overstates what actually happens and suggests that I am simply your marionette. Do you realize how insulting it might be to presume that you changed me just because you told me something? As though I were not in control of my own mind and my own decisions?
What is the difference? Sure, I’m not controlling you, but merely by informing you I change you from “unaware” to “aware” and from careless to cautious.
No. If I am a developer, you may have changed me from aware to unaware, if the message landed and I was open to receiving it. You cannot reasonably claim that you changed me from careless to cautious. To make that claim overstates your influence on me, and negates my own agency. You cannot change me without me being the agent of that change.
Notice, by the way, that you are precisely proving my point. I’m alerting you to what I consider to be a problem in the way that you’re looking at this, and you’re not changing your mind about it. And if I eventually persuade you, who would have made a change?
At least, this is what will happen if I do my job properly, and I might fail if I convey my message in a way you can’t understand it or even just deliver it at the wrong time. Just as I might miss something when I’m clearing the path for you in the first case.
People may not change even if you do your job properly.
I’ll try to go quickly about the points I saw in your last response:
1. sufficient & necessary: True, some problems may be avoided without that information, or occur with my warning. But this is the case for most preventative actions. When I clear a path in the room, you might stray off a bit and stub your toe – so it’s not sufficient, and you can avoid pain by walking very slowly and feeling your way through, so it’s also not necessary, yet if I follow you – in the case that I’m clearing a path in the room, you do agree I have taken a preventative action. So I don’t think that “sufficient” or “necessary” are relevant indicators here.
Difference between doing and helping – I think I get it. I still don’t see how is it applicable to reducing risk or preventing errors. Besides, with strong enough collaboration, it is pretty difficult to say that one side is doing the work while the other helps.
In other words, you’re saying that there’s nothing that you can do to control me without actually doing something, although you can foster an environment in which I’m less likely to hurt myself. The question is: as a tester, what is it that you actually to do other than to raise awareness? If you take some kind of management action, you are at that moment not in the testing role. If you change the product, or change the environment, you are at that moment in a developer role. Of course you can change roles at any time. There’s nothing wrong with that, as long as everyone is aware that you’re not in the tester role in that moment. In the moment when you’re doing something that is guaranteed to affect the landscape or the developer, you are not in the testing role.
For your example (sorry for missing them) – If I said nothing and nothing was done, I did not prevent anything.
If I raised an alert, someone changed something following my action, and a bigger problem was averted, I definitely did prevent this error. As did the one fixing the problem – I’m collaborating, not hoarding credit.
If I raised a concern, and nothing was done following it – Then yes, I have failed to prevent an error. That’s ok, since I’m not here (with regard to the subject of error prevention) to guarantee 100% prevention, only risk reduction.
So in the first case, you definitely provide prevention, and in the second you claim that you don’t provide prevention; only risk reduction. This is to explain success in one way and to explain failure in a different way.
I would like to add another scenario to this – Assuming someone else is designing a new feature, and just before they start, I say “please take notice of the fact that what you are about to change could also affect that old, seemingly unrelated feature”. Armed with that knowledge (that was very unlikely that this person would have known about if it wasn’t for me) – the design is different and an error was averted. Did I prevent an error? I would say yes.
I would say that you helped to prevent that person from shooting himself in the foot. Listen: what’s the problem with saying that you helped? What’s the problem with being modest? With acknowledging the limits of your influence, as well as the extent of it?
Lastly, you mention “action” as a game changer. Why do you exclude speaking up as an action? It might not qualify as what Austin called “speech act”, but it is an action nonetheless.
There are different kinds of actions, and different kinds of effects on different things, and there are different kinds of control. If you’ve read Austin, you probably understand the difference between speaking to an actor and controlling a marionette. They’re both actions. As a tester, you’re neither the director of the show nor the puppet master. In one case, you run the risk of insulting the director; in the other, you run the risk of insulting the performer. Yet even if you’re in neither role, you can help both of them.
I firmly agree with this post. The relationship between DEV and QA should not be like an artist and an art critic. It should be one of writer and copyeditor, each working together to create a quality product.
Getting QA early in the process helps quality greatly, involving them as soon as the stories are written, sussing out if the requirements and acceptance criteria are as black and white as possible, sussing out any areas of gray.
It seems to me that there’s somewhere a gap we won’t be able to bridge (at least, not in written form), so let’s leave it as is for the moment, since I think that everything I will say will be just repeating myself in slightly different words. So – thank you for the discussion, it did force me to think quite a bit, even if I still can’t agree with your message here.
I don’t actually care if you agree; it doesn’t bother me if you disagree. I’m not the boss of you. I’m delighted to prompt thinking about it, though.
Lastly, though – you raised another question, that must be answered – “what’s the problem with saying that I helped?”
Well, saying that I help isn’t a problem at all – Helping is one of my main functions as a tester – I help developers do a better work, I help support people by answering their questions or investigating on their behalf, I help product owners by providing them information they need to better decide in whatever issue they deal with.
I help a lot.
The problem is when “helping” is all that I do. I see the helper role as a role with no responsibility (if I gave the proper aid and something failed, I’m ok, ain’t I? ).
Michael replies: I don’t understand why the helper role means “no responsibility”. It seems to me that it would include, at least, the responsibility to help.
As a tester, I see myself mainly as a part of the development team (not necessarily the programming team), and I share the responsibilities and power of influence like any other team-member.
So, when something fails – it is my responsibility, the same way it is the responsibility of every other person on the team. Testing, gathering information and passing it on to those that need it, isn’t my goal, it’s my method of achieving my goal – releasing a good product. I don’t release the product my self, but neither does any one developer or the product manager. It’s something I do as part of a team.
Not every action I do as a tester is a testing action – I take an active part in some design meetings, writing algorithms or requirements and so on. But when I’m doing those actions, I do this as a tester, not only because I try to bring the same perspective I use when testing, or because I identify as a tester, but also because all of those things are what is expected from anyone in a testing role (in the environment I’m currently in).
Of course it’s true that not every action you do as a tester is not a testing action; just as not everything you do as a parent is a parenting action. I think there’s a real problem in failing to recognize that a role is a hat, and not a full-body tattoo; that roles can be picked up and put down where necessary; and that roles can be nested inside each other. I’ve written about that here. It seems to me clear that when you, Person Called Tester, are doing design and writing work, you are temporarily taking on a non-testing role; just as a goalie who runs downfield to score a goal is still the goalie, but is leaving the net unattended—or perhaps leaving the net to be tended by a non-goalie. That might be fine, and it might not; that evaluation would by a myriad of context factors. The point is to notice it happening; to be aware when the net is empty.
We can discuss the value, the cost, and the risk of strategic moves that involve shifting roles, of course. It would seem odd, though, if someone were to declare that such moves were immaterial.
Thanks for another insightful blog post Michael.
I have the same viewpoint as yourself. When talking about this subject, I like to relate this whole conversation to “time”.
If I am testing a requirement very early on when the idea for the new software or feature has been formed, and I ask a question that causes people (developers, designers or the product owner) to respond with “I didn’t think about that”, then some might say that a problem has been prevented in the design/the code/production, etc (i.e. some future activity to follow in time).
But at that point in time, we’ve actually just discovered a problem with the requirement (or idea) of the software.
The idea of prevention only relates to whether someone else (e.g. the designer or developer, etc) chooses to use that information (or remembers it!) later in time, when they are designing or programming the application.
Because of this, there is actually never any guarantee that discovering a problem or risk sooner will prevent it’s manifestation during later activities.
Hence why I don’t use the words “preventing a problem” anymore.
I used to say that though, when I was ignorant to this way of thinking… And it did used to feel good saying that – until I got stung with problems in the live environment that I had previously discovered and thought we had “prevented”.