The other day on Twitter, Cory Foy tweeted a challenge: “Having a QA department is a sign of incompetency in your Development department. Discuss.”
Here’s what I think: I’m a tester, and it’s time for our craft to grow up. Whatever the organizational structure of our development shops, it’s time for us testers to get out of the Quality Assurance business.
In the fall of 2008, I was at the Amplifying Your Effectiveness conference (AYE), assisting Fiona Charles and Jerry Weinberg with a session called “Testing Lies”. Jerry was sitting at the front of the room, and as people were milling in and getting settled, I heard him in the middle of a chat with a couple of people sitting close to him. “You’re in quality assurance?” he asked. Yes, came the answer. “So, are you allowed to change the source code for the programs you test?” No, definitely not. “That’s interesting. Then how can you assure quality?”
A good question, and one that immediately reminded me of a conversation more than ten years earlier. In the fall of 1996, Cem Kaner presented his Black Box Software Testing course at Quarterdeck. I was a program manager at the time, but the head of Quality Assurance (that is, testing) had invited me to attend the class. As part of it, Cem led a discussion as to whether the testing group should really be called “Quality Assurance”.
Cem’s stance was that individuals—programmers and testers alike—could certainly assure the quality of their own work, but that testers could not assure the quality of the work of others, and shouldn’t try it. The quality assurance role in the company, Cem said, lay with the management and the CEO (the principal quality officer in the company), since it was they—and certainly not the testers—who had the authority to make decisions about quality.
Over the years he has continued to develop this idea, principally in versions of his presentations and papers on The Ongoing Revolution in Software Testing, but the concept suffuses all of his work. The role for us is not quality assurance; we don’t have control over the schedule, the budget, programmer staffing, product scope, the development model, customer relationships, and so forth. And that’s okay; that’s management work, and that’s management’s job.
When we’re doing our best testing work, we’re providing valuable, timely information about the actual state of the product and the project. We don’t own quality; we’re helping the people who are responsible for quality and the things that influence it. “Quality assistance; that’s what we do.”
More recently, in an interview with Roy Osherove, James Bach also notes that we testers are not gatekeepers of quality. We don’t have responsibility for quality any more than anyone else; everyone on the development team has that responsibility.
When he first became a test manager at Apple Computer way back when, James was energized by the idea that he was the quality gatekeeper. “I came to realize later that this was terribly insulting to everyone else on the team, because the subtle message to everyone else on the team is ‘You guys don’t really care, do you? You don’t care like I do. I’m a tester, that means I care.’ Except the developers are the people who create quality; they make the quality happen. Without the developers, nothing would be there; you’d have zero quality. So it’s quite insulting, and when you insult them like that, they don’t want to work with you.”
Last week, I attended the STAR East conference in Orlando. Many times, I was approached by testers and test managers who asked for my help. They wanted me to advise them on how they could get programmers to adopt “best practices”. They wanted to know how they could influence the managers to do a better job. They wanted to know how to impose one kind of process model or another on the development team. In one session, testers bemoaned the quality of the requirements that they were receiving from the business people (moreover, repeating a common mistake, the testers said “requirements” when they meant requirement documents). In response, one fellow declared that when he got back to work, the testers were going to take over the job of writing the requirements.
In answer to the request for advice, the most helpful reply I can give, by far, is this: these are not the businesses that skilled testers are in.
We are not the brains of the project. That is to say, we don’t control it. Our role is to provide ESP—not extra-sensory perception, but extra sensory perception. We’re extra eyes, ears, fingertips, noses, and taste buds for the programmers and the managers. We’re extensions of their senses. At our best, we’re like extremely sensitive and well-calibrated instruments—microscopes, telescopes, super-sensitive microphones, vernier calipers, mass spectrometers. Bomb-sniffing detectors. (The idea that we are the test instruments comes to me from Cem Kaner.) We help the programmers and the managers to see and hear and otherwise sense things that, in the limited time available to them, and in the mindset that they need to do their work, they might not be able to sense on their own.
It’s not our job to take over anything, like writing the requirements documents. It’s absolutely fine for us to do any kind of work that we’re asked to do. When we take on such work, though, it’s important to be aware and to remind others that our testing work is not getting done.
Listen: if you really want to improve the quality of the code and think that you can, become a programmer. I’ve done that. I can assure you that if you do it too, you’ll quickly find out how challenging and humbling a job it is to be a truly excellent programmer—because like all tools, the computer extends your incompetence as quickly and powerfully as it extends your competence.
If you want to manage the project, become a project manager. I’ve done that too, and I was pretty good at it. But try it, and you’ll soon discover that quality is value to some person or persons who matter, and that your own standards of quality are far less important than those who actually use the product and pay the bills. Become a project manager, and you’ll almost immediately realize that the decision to release a product is informed by technical issues but is ultimately a business decision, balancing the value of the product and bugs—threats to the value of the product—against the costs of not releasing it.
In neither of those two roles would I have found it helpful for someone who was neither a programmer nor a project manager to whine to me that I didn’t have respect for quality; worse still would have been instruction on how to do my job from people who had never done that kind of work. In both of those roles, what I wanted from testers was information.
As a programmer, I wanted to know about problems the testers had found in my code, how they had found them. I wanted their help in identifying risk, and the ways in which the product might not work. I appreciated hearing about the clues that they noticed, and the steps I could take to point me towards finding problems myself.
As a program manager, I wanted to know what information the testers had gathered about the product. I wanted to know how they had configured, operated, observed, and evaluated the product to get that information. If our product were suddenly working differently from the ways in which it had always worked, I wanted to know about that. I wanted to know about inconsistencies within the product. The testers were able to tell me how our product worked in comparison to other products on the market; how it was worse, how it was better. I wanted to know how the product stacked up against claims that we were making for it.
So: you want to have an influence on quality, and on the team. You do that by helping the people who build the product. Want to know how help the programmers in a positive way?
- Tell the programmers, as James suggests in the interview, that your principal goal is to help them look good—and then start believing it. Your job is not to shame, or to blame, or to be evil. I don’t think we should even joke about it, because it isn’t funny.
- You’re often the bearer of bad news. Recognize that, and deliver it with compassion and humility.
- You might be wrong too. Be skeptical about your own conclusions.
- Focus on exploring, discovering, investigating, and learning about the product, rather than on confirming things that we already know about it.
- Report what you’ve learned about the product in a way that identifies its value and threats to that value.
- Try to understand how the product works on all of the levels you can comprehend, from the highest to the lowest. Appreciate that it’s complex, so that when you have some harebrained idea about how simple it is to fix a problem, or to find all the bugs in the code, you can take the opportunity to pause and reflect.
- Express sincere interest in what programmers do, and learn to code if that suits you. At least, learn a little something about how code works and what it does.
- Don’t ever tell programmers how they should be doing their work. If you actually believe that that’s your role, try a reality check: How do you like it when they do that to you?
Want to know how to help the managers?
- Provide them with the information they need to make informed decisions, and then let them make the decisions.
- Remain fully aware that they’re making business decisions, not just technical ones.
- Know that the product doesn’t necessarily have to hew to your standard of quality.
- It’s not the development manager’s job, nor anyone else’s, to make you happy. It might be part of their job to help you to be more productive. Help them understand how to do that. Particularly draw attention to the fact that…
- Issues that slow down testing are terribly important, because they allow bugs the opportunity to hide for longer and deeper. So report not only bugs in the product, but issues that slow down testing.
- If you want to provide information to improve the development process, report on how you’re actually spending your time.
- Note, as is so commonly the case, why testing is taking so long—how little time you’re spending on actually testing the product, and how much time you’re spending on bug investigation and reporting, setup, meetings, administrivia, and other interruptions to obtaining test coverage.
- Focus on making these reports accurate (say to the nearest five or ten per cent) rather than precise, because most problems that we have in software development can be seen and solved with first-order measures rather than six-decimal analyses derived from models that are invalid anyway.
- Show the managers that the majority of problems that you find aren’t exposed by constant, rote repetition of test cases, but by actions and observations that the test cases don’t cover—that is, by your sapient investigation of the product.
- Help managers and programmers alike to recognize that tools can be used for more, far more, than programming a machine to pound keys.
- Help everyone to understand that tools extend some kinds of testing and greatly limit others.
- Help to keep people aware of the difference between testing and checking. Help them to recognize the value of each, and that excellent checking requires a great deal of testing skill.
- When you’re talking about your work, emphasise that your job is skilled exploration of the product, and help managers to realize that that’s how we really find problems.
- Help the team to avoid the trap of thinking of software development as a linear process rather than an organic one.
- Help managers and programmers to avoid confusing the “testing phase” with what it really is: the fixing phase.
- When asked to “sign off” on the product, politely offer to report on the testing you’ve done, but leave approval to those whose approval really matters: the product owners.
Want to earn the respect of your team?
- Be a service to the project, not an obstacle. You’re a provider of information, not a process enforcer.
- Stop trying to “own” quality, and take as your default assumption that everyone else on the project is at least as concerned about quality as you are.
- Recognize that your role is to think critically—to help to prevent programmers and managers from being fooled—and that that starts with not being fooled yourself.
- Diversify your skills, your team, and your tactics. As Karl Weick says, “If you want to understand something complicated, you have to complicate yourself.”
- As James Bach says, invent testing for yourself. That is, don’t stop at accepting what other people say (including me); field-test it against your own experience, knowledge, and skills. Check in with your community, and see what they’re doing.
- If there’s something that you can learn that will help to sharpen your knowledge or your senses or your capacity to test, learn it.
- Study the skills of testing, particularly your critical thinking skills, but also work on systems thinking, scientific thinking, the social life of information, human-computer interaction, data representation, programming, math, measurement…
- Practice the skills that you need and that you’ve read about. Sharpen your skills by joining the Weekend Testing movement.
- Get a skilled mentor to help you. If you can’t find one locally, the Internet will provide. Ask for help!
- Don’t bend to pressure to become a commodity. Certification means that you’re spending money to become indistinguishable from a hundred thousand other people. Make your own mark.
- Be aware that process models are just that—models—and that all process models—particularly those involving human activities like software development—leave out volumes of detail on what’s really going on.
- Focus on developing your individual skill set and mindset, so that you can be adaptable to any process model that comes along.
- Share your copy of Lessons Learned in Software Testing and Perfect Software and Other Illusions About Testing.
Ultimately, if you’re a tester, get out of the quality assurance business. Be the best tester you can possibly be: a skilled investigator, rather than a programmer or project manager wannabe.
Note: every link above points to something that I’ve found to be valuable in developing the thinking and practice of testing. Some I wrote; most I didn’t. Feast your mind!