There’s a popular trope in the software development world these days that suggests that everybody on the team is responsible for testing.
With that idea in mind, some people take an extreme position: since everyone tests, no one needs dedicated testers any more. Developers can do all the testing; or business analysts can do all the testing; or the customers can do all the testing.
Then there’s another notion (which, exasperatingly, comes mostly from certain testers) that developers can’t test, and that every development team needs a dedicated tester, or a dedicated test team.
Both of these notions are, at their extremes, ill-considered and foolish. They’re instances of a pattern I sometimes call the Tyranny of Always.
It is absurd to say that developers can’t test. In the course of building a product, developers do plenty of testing: they write a bit of code; try it and see if it works; if it doesn’t, fix it, and if it does, move along. A developer isn’t able to produce useful code reliably without at least trying something about it every now and again. Yet it’s risky to assume that developers will have the time, motivation, incentives, and perspectives necessary to do all of the testing work.
To reduce the foolishness, let’s start with the reasonable parts. It’s true that everyone on the team might do testing work, or might have a role to play in helping others to test. It’s also true that testing means different things to different people. So does the quality of testing work.
Most people are not software developers, and most software products are built for people who are not software developers. People use software because they have a problem to solve. They want to accomplish tasks, to learn about or analyze things, or to be entertained. The user’s mindset is “I need something to help solve my problems.” Users aren’t interested in building software; that’s why they buy it, or ask developers to build it. Developers have motivation, knowledge, and skills to be able to say “I will make things that end your problems!” Given that the users know what they want, and the developers know how to build things, each group should be able to do high-quality testing work.
The quality of testing is subject to the Relative Rule. It’s embedded in a set of relationships between the testing work, the product and its state, and the different missions of testing in different contexts. The meaning, significance, and value of testing is a matter of perspective.
The perceived value of testing work is strongly affected by critical distance. Critical distance is the difference between perspectives — how close or far away we might be from what we’re observing and considering. I want to suggest that aside from users’ and developers’ perspectives on the product, there might be another perspective worthy of attention.
Close Critical Distance and the Builder’s Perspective
It’s fairly easy to see how being at close critical distance to the code can help testing. That’s the motivation for developer testing, or for “glass-box” or “open-box” testing. When you understand code, and coding, you will be more able to see problems in code. If you understand the functions in your product and the code that implements them, you’ll have ideas on code-related problems to test for, and how to test for them, and when to test for them. Even if you’re not familiar with the code specific to the product, familiarity with code in general will help you to review or to test it.
The person most familiar with the code — the person at closest critical distance — is the person writing it, mostly getting it right with the occasional stumble. All developers experience little misunderstandings, make little mistakes, encounter little problems. Just as writers notice and fix lots of their own errors in the course of writing, developers notice and fix plenty of problems quickly and easily as they write code. Disciplined developers apply approaches and tools in their work to help prevent such problems from evading them: TDD, or test-first development; checking built into the IDE; running the code after building it; contract testing… These things provide immediate feedback and alerts when there are problems in the code. This is testing at close critical distance.
On the surface, the testing that’s a part of disciplined programming might seem like all we need; who better to test something than somone who already has intimate knowledge of it? Wouldn’t the developer’s own testing be enough to show that “it works”? Why not get the developer to do all of her or his own testing?
One problem is that “it works” means “it appears to meet some requirement to some degree“. “It works” doesn’t mean “it meets everyone’s requirements entirely“, and it doesn’t mean “no problems, ever, for everyone“. Some problems are easy to see when we’re immersed in the activity of making something, in the moment of creation or very shortly afterwards. When we’re close to the work, other problems are harder to see because we’re close to it.
Making something and getting it done requires focus. Whenvever we’re focused on something, we are to a significant degree defocused from everything else. Envisioning success — which is necessary to built a product — requires us to some degree to dampen our concerns about the possiblity of failure. That’s not entirely a bug; that’s a feature when we consider the alternatives: “overthinking”, or “analysis paralysis”. Worrying about every dimension of trouble all the time can slow us down.
Of course, underthinking can be a problem too, so good writers and developers deliberately create critical distance between themselves and the work. Thoughtful creators periodically pause and pop out of the process of creating something to reflect, assess what they’ve created so far, and look for problems in it. This kind of evaluation requires a switch from a builder mindset to a critic mindset, or from an insider’s perspective to an outsider’s. It can be done, but the switch can take considerable mental effort.
Is there a way around that effort? Here’s one: step away for a while. We all have some experience of writing something (as I’m doing now) and looking at it several weeks later (as I’ll doubtless do in a few weeks) and shaking our heads: “In the name of all that is holy, what was I thinking when I wrote that?” Taking time away from the work is a reasonable heuristic for getting critical distance from it. Yet for most developers, in most development shops, time is in short supply. What else could we do?
Farther Critical Distance and the Developers’ Perspective
Getting someone else look at what we’ve created is a powerful heuristic for seeing things that we’ve missed. It’s why writers work with editors. It’s one of the motivations for pair or ensemble programming. It’s why responsible developers enlist the help of other developers for review, and why responsible managers foster that. It’s a really, really good idea to have the developer’s work reviewed by other developers.
There’s a social aspect at work too. It’s natural and easy for people to calibrate and align respect for others based on skills, knowledge, and expertise that they have in common. Close critical distance and close social distance are often bungee-corded together.
Developers tend to be really good at spotting problems in code — especially when it’s code written by other developers. If the product has problems that other developers can find, the product has problems. Who better than to notice problems in the code than people who have a deep knowledge of development? Why not get the developers, plural, to do all the testing?
Let’s start with this fairly obvious point: while a developer-reviewer’s perspective is different from the developer-builder’s perspective — which helps testing — developers are not always and entirely like users.
The Curse of Knowledge makes it hard for the insiders who are building a product to see it from the outside perspective of the people who will use it. David S. Platt, a developer and author of Why Software Sucks puts it perfectly: know thy user, for he is not thee. That’s why being at close critical distance can also hurt testing: when you know how the technology works from the inside, it can be easy to be oblivious to problems that outsiders are unhappy or concerned about. It can even be easy to dismiss those problems.
As a developer, even if you do a really great job of immersing yourself in the world of the users, it’s impossible to drop certain knowledge entirely. For instance, you might be able to empathize with someone who can’t read English. But if you’re reading this post in the original, whether you learned English as a toddler or as an adult, I can guarantee that you will never be able to see it from the same perspective as someone who has never been able to read English; who is just learning to comprehend bigger words and my occasionally convoluted sentence structure. SimiIarly, if you’re a programmer, you won’t ever be able to drop all of your knowledge about programming. You will always have a programmer’s perspective, to some degree.
Here’s a point that is sometimes a little less obvious to developers — the code is not the product. The product depends on the code, but that’s not the product from the user’s perspective. The product is a set of experiences, services and benefits that the code delivers to individuals and organizations — the people who buy the product, the people who use it, and the people that they serve. As David Platt says, “your user doesn’t come to you to buy software; they come to you to buy what your software does“.
Critical Distance from the User’s Perspective
The user’s perspective is at signficant critical distance from the work, from the coal face (or code face). Users are sensitive to — and, more importantly, affected by — problems that the builder doesn’t see, or doesn’t consider to be a problem.
Some products — APIs and other services, application frameworks and libraries, development tools… — are built for developers. Nonetheless, like a non-developer using a smartphone app, the developer who uses an API has a outsider’s perspective — different from that of the developers who built that API; people with insider knowledge.
Testing that only takes the developers’ perspective will be malnourished. If we involve the users, they’ll find problems that the developers won’t. If the product has problems that users can find, the product has problems. So… why not get the users to do all the testing that the developers don’t do?
This might be harder to answer. Kent Beck’s Extreme Programming, arguably the first book about Agile software development, proclaimed that there were only two roles; Programmer and Customer. The Programmers can evaluate the the code and the functions, making sure that they do what they’re supposed to; and the Customers can evaluate the user experience, making sure that they’re happy with it. What could be missing?
Here’s something that could be missing: the active and deliberate search for trouble.
Critical Distance from the Tester’s Perspective
Both building and using a product tend towards an optimistic mindset. As I’ve noted above, and in past posts in this series, most of the development team is focused on envisioning success. The developer wants to get something built. Meanwhile, someone who is using or purchasing a product is seeking to get value from a product. The user wants to get something done. Both users and the developers want everything to be okay.
Excellent testing requires a critical stance, dropping the idea that things are okay, and deliberately challenging it. For someone actively and optimisitically building the product, that requires a change towards a more pessimistic mindset to consider ways in which things may go wrong. That is: testing requires work that, to some degree, disrupts the developers’ primary ambition and task of getting things built.
Some organizations recruit users to help test. When it comes to testing, users who are typically focused on accomplishing a task successfully must also change their default stance. Even while testing from the user interface, testing is a diversion and distraction from users’ typical goals and motivations. Moreover, most users don’t go below the UI and deliberately develop an understanding of the product on a technical level — and why should they? That’s why they pay developers and software companies in the first place. Testing work disrupts the users’ primary tasks and ambitions of getting things done.
Testers are different. The primary mission of the responsible tester is to understand the status of the product — and especially problems that threaten its value, so that people can make informed decisions about it. When it comes to trouble, the tester’s primary task ambition is getting things noticed.
Testers are at farther critical distance from the code than the developers are. Testers are at farther critical distance from the product than the end users are. But testers are at close critical distance when it comes to problems — bugs, errors, omissions, risk. The tester’s primary role is to learn about the product in a way that supplements what the developers know about it and what the users might discover to their dismay. The tester’s job is find problems that matter before it’s too late.
Noticing problems requires us to challenge the product and people’s assumptions and beliefs about its quality. The critical stance is the tester’s default, with the goal of helping to defend the development team and the business from misplaced or unwarranted confidence. Rather than envisioning success, we’re anticipating failure. How might things go wrong? Where is risk apparent — and where might it be hiding? How might we be fooling ourselves? Starting from those questions, we examine things the team is producing and has produced, getting experience with them, looking at them critically, and questioning the validity of our beliefs about them.
Of course, developers necessarily consider the possibility of trouble as part of their work, and users occasionally encounter trouble. The point is that neither the builder role nor the user role is focused primarily on trouble. The shift to that mindset takes time and mental effort before the testing work can start — never mind the time and effort required to perform the testing. One workaround is to have someone occupying the testing role full-time.
The critic role can be applied usefully every step of the way, for anything that the development team is producing — including our intentions and designs for a product. My friend George Dinwiddie is the author of a heuristic that he calls The Three Amigos (whimsically named after an unfortunately terrible movie).
Rather than wait until a story was in the sprint to fully understand it, members of the team took time to discuss stories that were on deck for the next sprint. The product owner or an analyst, as representative of the business, would ask if a programmer and a tester had time to look at a story with him. The three of them made sure all their questions were answered before considering the story ready for development.
These three people roles were selected as the Three Amigos because they provide mutually orthogonal viewpoints on almost anything you may be building. The business representative thinks about what the business hopes to accomplish by building it. The programmer thinks about the details needed to implement it, such as what information is needed when and what technologies can accomplish the goals. The tester thinks about what might go wrong, either within the system or in the external context upon which it depends.
George is pointing out the value of having people from different roles in the same place at the same time, early in the process. When we’ve got a product owner, developer, and tester together, we have people at different degrees of critical distance between the problem space, the solution space, and the risk space. At the same time, we reduce social distance between the people and the roles. That also allows for flexible and instantly changeable roles and stances. The three roles form a collective, and that’s a great thing; each benefits from the knowledge and ideas of the others, and the capabilities of the collective are almost certainly greater than the sum of the capabilities of its members.
There is a way that all this can go wrong, though, and that’s when the tester relaxes the role and mindset of the critic. That can happen when testers don’t understand the significance of the critical stance. It can also happen when testers feel pressured to conform with the mindsets of others on the team; when testers are mandated to demonstrate that things are okay; when testers have insufficient technical and analytical skills; when “finishing the test cases” displaces the goal of searching for problems. Testers need to cultivate the critical mindset and investigate the product with the goal of finding trouble that matters — and to be effective, they need clients and organizations that recognize the value of testing work.
In Rapid Software Testing, we don’t say that every development group needs a dedicated tester. We don’t claim that developers don’t — or can’t — test. We don’t say that it’s a bad idea to engage users in the testing process; on the contrary. What we do claim is that having someone in a testing role is a powerful heuristic for obtaining critical distance quickly, addressing the mindset-switching problem, fostering testing skill, and getting focused testing work done.