Every day, in some discussion about testing, someone talks about the need for “balance between automated testing and manual testing”. This seems to me to be a supremely unhelpful way to think about testing work.
First, and once again, testing is neither manual nor automated. No one in any other cogntive, intellectual, investigative domain talks about their work that way; and no one in any such domain allows other people to talk about the work that way. That framing leads to serious social and economic problems; and I’m not kidding about that. Meanwhile, there are positive and productive ways to frame the tester’s interaction with the product and engagement with the work.
This time, though, I’d like to talk about the notion of “balance” — which is another category of unhelpful thinking in testing work.
When we’re confronted with a task to perform, it seems to me that we must consider what we need to do or apply get the job done. That’s not about “striking a balance”; that’s about fitting practices or tools to the task at hand.
For instance, when I’m considering going to the corner store just up the street to buy some groceries, I don’t talk about “balance between walking and taking a boat”. I don’t even talk about “balance between walking and taking the bicycle”. I consider whether the bike will save me time or effort; whether I’m in a hurry; whether I can carry the groceries more easily on the bike than on foot.
Similarly, if I’m considering travelling from my home in Toronto to Bucharest, I don’t talk about “balance between walking and taking an airplane”. If I want to get to Romania, the fact that there’s an ocean between here and there means that parts of that journey will involve taking a plane (or a boat); parts of it will involve walking; parts will taking a taxi or a bus to get the airport — unless I’m already at an airport hotel, and the airport is within walking distance.
For almost anything that we do, we can choose to apply tools to help us to extend, accelerate, enable, enhance, intensify, and amplify the activity. But we’re not obliged to use tools to any degree; nor are we obliged to ignore them. So instead of thinking in terms of “balance”, consider the task at hand.
- Maybe our task is to learn about the product with the goal of developing ideas about how to test it. We could do this without any kind of tools, and rely on our memory. Because memory is fuzzy and unreliable, we might want to create a list, or a mind map, or a table of features to examine. We could choose pencil and paper to help us visualize the list, or to supplement our memory, or to share our ideas with others. Or we might choose to use software tools to help with that.
Notice that for mind maps, coloured pencils and paper give a lot of freedom and flexibilty. Software tools tend to squash our ability to draw and illustrate our ideas free-form — but they make editing and revising the mind map much easier. Each approach affords advantages and disadvantages, but it would be silly to say we should do some of our mind maps on paper, and then balance that with other mind maps created with XMind. The issue is not balance; it’s about what we want and need to get the job done, in the moment. - Our task might be to explore the data that is being presented, stored, or processed by a Web-based product. We could choose to do this via direct interaction using only the interfaces provided to non-technical end users. That can be an important thing to do, because the user’s perspective is fundamental. The underlying data might correct, but problems in the user interface might lead to hidden data, or to confusion, or to frustration.
On the other hand, maybe aspects of the service never present themselves to end users via a GUI, but only to developers or other businesses. Even when there is a GUI involved, we can also use tools (like Fiddler) to collect HTTP requests and responses, and use other tools (like jq) to probe the details of the traffic and the JSONs that are being passed around. But again, this is not a question of balance; the issue is what we want to know, and how we want to get at it. - Our task might be to check specific facts about the output from some function in the product. We could do that directly and interactively for some simple, or visible, or low-risk functions. We’re actually doing that all the time, and we’re often not even conscious of when we’re doing it, and how much. In circumstances where the data is complex, hidden, and critical, it might be a good idea to write a bit of code to provide input to some specific functions, to trigger those function, to collect output from them, and to compare the output to specified and presumably desirable results — something that we call “automated checking“.
Our clients might give us a mission to do a lot of automated checking. We can also do plenty of valuable testing without ever doing any automated checking whatsoever. Again, this is not a question of balance; it’s about what we need to do to accomplish the mission.
Now, there is something that might look like balance, but isn’t: variety. It is important to consider variety in our models, our tools, our tactics, and our techniques. W. Ross Ashby — one of the central figures in the cybernetics and general systems movements — coined the Law of Requisite Variety — “only variety can destroy variety”. “When regulation is seen as a game between a regulator R and source of disturbances D, only variety in R can force down the variety due to D”. In practical terms, this means a controller of a system needs more states available to it than the system can be in.
Do you find that confusing? In Sensemaking in Organizations, Karl Weick points out that if you want to understand something complicated, you have to complicate yourself, so here’s an example that might help: if we’re going to keep the car on a twisty road, we must be able to observe the road and control the car to a degree of complexity greater than that of the road and its twistiness.
Applied to testing, the Law of Requisite Variety if we’re going to find a variety of bugs, in a variety of areas in the product, we’ll need a variety of models of the product; a variety of ways to recognize problems; a variety of ways to control the product; and a variety of experiments to perform. But that’s not about balance; that’s about diversity.
There’s no need to consider balance in transportation. There’s no need to balance movies with live theatre, nor music with visual arts. Similarly, there’s no need to consider “balance” in terms of whether or when we interact directly or indirectly with the product. Instead, let’s ask: what is the job we need to do, and how might tools help to extend our ability to get the job done?