A Context-Driven Approach to Automation in Testing

(We interrupt the previously-scheduled—and long—series on oracles for a public service announcement.)

Over the last year James Bach and I have been refining our ideas about the relationships between testing and tools in Rapid Software Testing. The result is this paper. It’s not a short piece, because it’s not a light subject. Here’s the abstract:

There are many wonderful ways tools can be used to help software testing. Yet, all across industry, tools are poorly applied, which adds terrible waste, confusion, and pain to what is already a hard problem. Why is this so? What can be done? We think the basic problem is a shallow, narrow, and ritualistic approach to tool use. This is encouraged by the pandemic, rarely examined, and absolutely false belief that testing is a mechanical, repetitive process.

Good testing, like programming, is instead a challenging intellectual process. Tool use in testing must therefore be mediated by people who understand the complexities of tools and of tests. This is as true for testing as for development, or indeed as it is for any skilled occupation from carpentry to medicine.

You can find the article here. Enjoy!

16 replies to “A Context-Driven Approach to Automation in Testing”

  1. Hello, Michael!
    Thanks for the descriptive and detailed document. I have some questions on it:

    Q1: You mention couple of times we need to avoid the use of the word “automation” as it is misleading. I don’t think it is misleading, but the interpretation it has was misleading for far too long.

    Michael replies: If the interpretation of a term is misleading, it’s not too far off to say that the term itself is misleading.

    The notion of “automate everything”, according to my understanding, comes from the belief that testing is a list of predefined actions and everyone can do it, why not automate it. And judging from the text I see our opinions overlap – “only reason people consider it interesting to automate testing is that they honestly believe testing requires no skill or judgment”. Having this in mind, removing the term automation is dealing with the symptom of the problem, but not the underlying cause – testing is commonly considered easy to do, step-by-step task that could be documented in detail and performed by anyone. Do we really need to avoid “automation” in our vocabulary or it would be better to focus on fixing the understanding other people have about testing.

    Sure, we can keep the word “automation” in our vocabulary. I think if you read more closely you’ll find that we mostly object to the term “test automation”.

    If you’re not speaking in the Rapid Software Testing namespace, you can use whatever words you like to mean whatever you like.

    Q2: The term “automation” is broadly used in engineering. We automate actions in car production, factories and almost every aspect of life. And yet, the problem we deal with in testing are not present in other areas, the limits of automation in factories, for example, are known – nobody expects a robot to invent the next model of BMW or asses that it works properly. Why is it such a problem in testing, isn’t it just a misinterpreted term, that was used in it’s wrong meaning for too long?

    I love this excerpt from George Orwell’s “Politics and the English Language”: “an effect can become a cause, reinforcing the original cause and producing the same effect in an intensified form, and so on indefinitely. A man may take to drink because he feels himself to be a failure, and then fail all the more completely because he drinks. It is rather the same thing that is happening to the English language. It becomes ugly and inaccurate because our thoughts are foolish, but the slovenliness of our language makes it easier for us to have foolish thoughts. The point is that the process is reversible.”

    Part of the problem is that testing was robbed of its meaning in the 1970s and 1980s by people who were interested in process models and formal methods and provably correct software. That’s where the confusion between checking and testing was born, too.

    Q3: For the sake of mutual understanding we use the term “automation” in order to explain tool supported testing to non-testers (managers, developers, marketing people, etc), how can we use the right terminology and avoid “automation” without giving them a lecture on what testing is and how deluded they are in their understanding about it?

    You might choose to start by avoiding the word “deluded”, no matter how well you might think it fits.

    Ask for a deeper conversation. You might like to try starting by talking (or asking) about automated checking and tool use in testing when someone refers to “test automation”. To some degree, you may have to explain why. Remember, no one says “automated management” or “automated programming”, but they do talk about management tools and programming tools. No one says “automated programming”, but they do talk about compilers.

    I observe that, in general, most testers are pretty sloppy in their talk about testing. Could you draw something like this? Can you map out test strategy in two minutes (or maybe longer)?

    Q4: A bit off topic, I support yours and James’ efforts in order to redefine testing terms, but don’t you think they have to be unified in something like a vocabulary, because now they are divided in multiple sources, books, papers, posts which creates kind of fragmented knowledge base. Have you considered the idea of compiling this into a book or something else?

    We don’t find that we’re redefining testing terms very often. To us, we’re more often going deeper than the shallow definitions. Here’s a case in point. Sometimes people claim that we’re redefining testing terms when we’re recovering them from the people who stole them.

    Ah, but I haven’t answered your question. Yes, we have considered a book. A book is a big, long-term project that we’re working on, but not working very hard at the moment.

    Thanks and good luck.

    Thank you for the comment.

  2. […] A Context-Driven Approach to Automation in Testing introduces a paper that helps you to identify the problems of test automation tools and use them in a helpful and productive way. This paper has 26 pages but I can assure you that reading this paper is a time well spent. […]

  3. IMHO this is a great paper describing how many professional testers operate – with excellent examples of your ideas to back them up. My only (very small) concern is that the title includes the term “Context-Driven” which, by it’s very nature, will restrict it’s readership. EVERY software tester who takes their craft seriously needs to be aware of the details and distinctions drawn out in this paper and not just those who “identify” themselves with the Context-Driven community.
    I hope one or both of you follow this up with presentations at various conferences this year (in the same vein that Michael did recently at EuroSTAR), as I see this as a great way to cross “community boundaries”.
    Thanks sincerely to both of you for putting your time into producing this White Paper.

  4. Bravo! Thank you Michael and James for this paper.
    I have seen too much time and money wasted on GUI automation tools that are misused trying to replicate intelligent testing by testers.
    I have seen success when testers use intelligence and collaboration to come up with test ideas, home grown scripts and various other tools to discover bugs.

    Susan Ward
    Software Tester
    Boston, Massachusetts

  5. Automation is also good way to test the software. But it all depends on the way you are doing automation.

    The point of the article is that automation is not “a way to to test the software”. Tools are means by which we humans extend or accelerate some of the tasks involved in testing software. In short, tools help us to test the software. Checking can be done without tacit knowledge; it can be done strictly by algorithms and behaviours. Testing needs tacit knowledge about intention, value, and social significance.

  6. Hi Michael,

    This is a wonderful paper that contains a huge amount of valuable information. One point that I would like to make is that although automation is a tool that should be used by testers where and when appropriate it is becoming a more and more important tool.

    Michael replies: I don’t think it’s accurate to say that automation is a tool, just as we don’t say that transportation is a vehicle. I believe tools to be important too, but I believe it’s also important to speak and write carefully about tools, with the goal of helping managers and testers who want to use them well.

    The trend to towards Continuous Delivery necessitates the use of automation (not just checks but for deployment, data generation, environmental clear down etc) more and more in the test world as we simply do not have the time to get bogged down in regression test cycles.

    I don’t think anyone disputes the general notion of the usefulness of tools for the purposes you mention here. We certainly don’t, and in the paper, we’re explicit about using tools for those purposes and lots of others.

    I perceive a problem in with what you’re saying, though, and I’d like to draw attention to it because I hear things like it a lot. If your problem is that you’re “bogged down in regression test cycles”, it’s not clear to me that the snap answer “use tools!” will help until we’ve done some analysis. For instance: are you seeing a consistent pattern of regression problems? Are you fixated on regression risk, at the expense of other risks? Are the programmers producing unreliable code because of unsustainable pace? Are you “bogged down in regression test cycles” because testing and review isn’t included in the development process? Do people think that testing can’t start until you’ve got a fully-built product? Tools amplify and intensify and extend whatever we are. If we’re expert testers, they can support our expertise. They can also help to make a dysfunctional process even more dysfunctional, even more quickly.

    Regarding the brittle nature of automation, automation approaches, frameworks and technical skills have improved in recent years among testers. This implies that we are no longer dependant things like record and playback UI tools. Automation is as a result becoming more robust and effective.

    Is automation becoming more robust and effective, or is testing becoming more robust and effective? Consider that question carefully, and consider a wide range of possible answers.

    When you talk about software development you speak of tools used by developers such as ‘autocoders’ etc.

    Note that we mention “autocoders” to emphasize the point that programming can’t be automated, and that programmers call “autocoders” compilers at least in part to emphasize that fact.

    There are an array of tools employed by developers that allow them to code more efficiently and effectively. I see automation in the same light from a testing perspective. While you maybe an exceptional tester without having automation skills I think that adding automation to your repertoire may make you a more efficient and effective tester.

    I’m glad you said “may”, and I agree. What’s most important is what fits for the individual and what fits for the organization. We typically have lots of coders in organizations, and those with programming skill can aid testers when necessary. People skilled at thinking critically about software, testing it, and applying tools thoughtfully seem more rare.

    Thanks again for a great paper.


  7. […] James Bach and Michael Bolton did an awesome job, making the difference between “testing” and “checking” , so I think it will be useless for me to repeat what they already said. If you are  not familiar with their work, I strongly recommend you do check “Testing and checking refined” and their white paper on automation in testing – A Context-Driven Approach to Automation in Testing. […]

  8. Thank you for the post. I loved reading it.

    Incidentally, my team also drafted a great post on Automation testing and I believe your readers will love to browse “How to Amplify the Impact of Virtual Reality with a Robust Test Strategy?” on Gallop blog.

    I would love to know your comments on our blog.
    Thank you,

    Michael (B.) replies: Thank you for the kind words. I looked at your blog, and didn’t much like what I saw, I’m afraid. Apart from everything else, it wasn’t really about “automation testing”. I sent you some comments directly, via email.

    Interested readers are welcome to look for the post and decide on their own.

  9. Hi. I agree with your perspective in this article A Context-Driven Approach to Automation in Testing It was an interesting read. We have recently posted a blog on Can Test Automation really help build sound Embedded Devices?’ I would like to get your views on the same.

    Michael replies: See my reply to another Michael just above.

  10. Hi,

    The paper on Context-Driven Approach to Automation in Testing by James Bach and Michael Bolton is an absolute enabler. The tools mentioned and the approach described is totally enriching. Thanks for sharing.

    You might like to check this post on “How to Amplify the Impact of Virtual Reality with a Robust Test Strategy?” would love to know your views.

    Thank you,



    Michael (B.) replies: I’m beginning to notice a pattern here.

  11. […] James Bach and Michael Bolton did an awesome job, making the difference between “testing” and “checking” , so I think it will be useless for me to repeat what they already said. If you are  not familiar with their work, I strongly recommend you do check “Testing and checking refined” and their white paper on automation in testing – A Context-Driven Approach to Automation in Testing. […]


Leave a Comment