DevelopsenseLogo

“Manual” and “Automated” Testing

The categories “manual testing” and “automated testing” (and their even less helpful byproducts, “manual tester” and “automated tester”) were arguably never meaningful, but they’ve definitely outlived their sell-by date. Can we please put them in the compost bin now? Thank you.

Here’s a long and excellent rant by pilot Patrick Smith, who for years has been trying to address a similar problem in the way people talk (and worse, think) about “manual” and “automated” in commercial aviation.

When it comes to software, there is no manual testing. My trusty Chambers dictionary says manual means “of the hand or hands”; “done, worked or used by the hand(s), as opposed to automatic, computer-operated, etc.”; “working with the hands”. It’s true that sometimes we use our hands to tap keys on a keyboard, but thinking of that as “manual testing” is no more helpful than thinking of using your hands on the steering wheel as “manual driving”.

We don’t say that Itzhak Perlman, using his hands, is performing “manual music”, even though his hands play a far more important role in his music than our hands play in our testing. If you’re describing a complex, cognitive, value-laden activity, at least please focus on the brain.

We might choose to automate the exercise of some functions in a program, and then automatically compare the output of that program with some value obtained by another program or process. I’d call that a check. (So did McCracken (1957), and so did Alan Turing.) But whatever you call the automated activity, it’s not really the interesting part.

It’s cool to know that the machine can perform a check very precisely, or buhzillions of checks really quickly. But the risk analysis, the design of the check, the programming of the check, the choices about what to observe and how to observe it, the critical interpretation of the result and other aspects of the outcome—those are the parts that actually matter, the parts that are actually testing. And those are exactly the parts that are not automated.

A machine producing a bit is not doing the testing; the machine, by performing checks, is accelerating and extending our capacity to perform some action that happens as part of the testing that we humans do. The machinery is invaluable, but it’s important not to be dazzled by it. Instead, pay attention to the purpose that it helps us to fulfill, and to developing the skills required to use tools wisely and effectively. Otherwise, the outcome will be like giving a 17-year-old fan of Top Gear the keys to the Ferrari: it will not end well.

People often get excited when they develop or encounter something that helps them to extend their capabilities in some way. Marshall McLuhan used “media” to describe tools and technologies—anything that humans use to effect change—as extensions of man, things that enhance, extend, enable, accelerate or intensify what we do.

He also emphasized that media are agnostic about our capabilities and our purposes. Tools not only accelerate what we do, they intensify what we are. The tool also changes the nature of the test that we’re performing, sometimes in ways that may escape our notice. Tools can extend or enhance or enable fabulous testing. But by accelerating some action, tools can enable us to do bad testing faster than ever, far worse than we could possibly do it without using the tool.

As Cem Kaner so perfectly puts it, “When we’re testing, we’re not talking about something you’re doing with your hands, we’re talking about something you do with your head.” Besides, as he and others have also pointed out, it’s exceedingly rare that we don’t use machinery and tools in “manual” testing.

Even in review—when we’re not operating the product—we’re often looking at requirement documents or source code on a screen; using a keyboard and mouse as input mechanisms for our notes; using the computer to help us find and sift information; using software and hardware to communicate with other people; comparing ideas in our head with the output from a printer. Is this stuff manual? I prefer to think of it as sapient, something that by definition can’t be performed by a machine (or, using a word coined by our friend Pradeep Soundararajan, “brainual”), and assisted by tools. But if you want to call that stuff “manual” testing, how do you characterize the act of creating automated checks? I imagine you would call that writing, or programming, not typing. Through macros, computers can effectively type, and save us time in typing; it takes people to write or to program. See the difference?

Several years ago, I used Excel to construct an automated oracle to model functional aspects in a teller workstation application, and identify which general ledger accounts should be credited or debited. Building the oracle took three weeks of solid analysis and programming work. As I was doing that work, I was learning about the system, performing tests, revealing problems, feeding back learning in the the development of my oracle. During this process, I also found lots of bugs. Sometimes my oracle helped me find a bug; sometimes tests that I performed interactively while developing the oracle helped me find a bug. Was I doing “automated” or “manual” testing?

I worked at another financial institution, where part of the task was to create tables of inputs and actions for functional integration checks, using FITnesse as the tool to help me select the functions, organize the data, and execute the checks. I identified risks. I designed checks to probe those risks. I entered those checks manually, via the keyboard.

The tool executed the checks, but I triggered its execution “manually”, by clicking on a button. I reviewed the output by eye, but the tool helped by allowing me to see unexpected results easily. I revisited our assumptions frequently, considering what the checks appeared to be telling us and what they might be leaving out. When we realized a lapse in our check coverage, I’d enter new checks by typing them in (although sometimes I laid them out in Excel first, and copied and pasted them in). So was I doing automated testing, or not?

Why is this a big deal to me? To me, there are two dominating reasons, and they complement each other.

When we think in terms of “automated testing”, we run the risk that we will narrow our notion of testing to automated checking, and focus our testing effort on the development and execution of automated checks. Checking is not a bad thing; it’s a good thing, and an important thing. But there are more aspects to the quality of a product than those that can be made subject to binary evaluation, which is what checks produce.

I’ve had the experience of being fascinated by checking to the point of distraction from a broader and deeper analysis of the product I was working on, which meant that I missed some important problems (thankfully, other testers found those problems before they reached the customer).  Qualitative analysis of a product isn’t about bits; it’s about developing a story. Yet investigating qualitative aspects can be aided by tools, which brings us to the second point.

Talk of “manual testing” often slides into talk of “manual exploratory testing”, as though exploration doesn’t make use of tools, or as though tools could not used in an exploratory way. This slipping limits our ideas about exploration and our ideas about tools. Exploratory testing is not tool-free. When we think in terms of “manual testing”, we run the risk of underestimating the role that tools can play in our interactions with the product and in the testing effort.

As a consequence, we may also ignore myriad ways in which automation can assist and amplify and accelerate and enable all kinds of testing tasks: collecting data, setting up the product or the test suite, reconfiguring the product, installing the product, monitoring its behaviour, logging, overwhelming the product, fuzzing, parsing reports, visualizing output, generating data, randomizing input, automating repetitive tasks that are not checks, preparing comparable algorithms, comparing versions—and this list isn’t even really scratching the surface. I’ve had experience of not pausing to think about how tools could help me, of neglecting to use them, and of using them unskillfully, and that compromised the quality of my work too.

To sum up:  “Manual testing” vs. “automated testing” is a false dichotomy, like “manual statistical analysis” vs. “automated statistical analysis”, or like “manual book editing” vs. “automated book editing”. In the sense that most people seem to use them, “manual” and “automated” refer to an input mechanism. People probably started saying “manual test” as a contrast to “automated test”, but just as testing isn’t manual, there’s no such thing as an automated test, either—except of the kind that we call a check, which still requires significant testing skill to design, prepare, interpret, and analyze.

The check might be automated—performed by a machine—but the testing that surrounds the check is neither manual nor automated. The person who performs the testing is neither an “automated tester” nor a “manual tester”.  Tools can aid testing powerfully, and in all kinds of ways, but they can also lead to distortion or dysfunction, so practice and develop skill to use them effectively.

If you think that all this is a silly quibble, take a page from my colleague James Bach: you don’t talk about compiling and linking as “automated programming”, do you?—nor do you talk about writing code as “manual programming”, do you?

Further reading:

The End of Manual Testing
The Honest Manual Writer Heuristic (a kind of “manual testing” I could endorse—but probably not the kind you think)

Further watching:

What’s Wrong with Manual Testing?

Subscribe to the Newsletter

27 replies to ““Manual” and “Automated” Testing”

  2. Regarding “manual testing”, if I go to dictionary.com (because I’m too lazy to walk to my bookshelf and get a *real* dictionary), I also see this definition:

    “involving or using human effort, skill, power, energy”

    I interpret that definition as one I could apply to testing without implying that I’m a highly-paid button pusher.

    In the larger sense of the post, I too, dislike (or loathe) the notion of “manual testing” vs. “automated testing”, and find the idea of “automated tester” laughable.

    But I struggle with the idea of describing testing as sapient. Not, of course, because it’s not – of course it is a sapient activity. I just prefer to call it testing (w/ no adjective) – no matter how the activity is carried out. In my experiences , I can’t recall a need to separate brain-engaged testing with automated (or computer-generated) tests. I do realize (painfully) that many testers appear to adore the distinction, so I think we have many points of agreement on this subject.

    Michael replies: I prefer to be able to say testing without the “sapient” part either; testing is sapient, and that’s that. When James Bach proposed the term (here, to distinguish sapient processes from non-sapient processes (like the mechanical execution of programmed instructions), I resisted—not because I disagreed, but because it seemed like an unfamiliar and weird word. Stuck for an alternative, I gave up and reluctantly adopted James’ word. Meanwhile, many people (mostly people fascinated by programming, and people fascinated by the role of test automation in the Agile world) seemed to believe that producing a body of competing code, and having machines execute that code was equivalent to testing the product. So, after a couple of years, the distinction between testing and checks was born. That greatly reduced the number of occasions in which I felt I needed to use “sapient”. But I can here: the checks themselves are done non-sapiently; the testing work surround them is sapient.

    Reply

  3. The subtext of the manual versus automated false dichotomy seems to be that manual is the poor, unprofessional relation of high quality, sophisticated automation. I wonder if part of the problem is a misplaced belief in the value of repeatabiilty, for which CMMI has to take its full share of the blame.

    The thinking would go, if something can be automated it is repeatable; it can be tailored to be precise, continually generating accurate, high quality results. Automated testing and “best practice” go hand in glove.

    In contrast, manual testing seems frustratingly unpredictable. The actions of the testers are contingent. I think that is an interesting word. I like to use it in the way it is used in philosophy and logic. Certain things are not absolutely true or necessarily so; they are true or false, necessary or redundant, depending on other factors or on observation. Dictionaries offer subtly different alternative meanings. Contingent means accidental, casual or fortuitous according to dictionary.com. These are incredibly loaded words that are anathema to people trying to lever an organisation up through the CMMI levels.

    I understand “contingent”, as a word and concept, as being neutral, useful and not particularly related to “repeatable”. It is sensible and pragmatic to regard actions as being contingent – it all depends. Others would regard “contingent” and “repeatable” as opposites; “contingent” then becomes evidence of chaos and unpredictability that can be cleaned up with repeatable automation. Some people regard “it all depends” as an honest statement of uncertainty. Others regard it a weak and evasive admission of ignorance. There has always been a destructive yearning for unattainable certainty in software development.

    To be clear, I am not decrying repeatability. I mean only that putting too much emphasis on it is unhelpful.

    Reply

  4. Your post got me thinking – thanks for that 🙂

    I’m with Alan on the use of sapient – or any other adjective tacked onto testing for that matter. In my context I don’t see the word “sapient” adding any value. When I hear people say it I automatically think they are trying to use uncommon words to make something sound more important than it is. Having said that I understand why it’s being done in the “community” and I like your way of thinking that it’s not something can be done by a machine.

    The link for the long and excellent rant by pilot Patrick Smith is definitely a good read. I feel like there this something in this quote (from near the bottom). There is an idea floating in there for me (just don’t know what it is yet)

    “I would like to see a drone perform a high-speed takeoff abort after an engine failure, followed by a brake fire and the evacuation of 250 passengers. I would like to see one troubleshoot a pneumatic problem requiring a diversion over mountainous terrain. I’d like to see it thread through a storm front over the middle of the ocean. Hell, even the simplest things. On any given flight there are innumerable contingencies, large and small, requiring the attention and often visceral appraisal of the crew. I can’t imagine trying to handle these things from the ground, thousands of miles away.”

    Reply

  5. […] esteemed colleague and sometime nemesis Michael Bolton has written a screed against the terms “manual testing” and “automated testing”. Check it out, […]

    Reply

  8. Sometimes definitely manual testing provides more accuracy than with those done by any automation tools. But there are some factors that definitely requires a software such as bug detection and flaw detection using a snapshot facility. That is the reason that nowadays automated testing is slowly taking place of manual testing.
    automated testing tool

    Michael replies: In our community, we would say “machine checking” and “human checking”, rather than “automated testing” and “manual testing” as you say above. I’d recommend you have a look here: http://www.satisfice.com/blog/archives/856.

    Reply

  9. Automated testing is designed, setup and operated by engineers. Manual testing is done by customers after each release. That is the state of affairs.

    Joemsie

    Michael replies: Is it? I think I’m describing something different.

    Reply

  10. Testing is about providing information about the product and I think it doesn’t matter how you provide this information.

    So I guess when Automated or Manual Testing terms were coined – the idea behind would be to distinguish the means of providing the information. Whether a code (automated) is providing the information or a Man (manual)

    Michael replies: In my view, this is giving far too much credit to the machine. The machine is not providing the information. A person is always providing the information, and the machine is a tool in that process. Why don’t we say that the pencil is writing the letter?

    Reply

  12. I don’t agree with some of your point of view that why we should not compare manual testing and automated testing. At this stage automated testing has been so much advanced that no more manual interaction is needed in testing. Automated testing can prove to be much reliable.Sooner or later manual testing would be eliminated.

    Michael replies: Okay, so let’s test that: show me a program that can write a well-worded and logically constructed argument that supports your claim, or refutes mine.

    By the way, on your site, the links to your privacy policy and terms and conditions pages (http://testing-whiz.com/) are broken sitewide as of 2:15pm GMT, June 27, 2013. I found that by looking myself for just a moment or two. Cheers, and good luck with your thing.

    Reply

  15. manual and automation testing which is most powerful tools in testing process in manual and automation testing or qtp testing

    Michael replies: And the most powerful keywords for SEO optimization.

    Pro tip: if you want me to preserve the links to your site, you must provide coherent content, full sentences, something interesting; that kind of stuff. Better luck next time!

    Reply

  16. Thanks for posting the manual testing for learn advanced concept mobility software testing learn load runner testing

    Michael replies: Thanks for posting the SEO search terms. You don’t mind if I remove the URLs, do you?

    Reply

  17. Software testing is changing. There are numerous tools that are making life much easier for developers and testers. I agree that Software Testers need to embrace these tools and adapt to the changing landscape. But, still manual testing will not go away completely, so there is need in proper testing software which will step by step help you to pass through, I`m currently using such called EasyQA, with it`s automatized functions it is way easier to do more in less time. Anyway many of the traditional software testing jobs that existed earlier will be replaced by automation tools. Testers who adapt and these tools will be the ones who will move to the next level.

    Michael replies: I appreciate the role of tools and the help they provide. I agree that testers should embrace tools. I’m glad that you’ve found something that will press buttons for you. But please try to grasp the point of this post: THERE IS NO MANUAL TESTING. There is manual pushing of buttons. There is manual typing of text. There is manual holding of a whiteboard marker as you sketch. But testing is a mental process that can neither be done manually nor through automation. Please stop thinking that the typing part, the pushing-buttons part, is any significant part of testing. Manual testing will never go away completely because it was never here to begin with.

    Reply

Leave a Comment