Heuristics of GUI Automation Tools

A correspondent on the Agile Testing mailing list asked recently

Shall automated acceptance tests use the GUI the app provides?

My reply sat in my drafts folder for a while, and I just found it. Too late for the conversation, really, so I’ll post it here.

My thought, as usual, is that automated acceptance tests checks should or should not use the GUI depending on

• the questions you want to ask and answer
• the business domain that you’re working in, and therefore the kind and magnitude of the risk you want to address
• the skill of your toolsmith (or the extent to which she can leverage the skills of others)
• the capabilities of your automation tools
• the extent to which you wish to model real users and observe what they observe
• the extent to which you want to deliver yourself automatically to a point where you can begin human-driven testing
• the extent to which tests have already been created for lower levels
• the extent to which you’re willing to value speed, precision, and volume over human observation and cognition
• the extent to which you can vary the data that you’re using–either systematically, pseudo-randomly, or randomly–in a productive way
• the extent to which you can create or use oracles that will allow the tool to help you recognize problems
• the extent to which you want to act in a confirmatory, rather than an investigative mode; (and therefore)
• the extent to which you’re willing to be vulnerable to confirmation bias; and
• the extent to which you’re can defend against automation bias.

In my observation and experience, the cost of automating tests checks increases and the value decreases—generally—the closer to the GUI you get. Automated tests checks at the unit level tend to be easily to comprehend, simple to automate, subject to falsifiable assertions, and immediately responsive to developers. Automated tests checks at the GUI level tend to be inadequate for recognizing problems that humans will spot easily, complex to program, much less clearly decidable, and harder to troubleshoot and debug, thus less immediately responsive to developers. There are bound to be exceptions.

You might get different answers, making better use of your tools and the people who use and program them, if you think of test automation in James Bach’s definition as “any use of tools to support testing”.

Postscript: There’s a considerably deeper treatment of this issue available from here.