“What Tests Should I Automate?”

Instead of asking “What tests should I automate?” consider asking some more pointed questions.

If you really mean “how should I think about using tools in testing?”, consider reading A Context-Driven Approach to Automation in Testing, and Testing and Checking Refined.

If you’re asking about the checking of output or other facts about the state of the product, keep reading.

Really good fact checking benefits from taking account of your status so that you don’t waste time:

  • Do I know enough about the product, and where there might be problems, to be sure that I’m not rushing into developing checks?

If the answer is No, it might be a better idea to do a deeper survey of the product, and scribble down some notes about risk as you go.

If the answer is Yes, then you might want to loop through a bunch of questions, starting here:

  • What specific fact about the product’s output or state do I want to check?
  • Do I know enough about the product, and where there might be problems, to be reasonsonably sure that this really is an important fact to check?
  • Is someone else (like a developer) checking this fact already?

Then maybe consider product risk:

  • What could go wrong in the product, such that I could notice it by checking this fact?
  • Is it a plausible problem? A significant risk?
  • Why do we think it’s a significant risk?
  • Is that foreboding feeling trying to tell us something?

Maybe if there’s serious risk here, a conversation to address the risk is a better idea than more testing or checks.

Assuming that it’s a worthwhile fact to check, move on to how you might go about checking the fact, and the cost of doing it:

  • What’s a really good way to check this fact?
  • Is that the fastest, easiest, least expensive way to check it?
  • Will the check be targeted at a machine-friendly interface?

Consider opportunity cost, especially if you’re targeting automated checks at the GUI:

  • What problems will I encounter in trying to check this fact this way, and doing that reliably?
  • What problems, near here or far away, might I be missing as a consequence of focusing on this fact, and checking it this way?
  • Is there some activity other than checking this fact that might be more valuable right now? In the long run?

Try thinking in terms of the longer term. On the one hand, the product and certain facts about it might remain very stable:

  • What would cause this fact to change? Is that likely? If not, is it worthwhile to create a check for this fact?

On the other hand, the product or the platform or the data or the test environment might become unstable. They might be unstable already:

Beware of quixotic reliability. That’s a wonderful term I first read about in Kirk and Miller’s Reliability and Validity in Qualitative Research. It refers a situation where we’re observing a consistent result that’s misleading, like a broken thermometer that reliably reads 37° Celsius. (Kirk and Miller have some really important things to say about confirmatory research, too.) (And, ironically, that link back there to “quixotic reliability” is consistently unreliable.)

  • Is there a chance that this check might lull us to sleep by running green in a way that fools us?

To address the risk of quixotic reliability and to take advantage of what we might have learned, it’s good to look at every check, every once in a while:

  • What’s our schedule for reviewing this check? For considering sharpening it, or broadening it?

A sufficently large suite of checks is incomprehensible, so there’s no point point in running checks that are no longer worthwhile:

  • What’s our plan for checking this fact less often when we don’t need it so much? For retiring this check?

The next questions are especially important if you’re a tester. Checks are least expensive and most valuable when they provide fast feedback to the developers; so much so that it might be a good idea for the developers to check the code before it ever gets to you.

  • Am I the right person to take responsibility for checking this fact? Am I the only person? Should I be?
  • Is checking this fact, this way, the earliest way that we could become aware of a real problem related to it?

Given all that, think about utility—the intersection of cost, and risk, and value:

  • Do I still believe I really need to check this fact? Is it worthwhile to develop a check for it?

After you’ve asked these questions, move on to the next fact to be checked.

“But asking and answering all those questions will take a long time!”, you might reply.

Not that long. Now that you’ve been primed, this is a set of ideas that you can soon carry about in your mind, and run through the list at the speed of thought. Compare asking these questions with how long it takes to develop and run and interpret and revise and review and maintain even one worthless check.

Now it’s true, you can save some time and effort by skipping the interpretation and review and maintenance stuff. That is, essentially, by ignoring the check after you’re written it. But if you’re not going to pay attention to what a check are telling you, why bother with it at all? It’s faster still not to develop it in the first place.

With practice, the questions that I offered above can be asked routinely. Don’t assume that my list is comprehensive; ask your own questions too. If you pay attention to the answers, you can be more sure that your checks are powerful, valuable, and inexpensive.

1 reply to ““What Tests Should I Automate?””

Leave a Comment