“What Tests Should I Automate?”

Instead of asking “What tests should I automate?” consider asking some more pointed questions. If you really mean “how should I think about using tools in testing?”, consider reading A Context-Driven Approach to Automation in Testing, and Testing and Checking Refined. If you’re asking about the checking of output or other facts about the state of the product, keep reading. Really good fact checking benefits from taking account of your … Read more

Experience Report: Katalon Studio

Warning: this is another long post. But hey, it’s worth it, right? Introduction This is an experience report of attempting to perform a session of sympathetic survey and sanity testing on a “test automation” tool. The work was performed in September 2021, with follow-up work November 3-4, 2021. Last time, the application under test was mabl. This time, the application is Katalon Studio. My self-assigned charter was to explore and … Read more

Worlds in Collision (keynote at ConTEST 2021)

Michael Bolton – Worlds in Collision (keynote at ConTEST 2021)

Listen to Michael Bolton at ConTEST 2021. What happens when builders and testers work together to develop GUI testing infrastructure?

In this keynote, Michael will give a preliminary experience report on observing, working, and collaborating with an expert tester (James Bach, co-author of Rapid Software Testing) — and an expert automator, Michael Mintz (author of Selenium Base) on a testing and learning mission.

The story is ongoing, and only beginning, but already we’re learning about key differences between world views and mind sets, and how we might go about resolving them. Check out online programs at Test Masters Academy: https://testmasters

What Should I Automate?

I get this question a lot: a tester who has just learned to program, or who has just learned about a new framework or tool set asks “Now that I’ve learned this, what should I automate?” Some people (mostly men, so it seems) go into hardware stores and see some fancy tool like a compound mitre saw. Unable to resist temptation, they imagine themselves building… something. So they buy the … Read more

Which Test Cases Should I Automate?

When someone asks “I have a big suite of manual tests; which tests (or worse, which test cases) should I automate?”, I often worry about several things. The first thing is that focusing on test cases is often a pretty lousy way to think about testing.  That’s because test cases are often cast in terms of following an explicit procedure in order to observe a specific result.  At best, this … Read more


Several years ago in one of his early insightful blog posts, Pradeep Soundarajan said this: “The test doesn’t find the bug. A human finds the bug, and the test plays a role in helping the human find it.” More recently, Pradeep said this: Instead of saying, “It is programmed”, we say, “It is automated”. A world of a difference. It occurred to me instantly that it could make a world … Read more

A Context-Driven Approach to Automation in Testing

(We interrupt the previously-scheduled—and long—series on oracles for a public service announcement.) Over the last year James Bach and I have been refining our ideas about the relationships between testing and tools in Rapid Software Testing. The result is this paper. It’s not a short piece, because it’s not a light subject. Here’s the abstract: There are many wonderful ways tools can be used to help software testing. Yet, all … Read more

Oracles from the Inside Out, Part 8: Successful Stumbling

When we’re building a product, despite everyone’s good intentions, we’re never really clear about what we’re building until we try to build some of it, and then study what we’ve built. Even after that, we’re never sure, so to reduce risk, we must keep studying. For economy, let’s group the processes associated with that study—review, exploration, experimentation, modelling, checking, evaluating, among many others—and call them testing. Whether we’re testing running … Read more

Oracles from the Inside Out, Part 5: Oracles as References as Media

Try asking testers how they recognize problems. Many will respond that they compare the product to its specification, and when they see an inconsistency between the product and its specification, they report a bug. Others will talk about creating and running automated checks, using tools to compare output from the product to specific, pre-determined, expected results; when the product produces a result inconsistent with expectations, the check identifies a bug … Read more

Very Short Blog Posts (15): “Manual” and “Automated” Testers

“Help Wanted. Established scientific research lab seeks Intermediate Level Manual Scientist. Role is intended to complement our team of Automated and Semi-Automated Scientists. The successful candidate will perform research and scientific experiments without any use of tools (including computer hardware or software). Requires good communication skills and knowledge of the Hypothesis Development Life Cycle. Bachelor’s degree or five years of experience in manual science preferred.” Sounds ridiculous, doesn’t it? It … Read more