On this page, I offer some notes and places where you can go for further information about testing in general — and about Rapid Software Testing approaches in particular. The material listed here is my own, except where noted otherwise.
Before we get started… what’s the big deal about heuristics?
“Heuristic” is a word that is poorly understood, but not hard to understand. A heuristic is, simply, a fallible means for solving a problem. “Heuristic”, as an adjective, means and implies “serving to discover”. All testing—indeed, all engineering—is based on heuristics; ideas and approaches and models that can work but that might fail.
Heuristics for Understanding Heuristics
Discussion of the Method (Koen) This book describes the engineering method as “the use of heuristics to cause the best change in a poorly understood situation within the available resources.” In testing, we’re trying to shift our notion of the status of the product from untested assumptions to observed facts. That’s an engineering process, and a heuristic process.
How to Solve It (Polya) The word “heuristic” was underappreciated until George Polya revived it in this book, originally published in 1945. How to Solve it focuses on the mathematical moving from what is known and building from that to solve problems.
In the Rapid Software Testing namespace, a test strategy is the set of ideas that guide your choice of tests. Here we mean “choice” and “tests” in the most expansive senses of the words. We’re referring not only to “tests that you choose to perform”, but also to the choices that you make in designing, selecting, performing, and interpreting your tests and test ideas.
In Rapid Software Testing, the Heuristic Test Strategy Model (HTSM, developed by James Bach and me) is a tool that we use to help us develop, think about, organize, and justify test strategy. The word “heuristic” does a lot of work here. Since it’s a model, the HTSM is a heuristic; a test strategy is a fallible means for solving the problem of how to test something; and testing itself is a heuristic approach for determining the status of the product.
The Rapid Software Testing Framework (developed by James Bach and me, with graphical help from Mary Alton) represents another way to think about test strategy and what motivates it.
If a test strategy is a set of ideas that guide your choice of tests, Lessons Learned in Software Testing (Kaner, Bach, & Pettichord) is a kind of second-order version of that; a set of ideas to guide your sets of ideas. Lessons Learned of the few truly good books on software testing.
Testing and Checking
The distinction between testing and checking is about the difference between testing itself and one element of it. Checking the output is relatively easy. Evaluating the product to find problems — and determining whether people will want to fix those problems or tolerate them — is much more challenging.
In the Rapid Software Testing namespace, testing is a heuristic process of evaluating a product by learning about it through experiencing, exploring and experimenting, which includes to some degree questioning, studying, modeling, observation, inference, etc.
Checking is an algorithmic process of operating and observing a product; applying decision rules to those observations; and then reporting on the outcome of those rules. That is, a check is not a test. A check is a part of a test that can be turned into a scripted process to be performed by a human or by a machine.
Testing and Checking Refined, written by James Bach and me, is a detailed description of how we think about the distinction and why it’s important.
The practical testability of a product is how easy it is to test by a particular tester and test process, in a given context. That means that testability isn’t a property of the product; it’s a relationship between the product, the context, and the tester.
People often refer to testability in terms of things built into the product; visibility (for example, log files) and controllability (for example, scriptable interfaces or APIs). These are instances of intrinsic testability. There is more to intrinsic tesability than that; and there are other dimensions of testability as well. These include
- epistemic testability: the gap between what (we think) we know about the product and what (we believe) we need to know
- value-related testability: how easy it is to determine what really matters to customers and other stakeholders
- project-related testability: how the project has been set up to make testing faster and easier
- subjective testability: how well the tester’s skill set and experience fit for the testing work that needs to be done
There’s a detailed explanation of these aspects of testability in Heuristics of Software Testability by James Bach. There’s a summary of that paper in my blog post Deeper Testing (3): Testability.
Test Cases and Alternatives to Them
Journalists don’t use journalism cases; researchers don’t use research cases; managers don’t use management cases; and designers don’t use design cases. In testing, why is there such a fixation on test cases? And how might we refocus on testing?
“Test Cases Are Not Testing: Toward a Culture of Test Performance” (James Bach and Aaron Hodder)
Breaking the Test Case Addiction (blog series)
Oracles: How to Recognize and Describe Problems
An oracle is a means by which we recognize a problem when we encounter one in testing. This concept is remarkably under-discussed, in my view. Here are a couple of posts that help to explain oracles and how we might apply them.
Oracles from the Inside Out (series)
Exploratory testing isn’t an activity unto itself. All testing, if it’s testing, is exploratory. That is, it involves navigating through an uncertain test space, and making choices while doing so.
In Exploratory Testing 3.0 (James Bach and Michael Bolton), you’ll find some of our current thinking on the subject, including a notion of deprecating “exploratory testing” and replacing it with “testing”.
“Manual” and “Automated” Testing
Testing is neither “manual” nor “automated”. We don’t visit manual or automated doctors; journalist don’t practice “manual journalism” or “automated journalism”; designers do “automated design” or “manual design”. Testing is the only profession, so it seems, that divides itself in this enormously unhelpful way.
Here’s a description of why I believe the distinction is problematic, and why we need to drop it: Manual Testing: What’s the Problem?
There are better, more positive ways to think and speak more about the tester’s activity and engagement. Read about those here:
Alternatives to “Manual Testing”: Experiential, Atteneded, and Exploratory
And here are two earlier complaints about “manual testing”: Manual” and “Automated” Testing, and The End of Manual Testing
In late 2020, I made a video aboiut “manual” and “automated” testing: What’s Wrong with Manual Testing?
Tools and “Automation”
In Rapid Software Testing, we embrace the role of tools, but we don’t want to turn them into a fixation. We would prefer to take a context-driven approach to automation in testing.
It’s a really good idea, in general, for testers to learn to code, if they feel so inclined. Here are at least three good reasons for testers to learn to code.
But if coding really, really turns you off, don’t worry. There are usually plenty of people who can code on the project.
We’ve also developed some ideas on how to select tools; you can find those here.
Measurement, Metrics, and Assessment
“Quality is value to some person(s).” —Jerry Weinberg
“Measurement is the art and science of making reliable and significant observations.” —Jerry Weinberg
In Rapid Software Testing, we are very skeptical of quantitative measurement of software quality or human performance. We are concerned about the danger of turning engineering into scorekeeping.
Three Kinds of Measurement and Two Ways to Use Them (article)
What Counts (article)
Assess Quality, Don’t Measure It (James Bach)
“Software Engineering Metrics: What Do They Measure and How Do We Know” (Kaner and Bond) If you only read one formal article on measurement, this 12-page paper might be your best bet.
Measuring and Managing Performance in Organizations (Austin)
Recording and Reporting
As you’re testing the product, it’s crucial to be able to keep track of your work, and to describe what you’ve found.
An Exploratory Tester’s Notebook
Rapid Software Testing Guide to Making Good Bug Reports (James Bach and Michael Bolton)
Breaking the Test Case Addiction Part 10 This part of the series addresses reporting, and includes several examples of test reports.
You can find an extensive list of presentations and courses that I’ve taught, including the slides and speaker notes for many of them, here.