Resources

On this page, I offer some notes and places where you can go for further information about testing in general — and about Rapid Software Testing approaches in particular. The material listed here is my own, except where noted otherwise.

Before we get started… what’s the big deal about heuristics?

“Heuristic” is a word that is poorly understood, but not hard to understand. A heuristic is, simply, a fallible means for solving a problem. “Heuristic”, as an adjective, means and implies “serving to discover”. All testing—indeed, all engineering—is based on heuristics; ideas and approaches and models that can work but that might fail.

Heuristics for Understanding Heuristics

Discussion of the Method (Koen) This book describes the engineering method as “the use of heuristics to cause the best change in a poorly understood situation within the available resources.” In testing, we’re trying to shift our notion of the status of the product from untested assumptions to observed facts. That’s an engineering process, and a heuristic process.

How to Solve It (Polya) The word “heuristic” was underappreciated until George Polya revived it in this book, originally published in 1945. How to Solve it focuses on the mathematical moving from what is known and building from that to solve problems.

Test Strategy

In the Rapid Software Testing namespace, a test strategy is the set of ideas that guide your choice of tests. Here we mean “choice” and “tests” in the most expansive senses of the words. We’re referring not only to “tests that you choose to perform”, but also to the choices that you make in designing, selecting, performing, and interpreting your tests and test ideas.

In Rapid Software Testing, the Heuristic Test Strategy Model (HTSM, developed by James Bach and me) is a tool that we use to help us develop, think about, organize, and justify test strategy.

The word “heuristic” does a lot of work here. Since it’s a model, the HTSM is a heuristic; a test strategy is a fallible means for solving the problem of how to test something; and testing itself is a heuristic approach for determining the status of the product.

The Rapid Software Testing Framework (developed by James Bach and me, with graphical help from Mary Alton) represents another way to think about test strategy and what motivates it.

If a test strategy is a set of ideas that guide your choice of tests, Lessons Learned in Software Testing (Kaner, Bach, & Pettichord) is a kind of second-order version of that; a set of ideas to guide your sets of ideas. Lessons Learned of the few truly good books on software testing.

Testing and Checking

The distinction between testing and checking is about the difference between testing itself and one element of it. Checking the output is relatively easy. Evaluating the product to find problems — and determining whether people will want to fix those problems or tolerate them — is much more challenging.

In the Rapid Software Testing namespace, testing is a heuristic process of evaluating a product by learning about it through experiencing, exploring and experimenting, which includes to some degree questioning, studying, modeling, observation, inference, etc.

Checking is an algorithmic process of operating and observing a product; applying decision rules to those observations; and then reporting on the outcome of those rules. That is, a check is not a test. A check is a part of a test that can be turned into a scripted process to be performed by a human or by a machine.

Testing and Checking Refined, written by James Bach and me, is a detailed description of how we think about the distinction and why it’s important.

Testability

The practical testability of a product is how easy it is to test by a particular tester and test process, in a given context. That means that testability isn’t a property of the product; it’s a relationship between the product, the context, and the tester.

People often refer to testability in terms of things built into the product; visibility (for example, log files) and controllability (for example, scriptable interfaces or APIs). These are instances of intrinsic testability. There is more to intrinsic tesability than that; and there are other dimensions of testability as well. These include

epistemic testability: the gap between what (we think) we know about the product and what (we believe) we need to know
value-related testability: how easy it is to determine what really matters to customers and other stakeholders
project-related testability: how the project has been set up to make testing faster and easier
subjective testability: how well the tester’s skill set and experience fit for the testing work that needs to be done

There’s a detailed explanation of these aspects of testability in Heuristics of Software Testability by James Bach. There’s a summary of that paper in my blog post Deeper Testing (3): Testability.

Test Cases and Alternatives to Them

Journalists don’t use journalism cases; researchers don’t use research cases; managers don’t use management cases; and designers don’t use design cases. In testing, why is there such a fixation on test cases? And how might we refocus on testing?

“Test Cases Are Not Testing: Toward a Culture of Test Performance” (James Bach and Aaron Hodder)

Breaking the Test Case Addiction (blog series)

Oracles: How to Recognize and Describe Problems

An oracle is a means by which we recognize a problem when we encounter one in testing. This concept is remarkably under-discussed, in my view. Here are a couple of posts that help to explain oracles and how we might apply them.

All Oracles Are Heuristic

FEW HICCUPPS

Oracles from the Inside Out (series)

Coverage: How to Describe the Thoroughness of Testing

In the Rapid Software Testing namespace, coverage is how thoroughly we have examined the product with respect to some model related to the product. Talking about coverage, therefore, requires us to talk about models related to the product, and to the systems in which the product exists.

I wrote the blog post Talking About Coverage to offer some suggestions on how to do that. A key point of that post is that many of the models we apply to coverage don’t yield well to counting and to measurement. From that, it follows that some things that bear on coverage could be measured, but overall test coverage can’t be quantified in a useful way. The good news is that, like quality, coverage can be discussed, described, and assessed.

The Product Elements section of the Heuristic Test Strategy Model includes a list of seven different dimensions by which we might evaluate and describe coverage.

There are as many different ways to represent coverage as there are ways to represent models of the product. Here are some examples of product coverage outlines in many different forms.

Some older articles address the topic of coverage, too.

Got You Covered: Excellent testing starts by questioning the mission. So, the first step when we are seeking to evaluate or enhance the quality of our test coverage is to determine for whom we’re determining coverage, and why.

Cover or Discover: Excellent testing isn’t just about covering the “map”—it’s also about exploring the territory, which is the process by which we discover things that the map doesn’t cover.

A Map By Any Other Name: A mapping illustrates a relationship between two things. In testing, a map might look like a road map, but it might also look like a list, a chart, a table, or a pile of stories. We can use any of these to help us think about test coverage.

Exploratory Testing

Exploratory testing isn’t an activity unto itself. All testing, if it’s actual testing, is exploratory. That is, it involves navigating through an uncertain test space, and making choices while doing so.

Some people refer to exploratory testing as “unstructured”, apparently because they have a limited concept of what “structure” means. (In general, it means a pattern that persists; the relevant senses from Oxford include “The arrangement and organization of mutually connected and dependent elements in a system or construct”; “The quality or fact of being organized in a particular manner; definite or purposeful arrangement of parts within a whole.”)

There are many things that lend structure to exploratory testing. Those who claim that exploratory testing is “unstructured” really mean that it’s unscripted.

In Exploratory Testing 3.0 (James Bach and Michael Bolton), you’ll find some of our current thinking on the subject, including a notion of deprecating “exploratory testing” and replacing it with “testing”.

It’s a good idea to have a mission in mind when you’re doing ~~exploratory~~ testing. Breaking the Test Case Addiction, Part 5 and Breaking the Test Case Addiction, Part 6 discuss ways to charter that mission.

“Manual” and “Automated” Testing

Testing is neither “manual” nor “automated”. We don’t visit manual or automated doctors; journalists don’t practice “manual journalism” or “automated journalism”; designers do “automated design” or “manual design”. Testing is the only profession, so it seems, that divides itself in this enormously unhelpful way.

Here’s a description of why I believe the distinction is problematic, and why we need to drop it: Manual Testing: What’s the Problem?

There are better, more positive ways to think and speak more about the tester’s activity and engagement. Read about those here:

Alternatives to “Manual Testing”: Experiential, Atteneded, and Exploratory

And here are two earlier complaints about “manual testing”: Manual” and “Automated” Testing, and The End of Manual Testing

In late 2020, I made a video about “manual” and “automated” testing: What’s Wrong with Manual Testing?

Deeper Testing (2) – “Automating the Testing“

Tools and “Automation”

In Rapid Software Testing, we embrace the role of tools, but we don’t want to turn them into a fixation. We would prefer to take a context-driven approach to automation in testing.

It’s a really good idea, in general, for testers to learn to code, if they feel so inclined. Here are at least three good reasons for testers to learn to code.

But if coding really, really turns you off, don’t worry. There are usually plenty of people who can code on the project, and relatively few who are focused on finding trouble. The latter people can get great help from the former group.

We’ve also developed some ideas on how to select tools; you can find those here.

Measurement, Metrics, and Assessment

“Quality is value to some person(s).” —Jerry Weinberg

“Measurement is the art and science of making reliable and significant observations.” —Jerry Weinberg

In Rapid Software Testing, we are very skeptical of quantitative measurement of software quality or human performance. We are concerned about the danger of turning engineering into scorekeeping.

Three Kinds of Measurement and Two Ways to Use Them (article)

What Counts (article)

Assess Quality, Don’t Measure It (James Bach)

“Software Engineering Metrics: What Do They Measure and How Do We Know” (Kaner and Bond) If you only read one formal article on measurement, this 12-page paper might be your best bet.

Measuring and Managing Performance in Organizations (Austin)

Breaking the Test Case Addiction, Part 8