There’s a lot of focus over the last several years on “shift left” testing — that is, testing that happens before we have a product to test. In the Rapid Software Testing namespace, we’ve come to call this prospective testing.
Wait… how can we test a product before we have a product? We can’t test the built product, but we can test the precursors to it. Prospective testing is focused on getting our intentions clear, figuring out what the business and its customers want from the product, and how we might go about building it. It involves review of requirement and specification documents; testing mock-ups, wireframes, prototypes, or simulators. It might involve testing API endpoints that will reside underneath a GUI that users will think of as “the product”.
Of course, each of the things that I’ve mentioned above might be considered a built product on its own, with its own set of precursors. Then that built product might represent a precursor to another product that we intend to build. Prospective testing can be applied to a prototype by evaluating a sketch about it; and prospect testing can be applied to the sketch by considering and evaluating things that people are saying as they’re drawing it. We can prospectively test ideas about the product, in our minds, via thought experiments.
Bugs in precursors might not be a big deal, because users won’t necessarily see them. As testers, we can’t prevent problems in the precursors we’re evaluating — the problems are already there — but we can prevent oblivion to those problems. Primary testing is a really good thing to do, because problems in the precursors stand a chance of persisting and contaminating the built product. It’s good to bring the problems to light early so that the designers and builders can prevent them from going futher.
Then, of course, there’s regression testing — testing motivated by risk due to changes made in a previously tested product. That’s a good thing too, because changes to the built and previously tested product could introduce new problems that cause quality to backslide.
But hang on… when was the built product previously tested?
The trouble we see is that talk — and worse, often, practice — has squeezed out the important testing work that happens on the built product, after prospective testing and before regression testing. The testing between those two is literally central, the most important “season” of testing. It’s so central that we’ve traditionally called it, simply, testing. In the course of writing our book, we’ve realized that, since the word “testing” is overloaded, we might need to be more explicit for people who are thinking about testing only in terms of prospective testing and/or regression testing. In the RST namespace, we’re calling it primary testing — testing now that we have a built product.
In primary testing, we’re actively looking for trouble. We’re obtaining experience with the product, exploring its nooks and crannies, and performing experiments on it. We’re developing and deepening our awareness and working mental models of the product of interest, as it has actually been built.
Primary testing tends to be focused on interactive, experiential, analytical work. It can be supported by lots of tools. It is productive in terms of discovering risks that we didn’t anticipate. It’s an opportunity to develop new ideas for automated regression checks — and to learn whether shallow checking is sufficient, or whether deeper testing is necessary.
Most importantly, in primary testing, we find lots of bugs. Those bugs are real, in that they are actually in the built product, and have therefore have a chance of being seen by the users.
Prospective testing is important, but it doesn’t account for emergence. It doesn’t account for the fact that some bugs will slip past us and into the built product, even in a highly disciplined development process. Automated checks are often derived from what we believe the built product will be like, and not from what it’s actually like. That is: automated checks without primary testing will be based on theories about the imagined product, not on experience with the built product. If you haven’t learned the product through primary testing, your regression checks will be probably be overfocused and unhelpfully shallow.
Both in talk and in practice, primary testing must not be crowded out.