Deeper Testing (3): Testability

Some testers tell me that they are overwhelmed at the end of “Agile” sprints. That’s a test result—a symptom of unsustainable pace. I’ve written about that in a post called “Testing Problems are Test Results“.

In Rapid Software Testing, we say that testing is evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, and plenty of other stuff—perhaps including the design and programming of automated checks, and analysis of the outcome after they’ve been run. Note that the running of these checks—the stuff that the machine does—does not constitute testing, just as the compiling of the product—also the stuff that the machine does—does not constitute programming.

If you agree with this definition of testing, when someone says “We don’t have enough time for testing,” that literally means “we don’t have enough time to evaluate our product.” In turn, that literally means “we don’t have time to learn about this thing that (presumably) we intend to release.” That sounds risky.

If you believe, as I do, that evaluating a product before you release it is usually a pretty good idea, then it would probably also be a good idea to make testing as fast and as easy as possible. That is the concept of testability.

Most people think of a testability quite reasonably in terms of visibility and controllability in the product. Typically visibility refers to log files, monitoring, and other ways of seeing what the product is up to; controllability usually refers to interfaces that allow for easy manipulation of the product, most often via scriptable application programming interfaces (APIs).

It’s a good thing to have logging and scriptable interfaces, testability isn’t entirely a property of the product. Testability is a set of relationships between the product, the team, the tester, and the context in which the product is being developed and maintained. Changes in any one of these can make a product more or less testable. In Rapid Software Testing, we refer to the set of these relationships as practical testability. This breaks down into five other subcategories that overlap and interact to some degree.

Epistemic testability. (Yes, it’s a ten-dollar word. Epistemology is the study of how we know what we know. Excellent testing requires us to study epistemology if we want to avoid being fooled about what we know.) As we’re building a product, there’s a risk gap, the difference between what we know and what we need to know. A key purpose of testing is to explore the risk gap, shining light on the things that we don’t know, and identifying places that are beyond the current extent of our knowledge. A product that we don’t know well in a test space that we don’t know much about tends to make testing harder or slower.

Value-related testability. It’s easier to test a product when we know something about how people might use it and how they might intend get value from it. That means understanding people’s goals and purposes, and how the product is designed to fulfill and support them. It means considering who matters&mash;not just end users or customers, but also anyone who might have a stake in the success of the product. It means learning about dimensions of quality that might be more important or not so important to them.

Intrinsic testability. It’s easier to test a product when it is designed to help us understand its behaviour and its state. When the parts of the product are built cleanly and simply, testing as we go, testing the assembled parts will be easier. When we have logging and visibility into the workings of the product, and when we have interfaces that allow us to control it using tools, we can induce more variation that helps to shake the bugs out.

Project-related testability. It’s easier to test when the project is organized to support evaluation, exploration, experimentation, and learning. Testing is faster and easier when testers have access to other team members, to information, to tools, and to help.

Subjective testability. The tester is at the centre of the relationships between the product, the project, and the testing mission. Testing will be faster, easier, and better when the tester’s skills—and testing skill on the team—are sufficient to deal with the situation at hand.

Each one of these dimensions of testability fans out into specific ideas for making a product faster and easier to test. You can find a set of ideas and guidewords in a paper called Heuristics of Software Testability.

On an Agile team, a key responsibility for the tester is to ask and advocate for testability, and to highlight things that make testing harder or slower. Testability doesn’t come automatically. Teams and their managers are often unaware of obstacles. Programmers may have created unit checks for the product, which may help to reduce certain kinds of coding and design errors. Still, those checks will tend to be focused on testing functions deep in the code. Testability for other quality criteria—usability, compatibility, performance, installabilty, or compatibility, to name only a few—may not get much attention without testers speaking up for them.

A product almost always gets bigger and more complex with every build. Testability helps us to keep the pace of that growth sustainable. A less testable product contributes to an unsustainable pace. Unsustainable pace ratchets up the risk of problems that threaten the value of the product, the project, and the business.

So here’s a message for the tester to keep in front of the team during that sprint planning meeting, during the sprint, and throughout the project:

Let’s remember testability. When testing is harder or slower, bugs have more time and more opportunity to stay hidden. The hidden bugs are harder to find than any bugs we’ve found so far—otherwise we would have found them already. Those bugs—deeper, less obvious, more subtle, more intermittent—may be far worse than any bugs we’ve found so far. Right now, testability is not as good as it could be. Is everyone okay with that?