Four Frames for Testing, Part 2: Four Kinds of Risk

In the first installment of this series, I introduced two key things that the business wants from development: a product of high value and low cost. In order for the business to get a high-value product, we must envision success so we can set about building it.

And yet… there’s risk.

It’s easy to assume that we’ve built a high-value product, and that cost to the business is low and will remain so. That can be a risky assumption. We’re human beings, and human beings make mistakes. People’s needs and desires — and our ideas about them — can change over time, too.

So, throughout development work, we must not only envision success, but also anticipate and confront the possibility of failure. To address that possibility, it’s a good idea to examine the product that we believe we’ve built. (Remember, from the first post in this series: “product” here refers to anything that anyone has produced, from an idea at the beginning to full-blown, deployed system at the end, and everything in between.)

A prudent and responsible development group will study the things we’ve built and our beliefs about them at every step of the way. We do that to find out if we and the business have really got what we really want.

Envisioning success is a wonderful thing. To achieve success, we must recognize that needs and desires might change, and there is potential of error. Anticipating the possibility of failure is pragmatic, just in case we have somehow failed to produce something of high value. All this prompts us to build in a diligent and disciplined way, which helps to keep value high and costs — including the cost of change — low. To help decide whether we really have the product we really want, we must test responsibly. So let’s decorate the bottom right and upper left corners of our table with those activities.

What Could Go Wrong?

Working in social groups — teams — is a powerful heuristic for allowing us to develop and refine our visions of success and our potential errors. Different people bring different perspectives — critical distance — to alert us to problems in our own thinking. Even when we work in groups, we’re still vulnerable to errors in designing, developing, building, and testing the product.

As we’re envisioning success, we might misimagine things. We might forget to consider people outside our own group — or even some people inside it. We might be unaware of things that certain people might want. Satisfying dimensions of quality always represents a set of tradeoffs on some level; some needs and desires might be in conflict with others. Getting consensus on what we intend, and for whom, can be hard. Even then, we might seem to agree on our intentions, but our understanding might be based on unquestioned assumptions and shallow agreement. Those misunderstandings can play out as we set out to build the product.

Even when we’re clear on our intentions, building on its own can bring certain kinds of routine and mundane problems. We might mistype; forget a step in some procedure we’re trying to encode; neglect to check a return value; lose count of something; make something happen too slowly, or too quickly; try to fit something into too small a space. Errors tend to prompt rework, cause delays and increase costs, posing risk to the business. Working in pairs or larger groups can help to address those risks. Tools and procedures can help too.

Studying the product can be harder or slower than it might need to be. Even we’re pretty clear on what we intend, and even when we’re building reasonably carefully, anticipating failure prompts us to consider that we might need to test more deeply. To do that efficiently, we need to be ready for deeper testing when decide that it’s necessary.

The motivation for testing responsibly is straightforward: even when we’ve been careful, diligent, and disciplined, there’s still a chance that problems might have got past us. This is especially true for complex systems with lots of emergent behaviours and attributes; for systems where there’s a high degree of change; for systems where there are many different kinds of users and patterns of usage; for systems that run on a wide variety of platforms.

So those, broadly speaking, are the risks. How do we and the business get what we want at each step, and how do we deal with those risks? We’ll look at that more detail in the next installment.