Conversations about testing in all kinds of places have been going pear-shaped for a long time. As Jerry Weinberg was fond of pointing out, the word “testing” is overloaded, lumping a variety of ideas and activities together. The word “testing” gets applied to different actitivities, performed by different people, working in different contexts, performing different tasks with different priorities, at different moments in the development process. No wonder that people in these different paradigms end up talking past each other.
Those differences and that overloading can lead to controversies. If we don’t consider these different contexts and context factors, the conversation won’t get any better, and neither will testing. Maybe we can address some of the controversies by starting from first principles, unpacking or “de-lumping” ideas about testing, and getting clear on what we’re talking about. That’s what I’m going to try to do in this long-ish series of long-ish posts.
What I’ll Be Talking About
Before we dig in, let’s start with getting straight on some terms that I’ll be using.
- When I say “product“, I’m referring to some kind of product or service. “Product” applies not only to deployed and running software, but also to anything that anyone might produce or provide along the way. That might include
- a feature
- a specification
- a function
- a
- a whiteboard diagram
- an automated check
- a requirements document
- a test design
- a one-line code change
- an idea
- …
- When I talk about “we“, in general, I’m referring to the development group or the project community — testers, developers, managers, documentation folk, DevOps builders and maintainers, customer support, or any number of roles I might have left out.
- We are all working for some organization or another; and sometimes our own organizations are serving others. Not all organizations are businesses, but in general I’ll refer to “the business” here to refer both to our direct clients and to their customers. I’ll make distinctions between those when they’re significant.
- When I’m talking about testing, I’m using our definition of testing in the Rapid Software Testing namespace. Testing is “the process of evaluating a product by learning about it through experiencing, exploring and experimenting, which includes to some degree questioning, studying, modeling, observation, inference, etc.” Remember that “product” applies to any product in the sense I’ve described above. The experiments that we perform might be “tests” in the conventional sense of configuring, operating, observing, and evaluating a running software product; for ideas or design artifacts, the “experiments” might take the form of thought experiments in analysis or review.
- Checking output from the product is an activity embedded in testing work. Checking (again, in the Rapid Software Testing namespace) is the process of operating and collecting observations about it; applying decision rules to those observations; and then reporting on the outcome of those rules; all algorithmically. That is, a check can be turned into a scripted process that can be performed by a human or by a machine. Checks, and checking, can be automated; testing can’t. Developing checks (which includes the process of designing and encoding checks) can be aided by tools, but can’t be automated either. Similarly, tools can help us to interpret the outcome of checks, but evaluating what the checks might be telling us (a bug in the product? a bug in the check? some other problem?) is a fundamentally human activity.
- Anyone on the project can take on the role of a tester, doing testing work, or performing tests. Some people have testing work as their primary responsiblity. In the Rapid Software Testing namespace, we usually call these, simply, “testers”. Sometimes we want to distinguish those testers from people in other roles who do testing work, when testing work isn’t their primary responsibility. We sometimes refer to those people as supporting testers, and refer to responsible testers when we refer to the former group. I’ll try distinguish between the two when the difference matters.
What Does the Business Want?
We’re software people. We’re providing services to an organzation or business of some kind. We’ll call it “the business”, even though the organization might not be a commercial venture; and we’ll treat “money” and “revenue” to represent value, cash or social credit of some kind.
So what does the business want?
The business wants to make money by providing products or services (we’ll just use “product” for now) that are valuable to people. These might include products that help people get routine stuff done more powerfully or efficiently; products that enable things that were never possible before; products that educate or entertain. The business can market and sell a high-value product, and bring in some of that tasty, tasty revenue.
For that to happen, the business needs people to envision something worth building, to design it, and to build it. That’s how the business gets something that it can sell as a product or service, and that’s how all that lovely money flows in.
At the same time, the business doesn’t want too much money to flow out in terms of salaries, supplies, support costs, and remediation when things go wrong. The business needs to keep the costs of development, support, and maintenance appropriately low, so that the business remains sustainable.
Now: in order for the business to get what it wants, at least one thing has to happen: we have to build the product.
Moving from an idea about something worth building to building it entails envisioning what a successful product will look like.
So far so good. But… there might be problems. (to be continued!)