Exploratory Testing on an API? (Part 4)

As promised, (at last!) here are some follow-up notes on previous installments in the series that starts here.

Let’s revisit the original question:

Do you perform any exploratory testing on APIs? How do you do it?

To review: there’s a problem with the question. Asking about “exploratory testing” is a little like asking about “vegetarian cauliflower”, “carbon-based human beings”, or “metallic copper”. Testing is fundamentally exploratory. Testing is an attempt by a self-guided agent to discover something unknown; to learn something about the product, with a special focus on finding problems and revealing risk.

One way to learn something about the product is to develop and run a set of automated checks that access the product via the API. That is, a person writes code to direct the machine to operate and observe the product, apply decision rules, and report the outputs. This produces a script, but producing a script is not a scripted process! Learning about the product, designing the checks, and developing the code to implement them checks are all exploratory processes. Interpreting and explaining and reporting on what happened are exploratory processes too.

When we use a machine to execute automated checks, there’s no discovery or learning involved in the performance of the check itself; the machine is doing the executing, and the machine doesn’t discover or learn anything. We do.

Checks are like the system on your car that monitors certain aspects of your engine and illuminates the “check engine” light. Checks are, in essence, tools by which people become aware of specific conditions in the product. The machine learns no more than the dashboard light learns. It’s the humans, aided by tools that might include checking, who learn.

People learn as they interpret outcomes of the checks and investigate problems that the checks seem to indicate. The machinery has no way of knowing whether a reported “failed” output represent a problem in the product or a problem in the check. Checks don’t know the difference between “changed” and “broken”, between “different” and “fixed”, between “good” and “bad”. The machinery also has no way of knowing whether a reported “passed” output really means that the product is trouble-free; a problem in the check could be masking a problem in the product.

Testing via an API is exploratory testing via an API, because exploratory testing is, simply, testing. (Exploratory) testing not simply “acting like a user”, “testing without tools”, or “a manual testing technique”.

Throughout this whole series, what I’ve been doing is not “manual testing”, and it’s not “automated testing” either. I use tools to get access to the product via the API, but that’s not automated testing. There is no automated testing. Testing is neither manual nor automated. No one talks about “automated” or “manual” experiments. No one talks about “manual” or “automated” research. Testing is done neither by the hands, nor by the machinery, but by minds.

Tools do play an important role in testing. We use tools to extend, enhance, enable, accelerate, and intensify the testing we do. Tools play an important role in programming too, but no one refers to “manual programming”. No one calls compiling “automated programming”. Compiling is something that a machine can do; it is a translation between on set of strings (source code) and another set of strings (machine code). This is not to dismiss the role of the compiler, or of compiler writers; indeed, writing a sophisticated compiler is a job for an advanced programmer.

Programming starts in the complex, imprecise, social world of humans. Designers and programmers repair messy human communication into a form that’s so orderly that a brainless machine can follow the necessary instructions and perform the intended tasks. Throughout the development process, testers explore and experiment with the product to find problems, to help the development team and the business to decide whether the product they’ve got is the product they want. Tools can help, but none these processes cannot be automated.

In the testing work that I’ve described in the previous posts, I haven’t been “testing like a user”. Who uses APIs? It might be tempting to answer “application programs” (it’s an application programming interface, after all), or “machines”. But the real users of an API are human beings. These include the direct users—the various developers who write code to take advantage of what a product offers—and the indirect users—the people who use the products that programmers develop. For sure, some of my testing has been informed by ideas about actions of users of an API. That’s part of testing like a tester.

In several important ways, there’s a lot of opportunity for testability through APIs. Very generally, components and services with APIs tend to be of a smaller scale than entire applications, so studying and understanding them can be much more tractable. An API is deterministic and machine-specific. That means means that certain kinds of risks due to human variability are of lower concern than they might be through a GUI, where all kinds of things can happen at any time.

The API is by definition a programming interface, so it’s natural to use that interface for automated checking. You can use validator checks to detect problems with the syntax of the output, or parallel algorithms to check the semantics of transactions through an API.

Once they’re written, it’s easy to repeat such checks, especially to detect regressions, but be careful. In Rapid Software Testing, regression testing isn’t simply repetition of checks. To us, regression testing means testing focused on risk related to change; “going backwards” (which is what “regress” means; the opposite of “progress”).

A good regression testing strategy is focused on what has changed, how it has changed, and what might be affected. That would involve understanding what has changed; testing the change itself; exploring around that; and a smattering of testing of stuff that should be unaffected by the change (to reveal hidden or misunderstood risk). This applies whether you are testing via the API or not; whether you have a set of automated checks or not; whether you run checks continuously or not.

If you are using automated checks, remember that they can help to detect unanticipated variations from specified results, but they don’t show that everything works, and they don’t show that nothing has broken. Instead, checks verify that output from given functions are consistent from one build to the next. That’s only one kind of oracle, Do not simply confirm that everything is OK; actively search for problems. Explore around. Are all the checks passing? Ask “What else could go wrong?”

Automated checks can take on special relevance when they’re in the form of contract testschecks. The idea here is to solicit sets of inputs and outputs from actual consumers of an API that represent specified, desired queries and results, and to agree to check the contract periodically from both the supplier and consumer ends. Because they’re agreed upon by both parties, contract checks are really very good for first-order functional checking. Nonetheless, remember that such checks are heavily focused on confirmation, and not on discovery of problems and risks that aren’t covered by the contracts. Testing doesn’t stop with the contract checks passing; it starts there.

That is, now that you’ve gone to the trouble of writing code to check for specific outputs, why stop there? I’ve used checks in an exploratory way by:

• varying the input systematically to look for problems related to missing or malformed data, extreme values, messed-up character handling, and other foreseeable data-related bugs;
• varying the input more randomly (“fuzzing” is one instance of this technique), to help discover surprising hidden boundaries or potential security vulnerabilities;
• varying the order and sequences of input, to look for subtle state-related bugs;
• writing routines to stress the product, pumping lots of transactions and lots of data through it, to find performance-related bugs;
• capturing data (like particular values) or metadata (like transaction times) associated with the checks, visualizing it, and analyzing it, to see problems and understand risks in new ways.

A while back, Peter Houghton told me an elegant example of using checking in exploration. Given an API to a component, he produces a simple script that calls the same function thousands of times from a loop and benchmarks the time that the process took. Periodically he re-runs the script and compares the timing to the first run. If he sees a significant change in the timing, he investigates. About half the time, he says, he finds a bug.

So, to sum up: all testing is exploratory. Exploration is aided by tools, and automated checking is an approach to using tools. Investigation of the unknown and discover of new knowledge is of the essence of exploration. We must explore to find bugs. All testing on APIs is exploratory.