Acceptance Tests: Let’s Change the Title, Too

Gojko Adzic recently wrote a blog post called Let’s Change the Tune on some of our approaches in agile development. In changing the tune, some of the current words won’t fit so well, so he proposes (for example), “Specifying Collaboratively instead of test first or writing acceptance tests”. I have one more: I think we should change the label “acceptance tests”.

“Acceptance tests” are given a central role in agile development. They are typically used to express a requirement in terms of an atomic example, and they’re typically expresses as automated output checks. That is, they’re expressed in the form of program code for a binary computer, code that helps us to determine whether some aspect of the product is functionally correct. When those checks produce happy results, say certain proponents of the lore, we know we’re done.

Yet acceptability of a product is multi-dimensional. In the end, the product is always being used by people to solve some problem. The code may perform certain functions exquisitely as part of product that is an incomplete solution to the problem, that is hard to use, or that we hate.

The expression of requirements and the determination of acceptability in terms of simplistic, binary decisions delegated to a computer seems to me like a bias towards processes and tools, rather than individuals and interactions. That is: it’s inconsistent with the very first point in the Manifesto for Agile Software Development.

Typically, acceptance checks are set very close to the beginning of a cycle of development. Yet during that development cycle, we tend to learn a significant amount about the scope of the problem to be solved, about technology, about risk, about trade-offs.

If the acceptance checks remain static, the learning isn’t reflected in the acceptance checks. That seems to me like a bias towards following a plan, rather than responding to change—another inconsistency with the Agile Manifesto.

Acceptance checks are examples of how the product should behave. Those checks are typically performed in very constrained, staged, artificial environments that are shadows of the envionments in which the product will be used.

“Acceptance tests”—examples to be checked—are not really tests, in the sense of testing the mettle of the product, subjecting it to the challenges and stresses of real-world use. Yet acceptance checks are often treated more as authoritative, definitive, specifications for the product, instead of representative examples. That sounds to me like a bias towards comprehensive documentation, rather than working software—yet again inconsistent with the Agile Manifesto.

Acceptance checks are often discussed as though they determined the completion of development. While the acceptance checks aren’t passing, we know we’re not done; when the acceptance checks pass, we’re done and, implicitly, the customer is obliged to accept the product as it is. That sounds to me like a bias towards negotiated contracts, rather the customer collaboration. That’s inconsistent with the fourth point in the four-point Agile Manifesto.

The idea that we’re done development when the acceptance checks pass is a myth. We might be done the first round of development — creating a product that can work. As a tester, I can assure you that a suite of passing output checks doesn’t mean that the product is acceptable to the customer, nor does it mean that the customer should accept it. It means that the product is ready for serious exploration, discovery, investigation, and learning—that is, for testing, deep testing—so that we can find problems that we didn’t anticipate with those checks but that would nonetheless destroy value in the product.

When the acceptance tests pass, the product might be acceptable. When the acceptance tests fail, we know for sure that the product isn’t acceptable. Thus I’d argue that instead of talking about acceptance tests, we should be talking about them as rejection tests.

Post-script: Yes, I’d call them rejection checks. But you don’t have to.

This post was lightly revised 2024-08-08.