Acceptance Tests: Let’s Change the Title, Too

Gojko Adzic recently wrote a blog post called Let’s Change the Tune on some of our approaches in agile development. In changing the tune, some of the current words won’t fit so well, so he proposes (for example), “Specifying Collaboratively instead of test first or writing acceptance tests”. I have one more: I think we should change the label “acceptance tests”.

Acceptance tests are given a central role in agile development. They are typically used to express a requirement in terms of an atomic example, and they’re typically automated. That is, they’re expressed in the form of program code for a binary computer, code that helps us to determine whether some aspect of the product is functionally correct. When those tests pass, say certain proponents of the lore, we know we’re done.

Yet acceptability of a product is multi-dimensional. In the end, the product is always being used by people to solve some problem. The code may perform certain functions exquisitely as part of product that is an incomplete solution to the problem, that is hard to use, or that we hate.

The expression of requirements and the determination of acceptability in terms of simplistic, binary decisions delegated to a computer seems to me like a bias towards processes and tools, rather than individuals and interactions. That is: it’s inconsistent with the very first point in the Manifesto for Agile Software Development.

Typically, acceptance tests are set very close to the beginning of a cycle of development. Yet during that development cycle, we tend to learn a significant amount about the scope of the problem to be solved, about technology, about risk, about trade-offs.

If the acceptance tests remain static, the learning isn’t reflected in the acceptance tests. That seems to me like a bias towards following a plan, rather than responding to change—another inconsistency with the Agile Manifesto.

Acceptance tests are examples of how the product should behave. Those tests are typically performed in very constrained, staged, artificial environments that are shadows of the envionments in which the product will be used.

Acceptance tests—examples to be checked—are not really tests, in the sense of testing the mettle of the product, subjecting it to the challenges and stresses of real-world use. Yet acceptance tests are often treated more as authoritative, definitive, specifications for the product, instead of representative examples. That sounds to me like a bias towards comprehensive documentation, rather than working software—yet again inconsistent with the Agile Manifesto.

Acceptance tests are often discussed as though they determined the completion of development. While the acceptance tests aren’t passing, we know we’re not done; when the acceptance tests pass, we’re done and, implicitly, the customer is obliged to accept the product as it is. That sounds to me like a bias towards negotiated contracts, rather the customer collaboration. That’s inconsistent with the fourth point in the four-point Agile Manifesto.

The idea that we’re done when the acceptance tests pass is a myth. As a tester, I can assure you that a suite of passing acceptance tests doesn’t mean that the product is acceptable to the customer, nor does it mean that the customer should accept it. It means that the product is ready for serious exploration, discovery, investigation, and learning—that is, for testing—so that we can find problems that we didn’t anticipate with those tests but that would nonetheless destroy value in the product.

When the acceptance tests pass, the product might be acceptable. When the acceptance tests fail, we know for sure that the product isn’t acceptable. Thus I’d argue that instead of talking about acceptance tests, we should be talking about them as rejection tests.

Post-script: Yes, I’d call them rejection checks. But you don’t have to.

7 replies to “Acceptance Tests: Let’s Change the Title, Too”

  1. Very good thoughts and well put!
    Firstly, I have always felt skeptic against acceptance tests written by those that aren’t accepting the product.
    Secondly, many automated acceptance tests I have seen are designed not to break just because a need to avoid problems when the software changes a lot. I.e., they are adjusted to be less powerful and less complex in order to fulfill the need of automation.

  2. Perhaps the term should be changed to ‘Acceptance to Test’ checks?

    I mean that as should these ‘pass’, testing can then begin.

    Sort of like an entry-level criteria to be met before more in-depth testing?

  3. I think there is a deeper issue, which is not about test. If you have ‘requirements’ or ‘specifications’, then you *can* have ‘acceptance tests’.

    On the other hand, if you view stories as an alternative to ‘requirements’, and stories are ‘a reminder to have a conversation’ (Mike Cohn), you cannot have acceptance tests.

    Unfortunately, and surprisingly, many in agile keep using the words ‘requirements’ and ‘specifications’.

    See my related post:

    The point – it’s not about tests, it’s about ‘requirements’.

  4. […] A finomhangolás eredménye egy id?ben specifikáció, „a készen van” definíciója, elfogadási teszt és egy kés?bb futtatandó funkcionális regressziós teszt. Én igazából nem akarom ezt elfogadási tesztnek nevezni, mert nagyon nehéz megindokolni azt, hogy miért szaknyelven írjunk, de mégis érthet?en és könnyen elérhet?en. Van egy teljesen más vita is arról, hogy az ellen?rzéseket automatikusan elfogadja a szoftver, vagy automatikusan visszadobja azokat a kódokat, amik nem felelnek meg az elvárásainknak. (😉 […]


Leave a Comment