Agile Testing

A correspondent on the Agile Testing mailing list asks a series of questions.

My answer is personal, and may or may not be shared by people on the list. I don’t want to get into a huge debate in that forum, and since the question comes up a lot anyway, I thought I would answer it and invite comment here.

1) Can we use Agile testing for Non Agile projects?

I have three ways in which I would answer the question. They depend strongly upon what “agile projects” means to you.

a) In an agile environment, there is a strong focus on unit testing through automated test frameworks, of which JUnit and its relatives are examples. Many agile environments use test-driven design, in which the developers write tests that are designed to fail until code has been written to pass those tests successfully. These tests, written by the developers, serve at least two highly important purposes. First, they help to drive the design of the product by providing a set of conditions that the unit-level code must fulfill to be considered complete. Second, they serve as powerful change detectors, so that the product can be refactored and retested with confidence and ease; after each modification to the code, programmers run all of the unit tests, and the unit tests have to pass before further work is done. In my observation and experience, this is an extremely powerful and robust method for developing software, and it doesn’t require an agile project to use it.

(In fact, expert developers have been following practices like these for a long time. In the early days of computing, there was a heavy focus on unit testing, at least in part because computer time was expensive; computers and development environments weren’t anything like as rapid and interactive as they are now; and also because the hardware wasn’t sufficiently reliable to trust without unit tests. Somewhere along the line these practices got lost, and the agile development community has done a lot to reinvent and restore them. Bravo.)

b) In an agile environment, there is a strong focus on developing user- or business-oriented tests up front; these are often called acceptance tests, and they’re typically written by customers or their representatives—testers or business analysts. In the same way that unit tests are created for the functional units within it, acceptance tests are oriented towards integrated or system-level views of the product. I observe that there are two slightly different nuances. Sometimes user tests are designed as examples for the way in which the product should behave (that is, they are example-focused). Sometimes, user tests used as benchmarks by which we might assess that a given feature or set of features is complete (that is, they are milestone-focused).

Often—but not always—these tests are both example- and milestone-focused. In an agile environment, typically these tests are automated. Agile teams often use frameworks such as FIT or FitNesse, in which descriptions of the intended behaviour of the product are intermingled with tables that give examples of intended functionality; developers code “fixtures” to allow the tables to interact with code from the product; and the framework then allows the user to exercise the code in question and to observe actual results. These tests tend to be very similar in nature to use cases, but are often more detailed and elaborate thanks to the tables and the fact that the code is runnable.

Agile teams might also use automation tools such as WATIR, which drives Web applications using (as of this writing, Internet Explorer and) the Ruby programming language. As with unit tests, when new functionality is implemented, these tests are run to demonstrate functionality; and when code is modified, these tests are re-run in order ensure that that which was working previously is still working.

I observe several potential problems with implementing this kind of testing. Outside of an agile context, these tests are difficult to implement when automation is part of the picture and developers are not inclined or mandated to colloborate with the testing process; FIT and FitNesse in particular require lots of help from the developers. Second, even in an agile context, these tests require rather more involvement from customers or customer representatives, and may depend on these people learning to use the tool when they don’t have the time or the inclination to do so. Third, these tests tend to be more confirmatory and less investigative; they tend to be focussed on demonstrating that the product works, rather than checking to see whether the product fails.

These processes can be implemented in a non-agile projects, and the risks above can be addressed, but it’s important to be aware of them.

c) There is another way of thinking about testing in a agile way (note the reframe and the absence of the capital “A”). I believe that can be done with agility in any project where the skills of the tester and the context of the project support it. Such testing would, in my view, include

• lots of quick tests that have very low cost but uncertain value
• extremely rapid feedback to the developers, but also to the rest of the project community minimal planning up front, in deference to learning more and focusing the practice as the project goes on and we learn more about it
• generally preferring test ideas over test scripts
• generally focusing on investigation vs. confirmation
• lightweight approaches to test documentation
• with respect to automation, tests that can be rapidly and easily developed and evaluated by a machine are automated; tests that have a high dependence on human cognition and evaluation may be executed and observed manually—and automation is used to assist that effort; and testers continuously develop and use automation skill to the extent that it helps themselves and the project.

Testers develop and use skills of critical thinking, general systems thinking, and context-driven thinking to identify risk, to create powerful tests, and to provide valuable information to management so that it can make informed decisions about the product and the project.

In summary, consider what testing might be like if it followed the Agile Manifesto, even if the rest of the project did not. This is the kind of testing that I teach and that I try to practice, and its most ardent and articulate exponent is James Bach. If you’re interested, see his Web site at http://www.satisfice.com. We call it Rapid Testing, but there is no trademark on the name, no copyright on the practices, and no patents on the skills; they’re mostly things that we’ve collected, developed, thought about, and written about that expert testers have been doing forever.

2) Are there any key changes to the normal testing process? If so can you please brief me those.

If you can tell me what you consider “normal testing”, I can identify some of the differences to the extent that I have not done so above.

3) What are the key success factors for agile testing?

In (a) above, I’d assert that the primary success factors include motivated developers who learn the approaches and follow them diligently and skillfully. These factors are somewhat organizational and highly personal.

In (b) above, I would consider as key the recognition and mitigation of the risks that I mentioned. These factors are somewhat personal but also organizational.

In (c) above, testers need to have independence—that is, some time and some freedom to develop new tests and new test ideas—, and I would also contend that testers must continuously focus on developing skill. That is, the success factors in (c) are somewhat organizational but mostly personal.

There were other questions:

4) How do we estimate Test Efforts for agile Testing? Can we use normal estimation models?

5) What is the typical % of testing effort in Software development Life cycle by using agile methodology?

6)Is there any difference in testing effort for Normal testing process and Agile testing process?

7) If the documentation is less in Agile testing, will it not impact the quality? I hope this will overcome by having daily meetings to update the status/issues.

I’ll answer these in subsequent posts.