Interviewing the Program

Testing, in the quick definition that James Bach and I use, is questioning a product in order to evaluate it. One way of questioning the product is to ask ordinary questions about it, and then to operate it—supplying it with input of some kind, and operating it in some way. The product “answers” us by producing output or otherwise exhibiting behaviour, which we observe and evaluate. Another way of questioning a product is to ask questions about some dimension of it. If we want to do excellent testing, it helps to remember that our product is not merely a program consisting of code that our team wrote. The product is a system. That system includes the program, the platforms upon which the program depends, the person using the program, and the task or business process that the program enables. The system also includes everything that we know about the system.

In the agile community, there appears to be a strong focus on writing tests, often before the code is written. For programmers, especially those using Test-Driven Development—a design activity and a programming activity—this makes a good deal of sense. The TDD experts suggest writing a test, and the writing the least amount of code possible to make the test pass. If the test doesn’t pass, refine the code until the test passes (along with all the previous tests), and then repeat the process. Add tests and code as you develop an understanding of what needs to be done next. That’s a fundamentally exploratory approach to writing code, learning things and applying that learning as you go.

For testing,though, preparing too many questions too far in advance is a risky business. Here are a few reasons why that’s so.

First, the programmers and the rest of the development team will learn things as the program is being developed. This will likely cause some stories or requirements to be reinterpreted, reframed, or rejected. Other requirements will be discovered, as we realize that our initial understanding was incomplete (and I’ll argue that it is almost always incomplete).

Another problem: when we’re preparing specific tests for something in the future, we’re not testing what’s available to us right now, and that testing may be tremendously important. It may not be code that we have to test; it may be requirements or specifications or designs or prototypes. But, in general, questioning—that is, testing—something that we have now tends to be more productive than questioning something that might be, some day.

Maybe the most significant problem is a mindset that I see in talk about agile development: “we write the acceptance tests first, and when the program passes those acceptance tests, we know we’re done.” This is a dangerous idea, especially for programs with which humans interact. When we say that a program that passes its acceptance tests will be acceptable, we presume implicitly that we know, in advance, all the questions that we will eventually want to ask of the program. We presume implicitly that the program needs only to answer those questions in order to be considered acceptable. Yet our purpose in testing should not be merely to confirm that the program can pass some tests. That a program can work isn’t a such big deal; our clients would like to know that the program will work. Our purpose, therefore, should also include identifying problems that threaten the value of the program. It could also include discovering things about the way that people might use the program, such that we can provide new features, modify existing ones, or propose workarounds for problems that the program cannot yet solve. These are not confirmatory processes, but investigative ones. So in addition to (and maybe instead of) the suite of confirmatory tests, we need to have real humans interacting with and exploring the program.

Think of it in terms of being technical recruiter, inside a bank, qualifying a candidate for a programming job. In consultation with the hiring manager, you determine that the ideal candidate requires experience with C++ and Java, has worked in the financial industry, and speaks English. The candidate should also have knowledge of advanced data mining techniques, statistical programming experience, and excellent references. You search through your database of applicants, and you find someone who indeed meets all of these requirements. Her résumé shows that she’s got the languages and experience, and her references check out. That is, the candidate already passes all of your acceptance tests. What happens next? Are you going to hire that person right away?

Or will you interview her? Get a team of people, with different priorities and different points of view, to interview her? Ask to see a portfolio? Put her through an audition process, in which she demonstrates some knowledge and some skill in applying it? Do you ask unscripted follow-up questions, based on the answers you get, the potential problems that you observe, and the concerns that you might have? Do you vary your suite of questions from candidate to candidate? Are your questions designed to confirm what your already know about the current candidate, or do you intend to probe—that is, test—the candidate’s qualifications?

Excellent software testing is like an excellent qualification and hiring process. When we seek to fill an important position, we don’t go by a mere checklist of attributes and hire the first person that can check each box. We don’t typically want someone who can work exclusively on one problem in one context. In most organizations, I would argue, we’re mostly looking for someone who has the skills and the maturity to adapt reasonably and responsibly to new situations and new problems. Our interview processes should reflect that. Similarly, when we test a piece of software, we’re not looking for something that can work exclusively for one person, on one platform, with one data set, in one context. Instead, a big part of our job is to show that the software will work for many people, using many platforms, dealing appropriately with whatever data we throw at it, in a multitude of contexts. Therefore our tests should not reflect merely our notion of what the program was going to be. Our tests should reflect everything that we know about the system so far, up to the result of the last test, up to this moment; what the program has become as we’ve developed it, and as we’ve learned.

So when you’re developing a product, are you going to fall for the narcotic comfort of the green bar—or are you going to test?