Why Checking Is Not Enough

Here is a specific, real-world example of testing where the focus doesn’t include explicit checking, and does not result in yes-or-no answers to predetermined questions.

This morning, I acted on a piece of email I received several days ago, offering a free upgrade to a PDF conversion package which I’ll call “PDFThing”. I’ll walk you through what happened, and parts of my thought process as it happened.

Since the email is addressed to me, and since it notes that I had purchased upgrade insurance, I presume that the company has all of the data needed to know which product was associated with that email address.

The mail message includes this text: “It’ll only take a few minutes, but we’ll need your serial number (also known as a license key) to deliver your upgade. [How do I find my serial number?]” (The text in square brackets was a link.)

I note that “upgrade” is misspelled. Spelling can be checked, and a spelling checker would have found that problem, but checks can’t guarantee correct spelling. Do you doubt this? What if I had said that Czechs can’t guarantee correct spelling? Or that cheques can’t guarantee correct spelling? You would have noticed right away, but a spelling checker would not have.

I see a more serious potential problem, though. If the company has data about me, why not provide helpful serial number information directly and immediately? Options include “Your serial number is…” or if they’re worried about someone intercepting the mail, “The serial number can be found in your version of PDFThing by…”, in a way specific to the version that is associated with that email address.

I click on the link. It takes me to an FAQ page that has a list of questions. Conveniently, the question titled “Where do I find my serial number for PDFThing?” already shows an answer:

“It depends on what version of PDFThing you have and where you bought it. If you bought PDFThing 7 from our website, your serial number (in alpha-numeric characters) can be found under the Help tab in the About section.”

I have PDFThing 6, though. And I purchased it at a store. So I apply an oracle: consistency with an implicit purpose. An implicit purpose of this answer is to convey information to users of *any* version of PDFThing, purchased *anywhere*. The answer doesn’t do that. Is this a problem? I don’t know if the product owner will consider this a problem at all, or a problem worth fixing, so I can’t provide a yes-or-no answer. What I can do as a tester is to note a possible problem, and move on.

I decide to open my existing version PDFThing, and I apply another set of consistency heuristics: consistency with history; consistency within the product; consistency with an implicit claim; consistency within the product. Maybe the serial number for version 6 is located in the same place as the serial number for version 7. I click on Help and then About. I find that the serial number is not in the place referred to by the FAQ text, so with respect to the product I own, that text is misleading and incorrect. Plus I apply the “consistency with comparable product” heuristic; many products put the serial number in the Help/About box. All in all, this looks more like a problem. Will the product owner see it that way?

I dimly remember that I received a copy of the serial number in the e-mail that I got when I first registered the product. I go on a hunt for that e-mail. It takes me a few minutes to find it, but eventually I do. I copy the serial number to the clipboard and I returned to the original e-mail and click on upgrade to download the product. My own impatience and exasperation suggests to me that there’s a problem here. Note that although you can test for an emotional reaction, you can’t check for it. At best, you can anticipate things like delays of a certain duration as the program is executing. Measurement theorists call that a surrogate measure—using one kind of measureument as a substitute or a stand-in for the thing that we’d actually like to measure.

When the product finishes downloading, I begin the installation process. I’m prompted for the serial number for my older version, which I provide. The installer accepts the serial number and prompts me for a directory into which the new version of PDFThing should be installed. I notice that the product is being installed into a folder that is specific to version 7. I suspect that the prior version of PDFThing is not being uninstalled, so rather than accepting the default directory, I browse upward. I find that indeed the new product is not replacing the old product. Problem or no problem? For a check to determine this, the decision rule would have to have been decided and programmed in advance.

I go to Add/Remove programs, and begin the uninstallation process for PDFThing 6. The uninstaller posts a dialog saying, “The following applications should be closed before continuing the install”, and in the window beneath, I see a reference to the title of an email message I’m drafting in Outlook. That makes sense; PDFThing 6 installs a toolbar in Outlook so that I can print PDFs directly from the program. Still, there’s no reference to Outlook itself, though. So is that the message that the designer of the program wants me to see?

I close the offending message window and save the message as a draft. I return to the uninstallation dialog, press Retry, and the uninstallation proceeds. It appears to make some progress. I switch to another window and continue working while uninstallation continues in background. After a brief interval, it posts the same dialog as before, but this time tells me that Microsoft Word should be closed before continuing the install. Is this the behaviour that the designer wanted?

I now wonder what would have happened had I not chosen to uninstall PDFThing 6. Would Outlook and Word have acquired a second set of toolbars for PDFThing 7? Would they be separate? Would the new one have replaced the old one? I could have perhaps have programmed checks for that, but would it have been worthwhile to do that? Wouldn’t eyeballing it be cheaper and faster? Maybe not; maybe there are bunches of registry entries and files and configuration settings and stuff connected to Outlook (and Word, and Explorer, and PowerPoint, and Excel) such that we’d really need a program to help us probe that. Would a check have wondered and raised that issue to programmers or designers?

The installation process continues. In the middle, a browser window appears, asking me why I’m uninstalling PDFThing 6. The options are “I don’t want to purchase or continue with the trial”; “I purchased the product and am uninstalling the trial”; “I’m upgrading to the latest version”; “I’m about to move my PDFThing 6 license to another computer.” It seems to me that the third option would be unnecessary if PDFThing 7’s installation program automatically removed PDFThing 6. So is this the uninstallation process that the product owner wants?

I answer the question (I’m upgrading), and the Web page offers a thank you for answering the question. In the interim, the uninstallation process seems to have terminated. Was it successful? I don’t know. Did the designers intend that uninstallation should end immediately? And what if I hadn’t had an active Internet connection; what would have happened then? Would checks raise these questions? Perhaps the development of checks might have, but the checks themselves would not have.

I return to the installer for PDFThing 7, and start it up again. Oddly, I’m not asked for my serial number this time. Has the product retained it from the last attempt? I don’t know. How would I find out?

The installation process carries on for a while, and at the end, I’m presented with a dialog that asks whether I want to buy the product or activate it. I choose the latter; I’ve already bought it. The activation window asks for the serial number. I provide it, and immediately I’m presented with this error dialog (which I haven’t altered):

Note that the dialog is titled “Information”; the name of the product isn’t provided. Look as well at the formatting of the message; it looks sloppy and unprofessional to me. Oh, and dammit, it IS the right serial number (it was accepted last time). Is this what the product owner wants?

I dismiss the dialog, and the activation dialog has a moving graphic indicating that the product is waiting from something. Otherwise, the product seems hung. Just in case, though, I click on the Activate button again. The “Information” dialog above appears again. There’s no choice apparent except to dismiss the two dialogs and get on the line to technical support, whereupon the costs will really begin to rack up.

Now you could say that, if I were a tester working for the PDFThing people, I should have received all of this information before beginning test execution, whereupon I should have prepared checks to be applied against the product. It’s a fine idea. But even when we’re working on the best imaginable teams in the best-managed projects, as soon as we begin to test test, we begin immediately to discover things that no one—neither testers, designers, programmers, nor product owner—had anticipated or considered before testing revealed them. It’s simply fatuous to suggest that everyone involved in the development of the product knows exactly what they will want or need from the outset. It’s even more fatuous to suggest that they should know such a thing. Software development is not simply construction according to prescribed plans. It is development. Like testing itself, it is a process of exploration, discovery, investigation, and learning.

It’s important not to confuse checks with oracles. An oracle is a principle or mechanism by which we recognize a problem. A check is a mechanism, an observation linked to a decision rule. That rule is based on a single application of a single principle. A check provides a signal, a bit, when the product’s behaviour or state is inconsistent with that principle. A check follows a rule; it does not apply a heuristic. Testing, which may include many checks, is not so restricted. Testing may produce a yes-or-no answer, but it may also produce an observation, a question, a concern, a dilemma, a new test idea, or a new check idea. Testing is not governed by rules; it is governed by heuristics that, to be applied appropriately, require sapient awareness and judgement.

Checking is an approach to making sure that we get the right answers, for questions and desired answers that we’ve already determined in advance. A passing check doesn’t tell us that the product is acceptable. At best, a check that doesn’t pass suggests that there is a problem in the product that might make it unacceptable.

Testing incorporates checking, but is a far richer set of activities: exposing ourselves to the unexpected, making new observations, spotting unanticipated problems, and raising new questions. Yet not even testing is about telling people that the product is acceptable. On the one hand, testing may have a different purpose. Cem Kaner, in the BBST course, lists

• Finding defects
• Maximizing bug count
• Blocking premature product releases
• Helping managers make ship / no-ship decisions
• Minimizing technical support costs
• Assessing conformance to specification
• Assessing conformance to regulations
• Minimizing safety-related lawsuit risk
• Finding safe scenarios for use of the product (workarounds that make the product potentially tolerable, in spite of the bugs)
• Assessing quality
• Verifying the correctness of the product

To which I would add

• assessing compatibility with other products or systems
• assessing readiness for internal deployment
• ensuring that that which used to work still works
• design-oriented testing, such as review or test-driven development
• understanding the workings of a poorly-documented product or library
• evaluating the usefulness of a new tool or service
• refining notions of risk

On the other hand, much of the time, we’re testing to help determine whether a product is acceptable for release. But decisions about acceptability are in the hands of managers, programmers, designers; those who build the product (and ultimately, acceptability is the decision of the product owner). Testing is about investigating the product to reveal knowledge that informs the acceptability decision. Sometimes that information comes in the form of binary answers to known questions; checks. Sometimes that information comes in the form of discoveries that pose new ideas, new risks, and new questions for those who are responsible for building and releasing the product.