A Tester Asks About Checking

In a previous comment, Sunjeet asks

Does not testing encompass checking? Can testing alone be efficient without doing any checking?

As I hope I made it clear in Elements of Testing and Checking, the development and analysis of checks is surrounded by plenty of testing activity, and testing may include a good deal of checking. Testing, I think, can be vastly more efficient if we consider the ways in which checking can be helpful. Cem Kaner, in his 2004 paper The Ongoing Revolution in Software Testing, said this:

I think the recent advances in unit testing (Astels, 2003; Rainsberger, 2004) have been the most exciting progress that I’ve seen in testing in the last 10 years.

With programmer-created and programmer-maintained change detectors (that is, checks -MB):
• There is a near-zero feedback delay between the discovery of a problem and awareness of the programmer who created the problem
• There is a near-zero communication cost between the discover of the problem and the programmer who will either fix the bug or fix the [check]
• The [check] is tightly tied to the underlying coding problem, making troubleshooting much cheaper and easier than system-level testing.

And I agree.

Should testers shun checking? Why not call checking as “confirmative testing“?

There is a role for testers to program and perform checks where cost is low and value is high, but I think that if practices associated with XP really begin to take hold, in the long run it behooves testers to get out of the checking business. That’s because the vast bulk of the checking work will be done by programmers; because checks at the system level tend to be time-consuming, error-prone, and expensive when performed by humans (and expensive to automate when not performed by humans); because they drive humans to inattentional blindness.

But there’s another reason: “confirmative testing” isn’t really testing; it’s confirming. It’s looking at a white swan and saying, “All swans are white;” at another and saying, “See? All swans are white;” and at yet another and saying “Just like I told you; all swans are white.” There’s an analogy in software, “It works on my machine.” “See? It works on my machine.” “Just like I told you; it works on my machine.” To find problems in a product, which is one of the key goals of testing, we need to get out of the confirmatory mindset.

I confirm that the exact problem is fixed – by exactly executing the steps mentioned in the bug report – by this I confirm that the bug and only that bug is fixed or not. Brainless? Yes… a machine could have done the same… yes BUT is this required …YES we might not be deriving new quality value from it but we are CONFIRMING existing quality info from it…

Let me suggest an alternative way of looking at this.

In an environment where the bug report is vague or your programmers are known or suspected of being unreliable with their bug fixes or you know that the programmer has not created an automated check, then what you’re describing might indeed be a very good idea. (When testing the fix, I might start by trying to reproduce the problem as exactly as I could, but I might also start with a slight variation on the problem to see if the general case has been fixed. Either way, I’ll likely end up performing a check to see if the special case of the problem has been fixed.)

Task 2 i do is …i look out for side effects …regression…new test ideas …execute more tests etc i.e. all the pillars of ET… How is task 2 useful without 1?

If you’re following exactly the steps mentioned in the bug report and the programmer has fixed the problem and the programmer has already set up an automated check for the problem, then what you’re doing is reproducing the programmer’s effort (and the machine’s effort) in performing a check, when it might be much more valuable to use your sapient skills and test. Note that I cannot decide either way. I’m not in your context or your immediate situation. But you can. To me, there’s at least one clear circumstance in which it would be greatly more valuable for you to focus on Task 2: When someone has already done Task 1.

2. Should i tell my lead/manager…buddy I am just a tester find a checker to do this! or get a machine to do this? if a machine needs to do this who is going to code/script a machine to do this …wont that be a tester himself?

In answer to the first question, let me ask this: Are you a tester or a checker? Again, I don’t know. The quality of the questions you’re asking suggest to me that you’re a tester; that is, you’re not accepting what I’m saying blindly, nor are you rejecting it out of hand. You’re thinking critically about the idea, even if I may have blown your mind at first.

Either way, I wouldn’t advise telling your lead or your manager that you’re refusing to do work. But thinking in terms of testing vs. checking, if you so choose, might trigger a productive conversation between you about the relative cost and value of the activities that you (and the programmers) are doing. It might indeed make more sense to get a machine to do the checking work—and for the programmers to insert some change detectors and take greater responsibility for the quality of their code, an idea that was one of the triggers for Extreme Programming and the Agile movement.

As for who does the programming, note the passage from the very posting upon which you commented:

“When someone asks, ‘Can’t we hire pretty much any programmer to write our test automation code?’, we can point out that the quality of checking is conditioned largely by the quality of the testing work that surrounds it, and emphasize that creating excellent checks requires excellent testing skill, in addition to programming skill.”

Thank you for your comments and questions.

See more on testing vs. checking.

Related: James Bach on Sapience and Blowing People’s Minds