The year was 2006. James Bach and I were running a workshop at the Amplifying Your Effectiveness conference (AYE). We were in one of those large-ish, high-ceiling conference rooms with about 15 programmers and software consultants.
We were showing them one of James Lyndsay’s wonderful testing machines. (You can find it here, but you’ll need Flash active to run it.) It looked like this:
At first, it’s all very confusing. When you press the buttons on the left, the red and blue balls on the right move in some way. The slider in the middle influences the range of motion somehow. In general, the mission of the exercise is to describe the behaviour of the machine.
Test cases are not testing. To illustrate this important fact, we give class participants the machines to investigate for a few moments, and then ask a question that James (Bach) asked in our AYE session.
“How many test cases,” he asked, “would you need to be able to understand and describe this product completely?”
Brows immediately furrowed. Clicking sounds from the buttons and murmured conversation between pairs of participants filled the room. “Two states to the power of five buttons with how many stops on that slider…?” “Wait, that button is just momentary…” “Seven hundred and sixt… no, that’s wrong.”
Whereupon, in a moment of perfect timing, a door opened, and Jerry Weinberg walked into the room. His walking stick and his bearing reminded me of Yoda and Gandalf and other sages and wizards.
“Hey, here’s Jerry Weinberg!” said James. “The world’s greatest living software tester! Jerry, how many test cases would you need to understand and describe this product completely?”
The room fell silent. Everyone wanted to know the answer. Jerry observed the laptop that James was holding. He didn’t touch the laptop, press a key, or move the mouse. He just looked for a few moments.
Then he said, “Three.” There was a pause.
Having worked with Jerry over a decade or so, James understood. “Three, Jerry?”, he asked dramatically, in mock astonishment.
“Hm.” (pause) “Yeah. Three,” replied Jerry. Another pause.
Then he peered at James. “Why? Were you expecting some other number?”
I will bet that if pressed, Jerry could have named the three cases he had in mind.
And what he was up to was in his reply (also in the “session based” “rapid” and related testing approaches.)
Clever man.
“Three” meaning himself, You and James? 🙂
Michael replies: Maybe. Were you expecting some other explanation?
[…] A Moment of Jerry Weinberg Zen Written by: Michael Bolton […]
Fun story. Your live portrayal was hilarious. Happy New Years.
Nope of course 🙂
“Having worked with Jerry over a decade or so, James understood.”
I’m not sure that I understand but based on my level of familiarity with your writing on the subject of software testing, I’m led to believe that Mr. Weinberg’s answer is to be taken as a mockery of the question itself, correct?
Michael replies: I’m not sure I’d say “mockery”, although the answer affords that interpretation.
After all, he did not interact with the puzzle at all prior to his answer. You mentioned, “When you press the buttons on the left, the red and blue balls on the right move in some way. The slider in the middle influences the range of motion somehow.”, but did he know these requirements to be true ahead of time? How are we defining “test case”? Does he have the same definition? And how is it that he could know that he would be able to understand and describe this product _completely_ in 3 test cases without any exploration and experimentation ahead of time? Why would the greatest _tester_ of all time come to an answer without actually doing any _testing_?
We can’t ask him any more, sad to say, but I have some pretty well founded beliefs about how Jerry might address your questions. I’ll try to channel him by various means.
Jerry didn’t worry about whether requirements or specifications were “true” or “correct”. What concerned him was whether the product the customer had was the product the customer wanted.
Jerry also didn’t think very highly of counting test cases. One of the earliest things he ever published was Computer Programming Fundamentals, written with Herbert Leeds. Jerry wrote the chapter on testing. In it, he tells the story of a programmer who, on several occasions, believed he had found the last bug in his program. Eventually, after each repair, another deeper and more damaging bug was discovered. As the epilogue to the story, Jerry says:
One of the lessons to be learned … is that the sheer number of tests performed is of little significance in itself. Too often, the series of tests simply proves how good the computer is at doing the same things with different numbers. As in many instances, we are probably misled here by our experiences with people, whose inherent reliability on repetitive work is at best variable. With a computer program, however, the greater problem is to prove adaptability, something which is not trivial in human functions either. Consequently we must be sure that each test does some work not done by previous tests. To do this, we must struggle to develop a suspicious nature as well as a lively imagination.
Jerry would acknowledge, too, that complete testing of a product (or a complete understanding and description of it) is not possible. To test that little puzzle completely would require us to put it into all of its possible states, from all possible states, at all possible hours of the day on all possible days, using all possible means of pressing the buttons and moving the sliders, including not only mouse and keyboard but also the control offered by screen-readers for blind folks, on every possible display, using every possible version of the Flash library, considering every possible user of the product at all ages and skill levels… Complete testing—in the sense of covering every variation in the system—is not available. And that’s okay. No one wants complete testing, and even if they did, they wouldn’t want to fund it.
Now, was Jerry’s reply mockery? That’s pretty strong; “gentle teasing”, perhaps. Since it was Jerry, instead of “mockery”, I’d say “test”; his answer is a test of James’ question.
There is more to testing than performing tests. Jerry was doing testing, by his lights (and by mine, too): gathering information with the intention of informing a decision. Testing also means challenging. Jerry was challenging James (or, rather, the character that James was playing) to consider the basis of the “how many test cases” question.
This was an instance of testing that all too few testers do, alas: probing the mission, questioning premises, prompting the questioner him- or herself to examine those premises. That, to me, is part of serious testing. To be sure, Jerry’s answer is of a different nature than interacting with the product, fingers on keyboards, but it is every bit as much testing activity as performing exploration and experiments on the product. Testing of the kind that Jerry was doing here can save a ton of time and effort doing testing of that other kind.
That’s the whole point of this cliff-hanger of a post, correct?
Jerry would want to honour whatever anyone might learn from an incident or story. Me too. One point in telling the story in the post is to prompt questions to reflect upon: what did Jerry mean? Why did he give the answer that he did? What questions do you have about the story now, or later? What questions could you apply to your own situations?
It seems like it worked, to some degree. Thanks for writing.
“…mockery of the question…”
It’s Michael’s blog, but I’ll chime in and he can correct.
Not “mockery” as much as illuminating the question’s deficiencies. This is the point of that kind of training exercise — I’ve seen Michael’s and James’ work — and Jerry’s answer. Deficient questions are a thing.
The hint is here, I think: “Then he peered at James. “Why? Were you expecting some other number?”” What were you *expecting*?
Information about what something does is pretty useless unless you define what you want, intend, and expect from the thing.
With his reply, Jerry actually made a little morality play with himself as the SUT, and James as the tester, illustrating exactly what the exercise in progress was illustrating: “Were you expecting some other number?” Nobody knows what to expect from the box, so any answer to “How much to test?” comes from somewhere other than what the box does.
I suspect Jerry could name 3 test cases in that situation that were about the testing system itself, which would likewise illustrate the point.
This is similar to the notion of “foretelling” from LeGuinn’s The Left Hand of Darkness, an art practiced to prove the “…perfect uselessness of knowing the answer to the wrong question”.
Any particular test case, number of test cases, testing stragegy, testing activity, or testing program, may produce robust answer (if well done) to the wrong question, thus be perfectly useless.
Michael replies: Thank you Jim. It’s an honour to have you reply to a reply on my blog — and what a wonderful reply you’ve given.
Michael: Thank you Jim.
Well thank you, kind host, and also for making a place where I can chime in sometimes, even though I’m too cranky and broken to wrangle my own web presence.
Fascinating. Thanks to both of you, Michael and Jim.