Testing, Checking, and Convincing the Boss to Explore

How is it useful to make the distinction between testing and checking? One colleague (let’s call him Andrew) recently found it very useful indeed. I’ve been asked not to reveal his real name or his company, but he has very generously permitted me to tell this story.

He works for a large, globally distributed company, which produces goods and services in a sector not always known for its nimbleness. He’s been a test manager with the company for about 10 years. He’s had a number of senior managers who have allowed him and his team to take an exploratory approach, almost a skunkworks inside the larger organization. Rather than depending on process manuals and paperwork, he manages by direct interaction and conversation. He hires bright people, trains them, and grants them a fairly high degree of autonomy, balanced by frequent check-ins.

Recently, on a Thursday the relatively new CEO came to town and held an all-hands meeting for Andrew’s division. Andrew was impressed; the CEO seemed genuinely interested in cutting bureaucracy and making the organization more flexible, adaptable, and responsive to change. After the CEO’s remarks, there was a question-and-answer period. Andrew asked if the company would be doing anything to make testing more effective and more efficient. The CEO seemed curious about that, and jotted down a note on a piece of paper. Andrew was given the mandate of following up with the VP responsible for that area.

Late that afternoon, Andrew called me. We chatted for a while on the phone. He hadn’t read my series on testing vs. checking, but he seemed intrigued. I suggested that he read it, and that we get together and talk about it.

As luck would have it, there was occasion to bring a few more people into the picture. That weekend, we had a timely conversation with Fiona Charles who reminded us to focus the issue of risk. Rob Sabourin, happened to be visiting on Saturday evening, so he, Andrew, and I sat down to compose a letter to the VP. Aside from changing the names that would identify the parties involved, this is an unedited version what we came up with:

[Our Letter]

Dear [Madam VP]…

[Mr. CEO] asked me to send you this email as a follow up to a question that I posed during his recent trip to the [OurTown] office on [SomeDate] on the current state of the testing at [OurCompany] and how our testing effectiveness should be improved.

The [OurTown]-based [OurDivision] test team has been very successful in finding serious issues with our products with a fairly small test team using exploratory test approaches. As an example, a couple of weeks ago one of my testers found a critical error in an emergency fix within his two days of exploratory testing in a load that had passed four person-weeks of regression testing (scripted checking) by another team.

Last week a Project Lead called me and asked if my team could perform a regression sweep on a third party delivery. I replied that we could provide the requested coverage with two person-days of effort without disrupting our other commitments. He seemed surprised and delighted. He had come to us because [OurCompany]’s typical approach yielded a four-to-six person-week effort which would have caused a delay in the project’s subsequent release.

Our experience using exploratory testing in [OurDivision] has demonstrated improved flexibility and adaptability to respond to rapid changes in priorities.

Testing is not checking. Checking is a process of confirmation, validation, and verification in which we are comparing an output to a predicted and expected result. Testing is something more than that. Testing is a process of exploration, discovery, investigation, learning, and analysis with the goal of gathering information about risks, vulnerabilities, and threats to the value of the product. The current effectiveness of many groups’ automated scripts is quite excellent, yet without supplementing these checks with “brain-engaged” human testing we run the risk of serious problems in the field impacting our customers, and the consequential bad press that follow these critical events.

At [OurCompany] much of our “testing” is focused on checking. This has served us fairly well but there are many important reasons for broadening the focus of our current approach. While checking is very important, it is vulnerable to the “pesticide paradox”. As bacteria develop a resistance to antibiotics, software bugs are capable of avoiding detection by existing tests (checks), whether executed once or repeated over and over. In order to reduce our vulnerability to field issues and critical customer incidents, we must supplement our existing emphasis on scripted tests (both manual and automated) with an active search for new problems and new risks.

There are several strong reasons for integrating exploratory approaches into our current development and testing practices:

• Scripted tests are perceived to be important for compliance with [regulatory] requirements. They are focused on being repeatable and defensible. Mere compliance is insufficient—we need our products to work.
• Scripted checks take time and effort to design and prepare, whether they are run by machines or by humans. We should focus on reducing preparation cost wherever possible and reallocating that effort to more valuable pursuits.
• Scripted checks take far more time and effort to execute when performed by a human than when performed by a machine. For scripted checks, machine execution is recommended over human execution, allowing more time for both human interaction with the product, and consequent observation and evaluation.
• Exploratory tests take advantage of the human capacity for recognizing new risks and problems.
• Exploratory testing is highly credible and accountable when done well by trained testers. The findings of exploratory tests are rich, risk-focused, and value-centered, revealing far more knowledge about the system than simple pass/fail results.

The quality of exploratory testing is based upon the skill set and the mindset of the individual tester. Therefore, I recommend that testers and managers across the organization be trained in the structures and disciplines of excellent exploratory testing. As teams become trained, we should systematically introduce exploratory sessions into the existing testing processes, observing and evaluating the results obtained from each approach.

I have been actively involved in improving testing, in general, outside of [OurCompany]. I am on the board of a testing association and I have been attending, organizing and facilitating meetings of testers for many years.

During this time, I have been exposed to much of the latest developments in software testing and I have led the implementation of Session Based Exploratory Testing within my department. In addition, over the past four years, I have been providing instruction in software testing both to the testers within my business unit and to companies outside of [OurCompany].

I look forward to the opportunity to talk with you about this further.

[/Our Letter]

Now, I thought that was pretty strong. But the response was far more gratifying than I expected. Andrew sent the message on Sunday afternoon. The VP responded by 8:45am on Monday morning. Her reply was in my mailbox before 10:00am. The reply read:

[The VP’s Reply]

Dear Andrew

Thanks very much for the email. I find this very intriguing! I believe the distinction you make between testing and checking is quite insightful and I would like to connect with you to see how we can build these concepts and techniques into our quality management services as well as my central team verification tests. I will get a call together with [Mr. Bigwig] and [Mr. OtherBigwig] so that we can figure out the best way to incorporate your ideas. Again, many thanks!!

[/The VP’s Reply]

A couple of key points:

• The letter was much stronger thanks to collaboration. Any one of the four of us could have written a good letter; the result was better than any of us could have done on our own.

• The letter is sticky, in the sense that Chip and Dan Heath talk about in their book Made to Stick: Why Some Ideas Survive and Others Die. It’s not a profound book, but it contains some useful points to ponder. The letter starts with two stories that are simple, unexpected, concrete, credible, and emotional (remember, the product manager was surprised and delighted). Those initials can be rearranged to SUCCES, which is the mnemonic that the Heaths use for successful communication.

• The testing vs. checking distinction is simpler and memorable than “exploratory approaches” vs. “confirmatory scripted approaches”. The explanation is available (and in most cases necessary), but “testing” and “checking” roll off the tongue quickly after the explanation has been absorbed.

• We managed to hit some of the most important aspects of good testing: cost vs. value, risk focus, diversification of approaches, flexibility and adaptability, and rapid service to the larger organization.

After reviewing this post, Andrew said, “I like the post a lot. Let’s hope we end up helping a lot of people with it.” Amen. You are, of course, welcome to use this letter as a point of departure for your own letter to the bigwigs. If you’d like help, please feel free to drop me a line.

See more on testing vs. checking, but especially this.