A Rapid Testing Success Story

One of the problems in our business is that people are usually reluctant to talk about testing, even when it’s successful. In order to discuss testing, they may have to cop to problems in their product, or in their development work. Even if they’re very happy with the things that they’ve learned in the course and put into practice, they may have to acknowledge that earlier forms of testing were less effective. No matter what, there’s usually some dirty laundry somewhere, so people are understandably leery about airing it on publicly visible clotheslines.

Here’s some feedback that I got recently from a customer, who was kind enough to allow me to make these comments public. It’s verbatim, except for the name change. A little context, without stepping on non-disclosure agreements: they’re a quite successful global company that makes commercial utility products. Like most commercial software companies, the work is done under severe time pressure, and product requirements are changing rapidly in response to market conditions. The groups that I’ve spoken to at this company are quite capable and engaged in their work, but the Rapid Testing course seemed to stir up the smoldering fire that was already there. My client wrote:

We are already starting to put into practice what you taught us – here is a mini case study:

Four of the group sat down last Friday and tested another product. Dan (name changed for confidentiality –MB) guided and made suggestions. None of the 4 knew the product under test. The product test lead spent half the day being a live oracle. Results: Another 50 defects. Several were crashes (buffer overrun- thanks Perlclip!). Many UI and usability defects. By the afternoon the team was starting to find more specific defects in what the product should do, but wasn’t doing. However, by this time they were getting very baked. This sort of testing is really hard work. However, the product lead was amazed by what was found, and the defects found per hour invested was – once again – orders of magnitude more effective than the testing that was currently going on with the product. We are going to cycle this much more frequently, and the same four are going to dig deeper on the same product later this week as well.

I was especially intrigued by the notion that this kind of testing is hard work. That sounded like a good sign. (Once a friend of my engaged a four-year-old kid in an elevator who was yawning. “Tired out?” asked my friend. “That’s how you know when you’ve had a good day.”)

Rapid testing seems very natural and easy to me, but I’ve been doing it for a while. I’m pretty convinced that it’s the kind of approach that gets easier with practice, so I asked the customer on a recent visit if he agreed. He nodded, and said that he thought that it was indeed getting easier for the team members who were doing it regularly. However, the approach is something of a paradigm shift, and people can easily slip back into the familiar. I’m glad that this company seems to have champions that will sustain the work.

By the way, these days, the three-day Rapid Software Testing course includes, at the client’s option, a fourth day of hands-on testing or consulting with the team or with individual members. I encourage clients to accept the offer because it’s useful to have a whole day to deal with the work in context. It’s fun for me, too, especially when I get to test something that’s new to me.

Many thanks to my anonymous client and his team. You folks know who you are.