Over and over, I’ve said that testing is better on many levels when it’s investigative, rather than merely confirmatory. Today I was browsing the Web. A question that someone had asked me about SwissAir led me to Wikipedia; from there I jumped to the article on Air Canada. I’m also a fan of Stan Rogers’ music, and so I looked into the incident in which he died—a fire that started in mid-air, and led to the deaths of 23 people on the ground in Cincinnati. Wikipedia claimed that the fire was triggered by an electrical short-circuit in the lavatory. I hadn’t heard that explanation; my understanding had been that the fire was caused by a burning cigarette in the lavatory garbage can. (My belief had been strengthened by the fact that onboard smoking bans came into force not long after the investigation into the fire.) Further research seems to confirm that an electrical problem was indeed the trigger—the direct cause of the fire. Other factors made it worse: “the delay before the crew detected and responded to the fire, the ineffective use of the fire extinguishers, the toxicity of the seat covers, and the difficulties encountered in evacuation.”
But then I read something that I found even more interesting: “Accident investigators tend to take an expansive approach when determining the ’cause’ of an accident. Aware that regulations are influenced by accident reports, investigators often seek to effect the greatest possible change. ‘It’s better if you don’t find the exact cause because then only one thing will get fixed,’ according to an NTSB investigator. Instead, for every serious accident the NTSB recommends a laundry list of changes in FAA regulations.” (That’s at http://content.cdlib.org/xtf/view?docId=ft8f59p27j&chunk.id=d0e2155&toc.depth=1&toc.id=d0e2100&brand=eschol ).
So the idea that our tests should pinpoint the problem is better seen as a heuristic—an approach that is often the right one, but that might be the wrong one for some purpose. It might be better not to pinpoint only one problem if it means that we fix a few of them instead.
That’s a really great point. I can’t count the number of times I’ve submitted a very specific bug report and maybe included a note that there’s likely to be a whole class of problems of this type. Invariably, the specific problem gets fixed instead of “problems of that type” because it’s easy to fix a single problem rather than track down all the instances of it.
I’ve seen two strategies for dealing with this:
1. Report a bug for every instance. Great for the bug count but – UGH!
2. Find a way to report the issue such that it highlights the general concern while still identifying specific instances. This requires a lot more thought on how to communicate the issue clearly, but ultimately tends to result in a more complete fix.
Of course in either scenario, I take it upon myself as the tester to investigate the conditions/instances where an issue occurs to make the lives of the person reading and fixing the bug easier because they would rather spend more time fixing and less tiem investigating, whereas for me – investingating is what it’s all about!
Michael – Thank you for the reference to this article.
>> But then I read something that I found even more interesting: “Accident investigators tend to take an expansive approach when determining the ’cause’ of an accident. Aware that regulations are influenced by accident reports, investigators often seek to effect the greatest possible change.
– Intuitively, I realize that “organizational factors” are a latent systemic contributor to issues identified during checking, er… testing. Need to bring these factors out into view.
– Perhaps a way to think about this, is to “do the opposite”. How would a plane crash investigation proceed? “root cause – plane hit the ground, investigation concluded.”
Do you know a good reference on the historical origins of accident investigations? Perhaps they also started out with a “confirmatory” bias?