People sometimes seem to struggle with a concept that’s central to testing, the concept of “oracle”. In the three-day Rapid Software Testing class, we define an oracle as a principle or mechanism means by which we recognize a problem. Sometimes I like to emphasize that oracles are fallible and context-dependent. When that’s so, I say that an oracle is a heuristic principle or mechanism means by which we recognize a problem. (Updated … Read more
A strong, active, local testing community is a wonderful thing. Kitchener-Waterloo, a hub of software and software-related companies small and large, has such a community, and an active association in the KWSQA. Each year, the KWSQA presents the Targeting Quality conference, which I’ve had the pleasure of participating in for a few years. Alas, this year I can’t make it—but maybe you can. This year’s presenters and workshop leaders are … Read more
Jerry Weinberg tells a story (yes, it’s me; I’m telling yet another Jerry Weinberg story) of meeting an old friend who looked distraught. “What’s the matter?” Jerry asked. The fellow replied, “Well, I’m kind of shellshocked. My wife just left me.” “Was that a surprise?” “Yes, it really was,” the fellow said. “I mean, we had had some problems, but I thought they were all settled.” Jerry paused for a … Read more
In the last post in this series, I noted some potentially useful structual similarities between bug reports (whether oral or written) and newspaper reports. This time, I’ll delve into that a little more. To our clients, investigative problem reports are usually the most important part of the product story. The most respected newspapers don’t earn their reputations by reprinting press releases; they earn their reputations through investigative journalism. As testers … Read more
We were in the middle of a testing exercise at the Amplifying Your Effectiveness conference in 2005. I was assisting James Bach in a workshop that he was leading on testing. He presented the group with a mysterious application written by James Lyndsay—an early version of one of the Black Box Test Machines. “How many test cases would you need to test this application?” he asked. Just then Jerry Weinberg … Read more
Calculating a ratio of passing tests to failing tests is a relatively easy task. If it is used as a means of estimating the state of a development project, though, the ratio is invalid, irrelevant, and misleading. At best, if everyone ignores it entirely, it’s simply playing with numbers. Otherwise, producing a pass/fail ratio is irresponsible, unethical, and unprofessional. A passing test is no guarantee that the product is working … Read more
Here’s a classic case of poor design and user experience. Most of us have seen something like it. It happened to my wife yesterday. It will happen to you again soon, probably. You’re making an online payment for some product or service. You press a button that says something like “Submit Payment”. A web page appears that says something like “Your payment is being submitted. Please do not close this … Read more
Here is a specific, real-world example of testing where the focus doesn’t include explicit checking, and does not result in yes-or-no answers to predetermined questions. This morning, I acted on a piece of email I received several days ago, offering a free upgrade to a PDF conversion package which I’ll call “PDFThing”. I’ll walk you through what happened, and parts of my thought process as it happened. Since the email … Read more
A fellow named Oren Reshef writes in response to my post on Worthwhile Documentation. Let me be the devil’s advocate for a post. Not having fully detailed test steps may lead to insufficient data in bug reports. Yup, that could be a risk (although having fully detailed steps in a test script might also lead to insufficient data in bug reports; and insufficient to whom, exactly?). So what do you … Read more
This week I had the great misfortune of reading yet another article which makes the false and ridiculous claim that exploratory testing is “undocumented”. After years and years of plenty of people talking about and writing about and practicing excellent documentation as part of an exploratory testing approach, it’s depressing to see that there are still people shovelling fresh manure onto a pile that should have been carted off years … Read more