A test case is not a test. A recipe is not cooking. An itinerary is not a trip. A score is not a musical performance, and a file of PowerPoint slides is not a conference talk. All of the former things are artifacts; explicit representations. The latter things are human performances. When the former things are used without tacit knowledge and skill, the performance is unlikely to go well. And … Read more
This post is adapted from a recent blast of tweets. You may find answers to some of your questions in the links; as usual, questions and comments are welcome. Update, 2017-01-07: In response to a couple of people asking, here’s how I’m thinking of “test case” for the purposes of this post: Test cases are formally structured, specific, proceduralized, explicit, documented, and largely confirmatory test ideas. And, often, excessively so. … Read more
This question came up the other day in a public forum, as it does every now and again: “Should you test against ambiguous/incomplete requirements?” My answer is Yes, you should. In fact, you must, because all requirements documents are to some degree ambiguous or incomplete. And in fact, all requirements are to some degree ambiguous and incomplete. And that is an important reason why we test: to help discover how … Read more
This is an expansion of some recent tweets. Do automated tests (in the RST namespace, checks) measure the quality of your product, as people sometimes suggest? First, the check is automated; the test is not. You are performing a test, and you use a check—or many checks—inside the test. The machinery may press the buttons and return a bit, but that’s not the test. For it to be a test, … Read more
Several years ago in one of his early insightful blog posts, Pradeep Soundarajan said this: “The test doesn’t find the bug. A human finds the bug, and the test plays a role in helping the human find it.” More recently, Pradeep said this: Instead of saying, “It is programmed”, we say, “It is automated”. A world of a difference. It occurred to me instantly that it could make a world … Read more
Want a quick idea for a burst of activity that will reveal both bugs and opportunities for further exploration? Play “Honest Manual Writer”. Here’s how it works: imagine you’re the world’s most organized, most thorough, and—above all—most honest documentation writer. Your client has assigned you to write a user manual, including both reference and tutorial material, that describes the product or a particular feature of it. The catch is that, … Read more
Testers don’t prevent errors, and errors aren’t necessarily waste. Testing, in and of itself, does not prevent bugs. Platform testing that reveals a compatibility bug provides a developer with information. That information prompts him to correct an error in the product, which prevents that already-existing error from reaching and bugging a customer. Stress testing that reveals a bug in a function provides a developer with information. That information helps her … Read more
In response to my recent blog post, 100% Coverage is Possible, reader Hema Khurana asked: “Also some measure is required otherwise we wouldn’t know about the depth of coverage. Any straight measures available?” I replied, “I don’t know what you mean by a ‘straight’ measure. Can you explain what you mean by that?” Hema responded: “I meant a metric some X/Y.” In all honesty, it’s sometimes hard to remain patient … Read more
One model for a software development project is the assembly line on the factory floor, where we’re making a buhzillion copies of the same thing. And it’s a lousy model. Software is developed in an architectural studio with people in it. There are drafting tables, drawing instruments, good lighting, pens and pencils and paper. And erasers, and garbage cans that get full of coffee cups and crumpled drawings. Good ideas … Read more
In testing, what does “100% coverage” mean? 100% of what, specifically? Some people might say that “100% coverage” could refer to lines of code, or branches within the code, or the conditions associated with the branches. That’s fine, but saying “100% of the lines (or branches, or conditions) in the program were executed” doesn’t tell us anything about whether those lines were good or bad, useful or useless. “100% code … Read more