In one forum or another recently, a tester, developer or DevOps person (I wasn’t sure which of these he/she was) asked: Do you perform any exploratory testing on APIs? How do you do it? Here’s an in-depth version of the reply I gave, with the benefit of links to deeper material. To begin: application programming interfaces (APIs) are means by which we can use software to send commands to a … Read more
Some might believe that checking and exploratory work are antithetical. Not so. In our definition, checking is “the algorithmic process of operating and observing a product, applying decision rules to those observations, and reporting the outcome of those decision rules”. We might want to use some routine checks, but not all checks have to be rote. We can harness algorithms and tools to induce variation that can help us find … Read more
Testers investigate problems and risk. Other people manage the project, design the product, and write the code. As testers, we participate in that process, but in a special way and from a special perspective: it’s our primary job to anticipate, seek, and discover problems. We testers don’t prevent problems in the product; we don’t design, build, fix, or manage the product. We’re evaluating things that other people are bringing to … Read more
In response to yesterday’s post on The Happy Path colleague and friend Albert Gareev raises an important issue: Until we sufficiently learned about the users, the product, and the environment, we have no idea what usage pattern is a “happy path” and what would be the “edge cases”. I agree with Albert. (See more of what he has to say here.) This points to a kind of paradox in testing … Read more
Last week on Twitter, I posted this: “The testing is going well.” Does this mean the product is in good shape, or that we’re obtaining good coverage, or finding lots of bugs? “The testing is going badly.” The product is in good shape? Testing is blocked? We’re noting lots of bugs erroneously? — Michael Bolton (@michaelbolton) January 31, 2018 “The testing is going well.” Does this mean the product is … Read more
There’s much talk these days of DevOps, and Agile development, and “shift left”. Apparently, in these process models, it’s a revelation that testers can do more than test a built product, and that testers can and should be involved at every step of development. In Rapid Software Testing, that’s not exactly news. From the beginning, we’ve rejected the idea that the product has to be complete, or has to pass … Read more
Here’s an easy-to-remember little substitution that you can perform when someone suggests “automating the testing”: “Automate the evaluation and learning and exploration and experimentation and modeling and studying of the specs and observation of the product and inference-drawing and questioning and risk assessment and prioritization and coverage analysis and pattern recognition and decision making and design of the test lab and preparation of the test lab and sensemaking and test … Read more
What does it mean to do deeper testing? In Rapid Software Testing, James Bach and I say: Testing is deep to the degree that it has a probability of finding rare, subtle, or hidden problems that matter. Deep testing requires substantial skill, effort, preparation, time, or tooling, and reliably and comprehensively fulfills its mission. By contrast, shallow testing does not require much skill, effort, preparation, time, or tooling, and cannot … Read more
This post is adapted from a recent blast of tweets. You may find answers to some of your questions in the links; as usual, questions and comments are welcome. Update, 2017-01-07: In response to a couple of people asking, here’s how I’m thinking of “test case” for the purposes of this post: Test cases are formally structured, specific, proceduralized, explicit, documented, and largely confirmatory test ideas. And, often, excessively so. … Read more
Several years ago in one of his early insightful blog posts, Pradeep Soundarajan said this: “The test doesn’t find the bug. A human finds the bug, and the test plays a role in helping the human find it.” More recently, Pradeep said this: Instead of saying, “It is programmed”, we say, “It is automated”. A world of a difference. It occurred to me instantly that it could make a world … Read more