Yesterday I described the moment of recognition of a problem, which tends to happen in the upper-left corner of this table: When I perceive a problem during testing, it’s my job to let people know about what I’ve found, and to arrive at a shared set of feelings and mental models about the problem, represented by the lower left quadrant. That is, I want to move from experience—my private, internal, … Read more
In the first post in this series, I introduced some of the factors in recognizing a problem when one occurs during testing. Let’s walk through some examples of recognizing and relating problems. Imagine that I’m a tester at Adobe, a few years back, testing a version of Acrobat. As it happens, my laptop’s screen is smaller than a desktop monitor, but at 1920 by 960 pixels, the resolution is quite … Read more
In Rapid Testing (and, we believe, in all worthwhile testing) learning about the product and finding problems that matter are central to the mission. In the role of a tester, I support the business and development team by finding and describing things I believe to be important problems that might threaten the value of the product. A skilled tester should be able do that at any stage—as soon as there’s … Read more
As James Bach and I have have been refining our ideas of testing, we’ve been refining our ideas about oracles. In a recent post, I referred to this passage: Program testing involves the execution of a program over sample test data followed by analysis of the output. Different kinds of test output can be generated. It may consist of final values of program output variables or of intermediate traces of … Read more
You do not need acceptance criteria to test. Reporters do not need acceptance criteria to investigate and report stories; scientists do not need acceptance criteria to study and learn about things; and you do not need acceptance criteria to explore something, to experiment with it, to learn about it, or to provide a description of it. You could use explicit acceptance criteria as a focusing heuristic, to help direct your … Read more
The longer I stay in the testing business and reflect on the matter, the more I believe the concept of “defects” to be unclear and unhelpful. A program may have a coding error that is clearly inconsistent with the program’s specification, whereupon I might claim that I’ve found a defect. The other day, an automatic product update failed in the middle of the process, rendering the product unusable. Apparently a … Read more
In the previous post, Lynn McKee recognized that, with respect to the Comparable Product oracle heuristic, “comparable” can be have several expansive interpretations, and not just one narrow one. I’ll emphasize: “comparable product”, in the context of the FEW HICCUPPS oracle heuristics, can mean any software product, any attribute of a software product, or even attributes of non-software products that we could use as a basis for comparison. (Since “comparable … Read more
People interpret requirements and specifications in different ways, based on their models, and their past experiences, and their current context. When they hear or read something, many people tend to choose an interpretation that is familiar to them, which may close off their thinking about other possible interpretations. That’s not a big problem in simple, stable systems. It’s a bigger problem in software development. The problems we’re trying to solve … Read more
Several years ago, I wrote an article for Better Software Magazine called Testing Without a Map. The article was about identifying and applying oracles, and it listed several dimensions of consistency by which we might find or describe problems in the product. The original list came from James Bach. Testers often say that they recognize a problem when the product doesn’t “meet expectations”. But that seems empty to me; a … Read more
In which the conversation about heuristics and oracles continues… Tony’s brow furrowed as he spoke. “No oracle comes with a guarantee that it’s giving you the right answer. That’s what you said. But surely there are some oracles that are reliable,” he said. “What about pure math?” “Pure math? All right. Here’s an example: what’s 61 plus 45?” “Duh. 106.” “Well,” I said, “for many computer systems prior to the … Read more