Ideas Around Bug Clusters

My colleague and friend Erik Petersen talks about bug clusters a lot. (I’ll get to discussing Black Swans in this space later, but I’m obliged to point out here that I disagree with Erik; I would argue that neither most bugs nor exploratory testing are black swans as he suggests.) I’ve been wanting to respond to the discussion for a while, and I think enough time has passed that I’m finally able to articulate my skepticism about the concept.

Note that, as James Bach says in his forthcoming book, skepticism is not the rejection of belief; it’s the rejection of certainty. My uncertainty with respect to bug clusters revolves around oversimplification of the notion. I think the idea as Erik explains it, while potentially powerful, begs important questions around modeling and factoring, and around cost versus value. My goal is to question the heuristic, and James and Pat Schroeder did with pairwise programming, not to dismiss it out of hand, but to arrive at a deeper understanding of how it might be useful and where it might be dangerous. In particular, I want to deprecate any notion that bug clusters might be a silver bullet, as Erik suggests—and I hope that when I’m done, he’ll appreciate the critique and respond to it. (That’s how things work in our community: Friends disagree. It’s important and okay to disagree. We try to work it out, and it’s nice if we do but okay if we don’t.)

First, modeling. Myers suggested that certain parts of the program tended to be far more problematic than others. The “parts” in question were, to him, areas of the program’s source code. To many people, the program is the source code. But consider Kaner‘s definition of a computer program, from his talk Software Testing as a Social Science. He says that a computer program is “A communication among several humans and computers who are distributed over space and time that contains instructions that can be run on a computer.” By this definition, the source code isn’t the program; it’s contained within the program. The program is the communication, and part of that communication contains source code.

What else is in the communication? What are the factors of a computer program, when we use Kaner’s definition as a point of departure?

A communication among separated agencies requires interfaces between them. There are several interfaces between the several people in the communication. There is a user interface for the end user, application programming interfaces for the programmers, testing interfaces for the testers. Typically there is documentation for each of these interfaces, too.

The communication between people is mediated by communication among computers and related systems. That mediated communication involves hardware (which may include computers, switching devices, processors that we may or may not think of as computers), protocols (which circumscribe the extents and limits of aspects of the communication), software (that may process the communication in some way, enhancing it or encapsulating it or compressing it or distorting it), and firmware (which itself may involve interfaces, data, programs, and protocols).

The product that enables the overall communication may be modeled in terms of structure (including computers, programs, modules, objects, classes, data objects within the classes, functions within the classes, and, yes, lines of code that enable those functions); in terms of functions that the product performs; in terms of data on which the program interacts; in terms of platforms upon which the program depends; in terms of operations that describe the way in which people use the program; and in terms of time and its interaction with the program.

This is just one set of ways of modeling or mapping the product. For the purposes of testing—questioning the product in order to evaluate it—we could conceive of many others. We could come up with a complexity map, or a map of Which Programmer Wrote Which Stuff, or Which Groups Specified Which Requirements. We could list our ideas about the program in terms of “things that involve localization and things that do not”; “things that involve currency and things that don’t”; “database interactions and not-database-interactions”; “dialog layout”; “workflows”; ad infinitum. We could come up with lists of risks, subdivided into various categories. All of these models and maps provide us with ideas about relationships, about what might constitute “near” or “far away“—and consequently, potentially useful but potentially overwhelming numbers of ideas about the points around which things could be said or imagined to cluster.

I have no problem believing in the notion of bug clusters. Certainly people tend to be good at some things, but not so good at others. Our conceptions of something might be very clear in some areas, more vague in others. It’s reasonable to see an error in one place and infer that may be errors close to it. But “close” using what map? Proximity is multivariate.

Second, cost versus value. When I find a bug, it’s wonderful to be able to use that information to find another one, so I often look for other bugs “clustered” “near” this one, based on whatever model I’ve been using or whatever factors I choose—consciously, subconsciously, or unconsciously—to apply to my understanding of what I perceive to be the bug. If I’m really on my game as a tester, I must also be ready to suspend certainty on whether my client will even agree that this is a bug. The factors that I perceive might be very important, and those models might be very powerful—or they might be total red herrings. So at the same time, I also have to consider the the opportunity cost of this activity, and the risk of not finding other bugs which might be more serious, or other information that might be more important in other, “far away” areas of my current model, or in different models altogether. It might be a better use of time to report the first problem that I find, noting my models and my suspicions, and let others (the designers, the business analysts, the programmers, the product managers) seek “related” problems using my models or their own.

The Bug Cluster Heuristic says to me “If you see a problem, suspect similar problems near it.” This heuristic can useful to me, but it depends on my models and my notion of “near”, and it depends on my considering carefully the cost and the value of further investigation, versus the value of obtaining broader or deeper coverage. A momentary pause for reflection on those questions could be time well spent.