What Colour Is Your Box?

At the Software Testing Club, Phil Kirkham asked a question headed “What Colour Tester Are You?”

Another totally unscientific survey – how many readers of this site would consider themselves to be black box testers, white box testers or grey box testers ?

Or if you are a test manager, what colour testing do the testers you are in charge of do?

I whimsically replied, “I deny the existence of the box,” a statement for which Michele Smith (quite justifiably) asked for clarification. Here’s something pretty close to my reply.

I’m not sure the distinction between shades of boxes is terribly helpful. The box metaphor is helpful in one way: it reminds me to think about constraints by which I don’t want to be bound. I can almost always do something about some of those constraints. Meanwhile, there are some constraints that I can only ever circumvent to some degree, not completely.

The metaphor of the black box is intended to represent a system for which we have no knowledge of the internals. As a tester, I’m rarely in that position unless I want to be, and I usually don’t want to be. Simply by asking for it, I can usually pop the lid off the box entirely by asking for the source code itself, or for other useful forms of information about the internals. If I can’t see the code, at least I can ask lots of questions; I use the Heuristic Test Strategy Model as a point of departure for them. By using tools of various kinds I can always peer inside the box to some extent—listing out the imported and exported functions, inspecting the resource table, or examining the compiled binary in a text editor for clues. By using knowledge and experience, I can make reasonable inferences about the internals. But those determinations don’t tell me what’s going on inside, and that might be crucially important. Thinking solely in terms of the black box limits me. I want out.

(The metaphor of the white box, to me, is silly, since a white box is just as opaque as a black one. Note that this is an example of matching bias; the opposite of black is white for most purposes, so people talk about white-box testing and forget what the metaphor is supposed to represent. Glass-box testing covers the intended meaning better for me.)

So let’s look at the glass box, then. The glass box is clear, allowing us to see everything in its innards. That’s certainly a nice idea, but it prompts some questions. What does “everything” mean? Is it really important to see everything? Could we see everything even if we looked?

Consider: we generally consider the glass box to be the source code for an application, consisting of instructions that our developers have written. Yet our developers make calls to application frameworks, third-party libraries, operating system functions, code that other developers wrote, hardware interfaces, and so on. Our code may provide services and functions for other code. So we’re not really seeing everything; we’re seeing what there is to see, and how it might refer to other black or glass boxes. We might be able to take advantage of our knowledge of the the internals of our code to use its interfaces or to instrument it in some way. But even if we can read the source code, we can’t necessarily determine the significance of the function for some task, nor can we anticipate completely how it might go wrong. We might be able to determine if the code has problems in it, but we can’t determine if the code was written in an optimal way, since we’re residents of our own state of the art, as Billy Vaughan Koen might say (see his book Discussions of the Method). We can’t completely recognize the programmer’s intentions, her mental shortcuts, her brilliance, or her blind spots; the code might suggest things about that, but it doesn’t tell us conclusively. Even if we can anticipate the platforms on which the code is supported, we can’t represent them all in the lab, and we can’t determine if a change to those platforms will render our code useless. Visibility into the glass box might be helpful, but it doesn’t necessarily and on its own tell us about what might be important to someone who matters. Indeed, thinking too much in terms of the glass box may dazzle or distract us into thinking that the box itself is the important thing, rather than the value that the box provides with respect to the rest of the system and its human users.

The box that I’m currently looking, black or glass, at is always connected to other boxes. No box ever really stands alone; its behaviour is ultimately interesting only with respect to other boxes. Some of the more important black boxes here, boxes into which I cannot truly see, are the minds of the people who wrote the code and the minds of the people who use it. Thinking solely in terms of the glass box limits me, just as surely as thinking solely in terms of the black box does.

So: am I a black box tester? If I feel that I’m in that mode, one of my first impulses is to consider whether it’s important, for the current question I have, to figure out what’s going on inside. Am I a glass box tester? If I’m in that mode, one of my first impulses is to consider what I’m not noticing and how this box interacts with other boxes, some of which may be glass, some of which may be black, and some of which may be frosted, transparent, translucent, made of Lexan, made of cellophane, or made of diamond, all in every colour of the rainbow.

So whatever the colour of the box, whatever its visibility, the value of my examination is limited if I’m only looking at its insides or its outsides. As much as I can, I want to comprehend them both. In particular, if I’m trapped in a box, I want to escape before the food and oxygen run out.