Boundaries Unbounded

This post started as a LinkedIn post, which got started as a comment replying to this poll:

It’s depressing to see ideas about testing and risk reduced to dopey formulas that are great for softball “certification” exam questions, but terribly limited for investigating and revealing product and business risk.

Here’s how we describe “boundary” in Rapid Software Testing:

A means by which something is classified or filtered

This means that a boundary could be a dividing point between two contiguous regions of behaviour or data. The latter is expressed in the very widespread notion of “boundary value”, but let’s not limit boundary analysis and testing to data alone. Other elements in looking at boundary behaviour might include

• a principle for assigning things to particular categories or buckets
• an interface that provides a “viewport” function that presents us with some subset of a larger set
• the earliest or latest time that a process can be executed
• the point at which a process blocks something from happening

We could think of plenty of other examples, and you can too.

With that in mind, none of the multiple-choice answers offered for the question are the boundary values for testing that field. Those are boundary values that one could associate with the given description.

What’s the difference?

Here’s what: the description is one thing, and the actual behaviour of the product is another.

In fact, we don’t know what the actual boundary values are for that field until we’ve tested. And even after we’ve tested, there’s no way of being absolutely sure we’ve identified all of the places were behaviours or classifications could change.

And here’s how we describe boundary testing in Rapid Software Testing:

Testing focused on boundary-related risk.

If we intend to do boundary value analysis and boundary testing, here are just a few things we might want to consider.

One good first step is to avoid thinking of the description as the only source of information in the design of the test. The field and its value aren’t the only factors that could be in play. So consider asking: is this all there is to say about this field? How does this statement relate to other statements? To statements that are missing?

The field is part of a system that almost certainly has interacting and interdependent elements that might affect how “obvious” boundaries are presented and processed. Knowing that should prompt us to consider not only the statement given, but other explicit and unstated conditions related to it. So what’s the purpose of this field.

For instance, maybe “2004” is part of a larger set of boundary conditions; maybe dates later than June 30, 2004 should be rejected and dates on or before it should be accepted. As I write this, it’s July 1, 2023, and June 30, 2004 was one day more than 19 years ago. Is 2004 the big deal, or is it “19 years ago” that’s important? This boundary value (and “the” right answer) might change over time.

How is the input field implemented? As a drop-down box? If so, the values 1899 or 2005 might not even be available as choices. Is the field implemented a text box, or as a combo box, which affords selection both from a list and free-form text entry? If so, there may be boundaries on the length of the input string. Maybe there is weird behaviour or bad error handling to be revealed when the given number is shorter than three digits, or longer than four.

There might be a boundary between 2004 and 2005. Isn’t there a potential boundary between 2004 and 2004.00000001?

Maybe the product fails to handle a boundary between 0 and -1, or between “things that are numbers and things that are not”. For instance: the Ruby programming language contains a function, to_i, that converts strings (like “143” or “-2”) to integers (143 or -2). This function does two things that could prove surprising. First, when presented with a string like “3.9”, the function truncates the number rather than rounding it. Second, when presented with a string that doesn’t fit the function’s pattern matching, the function doesn’t return a result indicating an error or an exception — it returns 0!

Famously, a couple of decades back, there was a meaningful boundary between 1999 and 2000. Remember that? There’s another one like it coming up, in the early months of 2038: the Epochalypse.

Does input for our field get filtered or constrained—restricted to digits only? Where might input constraints be implemented? In the front end? Could those limits be evaded by going behind the browser and submitting a backend request directly? Does the backend do data validation and error checking?

Perhaps error checking and exception handling deal properly with input up to certain values or certain lengths, but fail when overwhelmed. (I just now observe that today, the note-taking tool Google Keep cheerfully handled reminder dates up to the year 275760 and saved them, but any year later than that triggered the time and date stamp to be displayed and saved as “NaN Nan, 12:00NaN AM”. And yet according to my notes, eariler on 275749 was the last year before the NaN effect kicked in, so there must be other variables involved.

In complex systems there may be processing upstream and downstream of the function being described, such that our input field seems to handle things properly, but things get messed up when the data comes from upstream, or when we try to post-process, save, or retrieve the data.

There might be time-oriented boundaries. This year, the boundary between 1902 and 1903 might be significant if the system processes the difference between a century ago and today, using only two digits for storage or display. There might be timing boundaries, such that an answer must be supplied within a given period — or not too soon.

Et cetera.

It’s easy take a number that you see in a description and add one to it or subtract one from it. A junior-school child could do that.

If we want to test, we must not limit our theory of error to the idea that the developer might have typed a less-than symbol (‘<‘) instead of a less-than-or-equal-to symbol (‘<=’) to process an integer field.

To test, we must also strive to discover behaviour related to boundaries that we haven’t been told about. Such behaviour could be surprising to the people building the product and potentially threatening to its value. To find it, we must learn the product deeply, and periodically explore and experiment with it.