Out of Order

To testers (and to designers, developers, and managers, and others who might do testing), this is your periodic reminder to do things out of order.

By “do things out of order”, I mean: perform sequences of actions that do not follow the designer or the developer’s imagination of how things should happen; how things normally happen; what the customers might want; or what’s easiest to process.

Here’s an example from Quicken, a product that has had this problem since forever. When creating a new invoice, there’s a combo box (text field and drop-down) from which I can conveniently select an existing customer, or type in the name of a new customer.

However, when I’m preparing an invoice, I often go straight to entering the customer’s name and address data, a little bit farther down on the form; then I enter the details for the invoice in the usual way. However, on attempting to save the invoice, a prompt appears telling me to fill in the data for the combo box. When I do that by typing the name of the company, the address information that I’ve already entered disappears, replaced by only the company name!

Now, it might be easy to roll your eyes and say “Quicken!” But that’s just today’s installment in the series. I’ve encountered this issue in all kinds of places over the years.

So how about your product? What assumptions govern your model of the user’s “natural” flow? What happens when the user doesn’t share the mental models of the designers or developers?

What happens when you vary your own patterns of behaviour while testing? What happens when you change your mind or make a mistake?

Before creating automated checks that follow some assumed order or sequence, do you also test to find out what happens for other ones? In addition to reasonable data, do you also try to submit data that might be entered by mistake, or maliciously?

Some people might suggest that “no user would ever do that”. What they really mean is “no user that I’ve thought of, and that I like, would do that on purpose, in a way that I’ve imagined so far”. In other words, “I wouldn’t do that.” As the builder or designer of the product, of course you wouldn’t. But we don’t make products for builders, designers, or insiders on the development team. We make them for customers, users, and outsiders. That applies to documentation, too.

If the consequence of some variation from the designer’s intention is frustration, confusion, annoyance, or (especially) data loss, report it as a Bad Bug. But you won’t get to find out if you (or your automation) perpetually follow the “normal” order of things.