Testing, Checking, and Changing the Language

In the course of trying to describe distinctions between testing and checking, a number of questions have come up:

• Do you want to change the language?
• Won’t saying “check” be confusing?
• Won’t this undermine our goal of industry-standard terminology?
• Won’t calling certain kinds of tests “checks” fly in the face of years of documentation and books?
• Isn’t this yet another case of you wanting testing to be done the same way everywhere?

In addition to the gratifying remarks, there has been a spate of similar questions or comments, in replies to the blog post, on Twitter, and various places around the blogosphere.

Then, this evening, I got this:

I think it’s fine to emphasize the value of exploration vs asserting, However I find your attempt at creating a new dictionary to solve the worlds problems incredibly naive and manipulative.

Words mean what people on average choose them to mean; they don’t even obtain their meaning from mainstream dictionaries let alone ones concocted on a blog.

What was the bloody problem with just using the phrase ‘exploratory testing’? I think what you are trying to do is create a ‘newspeak’ in which naughty thoughts are harder to express.

Apparently I blew Anonymous’ mind.

I’ll get to Anonymous way below, but for everyone else, here are some answers to the questions above [her|his] post. To all, I hope we can continue the conversation and work things out in a way that helps to advance our several crafts.

Do you want to change the language?

That might be cool, but it’s not my goal. I made a proposal. Here’s what the Compact Oxford Dictionary says about “propose”:

• verb 1 put forward (an idea or plan) for consideration by others.

Right: consideration. The idea is on the table, and through conversation with various people, maybe we can sharpen the idea and make it more useful. I’d love it if people started expressing themselves more precisely sometimes—calling “unit tests” “unit checks” when they get that way, for example—but as I said in the first post in the series, that’s not my goal. I said then, “I can guarantee that people won’t adopt this distinction across the board, nor will they do it overnight. But I encourage you to consider the distinction, and to make it explicit when you can.” Got that? I’m encouraging, not commanding. I’d be completely happy if people—testers and programmers—were simply conscious of the distinction, and were able to make more informed choices based on that.

(And more on that in tomorrow’s post.)

For all those who keep asking, I said it again in the comments to the first post: “there’s no authority who can enforce the distinction. The question is whether the distinction works for you, if it’s helpful, if it triggers a different way of thinking. In addition, there’s no particular harm in using “test” and “check” interchangably in common parlance.” I said it again later in the comments: “@Gerard: I’m not really interested in “purifying” the usage, although it was fun to hear so many people picking up on the idea at Agile 2009…Nor is there a right or wrong way to use the words. If people adopted testing vs. checking universally, that would be cool, I guess, but it’s a pretty unrealistic to believe that they would. I’m glad you agree the distinction is useful.”

And I said it again in the comments to the first post: “@Declan: First, as both the original text and the comments outline, I’m not really interested in changing people’s language. Got something that is being run entirely by automation and that receives no human evaluation? Want to call it a test? Go ahead; I wouldn’t want to stop you.”

And I said it again in the comments to the second post: One more thing, though. I don’t want to enforce orthodoxy on speech; that way lies the ISTQB and the SWEBOK and all that. In casual conversation, it makes little difference what word you use. It’s when you want to think critically about what you’re up to that the distinction (and not the label) matters. “I want to check to make sure that these are good tests,” is a fine thing to say, even if what you’re doing is really very testerly, and far more than checkish.

The third post introduced both “test” and “check” with “for the purposes of this discussion”. Plus there’s all the stuff that’s new to this post, so I will now say this: please use whatever words you like. If either one of us is unclear to the other, we’ll talk it over—probably very quickly—and we’ll work it out.

Won’t saying “checks” be confusing?

I don’t think so. I think not having a word for checks is confusing.

Won’t this undermine our goal of industry-standard terminology?

Nah. First of all, that may be your goal, but it’s not ours; leave me out of it. The notion of industry-standard terminology is, from my perspective, silly and misguided; I don’t advocate it, and I often speak out against it. To begin with, is there even a testing industry? There isn’t, any more than there’s a “writing” industry. Testing, like writing, is done in all kinds of different business contexts, organizational models, development paradigms, spoken and written languages, social structures, and for all kinds of purposes within those situations. There’s too much diversity in the world for there to be any “standard” terminology. And that’s a good thing. Diversity is complex and messy, but it also addresses people’s different values, supports different context, and engenders innovation.

One more thing—and here I address the English-speaking people: if there is to be industry-standard terminology, what language should it be in? Why not Mandarin? Why not Spanish? Why not Hindi? If you say that the English words should be interpreted into other languages, how can you be sure that the nuances of your terms show up in the interpretation?

Won’t calling certain kinds of tests “checks” fly in the face of years of documentation and books?

If people were actually to adopt the term “checking”, maybe it would. I’m not holding my breath. But you know, I’m optimistic about the bright people in this business. I think people who are genuine students of their crafts would be able to cope by making an instant mental translation, and those who aren’t are unlikely to read books anyway. The world didn’t come to an end when people started calling functions “messages” or “methods”. Nobody panics when a development project deals with several meanings of “integer” simultaneously. Boris Beizer introduced “The Pesticide Paradox”; how did people react? The way they always do as they make sense of the world: “Oh, that’s what you mean.” “Oh, that’s what you call that around here.” “Really? At KnitWare, we called it this.” “That’s cool; I never thought of it that way.”

Within a project, we sort this kind of stuff out all the time, quickly and efficiently. There’s even a name for what we’re developing: ubiquitous language, by which Eric Evans means a shared language specific to a particular team in the context of a particular project. (Why hasn’t that term been adopted as an industry standard? Answer: even though everyone does it, the standards-bearers won’t brook it, because they refuse to deal with the fact that non-standard behaviour is standard.) There will always be new people arriving on the scene with new words. The same old people will coin neologisms. Other people will get confused by certain unfamiliar terms. New technologies will show up with new labels. We’ll occasionally run into archaic words and technologies. (Only a few years ago, I had to write a little routine to translate between ASCII and EBCDIC. Don’t know what I mean? Look it up.) It’s important to notice these patterns, but sorting them out is not a big deal.

Isn’t this yet another case of you wanting testing to be done your way, all the time, everywhere?

Of course not. I don’t believe that testing should be done the same way everywhere. I do have biases in favour of testing being done as inexpensively, as rapidly, and as thoughtfully as possible; in favour of testing that is oriented towards value for people, rather than technological fetish; in favour of documentation being pared down to the minimum required to completely satisfy the mission and the client; in favour of the elimination of waste at every turn; in favour of a wide diversity of oracles and coverage models; in favour of the empowerment and skill and responsibility of the individual tester; towards principles like those expressed in the Agile Manifesto (although I don’t claim Agility^TM).

Not everyone agrees with those principles and those biases. That’s okay; it takes all kinds of communities to make a world, and (as Cem Kaner says) there’s lots of room for reasonable people to disagree reasonably, since there are so many different forms of experience and context to inform our points of view.

Some people say that we context-driven testing advocates believe that all testing should be context-driven. That’s ridiculous, and we’ve said so (see the Commentary). Reasonable people should be able to recognize that the claim of context-driven imperialism is oxymoronic; to be a context-driven thinker absolutely requires us to identify and acknowledge circumstances where context-driven approaches aren’t a good idea (see Four Attitudes Towards Context, linked here).

As I’ve said numerous times, in various forums (you could look them up), the testing that we perform for a financial institution would be ludicrous overkill for an online dating service; the testing that we do for medical devices would be far too slow and detailed for computer games. And as I’ve said numerous times, the quality of testing is like the quality of anything else: value to some person who matters. Do I disagree with you, or you with me? If you want, you can dispatch the disagreement right away by saying that I don’t matter. On the other hand, if I do matter to you, let’s talk about it and work it out.

If after that, we’re still not in agreement, that’s okay too. At one point in the SHAPE Forum (may it rest in peace), a maintenance programmer bemoaned the irrationality he saw in others’ source code, and ask how he could deal with it. Jerry Weinberg responded that “your first step is to stop thinking of it as ‘irrational’, and to start thinking about it as ‘rational from the perspective of a different set of values'”. Or as philosopher/risk manager/stepbrother Ian Heppell once elegantly put it, “Most arguments seem to be about conclusions, when they’re really about premises.”

Now:

Dear Anonymous,

I’m sorry that I blew your mind.

I think it’s fine to emphasize the value of exploration vs asserting, However I find your attempt at creating a new dictionary to solve the worlds problems incredibly naive and manipulative.

I wonder if you’ve heard about the Association for Software Testing’s dictionary project. You can read about that here and here. My vision for the dictionary is that it be based on the Oxford English Dictionary model, as described in Simon Winchester’s fabulous book, The Meaning of Everything. From the outset, the OED was designed to be descriptive, not prescriptive. The idea was to produce the story of each word, and to track where and when each one had appeared, and to follow its different paths through history, language, and culture.

We don’t intend to solve the world’s problems, nor testing’s. A dictionary won’t do that, but it’s possible that recognition of alternative points of view might be an interesting first step. As Cem Kaner puts it in the linked post, “We are not imposing definitions on the community, we are honoring the fact that different approaches to testing yield different language usage. We are not advocating a best definition, we are advocating a good practice, which is to adopt your client’s vocabulary (at least, to adopt it when talking with that client).”

Oh, and it took 71 years to complete the first version of the OED. The third is scheduled to be completed in 2037. With the idea on the table at the AST for the last three years, and with no editor yet chosen, we’re following the OED development model very well indeed. But I digress.

I recognize that you may have an emotional investment in certain words. If that’s true, that’s okay. I do too, sometimes. But I realize that if I want to keep using a certain word, or a certain pronunciation, or a certain turn of phrase, no one can stop me. And no one can stop you either, Anon. You are an adult, you can choose your own path, and you can only be manipulated if you’re willing to be manipulated. Peace be upon you. Namaste. Shalom. Really.

As I said in the comments to the first post, “there is no the definition of testing. It is always someone’s definition of testing—just as there is no property of quality that exists in a product without reference to some person and his or her notion of value.”

Then you said,

Words mean what people on average choose them to mean; they don’t even obtain their meaning from mainstream dictionaries let alone ones concocted on a blog.

Two questions on that.

First, if you’re right about that (and I think you are), how is it that language evolves? I think there are several reasons. Sometimes a new technology appears, and we need a new name for it and the stuff around it. As I tweeted this very evening (note the neologism),“Wife now: “I’m going to CASECamp. I sent stuff out on Crowdvine, but I should have Twittered it.” Sentences unintelligible one year ago.” Sometimes it’s because people notice things that they want names for, and they name them. Sometimes it’s because people want to break something complex into simpler components. Sometimes it’s because people want to lump a bunch of elements into a single concept or model. I’m trying to de-lump a previously lumped part of testing.

Second, if you’re right (and I think you are) that words don’t obtain their meanings from the ones concocted on a blog, then we’re all safe. So what’s the problem?

But there is one thing I’d gently dispute. Words don’t acquire meanings based on averages; they acquire meanings as soon as two or more people are willing to share that meaning. One ex of mine had a word, “dido”; it meant “idiosyncracy or habit of the type exhibited by a cat, or a person acting like one”. She and I also developed “insurance pee”, meaning “an activity in which your kids (or your spouse, or your self) should indulge just before leaving on a long highway drive.” Another ex used “fleffing”, meaning “to dither”.

So far, a handful of people like the word “check” as I’ve described it. Some don’t. Others are willing to consider it. Most people, so it seems, just don’t care. What’s the big deal?

What was the bloody problem with just using the phrase ‘exploratory testing’? I think what you are trying to do is create a ‘newspeak’ in which naughty thoughts are harder to express.

Perhaps you don’t remember the somewhat epic controversy over the term “exploratory testing”. “You mean ‘ad hoc’ testing, right?” “You mean testing without knowing what you’re doing, right?” “You mean testing without documentation, right?” This went on for years. Well, thanks to Cem’s coining of the term (we believe that he was the first, in the first edition of Testing Computer Software), James Bach’s vigourous articulation of the idea, and a bunch of other material written by Jonathan Bach, Elisabeth Hendrickson, Mike Kelly, Jonathan Kohl, James Lyndsay, Brian Marick, Bret Pettichord, Harry Robinson, Rob Sabourin, James Whittaker, and many others, including me, we now have a really substantial body of thought on the subject so that people can discuss it, learn it, teach it, compare it different contexts, figure out how to do it better. “Exploratory testing” even has its own erratic Wikipedia entry—the hallmark of every powerful idea.

Similar controversies have been expressed over “agile” and “extreme programming”. I’ve observed (only half-jokingly) that no one who had ever played rugby would name a development model “Scrum”. But people appropriate words all the time. Nothing new there either. In 1661, Boyle was talking about the “spring of the air”, and Hobbes was launching vitriolic attacks not only on Boyle’s conclusions, but on the whole premise of “experimental philosophy”, another neologism of the age. Controversy is nothing new. The world is a story that we’re all editing as we go.

So thanks for your participation; it drives me to clarify the idea, to make it more useful. Feel free to post a more complete expression of your concerns on your blog, point me to it, and we’ll work it out. Or, as above, you can simply declare that I don’t matter.

Newspeak, in Orwell’s 1984, was not a language designed to suppress naughty thoughts; its ultimate purpose was to make it impossible to express any thoughts at all. Suppression of naughtiness was simply a beneficial (to Big Brother) side effect. My purpose is the opposite of that. My purpose is to foster more thinking, deeper thinking about our fascinating and complex craft. If some consider such thoughts naughty, great. Testing could use some titillation.

See more on testing vs. checking.

Related: James Bach on Sapience and Blowing People’s Minds