DevelopsenseLogo

Testing is…

Every now and again, someone makes some statement about testing that I find highly questionable or indefensible, whereupon I might ask them what testing means to them. All too often, they’re at a loss to reply because they haven’t really thought deeply about the matter; or because they haven’t internalized what they’ve thought about; or because they’re unwilling to commit to any statement about testing. And then they say something vague or non-committal like “it depends” or “different things to different people” or “that’s a matter of context”, without suggesting relevant dependencies, people, or context factors.

So, for those people, I offer a set of answers from which they can choose one; or they can adopt the entire list wholesale; or they use one or more items as a point of departure for something of their own invention. You don’t have to agree with any of these things; in that case, invent your own ideas about testing from whole cloth. But please: if you claim to be a tester, or if you are making some claim about testing, please prepare yourself and have some answer ready when someone asks you “what is testing?”. Please.

Here are some possible replies; I believe everything is Tweetable, or pretty close.

  • Testing is—among other things—reviewing the product and ideas and descriptions of it, looking for significant and relevant inconsistencies.
  • Testing is—among other things—experimenting with the product to find out how it may be having problems—which is not “breaking the product”, by the way.
  • Testing is—among other things—something that informs quality assurance, but is not in and of itself quality assurance.
  • Testing is—among other things—helping our clients to make empirically informed decisions about the product, project, or business.
  • Testing is—among other things—a process by which we systematically examine any aspect of the product with the goal of preventing surprises.
  • Testing is—among other things—a process of interacting with the product and its systems in many ways that challenge unwarranted optimism.
  • Testing is—among other things—observing and evaluating the product, to see where all those defect prevention ideas might have failed.
  • Testing is—among other things—a special part of the development process focused on discovering what could go badly (or what is going badly).
  • Testing is—among other things—exploring, discovering, investigating, learning, and reporting about the product to reveal new information.
  • Testing is—among other things—gathering information about the product, its users, and conditions of its use, to help defend value.
  • Testing is—among other things—raising questions to help teams to develop products that more quickly and easily reveal their own problems.
  • Testing is—among other things—helping programmers and the team to learn about unanticipated aspects of the product we’re developing.
  • Testing is—among other things—helping our clients to understand the product they’ve got so they can decide if it’s the product they want.
  • Testing is—among other things—using both tools and direct interaction with the product to question and evaluate its behaviours and states.
  • Testing is—among other things—exploring products deeply, imaginatively, and suspiciously, to help find problems that threaten value.
  • Testing is—among other things—performing actual and thought experiments on products and ideas to identify problems and risks.
  • Testing is—among other things—thinking critically and skeptically about products and ideas around them, with the goal of not being fooled.
  • Testing is—among other things—evaluating a product by learning about it through exploration, experimentation, observation and inference.

You’re welcome.

Subscribe to the Newsletter

25 replies to “Testing is…”

  1. At the top of my white board at work is James Bach’s quote on testing:

    “Testing is an infinite process of comparing the invisible to the ambiguous in order to avoid the unthinkable happening to the anonymous.”

    Michael replies: A great one, underscoring the point that there are many. Thanks for mentioning it.

    Reply

  4. Michael,

    [You’re welcome.]
    Thank you!

    I thought this was a really good post. It took me a day to digest it and think up some questions….

    [please prepare yourself and have some answer ready when someone asks you “what is testing?”]
    What is your “elevator pitch” (http://en.wikipedia.org/wiki/Elevator_pitch) answer to “What is testing”? Let’s assume a 15 second elevator ride (or enough time for 1-2 brief sentences). What would your answer be to:
    • a non-technical person
    • a technical person (but not a tester)
    • someone who has been testing for a very short time
    • someone who has been testing for a very long time
    • a context-driven tester

    Michael replies: I’ve given you 15 or so answers. Now it’s your turn; then we’ll see about mine.

    Do you think there is a difference (subtle, or not) between asking “What does testing mean to you?” and “What is testing?”

    The second question is always the first question in disguise. That is, there is no neutral, theory- and experience-independent answer to the second question.

    Also, it seems to me that some of these answers are products of, results of, purposes of, or subsequent actions to testing. For example, “Testing is—among other thing—raising questions to help teams to develop products that more quickly and easily reveal their own problems.” To me, this seems like an action (raising questions) followed by a purpose (to help teams…). Although, Oxford says that definition is, “a statement of the exact meaning of a word”, so perhaps this is ok. Thoughts?

    Yes, please.

    Anyway, I asked the last questions because I wonder where one should draw the line. If “Testing is…” this thing, that produces this product, that results in this other thing, and is performed for this purpose…how far can you extend the definition before it is no longer reasonable? Would it be reasonable to say, “Testing is – among other things – an occupation.” Or “Testing is – among other things – a way to feed your family.” Or “Testing is – among other things – a source of happiness” (since “evaluating a product by learning about it through exploration, experimentation, observation and inference…informs quality assurance…which helps our clients to make empirically informed decisions about the product, project, or business…which keeps us employed…which earns us pay…which allows us to feed our families…which makes us happy”).

    No. I mean, yes. What would you do if I were to provide a pat answer to that? Come on, Damian; you’re too sharp a thinker to stop there.

    (For those of you who might think I’m being uncivil to Damian, let it be known that I meant what I said in my last sentence above. Being an advanced student of Rapid Testing, as Damian is, isn’t easy; after a certain point, we require powerful thinkers such as he to bring dessert and a couple of side dishes to the meal—whereby they demonstrate their prowess and extend their reputations. Stand by for Damian to go deeper now.)

    Thanks, again!

    Reply

  5. [Stand by for Damian to go deeper now]
    Fair is fair, and that’s fair.

    [Do you think there is a difference (subtle, or not) between asking “What does testing mean to you?” and “What is testing?”]
    [The second question is always the first question in disguise. That is, there is no neutral, theory- and experience-independent answer to the second question.]
    Paraphrased: “It’s the context, stupid”. I can’t believe I sometimes still forget that. Anyway, I now think the question “What is testing?” has an implied “…to you, in some context” appended to the end. Maybe a “within reason”, too. Taking that into consideration, the question becomes “What is testing…to you…in some context…within reason?” If so, then there are many possible answers. But, I don’t think that means that all answers are acceptable. I think that all answers can be challenged and dissected for merit, and only those answers which are shown to have merit are acceptable.

    Michael replies: What you’re doing here is applying The Relative Rule to the list, which is a good thing to do. Now: apply The Relative Rule to the idea of “merit”.

    [Oxford says that definition is, “a statement of the exact meaning of a word”, so perhaps this is ok. Thoughts?]
    [I asked the last questions because I wonder… how far can you extend the definition before it is no longer reasonable]
    [Yes, please.]
    I started (as I often do) with the definition. In this case, I explored the definition of “definition”. This lead me to the definition of “meaning” (“what is meant by a word, text, concept, or action.”), which lead me to the definition of “meant” and “mean” (“Intend to convey, indicate, or refer to (a particular thing or notion); signify”). I mulled these ideas over for a while, and while I felt like I was learning, I did not feel like I was progressing in a meaningful way. So, I switched tactics. Sometimes I use Google as a way to show me what others have researched in an area by typing in a term plus “vs.” and seeing what Google will suggest to complete the query. When I typed “definition vs.”, Google suggested (among other things) “meaning”. So, I searched for “definition vs. meaning” and read through the first 2 search results which seemed…meaningful…to me. The first indicated that while “definition” and “meaning” might be technically different, they are very closely related. The second had an answer that supported this idea and satisfied me.

    So, “how far can you extend a definition before it is no longer reasonable”? My answer is, “A definition can be extended to the point that the intended use of the definition is fulfilled.” That is, if my short, explicit statement (definition) does not convey (at least a very good approximation of) the idea that I am trying to convey to the audience I’m trying to convey it to, then I may need to extend the definition further to bolster its meaning by describing and explaining the idea. Just as you have done with your 18 “Testing is” statements, above. If 1, or 2, or 5 of these statements does not suffice…if they do not convey the idea that is trying to be conveyed (what testing is)…then there are more statements that can help describe and explain the idea further. And on and on, until the idea is approximately conveyed – and the intended use of the definition is fulfilled.

    There is another idea at work here. When we say “X is…” X can be many things; it can be many things at the same time; and those things can represent problems or contradictions with respect to each other. And yet those problems or contradictions don’t necessarily invalidate any particular idea.

    [What is your “elevator pitch” answer to “What is testing” to…]
    [I’ve given you 15 or so answers. Now it’s your turn; then we’ll see about mine.]
    For me, this was actually the toughest question to answer. I typed up the thoughts and process I went through as I attempted to answer it, but it became rambling and I deleted it. In the end, I decided that the definition of “testing” that you and James proposed in the “Testing and Checking Refined” blog post (http://www.satisfice.com/blog/archives/856) is still my favorite. I used this definition with the addition of a purpose, to help bolster its meaning, and approximately convey the idea more completely. And so, for 4 of the 5 audiences, my “elevator pitch” answer would be, “Testing is – among other things – the process of evaluating a product by learning about it through exploration and experimentation, which includes to some degree: questioning, study, modeling, observation, inference, etc. partly in order to inform a customer so that they can make more empirically informed decisions.” For the non-technical person, I imagined that they would probably know little/nothing about software development or even technology, at all.

    And some technical people might struggle, at first, with the word “empirically”. As someone once said, “You can’t win; you can’t break even; and you probably can’t even quit the damned game.”

    While the answer above might suffice, I decided to explain testing to them by using an analogy to something they were familiar with, instead. To them I’d answer, “Testing is – among other things – similar to investigative journalism. As investigative journalists deeply investigate stories and report their findings to consumers , software testers deeply investigate computer programs and report their findings to customer, as well.”

    Yes. I like that approach, and I love that metaphor in particular.

    I hope you find my mashed potatoes, asparagus, and apple pie satisficing.

    Yes. Okay, spectators: see what I meant about Damian going deeper with this?

    Reply

  6. To a question if someone ask what other defination testing can be!I would say it is a mix of art,craft,scientific and engineeting field.It can be related to craft and art as testing needs audience to appreciate ,test strategy,plans etc etc…..Lastly it can be called scientific and engineering discipline as testing deals with lot of mathematics,scientific information and also hypotehical constructs.However,this defination might be more generic.i hope someone might agree .

    Reply

  7. Testing is—among other things—learning, about things someone wants to know about.
    Testing is—among other things—finding out if wishes came true.
    Testing is—among other things—something you like when you have experienced it.
    Testing is—among other things—a messy business (from Ruud Cox)

    Reply

  8. Unfortunately, for many companies, time-to-market and “just good enough” are the driving principles behind their engineering schedules. Therefore:

    Testing is – among other things – resolving and/or mitigating only those product defects which would cause customer to ask for their money back.

    Michael replies: But testing is neither of those. Testing neither resolves nor mitigates defects. Testing might reveal problems and raise awareness and understanding of them. That awareness or understanding might inform decisions by developers or managers to mitigate or resolve problems; but awareness and understanding do nothing on their own without further action.

    Reply

  10. Testing is—among other things—questioning the product both literally and figuratively to help foster a sense of trust with aspects of it.

    Not sure of my use of the words literally and figuratively. They are supposed to represent interacting with the product in a real sense, and questioning through speech.

    Michael replies: Thanks for the comment, Mark.

    I like the notion of questioning the product as you’ve expressed it. I have some concern about fostering a sense of trust. As a tester, I’m not really interested in fostering a sense of trust. There’s usually plenty of that in the air. I’m interesting in finding problems; determining when trust is unwarranted.

    Reply

  12. I too wasn’t sure of the choice of word trust when writing this. I don’t think of trust as in certainty or building confidence, but more bridging the information gap between what we know and what we need to know.

    I’ve re-framed my initial definition:

    Testing is—among other things—questioning the product both literally and figuratively to help understand where trust is unwarranted.

    Reply

  13. Great blog post, and definitely thought-provoking.

    My answer would be (and this is something I’ve actually told people)…I see the testing role as similar to the CIA for a President. Our job ultimately is to provide information to decision-makers so that they can make good decisions about (1) where to put development resources (which bugs are out there? what should we fix first), and (2) whether a product is suitable for end-users/customers. Our job is NOT to make the actual decisions, but to provide information and recommendations.

    Michael replies: I appreciate the spirit of the reply. I might change “as similar to the CIA for a President” to as similar to the intended role of the CIA for a President”. Alas the organization has a history of going beyond the gathering of intelligence, so it seems.

    Reply

  14. […] Testing to me is never a dull job. It is playing to learn, exploring new stuff, crime scene investigation, solving puzzles, work with many different people and help people create beautiful solutions to complex problems. Maybe I need to explain what I think testing is. Testing is evaluating a product by learning about it through exploration and experimentation. Testers light the way by collecting information to inform decisions by people who make the product or decide about the product. As a tester people pay me to learn, how cool is that? Michael Bolton recently tweeted an interesting series of what testing can be. He collected his tweets in a must read blog post. […]

    Reply

  15. I like this blog post a lot. It really shows how complex testing actually is.

    What’s usually your first response if someone asks you this question, Michael? (if you had less than 30 seconds to answer 🙂

    It feels like there is something missing too, surrounding the different perspectives regarding different testing activities (done by different people) – E.g. what would testing mean to an SDET, or a developer reviewing code…

    Each testing activity is focused on different risks and because of the different contexts between each of the risks, some statements on what testing is might not be applicable. (I guess this rules out my earlier question of which statement would be your top response…..).
    Should the “type” (for want of a better word) of testing be detailed in each statement too, to give more context to the person that asked the question?

    Reply

  20. Your post is fantastic!

    It describes everything in detail about testing and can clear the doubts about testing for any person.

    Michael replies: Thank you for the kind words. I don’t think it describes everything in detail, though; and lots of doubts may remain. That’s my experience, at least.

    I would like to give my suggestions also. Testing is – among the other things- a process of validating and verifying that an application or program meets the business requirements or not.

    I’d prefer not to think of it that way, unless the focus is on “or not”. People tend to find what they seek. In general, you look to validate or verify, that’s what you’ll focus your attention on, at the risk of distracting yourself from problems. On the other hand, if start by assuming that there are problems in the product and—after a diligent search—you don’t find them, you get the verification for free. This is consistent with science: we can’t prove a theory to be correct, but we can disprove it.

    However, a clear automation strategy can ensure the right return on investment.

    Alas, I don’t agree for several reasons. A strategy might help to guide our testing, but neither strategy nor testing can ensure anything. Tools can help us to test, but tools only extend what we are. If we’re lousy testers, tools will help us do lousy testing faster and worse than we’ve ever done it before. Finally, testing is not an investment. It does not produce a return. This is similar to the way in which social media marketing is not an investment.

    If you mean that wise use of tools can help us test, that’s different.

    Reply

Leave a Comment