Questioning Test Cases, Part 2: Testers Learn, But Test Cases Don’t Teach

In the last post, my LinkedIn correspondent provided a couple of reasons why she liked writing test cases, and why she thought it necessary to write them down in the early stages of the project. Then she gave a third reason:

When I’m on a project and I am the only one who knows how to test something, then I can’t move on to something new. I’d still be testing Cut/Copy/Paste if I were not able to pass that script on easily to a new recruit.

I often hear this kind of justification for test cases: “We need test cases so that testers can learn quickly how to test the product.” To me, that’s like saying that we need driving cases so that people can learn quickly how to drive in a new city. In a new city, a map or a general set of ideas might be helpful. Driving cases won’t, because to drive through a new city well, you have to know how to drive. In the same way, test cases are unlikely to be anything more than negligbly helpful. In order to learn quickly how to test a new program well, you have to know how to test.

It also seemed to me that she considers writing to be the One True Way to impart information, and reading to be the One True Way to receive it. Now, if I’m wrong about that, it would have to be because I’ve misinterpreted something she’s written. Fancy that!

So for the sake of discussion, let’s say I were to to hire (say) an older domain expert, making a career change, who had no computer experience, and let’s use Cut/Copy/Paste as an example. With the new recruit, I’d do what I’ve done many times in coaching sessions. I’d sit with her down with a copy of Microsoft Word, and open up a document that had some text and some images in it. Then I’d say something like this:

“We’re going to learn how to move text around, cutting and pasting and making copies of it so we can save time on retyping stuff that the machine can do for us. Okay. Put your cursor there, and press Shift-RightArrow there. Good. See what happened?”

(There would be lots of pauses. I’d pause after each question to give the trainee time to answer; frequently to let the trainee experiment; and often to let the trainee ask questions of her own. I won’t do that explicitly here; I’ll let you put most of the pauses in yourself, as you read. For sure, assume that there’s one at least after every question, and usually after every sentence.)

“So try doing that a few more times. What happened? So you’ve highlighted the text; people call that marking or selecting the text. What happens when you add the Ctrl key, and press Ctrl-Alt-RightArrow at the same time? Right: it selects a word at a time. Now, try pressing Ctrl-X.

“Whoa, what happened just there?! Right: pressing Ctrl-X that makes the selected text go away; you can see that. What you might not know is that the text goes onto the Windows Clipboard, a kind of temporary storage area for all kinds of stuff. So you haven’t erased it; it’s more like you’ve cut it out, and the computer still has a copy of it stashed somewhere.

“Now, put your cursor here, and hit Ctrl-V. What did that do? Yeah; it pasted the text you cut before. That is, it retrieved the text from the clipboard, and made a copy right where you pressed Ctrl-V. So, Ctrl-X cuts, and Ctrl-V pastes a copy of the thing you cut.

“Try that again. What happened? Right; another copy. So it’s a little different from regular cutting and pasting, in that you can make many copies of the same thing. When might that come in handy? Good idea! Yep, that’s a good idea too. Try it a few more times, in other places.

“What does Ctrl-V do? What does Ctrl-X do?

“Try selecting something else, and pressing Ctrl-X again. What happens when you paste now? Can you get the old one back? Well, there might be special add-ons that allow you to keep copies of lots of things you pasted, but normally, you can only paste the last thing you put on the Clipboard.

“How are you going to remember that Ctrl-X means “cut”? That’s a great idea! Myself, I think of the X as a pair of open scissors, ready to cut—but your idea is great, and better than that, it’s your own, so you’ll remember it.

“How are you going to remember Ctrl-V as paste? Yeah, it’s tough to come up with a way to remember that, isn’t it? I think of it like the tip of one of those old glue bottles from grade school. On the other hand, what’s in between X and V on the keyboard? Now, go select some text again. Remember how? Good.

“This time, try pressing Ctrl-C, instead of Ctrl-X. Right; nothing seems to happen. And yet… move your cursor here, and try pasting. Yeah; it’s a copy, even though the original stays where it was. So Ctrl-C is Copy–C for copy, which makes some kind of sense, for a change—right there in between cut and paste. And there’s another way to remember; try pressing Alt-E. E stands for Edit, that’s right. What do you see listed on the menu, there?

“Now… there’s some documentation for these cut-and-paste features; it’s in the Windows documentation. Press F1. Great. Now look for ‘Keyboard Shortcuts’.

“There’s a list. How many different ways can you find of marking, cutting, and pasting text? Make me a list, and I’ll be back in ten minutes. Be ready to show me your report, and to demonstrate the ones you’ve discovered. Oh—and one more thing: create a list of at least five circumstances in which you might use this stuff, and we’ll talk about that too.”

It’s taken quite a while to type this example. It would take considerably less time to do it. Moreover, the trainee would learn more quickly and more deeply, because she’d be interacting with the product herself, experiencing the actions in her body, literally incorporating them. The questions, the answers, and the interactions make the learning more sticky. The practice makes it more sticky still.

I try to encourage people to create mnemonics to help remember. While conversing with the trainee, I’d also be observing her. She’ll be making plenty of mistakes; people do that when they’re learning. In fact, people should be encouraged to do that when they’re learning. (Brian Marick’s book Everyday Scripting in Ruby encourages deliberate mistakes, which is one of the reasons I like it so much.) When I see mistakes, I’ll give her a chance to feel the mistake, puzzle out a way around it, and then—if she needs it—help her get out of it. (“Let’s figure out how to fix that. Hit Alt-E again. What’s listed on the Edit menu? So what’s the keyboard shortcut for Undo?” “Oops, you pressed Ctrl-Z one time too many… What does Ctrl-Y do?”).

After a while, I won’t tell her how to do everything right away. I’ll start asking her to figure out some things on her own, and only help out when she’s stuck. That way she’ll be learning it for herself, which is more challenging but which has more effect. I’ll keep trying to raise the bar, giving her more challenging assignments, increasing her independence while also increasing her responsibility to report. When I see her make a mistake, I might correct her right away, or I might let her continue with the mistake for a bit until she encountered some kind of minor consequence. That makes the learning even more sticky.

Finally, I’d cut the conversation short if she told me that she already knew how to cut and paste—but I’d get her to show me that she knew it, and I’d give her some more exercises so that I could observe her.

The trouble that I see with “passing on a script to a new recruit” is that most test scripts I’ve seen are very long on what to do or how to do it, and correspondingly short on why to do it and how to generalize it. That is, they’re short on motivation, and they’re short on risk, and they rarely include a mission to learn. In our teaching, when we’re working with test groups, and when we’re coaching individual testers, James and I put less emphasis on techniques. Instead, we focus on making sure that people have the skills to know what to do when they receive a testing mission, whether that mission is written down in detail or not.

If there are specific things that I want a tester to look for, I’ll provide specific instructions (either written or spoken), but mostly I want to set out on their own to discover things. I want to encourage testers to vary their work and their observations. Variation makes it more likely that they will find problems that would be out of sight if we were to focus on the scripts. As a bonus, giving testers a concise, open-ended task keeps writing to a minimum, which saves on development cost, maintenance cost, and opportunity cost.

We’ll finish up this series in the next post.

10 replies to “Questioning Test Cases, Part 2: Testers Learn, But Test Cases Don’t Teach”

  1. Good stuff that needs to be continually broadcast till people get the message. It’s basic, in the sense that it’s of fundamental importance. Documentation doesn’t get the job done. You need an intelligent understanding of the problem that can be applied and refined. It should be obvious, but sadly it isn’t.

    What really gets me about the need for skiploads of documentation is that it comes at a huge price. The time I’ve had to spend pointlessly perfecting stuff that wasn’t needed could have been better spent understanding the application, the business problem and the technology. I’ve sometimes had the dispiriting insight that I knew my way round the documentation far better than I knew the application.

  2. Adding motivation to a test directive is a skill I haven’t really attempted to hone. I’m thinking I should start. In my current situation this will mean more description and motivation added to test cases (and perhaps making the test cases with steps less explicit). The challenge will be on how to translate my experience and domain knowledge in a fairly specialized field from my brain to the written word or a training session. I fear I’m a better tester than a teacher. Another skill I to acquire/improve.

    How about supplementing the written word with drawings, tables, diagrams, flowcharts, photos, references, and above all, conversation?

    Why do you feel that you require more description in the test cases? Could the description somewhere else? In a separate document? In a training session? In a cycle of briefings and debriefings, such that the tester (not you) takes notes? In the skill set of the tester?

    One nitpick: This isn’t the first time I’ve heard you compare testing to driving with respect to test cases/scripts and each time it grates. I think I understand your motivation, but the analogy does nothing to convince me. I know no analogies are perfect but I can’t get over (my perceived) imperfections.

    For the moment, I’d like you to think very carefully about what a perfect analogy would be. Then, below, I will address the issue of imperfections.

    * If the person already knows how to drive but is unfamiliar with a particular city, learning would definitely involve a map, but also specific directions on getting from point A to point B.

    That’s not necessarily true. Imagine the instances in which you’ve given instructions, or received instructions from other people, to get from one place to another. If you’re aware that the roads are clearly marked, you might only provide a map for part of the journey. You might not provide a map at all, but a list of written instructions. You might deliver instructions verbally, and let the driver take the notes that he or she perceives that she needs. Depending on the driver, you might choose one style of instructions over another. When you know a driver is very self-sufficient, an address might do.

    In addition, the level of specificity is an open question. You provide specific instructions to the exact degree that you’re worried that the person might not know what to do, or might know what to do and forget, or might encounter some risk along the way. If you’re like most people, when you’re giving someone driving directions, there’s a ton of detail you’re leave vague, or leave out altogether. You might use fungible units, advise them to follow a certain road “for about ten minutes” or “about ten miles” when you know they’re driving at highway speed. You might include a significant amount of detail, but consider: how much detail don’t you include? If a person already knows how to drive, would your instructions include checking the mirrors while reversing into a parking space? Suppose you included instructions to park at a particular parking lot; what would happen if that parking lot were full? What would you expect of an experienced driver?

    * Especially if the goal is cover as many roads as possible.

    Perhaps. But I perceive you’re making a basket of assumptions here, and I think you might want to reconsider some of them. Whatever your map of the product, I will guarantee you that the map is inaccurate in many respects. That’s why we test: because the products and our maps of them don’t agree. The object of testing is not only to cover the map, but to construct the map where it’s missing bits, and to invalidate it where we think it’s accurate. To find out what’s really there in the product, what’s absent, what’s dodgy, what’s congested, what’s ill-considered. We not only want to cover the roads that are there, but to discover the risks and to discover where there are unexpected roads. To the extent that someone wants to understand something new about the product, or understand what we’ve actually done in testing, I think we should be emphasizing recording what we’ve done, rather than taking enormous amounts of time to identify, in detail, what we hope to do.

    * Once the driver is more familiar with the city, the specific directions wouldn’t be needed (or only needed when the city developers add a new subdivision or change up the roads without providing a map).

    Now you’re talking. Except a great way for the driver to get familiar with the city is to build his own map, with the added benefit that he can then compare it with yours, so that you can see what’s missing on your map, or so that you can get an idea of how he’s exploring and accounting for his work.

    * If the person doesn’t know how to drive and they are in a new city, the same thing applies, but there would be even more guidance and specific instructions on how to do things (e.g. changing lanes and parallel parking). Hopefully the first few sessions are in a parking lot 🙂

    One point is that you don’t send a novice driver unaccompanied, untrained, and unsupported out driving at all, never mind to an unfamiliar place. Another is that you cannot (or at least should not) expect a script to stand in for direct, personal supervision, observation, and evaluation. Finally, If I can’t trust the person to drive, I can’t trust them to drive in an unfamiliar place, and I certainly can’t trust them to drive while reading a script. If someone doesn’t know how to change lanes, I’m not sending them out alone. Period. Correspondingly, I’m not going to waste my time and insult the intelligence of an experienced driver by giving them far more detail than they really need.

    On the other hand I may be completely missing the point of the analogy.

    You’re not missing the point of the analogy; I’d argue you’re missing the point of analogy altogether. Metaphor and simile are associative speech. The purpose of associative speech is to get you thinking. With respect to “imperfection” of the analogy, you are confusing comparison of a handful of factors with comparison of all factors. You are applying a kind of binary fallacy, thinking that a comparison between two things need be complete and absolute. It needn’t.

    Please try to understand this: If I were to tell you that my dog were like my car in that they both require a lot of maintenance, would you expect my dog to have a gas tank? Would you expect me to take the dog to my mechanic when it appeared ill? If I said that picking up the guitar again after a couple of years was like riding a bike for me, would you envision a musical instrument with wheels and handlebars? If I were to suggest that my former boss was a pain in the ass, would you expect everyone at work to have trouble sitting down? If I said that a friend is as honest as the day is long, would you think that I measured honesty in hours? If I said that your newer testers were green, would you contradict me because they didn’t match a certain Pantone colour?

    Do you know why analogies are not perfect? It’s because analogies are intended to make people think about the ways in which something is like something else, while intentionally ignoring the ways in which they’re different. That is, they’re not supposed to be perfect. They’re supposed to draw your attention to some factor of something, so that you can consider that factor in a new or different light. We use metaphor all the time. For example, notice that I can use that expression, “in a new or different light”, and you don’t get confused or think “Hmmm… incandescent or flourescent?”

    Similes and metaphors are a form of modeling. A model is a representation—literally, a re-presentation—of something in terms of something else. Typically, we use models to frame complicated or unfamiliar things that we don’t understand in terms of simpler or more familiar things that we do understand. They are imperfect; they’re designed to be imperfect. They’re intended to be imperfect. Were there a one-to-one correspondence between every aspect of a model and the thing being modeled, there would be no simplification and no point in using the model. It would be like Steven Wright’s full scale map of the United States: “It took me all summer to fold it.”

    Metaphor and simile trigger the capacity of the human mind to forge new links between ideas. Look above: you complained about the simile, and then proceeded to use it to build your argument. The simile did its job.

    I’m looking forward to part 3 and to your talk at the Targeting Quality conference.

    Thanks. Me too.

    —Michael B.

  3. I have been fighting excessive test documentation on my projects for many years. Sadly, I often find the greatest resistance comes from testers themselves rather than from management. Yet those same testers will tell me proudly that they often “go off script” and that’s how they find the most important bugs.

    Michael replies: Yes; a great observation. Old habits die hard, and awareness of the contradiction you’re describing seems to take time to grow.

    I don’t get it, but it’s an entrenched culture that goes right back to the evolution of software testing — which is frequently mirrored in the evolution of an individual tester. Many of us started out doing exploratory testing because that was the only thing we knew. Then someone came along or we read a book, and we discovered we weren’t doing testing “properly” because our explorations weren’t repeatable or structured in a conventional way. Some of us learned that stuff, reaped a little benefit but also learned the limitations of Big Documentation the hard way and got past it: back to exploration. But the testing world as a whole has got itself stuck in the Big Doc mold.

    Mold—are you referring to a gizmo of a certain shape that lends its structure to something fluid, or to growths that do a have a system function but that thrive in dark, damp places and are often involved in decay or disease? Probably both.

    I was, for a time, subject to the history you describe. The principal thing I learned was that Big Documentation was a great idea, and the illusion of control made me feel better, but that it didn’t work. So now I think—and very occasionally say to people—”big documentation is a phase you’re going through.” It’s better, I think, to trigger the recognition, because people in the throes of Big Documentation Belief will only learn by experience and feeling the sting that comes with failures. Like, I hasten to add, the rest of us.

    Like project management process ceremony on some projects, this testing process ceremony takes on a life of its own. It becomes a separate system that has little to do with moving a project forward to produce useful software, and everything to do with providing the illusion of control.

    Yes. Once not too long ago, I spoke too quickly. Jerry Weinberg was in the room, and corrected me. “We’re supposed to be a service to the project, not an obstacle,” I said. “Our role is to give people information as quickly and as accurately as possible to help them assure the on-time completion of the project.” Jerry peered at me over the glasses and said, “the successful on-time completion of the project.”

    Nice to see you here; thanks for writing.

  4. Your post reminds me the times when I was being introduced to any project at the first time. In my previous organization I still remember that the first project of my career had been explained by one senior Tester and I was supposed to replace him.

    We sat in front of the computer and he explained the flow and modules of the product verbally and then I started exploring the product and was questioning him anything I didn’t get. There was no documentation at all for that project.

    it’s a great example, but was there no documentation at all? Were there prior bug reports? User manuals? Marketing materials? Comments in the source code? And did you take notes or draw sketches as you built your understanding? Did you generate data? Collect or develop tools? It helps, I’ve found, to think expansively of what might be available, or what we might be generating.

    After some months I was supposed to be transfered to another project, so before that they asked me to write a Project Manual which can later be used by a Project analyst for an upcoming changes in the same project.

    I wrote an oracle—a spreadsheet that would provide a comparable result for certain transactions—a few years ago, for a big bank. As part of the documentation for the oracle, I included some notes on how to think about the transactions generally. It was the kind of document that I wished that I had had when I started on. Yet I’ll probably never know for sure who used that document or who found it helpful—except that the writing of that background information was helpful to me. That documentation wasn’t test cases, but it helped me to solidify ideas on how to design and perform tests in whatever format.

    Second instance in which there was an urgent need to get involved in one project as the developers were complaining about the previous tester who was not submitting proper information. So again here too I sat with that tester and he explained me the product verbally.

    What was shocking here for me was, when I went through some existing functional documents, I found that the current functionality of one module was totally different than what was originally asked by a client/written in those documents. We then had to arrange a meeting.

    Yes. Alfred North Whitehead once said, “Seek simplicity—and learn to distrust it.” The same can be said usefully for documentation, I think.

    In the final case when I had resigned from the job and I need to train a new tester who was supposed to replace me. I didnt have a time even to train him separately as I had a lot of things to complete before I leave. Everyday I used to call him when I was actually testing the AUT so I can save the time and can also introduce him to the project. What I learned here is inbetween the process of explaning him, I was also getting more ideas of different testconditions and could look the AUT with different angles.

    Yes. Explaining something in whatever way we’re rendering the explanation is a trigger to our own learning, I’ve observed.

    Here my question is, what should be the approach of a New tester who is being introduced to the project first time and another tester is handing over the project to him/her? Sometimes the new testers want everythings to be explained and do not try to think on thier own initially and Company officials also wants you to make sure that you pass or document every little information before you exit the project or an organization.


    I think the new tester should work in whatever way suits him/her, the experienced hand, and the project overall. More to come in later posts. Thank you for contributing your experience here; that’s valuable.

  5. With respect to my critique of the analogy, you are, of course quite right. I read more meaning into it than was appropriate, taking it too far. This isn’t the first time I’ve had difficulty with an argument from analogy. I wonder what that says about me…

    Michael replies: To me, it says at least two things: 1) It says that you’re aware of the issue, and you can learn with practise. 2) That, irrespective of your difficulty, you actually use the analogy to reason your way through things quite well (as you demonstrate below). The key, I think, is to think of analogy as a means of expanding your models.

    I’ll say no more about the analogy beyond this: I use directions all the time (mainly google map direction printouts – they litter my car floor). I also use maps. When I am on vacation I even pay guides to tour me around and highlight interesting things. I feel like I am always learning about the places I go, with each discovery method teaching me differently than the others.

    I think that last sentence is most important: there are many different ways to approach a learning assignment—and that testing is a learning assignment, both for us and for our clients.

    You said: How about supplementing the written word with drawings, tables, diagrams, flowcharts, photos, references, and above all, conversation?

    Yes, supplemental information is critical. In our context, this includes relevant parts of standards, industry guidance and sometimes UI mockups, among other things. Our test case repository system makes this a using auxiliary information while executing a test little clunky, but not overly.

    Conversations about test cases help immensely, especially around functionality that is unclear or complicated.

    Aside: Not relevant to communicating one tester to another, but I spend 4 solid days last month sitting in meetings where the two project developers hashed out how new functionality was going to be designed. I wrote down test ideas for as much stuff as I could, occasionally popping into the conversation with questions like: “So if you implement it like that, will I’ll be able to verify it is working by running scenarios A, B, and C?”. I don’t have the functionality to test yet, but I’m really excited to see how working with the developers are such an intimate level will effect the quality of the code come release date.

    Sounds like a great experience. I hope they’d say the same. Keep us posted!

    You said: Why do you feel that you require more description in the test cases? Could the description somewhere else? In a separate document? In a training session? In a cycle of briefings and debriefings, such that the tester (not you) takes notes? In the skill set of the tester?

    Some of our test cases that have no description/synopsis or overview, just a list of steps and the anticipated behaviour. I’ve created some of them and inherited others. The automated tests (checks) are the worst – I don’t know what the test case is doing let alone its motivation! In this situation, I have a hard time determining if the test case is worth running. Adding a What (and a Why) is another tool to help me answer and be accountable for the If and When.

    I like clear code, but I have to recognize that “clear” is subject to the Relative Rule: “clear” means “clear to some person(s), at some time.” There are people who claim that the code of an automated check should be self-documenting. A possible pitfall I’ve noticed when I write self-documenting code is that I understand it—myself. That doesn’t guarantee that anyone else will understand it. Pairing helps with that. Review helps with that. And the occasional comment helps with that too.

    I could put the description somewhere else; to be honest I’ve never questioned their location other than with the test case. It could go in a training session with the training session notes “linked” with the test case. It could also go in briefing and debriefing session notes (which would mean we would need to start briefing and debriefing sessions).

    Something to ponder: I wonder how much it could go into the culture of the project.

    This blog edifies without fail. Thank you for that.

    If that’s so, it’s at least in part due to the quality and thoughtfulness of the comments. So you’re welcome, and thank you.

  6. There is at least one context where writing detailed test cases may (unfortunately) be necessary. If your testing results are audited by an outside party, they will likely require a certain amount of detail with regards to how you generated the results listed. That being said, I’m not a fan of spending a huge amount of time on that documentation, so I’m trying to figure out how to meet the needs of auditors, while at the same time reducing the time spent on writing test cases.

    How about working with the auditors? When managers ask, “Why would you do that?”, try answering, “It might save time and money to discover what the auditors are really looking for, so we can save time on documentation and apply the savings to testing.” If they object, ask if they’re aware of “The Least Burdensome Provisions of the FDA Modernization Act of 1997: Concept and Principles; Final Guidance for FDA and Industry“.

  7. Matt – as a former IT auditor I’ve got to say that Michael’s correct. Good auditors should be looking for evidence of test results, not test documentation. Of course there are plenty of bad auditors around, but I think that’s usually a result of inexperience or an unhealthy corporate culture. In any case the best response is to speak to them as Michael suggests.

    If you want some more ammunition you could tell them to look at the Sarbanes Oxley guidance material from the Information Systems Audit & Control Association. It’s available to members only on their website. Get in touch with me if you want a copy.

    The Institute of Internal Auditors has also provided guidance on Sarbox.

    Sarbox may apply only to the USA, but its requirements are fairly stringent, and you’d have a very strong argument if you could show that not even it requires detailed test scripting. The trouble is that Sarbox has the reputation for demanding extensive documentation, and some auditors go by the reputation rather than the legislation.

    Or look at this article by Kanwal Mookhey in the Internal Auditor magazine.

    And this article I wrote, in which Kanwal gave me some more explicit quotes on this subject.


Leave a Comment