Back in the 1980s and 1990s, Quarterdeck Office Systems (later Quarterdeck Corporation)—a company for whom I worked—was in the business of creating multitasking and memory management products to extend and enhance to Microsoft’s DOS operating system. The ideas that our programmers developed were so good and so useful that similar ideas were typically adopted by Microsoft and folded into DOS a year or so later. After each new version of the operating system, people would ask us “are you concerned about Microsoft putting more memory management stuff into DOS?” Quaterdeck’s reply was always that, as long as Microsoft supported DOS, we would find ways to improve on memory management—and that we were delighted that Microsoft had legitimized the category.
I wasn’t lucky enough to attend Dr. James Whittaker’s presentation at EuroSTAR 2008, in which he described the concept of touring the software as a way of modeling and approaching exploratory testing. Fortunately, Dr. Whittaker has presented a number of these ideas as part of his recent Webinar “Five Ways to Revolutionize Your QA” on the UTest.com site, which came to my attention on April 1, 2009.
The touring metaphor in testing has been around for a while. I learned about it through James Bach’s Rapid Software Testing course, which I started teaching in 2004, and of which I’ve been a co-author since 2006. In 2004—that’s the first version for which I have my own copies of the course notes—Rapid Software Testing included several ideas for tours:
- Documentation Tour: Look in the online help or user manual and find some instructions about how to perform some interesting activity. Do those actions. Improvise from them.
- Sample Data Tour: Employ any sample data you can, and all that you can. The more complex the better.
- Variability Tour: Tour a product looking for anything that is variable and vary it. Vary it as far as possible, in every dimension possible. Exploring variations is part of the basic structure of my testing when I first encounter a product.
- Complexity Tour: Tour a product looking for the most complex features and data. Look for nooks & crowds where bugs can hide.
- Continuous Use: While testing, do not reset the system. Leave windows and files open. Let disk and memory usage mount. You’re hoping that the system ties itself in knots over time.
But the idea had been around before that, too. Tours were also mentioned in the Black Box Software Testing course, co-authored by James and Cem Kaner, which I attended in 2003. They were part of a larger list of test ideas called “Quick Tests”, which included other things like interruptions (starting activities and stopping them in the middle; stopping them at awkward times; performing stoppages using cancel buttons, O/S level interrupts, ctrl-alt-delete or task manager, arranging for other programs to interrupt, such as screensavers or virus checkers; suspending an activity and returning later) and continuous use (while testing, avoiding the resetting of the system; leaving windows and files open; letting disk and memory usage mount, hoping that the system ties itself in knots over time).
Note that concept of touring wasn’t terribly new in the BBST course notes and appendices either; skilled testers had been using them for a long while before . In 1995, Cem Kaner noted that the user manual is a test planning document; as he said in Liability for Defective Documentation, “It takes you on a tour of the entire program.” Elisabeth Hendrickson gave a presentation at STAR East in 2001 called “Bug Hunting: Going on a Software Safari”, which gave an overall list of test ideas using the metaphor of a tour. The idea of describing tours of a specific aspect or attribute of the product (namely the menu) appeared in an article by James Bach in the Test Practioner in 2002.
Much more serious work based on the concept of tours happened in 2005. Mike Kelly did some work with James Bach, and blogged some ideas about what they had discussed in an August 2005 blog post. Mike amplified upon that in his more complete list of tours (using the mnemonic FCC CUTS VIDS) in September 2005.
- Feature tour: Move through the application and get familiar with all the controls and features you come across.
- Complexity tour: Find the five most complex things about the application.
- Claims tour: Find all the information in the product that tells you what the product does.
- Configuration tour: Attempt to find all the ways you can change settings in the product in a way that the application retains those settings.
- User tour: Imagine five users for the product and the information they would want from the product or the major features they would be interested in.
- Testability tour: Find all the features you can use as testability features and/or identify tools you have available that you can use to help in your testing.
- Scenario tour: Imagine five realistic scenarios for how the users identified in the user tour would use this product.
- Variability tour: Look for things you can change in the application – and then you try to change them.
- Interoperability tour: What does this application interact with?
- Data tour: Identify the major data elements of the application.
- Structure tour: Find everything you can about what comprises the physical product (code, interfaces, hardware, files, etc…).
Dr. Whittaker does suggest some interesting notions of his own for tours in the UTest talk:
- Money tour: Test the features that users purchase the app for (which is rather like Mike’s “user tour” above, I guess)
- Rained-out tour: Start and stop tasks, hit cancel, etc. (rather like Kaner’s notion of “interruptions” above)
- Obsessive compulsive tour: Perform tasks multiple times, perform tasks multiple times, perform tasks multiple times
- Back alley tour: Test the least-used features
- All-nighter tour: Keep the app open overnight (like Kaner’s notion of “continuous use” above)
In his related blog post, “The Touring Test”, Dr. Whittaker says, “At Microsoft a group of us test folk from around the division and around the company are experimenting with tour-guided testing.” Cool. He also says, at the top of the post, “I couldn’t resist the play on Alan Turing’s famous test when naming this testing metaphor.” The idea that he named tours independently is a little surprising, but when we think about the practice of skilled exploratory testing, the “touring” metaphor might be obvious enough to have been arrived at independently. These things happen.
For example, in 2005, James Bach showed me a bunch of test techniques that he called “grokking”. (Grokking is a word invented by Robert Heinlein that describes deep, meditative contemplation and comprehension.) I thought “grokking” wasn’t the right name for what James was describing, because the tests depended extremely rapid cognition and removing information, the very opposite of reflective contemplation. I was reading Malcolm Gladwell’s book Blink at the time, and I suggested that we label the techniques blink testing. Only later, when I was researching the history of similar observational approaches for an article I was writing on blink testing, did I find a reference to astronomer’s tool from the 1920s: it was called a Blink Comparator. It was fun to note that discovery, a little sheepishly, in the article. So I can understand how it’s easy for people to use the same label for an idea.
But then something else came up.
In the Webinar for uTest.com, Dr. Whittaker also presents the concept of a “low-tech testing dashboard”, in which he suggests using a whiteboard and coloured markers to report on project status. This suggestion isn’t just a variation on the Big Visible Charts that are recommended in the Agile literature; it’s strikingly similar to an idea presented James Bach at STAR East in 1999 in a talk called “A Low-Tech Testing Dashboard“, posted on his Web site since around that time, and also part of the Rapid Software Testing course (pages 136-146).
I am delighted that authors as well-respected as Dr. Whittaker and that companies as prominent as uTest and Microsoft are endorsing and helping to spread ideas on tours and dashboards. I think they’re worthwhile approaches, and I believe that such endorsement helps in the wider effort to get the ideas accepted. Yet I also believe that it would be a friendly and respectful gesture if Dr. Whittaker’s presentation included acknowledgement of prior work in field that it covers. It would be similarly helpful if books like Dr. Whittaker’s How To Break Software or Page, Johnson, and Rollison’s How We Test Software at Microsoft contained bibliographies so that we could more easily find references to some of the ideas presented.
What does the community think? How important is it to acknowledge earlier work?
Honestly, I don’t think the community cares a lot about who was first to think of it. The community is just interested in the ideas, techniques, etc. rather than acknowledging the right person.
But personally I do think it is good practice and a matter of being well mannered, that authors mention their sources.
You mean, James Whittaker wont mind if I convert his book How to Break Software into a class without owing any credits to him and make good money, he wont mind?
The question I ask myself if I do that: What hidden message am I passing on to the community if I do so?
No, I’m saying the community probably won’t mind. And with community I’m thinking of people that are primarily interested in the content, like the attendees of your class.
Of course the community of fellow presenters and authors will be less thrilled.
I’m a fan of providing credit. I do my best to read what’s going on in the industry and make sure I can cite the correct sources when it makes sense to.
However, I will say, sometimes it can be difficult to get the right source. If Scott Barber introduces my to an idea on performance testing, but he cites some other work, then I often can’t remember who he cited. I’ll cite Scott.
I’d leave it up to the reader to work backwards from there. That is, it’s the reader’s job – if they care – to go open Scott’s paper and see who he cited. I think that speaks to Arjan’s point that most testers only care about the ideas. If I were in an academic setting, I’m sure I’d hold myself to a higher level of historical research, but I suppose I’m ok with citing the source where /I/ got the information.
It’s also difficult for topics I’ve been talking about or have been exposed to and using for a long time. When I talk about ET, it’s hard to cite. Not because I don’t know who did what (most of the time I think I know), but because it /feels/ like mine because I use it so much. Because of that, I often myself using blanket statements like “when working with James…” or “some time ago Kaner…”. It can be difficult to do it all the time.
That said, if you know it’s not your idea and you know where it came from, give the credit. Even if it’s just a footnote or a link.
In the immortal words for Robert A. Heinlein, “There ain’t no such thing as a free lunch.”
Everything comes at a cost, even plagiarism. If you want it in a word, plagiarism is stealing. The plagiarist robs the unwitting collaborator of their intellectual property, and profits from their work. There’s a reason we have bibliographies and works cited pages and foot notes and reference lists; it’s not up to the reader to determine where the author’s work came from. Think of it as a paper trail – I cite this blog, the blog cites other authors, those authors cite earlier works and so on. I’m not suggesting that the author has to cite the original source of the idea because that isn’t always certain; I’m merely stated that you shouldn’t present someone else’s work as your own, or fail to mention where you got the idea.
Until I read this blog, I could only assume that Touring and low-tech dashboards were Microsoft inventions.
At any rate, I believe that honesty and integrity are very important, and should never be compromised for the sake of perpetuating a useful idea. Just my 2 cents.
It’s a good habit to cite where you’ve got an idea or even just an inspiration from. However, sometimes, it’s not practically possible to find the right name.
And we also do not want to cloud the message completely by injecting a full path back to the original idea or result at every mentioning.
But you can *always* include a phrase, wording or hint, that this is something you’ve been inspired to by someone else, and doing so shows you’ve got manners and integrity.Speaking of Microsoft in particular, it appears that most of what they do is invented by others: don’t we all ‘know’ that windows was an apple invention ? (I also remember something called ‘Gem’ that was out prior to windows 3.1). Word and Wordperfect (wordstar).. etc etc – sometimes I think of what they actually did themselves, apart from DOS, of course – no wait – wasn’t there something called unix at the time ?!? Early DOS shows many similarities.. 😉
I guess we in our business reinvent the wheel all over time and again. It’s difficult to keep track. And ultimately, who can write a book or a lecture or class or anything today without admitting to be, (quote Newton:) “standing on the shoulders of Giants” ?
Not too long ago I read a most creditable story that someone patented the wheel recently in Australia (http://digg.com/d13xYh – believable ?). I guess, that for those without integrity and manners – honour of invention is up for grasps.. unfortunately..
One of the lines of experimentation that I keep hearing about, when people tell me about tours, is research that purports to show that Tour X is better than Tour Y. Maybe the Complexity Tour is better than the Fed-X tour. Maybe the Variables tour is better than the Error Message tour.
I heard that again, recently, as a comment on a set of lecture notes I’m creating for a course on Test Design. (And again, in a related context, today–which motivated me to finally publish a comment.)
My course (on video) starts with a demonstration of a feature tour and then gives an inventory of tours that I’ve learned from Bach, Bolton, Kelly and Hendrickson over the years. Why, the commentator asks, do I not explain which tours are better and which are worse.
And indeed, there are people who believe that Research Has Been Done that answers this question to some degree, and that More Research Could Be Done to explore it more thoroughly. An enterprising researcher could probably sell this idea to a corporate research department or even a scientific-research funding agency, on the grounds that we might be able to establish some Best Practices for Touring. I’m sure this can be dressed up and made to sound impressive.
However, I think this is a shallow and unproductive line of work. I have never heard it from anyone who (in my view) knows much about touring. Or to put it differently, anyone who suggests it is telling me that they don’t understand what tours are or why we do them.
A tour is a directed search through the program. Find all the capabilities. Find all the claims about the product. Find all the variables. Find all the intended benefits. Find all the ways to get from A to B. Find all the X. Or maybe not ALL, but find a bunch.
This helps the tester achieve a few things:
(1) it creates an inventory of a class of attributes of the product under test. Later, the tester can work through the inventory, testing each one to some intended level of depth. This is what “coverage”-oriented testing is about. You can test N% of the program’s statements, or N% of the program’s features, or N% of the claims made for the product in its documentation–if you can list it, you can test it and check it off the list.
(2) It familiarizes the tester with this aspect of the product. Testing is about discovering quality-related information about the product. An important part of the process of discovery is learning what is in the product, how people can / will use it, and how it works. Tours give us different angles on that multidimensional learning problem.
(3) It provides a way for the tester to explain to someone else what she has studied in the product so far, what types of things she has learned and what basics haven’t yet been explored. In a field with few trustworthy metrics, this gives us a useful basis for reporting progress, especially progress early in the testing effort.
So which tour is better?
From a test-coverage perspective, I think that depends a lot on contract, regulation, and risk.
(i) To the extent that you have to know (and have to be able to tell people in writing) that all the X’s have been tested and all the X’s work, you need to know what all the X’s are and how to find them in the program. That calls for an X-tour. Which tour is better? The one you need for this product. That probably varies from product to product, no?
(ii) Some programmers are more likely to make X-bugs than Y-bugs. Some programmers are sloppy about initializing variables. Some are sloppy about boundary conditions. Some are sloppy about thread interactions. Some are good coders but they design user interactions that are too confusing. If I’m testing Joe X’s code, I want to look for X-bugs. If Joe blows boundaries, I want to do a variable tour, to find all the variables so I can test all the boundaries. But if Joe’s problem is incomprehensibility, I want to do a benefit tour, to see what benefits people _should_ get from the program and how hard/confusing it is for users to actually get them. Which tour is better? The answer is “yes”.
(2) From a tester-learning perspective, people learn differently from each other. If I set 10 people with the task of learning what’s in a program, how it can be used, and how it works, those people would look for different types of information. They would be confused by different things. They would each find some things more interesting than others. They would already know some things that their colleagues didn’t. Which tour is better? The tour that helps you learn something you’re trying to learn today. Tomorrow, the best tour will be something different.
Testing is an infinite task. Define “distinct tests” this way: two tests are distinct if each can expose at least one bug that the other would miss. For a non-trivial program, there is an infinite number of distinct potential tests. The central problem of test design is boiling down this infinite set to a (relative to infinity) tiny collection of tests that we will actually use. Each test technique highlights a different subset of this infinity. In effect, each test technique represents a different sampling strategy from the space of possible tests.
Tours help the tester gain insight into the multidimensional nature of this complex, infinite space. They help us imagine, envision, and as we gain experience on the tour, prioritize the different sampling strategies we could use when we do more thorough, more intense testing after finishing the tours.
So which tour is best? The ones that give the testers more insight and that achieve a greater stretch of the testers’ imagination. For this, some tours will work better for me, others for you.
The best tour isn’t necessarily the one that finds the most bugs. Or covers the most statements (or take your pick of what coverage attribute) of the product. The best tour is the one that helps the individual human tester learn something new and useful. (That’s why we call it exploration. New and useful knowledge.) And that depends on what you already know, on what risks and complexities characterize this program, and what the priorities are in this project.
A tour that is useful to you might be worthless to me. But that doesn’t stop it from being useful for you.
Rather than looking for a few “best” tours, I think it would be more interesting to develop guidance on how to do tour X given that you want to. (Do you know how to do a tour of uses of the program that trigger multithreaded operations in the system? Would it be interesting to know how?)
Rather than looking fro a few “best” tours, I think it would be more interesting to develop a more diverse collection of tours that we can do, with more insight into what each can teach us.
Rather than seeking to objectify and quantify tours, I think we should embrace their subjectivity and the qualitative nature of the benefits they provide.
Pseudo-academics and best-practicers trivialize enough aspects of testing. They should leave this one alone.
Michael replies: Thanks for the comment, Cem.
[…] few years ago, Michael Bolton wrote a blog post on “Testing Tours and Dashboards.” Back then, it had recently become fashionable to talk about “tours” as a […]
[…] I saw him recording ‘test ideas/cases’ as he was exploring. This immediately took me back here – A very nice post by Michael Bolton on Testing Tours. What made me even more #soproud was the […]
Some of the first testing books I read were James Whittaker’s including his book on Exploratory Software Testing. The material was new to me at the time but I later found out it wasn’t the most original nor did it provide much in depth information about the origination of the term and how it was truly applicable. Instead it seemed to say to the reader: these tours are the best, don’t you think? (I dont remember any mention of Mr. Bach.)
I read Whittaker’s blog posts about a Tester Dashboard, again not until I read this post did I realize that was also a Bach idea. It seems weird that someone who has a PhD. and had to do a dissertation wouldn’t naturally reference sources of ideas or inspiration for ideas. (Steve Jobs was known for stealing ideas and making them his own, so it’s not unheard of.)
Looking back you can see the lack of references in most of his books; perhaps the market he is targeting doesnt care? That doesnt make it right mind you. Acknowledging earlier work doesn’t seem like it would take that much extra effort and pays respect to those who came before you.
[…] can visualise the areas they are talking about. Michael Bolton wrote a cracker of a blog here, (http://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards/) and also take a look at the comments, specifically Cem […]
Michael says: This pingnback came from http://www.theagiletribe.net/index.php/2011/12/20/the-quick-and-the-dead/, which now appears to be… dead.
[…] Bolton, M. (2009). Of testing tours and dashboards. http://www.developsense.com/blog/2009/04/of-testing-tours-and-dashboards […]
[…] Michael Bolton’s reaction to Whittaker’s research and book. […]
[…] 3 explains many different types of mobile testing tours (if you haven’t heard of these, try here and here as a start pointing). I had done some research into testing tours prior to reading this […]
[…] test. Other great sources of tests idea’s are the domain testing workbook from Cem Kamer, testing tours, the heuristic test strategy model, You Are Not Done […]
[…] tours in the mentioned books, as well as lot of literature about which’s the best tour, keynotes from testing mesias, or implementation examples (in this case, from Google). Have you applied any of these tours […]
[…] Initially any observation of the product gives us information that requires us to change our mental model to be able to explain the product. Any new observed element will be added to the model. Within just a few moments “random” observations will not be a very efficient source of information to build, refine and refactor our model. At this stage a more systematic approach to observing is necessary. One popular such approach is touring the software. […]
[…] kicked off Untapped feature tour […]
[…] also did my hunt. My charter was to do the feature tour. When you first hit the application, there were instructions (only the first time, that was […]
[…] This insight is invaluable for the testers to understand how the user would interact with the feature once it has been developed. They can form realistic scenarios when they come to explore the system and run their testing tours. […]