DevelopSense Blog

Automation Bias, Documentation Bias, and the Power of Humans

2009-06-30T12:56:00.003-05:00

A few weeks I went down to the U.S. Consulate in Toronto to register Ariel, my daughter, as an American citizen born abroad. (She's a dualie, because she was born in Canada to an American parent: me. I'm a dualie too, born in the U.S. to Canadian parents. Being born a dual citizen is a wonderful example of a best practice. You should follow it. But I digress.)

The application process is, naturally, fraught with complication and bureaucracy. There's also a chilling and intimidating level of security; one isn't allowed to bring anything electronic into the Consulate at all. No cell phones, no PDAs, and certainly no laptop computers. That means no electronic records, and no hope of looking anything up. So one has to prepare.

There's a Web site for the Consular services. One of the first items that one sees on the site is a link for telephone inquiries. Note a couple of things here: the telephone services are for visa information, not for general information; and that visa information costs USD$0.90 per minute for a recorded system with no operator. (Oddly, that's the price for calls from the U.S.; calls from Canada are cheaper, at CAD$0.69 per minute.) I didn't test that.

With only a little digging, I was able to find information related to registering a birth abroad. I gathered the information and documents that I figured I needed, and took it all down to the Consulate. I was getting ready to travel the next day, and so in typical fashion, I pushed things out to the noon deadline for receiving applications. I watched the clock on the car anxiously, parking at 11:53 and getting to the Consulate at 11:55. "Wow, that's pushing it," said the security guard. "Last one today."

When I spoke to the friendly, helpful lady behind the counter (I mean that; she was genuinely friendly and helpful) she they told me some things that the Web site didn't.

The application form itself is online, and these days it's one of those PDFs that has input fields, so everything can be nice and tidy. Again, though, there are some fields in the form that have several possible answers. There is some helpful information available, but I still had questions.
The consular officers want to see original documents, but accept and keep only photocopies of them. You need to come with your own photocopies. If you don't, it costs you $1.00 per document—and there are lots of documents. This isn't noted anywhere on the Web site that I could see.
On one of the Web pages listing documentation requirements, it says "In certain cases, it may be necessary to submit additional documents, including affidavits of paternity and support, divorce decrees from prior marriages, or medical reports of blood compatibility." Well, what cases? The page doesn't tell me, and getting it wrong means an extra trip. The lady behind the counter reviewed what I had brought, answered a number of questions, and told me exactly what to bring next time.

As I travel around, I sometimes see an implicit assumption that documents tell us all we need to know. Yet documents are always a stand-in for some person, an incomplete representation of what they know or what they want. They're time-bound, in that they represent someone's ideas frozen at some point in the past. They can't, and don't answer followup questions. As Northrop Frye once said, "A book always says the same thing." Yet if we look more closely, not even ideas that are carefully and thoroughly debated can be expressed unambiguously. That's why we have judges. And lawyers.

The next thing that happened emphasized this. After I left the Consulate, I returned to my car. At the collection booth, the posted time was 12:20. I'd been less than half an hour, which is good because parking at that garage costs $3.00 per half-hour. I handed the attendant my ticket. The charge was $6.00.

"What?! I've only been gone for 25 minutes."

She looked at the ticket. "Sorry, sir. You checked in at 11:40."

"No way," I said. "I know what time I checked in; I was running late. It was at least 12 minutes later than 11:40. I got to the entrance to the Consulate, just over there, at 11:55. No way I could have taken 15 minutes to walk 75 metres!" She showed me the ticket. It said 11:40. "That's impossible. I want to check the clock."

The difference was only $3.00, but I was furious. I exited the garage, drove around to the entrance and check the display. It read 12:24, the correct time. I pushed the button and pulled out a ticket; it too read 12:24. To her credit, the attendant appeared and checked the clock, and asked to see the ticket I had just printed. "12:24. I'm sorry, sir, there's nothing I can do." Quite true, no doubt.

In this case, the (clearly fallible) machinery and the (clearly fallible) documentation were more credible than I. I didn't check the ticket on the way in. And yet I know when I arrived, and I know that there must have been some kind of failure with the machinery. A one-off? A consistent pattern? Happens only at a certain time of the day? A mechanical problem? A software problem?

All the way home, I pondered over how the failure had occurred, and how one might test for it. But what impressed me most about my experience with the Consulate's Web site, and the consular officer, and the the parking ticket machine, and the parking attendant, was the way in which we invest trust, to varying degrees and at various times, in machines and in documents and in people. When is that trust warranted, and when is it not?

Postscript: Just now, as I attempted to publish this post, the net connection at this hotel was suddenly unavailable. Again.

Exploratory Testers' Meetup, June 3

2009-06-30T03:47:00.002-05:00

Thanks to the energetic James Lyndsay, a bunch of us are meeting at the conclusion of his Exploratory Testing class and my Rapid Testing class on Friday, June 3 2009, in London, UK. I expect we'll be there somewhere between 5:30 and 9:00pm. The venue is the Prince Arthur pub, 80-82 Eversholt Street, Euston, London, NW1 1BX, right across the street from Euston Station, north of Euston Road. Three large tables have been reserved. All are welcome; please spread the word!

After that, I'm off to The Magnet, a pub near the Archway station, where a gang of folks gets together every Friday from 9:30pm until late to play Irish traditional music. I'll be armed with my mandolin. All are welcome there, too!

James Lyndsay Mea Culpa

2009-05-27T15:56:00.004-05:00

In a recent posting, I made a mistake: I erroneously stated that James Lyndsay, the genial host of the London Workshops on Exploratory Testing (LEWT), had not attended a LAWST conference before setting up LEWT. Except I was wrong: he had. Shame on me for not checking.

If you're not aware of James' work, you would do well to know about it. He's the author of a rich set of exploratory testing puzzles that take the form of engimatic black box machines. James Bach and I have used an early version of one of these machines in the Black Box Software Testing course for several years. I can't, won't, tell you much about them here, since the whole point is to encounter and explore for yourself. But I can tell you that they're intriguing and stimulating, and they help to sharpen the questioning processes involved in excellent testing.

James also took the Best Paper honours at STAR East this year for The Irrational Tester: Avoiding the Pitfalls, in which he presented "his view of bias—why we so often labor under the illusion of control, how we lock onto the behaviors we're looking for, and why two people can use the same evidence to support opposing positions." These are important and under-explored issues in the testing business.

James is presenting a class on Exploratory Testing in Berlin on June 4-5, and in London, July 2-3. That happens to overlap the latter two days of a course I'm teaching. Nonetheless, I'm delighted to recommend his.

Automation and Coverage Part II

2009-05-25T15:33:00.004-05:00

Last week posted a blog entry on automation and coverage, in which I questioned the usefulness of trying to cover "everything" with automated tests, comparing them to the CCTV cameras that are in use all over the place, but especially in Britain. Despite the limitations of such schemes, there might also be some useful aspects. What might they be?

For certain areas that we decide to cover with a camera, like public streets or open squares, we won't necessarily need to monitor all of the images all the time, but if the data were stored, we could review it—especially after trouble had occurred—to try to find out what went wrong and who the instigators were. This is like a continuous logging system for a program under test.
For places that are hidden or potentially dangerous, like subway underpasses, we might want to set up a motion-activated camera. This is like an event-based logging system, in which we record some aspect of the system state based on some occurrence that we anticipate.
For high-traffic areas like urban highways, where we want to direct lots of attention about the flow of vehicles, it might be a good idea to set up a number of cameras and monitors at various points on the roadway, and cycle through the images every few seconds. Even if we don't see the incident as it happens, CCTVs allow us to spot trouble fairly shortly after it happens, since traffic will tend to change its behaviour, typically bunching up behind a blockage. Sophisticated cameras would allow us to pan and zoom, inspecting the specific nature of the blockage, and will help us determine how to respond. This another form of logging, more like polling. We can perhaps combine it with probes or indentifiers on the data to make it easy for us to follow a record or data packet that attracts our interest.
The same kind of cameras can follow and monitor the (mis)behaviour of drivers, such as those who are blatantly speeding or breaking the rules. This is like a monitor that allows us to track the progress of a particular data set through an application.
We can set up a particular camera at a particular interface that allows us to stop traffic and alter it or its behaviour in some way before we send it on. I think of tools like Burp Proxy or Fiddler in this way.
In the last few years, more and more people have obtained portable cameras and camera phones; most of those devices do video, too. See something interesting as you're testing? Use a literal camera, an inexpensive point-and-shoot model, to record some aspect of your activities as you're testing, or as you fill up a white board or a set of post-it notes during a meeting.
Tourists and amateur newshounds, on the spot as an event occurs, can often take pictures that later turn out to be valuable. A tester can take advantage of the data capture tools that the operating system provides. Windows Vista comes with the Snipping Tool, and previous versions of Widows include Print-Screen (to print the screen to the printer), Shift-Print-Screen (to place a copy of the entire screen on the Widows clipboard), and Alt-Print-Screen (to place the topmost window of the screen on the clipboard.) Since these tools are readily available, it's as though everyone had a camera of his or her own.
A real video camera, or a screen recording tool like BBST Assistant, can allow you to record the actions of an end-user as he or she works her way through the application.
Many applications allow you to go forwards and backwards. See if you can get your application into a tangled state by performing activities, backtracking with Ctrl-Z, and then perform the activities again, the same way or with variation. Then backtrack again. Then do it again, perhaps brancing to a different point. Set up a monitor of some kind to help you record and observe what happens. The idea is to get the application to get confused and to tie itself in knots.
Ask the programmers about the built-in error checking in the program, and ask if the error -checking routine logs its actions, perhaps by a configuration switch.
A program can be instrumented to provide interfaces to coverage tools, or to requirement tools such as Fitnesse. Not only can you get a shapshot of results from the running program, but you can do it again and again.
Psychologists often put the patient in a room with a one-way mirror with a camera behind it, maybe hooking some electrodes up to his skull, and interview him or merely watch his behaviour. A debugger is a kind of equivalent of this approach.
When there's an important event happening, news stations send television crews to record and report. Sometimes that footage is helpful in determining what's going on, but note that the film crew is constantly making choices about where to place the cameras and what to observe. That is, every observation is a subjective observation at some level.
Note that the television show or movie that you see is typically the product of extensive preparation, rehearsal, gobs of technical equipment, and sophisticated editing. If you choose to automate and record everything you will generate a lot of stuff that will end up on the cutting room floor. Who gets to do the editing?

These are just a few examples; I'm sure you'll come up with many more (and they're welcome here). Note the kind of automation that I've been talking about here involves little to no direct, continuous control or monitoring people, and note that most of the approaches can help to reduce dependence on elaborate prescriptive or retrospective documentation. Not all of these approaches will fit your context, but there are several here that might. The idea is not to adopt them all, but to consider one or more possibilities that might be helpful in your context.

Bangalore Workshop on Software Testing

2009-05-19T21:58:00.005-05:00

In 1999, Cem Kaner and Brian Lawrence came up with the idea of having testers and test managers meet to talk about some of the problems that seemed to bedevil all of them. This was, for its time, a radical idea for the testing community. Here's what they said, after the second LAWST but before the third:

This is a process developed by Cem Kaner and Brian Lawrence for technical information-sharing across different groups. It's not very new. We adapted it from some academic models and models from other industries. Our goal is to facilitate sharing of information in depth. We don't see enough of that happening today.

You can read the rest of the report here, at Cem Kaner's site.

These conferences were to be different. They were to be small, fewer than 25 people, aggressively non-commercial, based on experience report and open, facilitated discussion. The ideal was to inspire further research and to publish papers.

Several LAWSTs were held, and then others began to happen, including (a very partial list here), the Software Test Managers' Roundtables, the Workshops on Peformance and Reliability, the Workshops on Teaching Software Testing, the Workshops on Heuristic and Exploratory Techniques, Software Testing in Financial Services... Having attended a LAWST, James Lyndsay started a LAWST-inspired meeting called the London Exploratory Workshop on Testing (LEWT) that has a less formal structure (as far as I know, it remains the only LAWST-inspired meeting to be held in the back of a pub, although on the last day, WHET IV moved to the upstairs of one in 2007). Fiona Charles and I started the Toronto Workshops on Software Testing in 2005, more formal than LEWT but less formal than LAWST.

Cem and Brian's gift to the community continues to expand all over the place. Back in 2007, Pradeep Soundararajan took me to a meeting of a group of testers that he had arranged in Koramangala, Bangalore. I blogged about that here, encouraging Pradeep and the rest of the group to build a community of inspired testers.

My observation is that when you're waiting for something good to happen (all over the world, but especially in India), nothing at all seems to happen for a while... and then all of a sudden, everything happens.

In this case, the Bangalore Workshop on Software Testing happened, with the help of the Edista Testing Institute and Test Republic. And a number of skilled Indian testers prepared presentations, delivered them, and questioned each other. And Pradeep wrote about it all here.

By Pradeep's account, a ton of ideas emerged in the discussion. You can read about those in the report. The overall theme that I see when I read the report is a group of testers using ideas that they've learned from elsewhere, and then (as James Bach describes it) inventing testing for themselves. That is, using the ideas that they've read or heard about, trying them in context, seeing if they work, tuning the stuff that does, adapting the stuff that might, and rejecting the stuff that doesn't. And above all, questioning all the time. All in all, a terrific report on what must of been a terrific meeting. I'm now deeply envious of everyone who was there. Bravo, each and every one of you. Bravo, Edista and Test Republic. Bravo, Pradeep—and Bravo, India!

Do you want to attend a gathering like this, scaled up to a more formal conference setting, but that preserves the focus on experience reports, discussion, and learning? Do you want to go to a testing conference and confer? Do you want to explore and investigate what other testers are doing, rather than hearing a canned talk with questions only allowed if we have time? If the answer to any one of those questions is Yes, then I give you my highest recommendation: go to CAST, the Conference for the Association for Software Testing, in Colorado Springs, CO, June 13-16.

Posted: Presentation Notes from STAR East

2009-05-19T20:10:00.003-05:00

At the STAR East conference, produced by Software Quality Engineering in Orlando, FL, I gave a keynote address on Testing and Noticing. I also gave a half-day experiential workshop on Difficult Testing Questions and How to Answer Them, and a track session called Insource or Outsource Testing: Understanding Your Context.

A number of people have asked about the source for the video that I showed. It can now be revealed here: http://www.youtube.com/watch?v=ubNF9QNEQLA For reasons that I hope are obvious to those who were in attendance, I declined to provide this before the talk. :)

All of my talks since March 2005 are listed at http://www.developsense.com/past.html, and I've provided the notes for a fair number of them. If something is missing that you'd like to see, drop me a line and let me know.

How Far Back Does This Go?

2009-05-19T12:16:00.004-05:00

For almost as long as I've been a tester, with occasional lapses into process enthusiasm, I've been questioning the value of test automation as a presumed good, especially when the automation is deployed against the highest levels of the application. Automation is a tool, and there is great value in tools. But with that value comes risk.

The Agile Manifesto, properly in my view, emphasizes individuals and interactions over processes and tools, and it emphasizes working software over comprehensive documentation. The Manifesto notes that "while there is value in the things on the right, we value things on the left more."

McLuhan had some remarkable observations on tools, which he considered a subset of media. I wrote about the value of McLuhan thinking for testers here. McLuhan famously identified writing—in particular the phonetic alphabet—as a technology. In his Laws of Media, he points out that one of the effects of a medium is to extend or enhance or enable or accelerate or intensify some human capability in some way. Another effect occurs when the medium is stretched or "overheated" beyond its original or intended capacity; it reverses into the opposite of its enabling or extending effect.

I was introduced to McLuhan's work largely through a CBC Radio program called Ideas. In 1988, David Cayley covered a conference on Orality and Literacy, co-sponsored by the McLuhan Program at the University of Toronto and the Toronto Semotics Circle (for a description of the program and of the conference, go here, and then search for "literacy"). The conference was set up to question the idea of literacy as the centre of education—not to reject it, but to question it.

The motivation behind the questioning was to understand better the role of literacy in education and in the world in general. Some scholars pointed out that literacy has to be seen in its human context, as an extension of oral discourse, because it is as listeners and speakers that we evolved, not as readers and writers. As David Pattanayak, the director of the Central Institute of Indian Languages at the University of Mysore put it, "What I am worried about is that there are 800 million illiterates in the world, and for those 800 million illiterates, there is nobody to speak. We are speaking as though literacy is responsible for everything—for family welfare, GNP increase, for modernization, for all kinds of things, but I don't think that is correct...The whole question is that there are illiterates and there are literates, and we should be looking for interaction among the illiterates and the literates, rather than trying to prove the superiority of one over the other." Why? Because literacy is one means to an end; it helps with many things, but it certainly does not guarantee the accomplishment of our human goals. As he later goes on to say, "Literacy without social concern is meaningless."

To me, this has a direct connection to our business and to our fascination with written and/or automated tests. For many years, we've tried to improve testing by trying as hard as possible to remove the messy human bits from it. James Bach describes this in his essay in The Gift of Time. We'll call it the Chapel Hill approach to software testing; papers on it are collected in Hetzel's Program Test Methods (something much closer to being the first book on software testing than Myers' The Art of Software Testing, by the way). The Chapel Hill approach to reducing testing's problems with those unreliable humans, says Hetzel, is to lean hard on media, improving "written specification methods" and using "unambiguous testable specification languages", rather than treating testing as an open-ended investigative process.

A bunch of us, led by the work of Jerry Weinberg, Cem Kaner, and James have been questioning the value of putting media at the centre of testing for qute some time now. Questioning the value of written artifacts isn't exactly new; it goes back still farther than that. How far? How about the Greek philosophers—Plato, and Socrates?

There's a version of Plato's dialogue Phaedrus online. The part that most interests me starts with the paragraph that you can find by bringing up the link and using your browser's search function to look for "Theuth". Read that paragraph and a little further, and you come across this, when Socrates talks of written-down speeches:

You would imagine that they had intelligence, but if you want to know anything and put a question to one of them, the speaker always gives one unvarying answer. And when they have been once written down they are tumbled about anywhere among those who may or may not understand them, and know not to whom they should reply, to whom not..."

Wait, it gets better. I recommend reading this bit slowly, savouring it:

Until a man knows the truth of the several particulars of which he is writing or speaking, and is able to define them as they are, and having defined them again to divide them until they can be no longer divided, and until in like manner he is able to discern the nature of the soul, and discover the different modes of discourse which are adapted to different natures, and to arrange and dispose them in such a way that the simple form of speech may be addressed to the simpler nature, and the complex and composite to the more complex nature-until he has accomplished all this, he will be unable to handle arguments according to rules of art, as far as their nature allows them to be subjected to art, either for the purpose of teaching or persuading;-such is the view which is implied in the whole preceding argument.

My emphasis—context-driven thinking! Wait; it gets better yet. Really slowly, now:

But he who thinks that in the written word there is necessarily much which is not serious, and that neither poetry nor prose, spoken or written, is of any great value, if, like the compositions of the rhapsodes, they are only recited in order to be believed, and not with any view to criticism or instruction; and who thinks that even the best of writings are but a reminiscence of what we know, and that only in principles of justice and goodness and nobility taught and communicated orally for the sake of instruction and graven in the soul, which is the true way of writing, is there clearness and perfection and seriousness, and that such principles are a man's own and his legitimate offspring;-being, in the first place, the word which he finds in his own bosom; secondly, the brethren and descendants and relations of his others;-and who cares for them and no others-this is the right sort of man; and you and I, Phaedrus, would pray that we may become like him. (My emphasis added all over the place, there.)

What happens when we apply all this to testing? To me, this says

that conversation, rather than documentation, is central to the work that we do;
that notions of correctness are pointless unless they're based on value;
that we need to study and practice testing, or anything else that we wish to understand;
that we must be cautious and think critically;
that the focus on nomenclature and unquestioned bodies of knowledge proffered by the ISTQB and other certifiers is foolish; and
that we should aspire to the values that Socrates proposes.

To me it also says that if we want to be great testers, it would be a good idea to study philosophy, focusing (as James suggests)(James Bach, not William James, although he'd probably agree) on ontology (how we conceive of the world, the nature of being) and epistemology (how we know what we know).

And, yes, there is irony too. Here's Plato, griping through Socrates about how dangerous writing is, and he's written down this dialogue. What does that tell us? To me, it suggests that everything that we have to say about what we think and what we do is based, not on absolute principles of truth and certainty, but on heuristics and skepticism.

What does this say to you?

Automation and Coverage

2009-05-17T23:54:00.005-05:00

If you don't read the forums on the Software Testing Club, I'd recommend that you consider it. In my view, the STC is one of the more thoughtful venues for conversation about testing. (I'd recommend subscribing to the Software Testing mailing list, too.)

A correspondent recently posted a request for help in recommending an automation approach. I answered something like what follows:

Need to get a code coverage of at least 70% and as it stands, Selenium is probably only covering about 20% as it just tests basic user flows.

I'm going to try to help in a different way, by encouraging you to question some assumptions.

What does "100% code coverage" mean to you? 100% of the code your programmers write? Or 100% of the code that is invoked by the application? Remember, much of the code that your customers will use is in the platform—the code on which your application depends and that depends on the browser(s) and the operating system(s) and the application framework(s) and the plugin(s) and the version(s) of each and the...)

But more importantly, what does it mean to cover the code? To execute each line? Each branch? Each branch, with each condition that might drive code down that branch? Have we covered the code if we run it and it doesn't crash? If it doesn't cause a problem that we observe right away? If it doesn't cause a problem that some automaton can observe? If there's a problem that our test automation hasn't been programmed to detect, how are we going to find out about it?

Let's use an analogy. Suppose that you worked for a police force, and you wanted to monitor what was going on in the city by setting up closed-circuit TV cameras. (This is being done in many cities in the United Kingdom these days. What would it mean to cover everything with CCTV cameras? Would you cover the entire city with cameras? Every street? If you did, would you be able to see inside buildings? Down every alleyway? Inside every rubbish bin? How expensive would this become? How long would it take to install the equipment?

What sorts of problems would you use the cameras to detect? Murder? Assault? Drunk and disorderly conduct? Embezzlement? Pickpockets? Littering? Parking violations?

So suppose you sorted out these problems and you went to the effort of setting up a bunch of cameras. How would the cameras know the difference between bad behaviour and behaviour that wasn't so bad? They wouldn't; they'd need human monitors. How many? Would the humans get bored or inattentive watching a bunch of non-events? Would it be more helpful to have some of those humans outside, on the street, actively helping to keep the peace and stopping trouble before it happens?

Now, these questions may sound ridiculous, and to some degree they are. But here's an important question: at what point would they not become ridiculous? Clearly we couldn't obtain 100% monitoring of what was going on in the city, so maybe we'd settle for 70%. But 70% of what?

How do we make sure that crime isn't bursting out all over the place? What kinds of problems aren't observable by cameras? Are there senses other than vision that humans might bring to bear in identifying problems? Might we find out about certain kinds of problems earlier if we could smell smoke or sewage? Might we be able to identify criminals if we could listen in on their conversations?

But maybe all this monitoring isn't as helpful as it might be. A city with little crime isn't the result of total vigilance by the police; it's the result of various things—many of them little things, many of them social conventions—such that there's little need to monitor or investigate. When people are well-fed and well-housed and well-educated, when they all have valuable jobs to do, when they're interdependent and feel responsible to one another, the crime rate goes down. When issues are debated and decided in public, based on consensus, people recognize the value in the rules and conventions, and generally feel less inclined to try to get around them. People tend not to cheat when they feel they're getting a fair deal. Security comes with freedom, responsibility, visibility, and trust. That's not infallibly true, but recognizing it can save a lot of effort.

Similarly, in a software development project, some practices are going to make it possible to reduce the emphasis on automation coverage at the GUI level. If your programmers are unit testing their code thoroughly, certain kinds of high-level code coverage aren't going to be so necessary; in fact, they'll likely be better tested than if they're tested via the GUI, since risks and questions about the code can be targeted specifically. If testers are performing tests (interactively or getting help from automation) at some level below the GUI, there's code coverage happening there too. If testers are operating the product as regular users do, the testers are obtaining some of the kind of coverage that users need. If testers are operating the product according to extreme, unusual, harsh, complex, challenging scenarios, then they're getting even more of the coverage that users will need.

The thing that usually gets left out of these discussions is that coverage is multi-dimensional. For testers, the issue isn't just code coverage (statement coverage, or branch coverage, or branch-condition coverage, or...). The issue is test coverage, testing for all kinds of dimensions of value, and code coverage is a manifestation of that. Neither test automation nor code coverage tools will tell you anything about whether a product follows the user's workflow, or whether an error message was informative and clear, or whether the product has a useful logging system. Neither test automation nor code coverage tools will report on missing functionality, nor will they point out that your product doesn't support some relevant standard.

It's not that these tools are unhelpful; on the contrary, they can be very helpful. It's just that they're not the be-all and end-all of testing. Test automation (which we define as any use of tools to support testing) makes many impossible things possible, and makes many hard things easy. One of most useful things that code coverage does do: it points us to places where we haven't run the code at all, and therefore haven't tested. That might be very interesting.

I've written a number of articles on the distinction between code coverage and test coverage. Two of them are here:

http://www.developsense.com/articles/2008-10-GotYouCovered.pdf
http://www.developsense.com/articles/2008-11-CoverOrDiscover.pdf

In addition, Brian Marick (a long time ago, in tester years) wrote an article on how to misuse code coverage tools. That's here: http://www.exampler.com/testing-com/writings/coverage.pdf.

To London, to London to visit... some testers

2009-05-17T00:16:00.003-05:00

I'll be in London (the U.K., not London Ontario), June 17 2009, to present a keynote, "Two Futures of Software Testing" to the British Computer Society (BCS) Specialist Group in Software Testing (SIGIST; they must have bought a vowel). In the talk, I project a dark future for testing, in which the goal is Making Sure That Tests Pass, and in which processes and tools rule the roost—chillingly reminiscent of what many testers already see every day. I also project a bright future for testing, in which the goal is learning about the product or service we're testing, and providing valuable information to management, and in which the central figure is the mindset and the skill set of the individual tester, where everything else is support for that.

The night before, June 16, I'll be chatting about testing, sharing stories, and sampling real ale in some pub (location as yet to be determined) with Rob Lambert and a crew of other testers. You're most welcome to join us; contact me and I'll send you the details as they become available.

I'll be back in London again July 1-3 2009 to present a relatively rare three-day public session of Rapid Software Testing. The class will be held at the Crowne Plaza Hotel St. James, and it's presented under the kind auspices of ElectroMind. Please contact Stephen Allott (stephen.allott@electromind.com) for information on pricing (package deals available) and registration. (Note that the June 8-10 event in Southampton, sponsored by iMeta, has been rescheduled.)

Not long after that, July 13-16, I'll be at CAST 2009 in Colorado Springs, Colorado. There are a lot of very good conferences around the world, but this one is special; it's by testers, for testers, and the focus is on conferring and learning from one another. Jerry Weinberg, Cem Kaner, James Bach, Scott Barber, Fiona Charles, and Jonathan Koomey (the author of Turning Numbers Into Knowledge) will be presenting. Not to be missed. If you're going, please spread the word among colleagues. If you're not going, I'm indeed sorry... but please spread the word anyway, would you?

An Experience Report from India

2009-05-14T10:12:00.003-05:00

I don't know how this slipped in under my radar, but it did until a couple of days ago.

Sharath Byregowda is a software tester in Bangalore, and he provides a marvelous experience report here. As I read the report, I'm delighted on a number of levels.

First, it's India! India tends to be a very conservative place when it comes to testing, with many test organizations preferring scripted, document-heavy, bureaucratic and clerical approaches. Not that it would have been their idea, necessarily. A lot of Indian testers are smarter than that, but many test organizations there find themselves obliged to follow the testing missions set by companies here in the West.

If finding important problems quickly is the goal, those approaches don't work very well. They focus on repetition, confirmation, validation, and verification. Those things are important, to be sure, but one would think that an organization that was aware of potential problems would do everything in its power to thwart those problems before the code left the shop. Why lengthen the feedback loop? If I were running a development group, I would try to make sure that my outsource lab would be in a position to tell me only things that would surprise me.

Second, Sharath is a devotee of the Rapid Testing approach. Sharath took a Rapid Software Testing course through the Edista Testing Institute in Bangalore. The course was presented by Pradeep Soundararajan, who is in turn a student of me and of James Bach.

Third, Sharath is a graduate of the Black Box Software Testing Foundations course. That course was co-authored by Cem Kaner and James Bach. It's under continuous development, and each session is strongly influenced by the interaction between participants themselves. That's remarkable since the course is delivered entirely online. (It's available free to members of the Association for Software Testing.)

Fourth, I'm delighted that Sharath is blogging about his actual experiences working with actual clients. That's important. Often our clients (or the testers who work for them) are sometimes reluctant to have people go public because...well, because Rapid Testing finds a lot of problems quickly, and no one really likes talking about problems.

Michelle Smith, another Rapid Testing student, has also provided a great experience report on how she trains testers. You can read it here, and you can read James Bach's response here.

Well done, Sharath! Well done, Michelle!

Active Learning at Conferences

2009-05-09T10:00:00.004-05:00

I was at STAR East this past week, giving a tutorial, a track session, and a keynote. I dropped in on a few of the other sessions, but at breaks I kept finding myself engaged in conversation with individuals and small groups, such that I often didn't make it to the next session.

At STAR, like many conferences, the track presentations tend to be focused on someone's proposed solution to some problem. Sometimes that solution is highly specific to a given problem that isn't entirely relevant to the audience; sometimes it's focused on a particular tool or process idea. The standard format for a track presentation is for a speaker to speak for an hour with at most a couple of minutes for questions at the very end. Typically someone is speaking because he has energy for a particular topic. So, with the best of intentions, he puts a lot of material into the talk such that there's a morning's worth of stuff to cover in an hour. Trust me: I know all about this, and alas my victi...I mean, my audiences do too.

So over the last several years, I've been trying to learn things to change that, and two annual conferences have helped to show me the way. The first, starting in 2002, was the annual AYE Conference, at which PowerPoint is banned and experiential workshops rule. The second is the annual Conference for the Association for Software Testing, which I attended in 2007 and chaired in 2008. For me, the key idea from which everything else follows is to transform the audience into the participants.

There are two basic types of sessions at CAST. One is the experiential workshop, which typically begins with an exercise, puzzle, or game that is intended to model some aspect of some problem that we all face. At the end of the exercise, the participants discuss what happened and what they've learned. Sometimes there's another iteration or stage of the exercse; sometimes the discussion continues until time or energy is up. This is almost always far more memorable, more sticky, than someone's story. The lessons learned are direct and personal. Instead of receiving lesson or hearing about an experience, we've lived through one.

The other kind of session at CAST is the experience report. A speaker is given a specifically limited time to tell her story. Participants may ask clarifying questions ("What does CRPX stand for?" "I'm sorry, when you said 'we finished in two', did you mean two days or two weeks or two iterations?"). Other than that, participants stay quiet so that the speaker can tell her story uninterrupted. Then at the end of the talk, there's a discussion in which all of the participants have the chance to question, contextualize, and respond to the presentaton. Conversation is moderated by a trained facilitator whose job it is to direct traffic, ensure that everyone gets a chance to be heard, and to make sure that the conversation isn't dominated by a handful of people. Being an AST facilitator can be a challenging job, keeping order while co-ordinating the threads of the discussion and the queues of questions or comments, often with energetic people in the room.

And the energy is contagious. Participants and speakers alike are mandated to challenge tropes with their own experience, to identify dimensions of context that frame their experience, and to teach and learn from each other. When a session's time is up, if there's energy for a particular topic, the conversation continues and we change the break time, move to another room reserved for the purpose, or break out into groups for lunches or hallway conversations. People get engaged in the conversations; they discover new colleagues

This presentation-and-discussion format is a scaled-up version of the LAWST-style workshops, a set of peer conferences which were started by Cem Kaner and Brian Lawrence in 1999 for the purpose of getting skilled testers in conversation with one another to address a specific question about software testing. At LAWST-style workshops, the typical attendance is 20 people or so. When the Association for Software Testing held its first conference in 2006, many people wonder whether the format would scale up to rooms of 100 people or more. Thanks in part to the lessons learned in the peer conferences, and also thanks to the skill of the facilitators, there have been many vigourous discussions—yet everyone who wants to be heard can be heard, even for the keynote presentations.

This year CAST will happen in Colorado Springs, Colorado, July 13-16. There are some very impressive speakers and tutorial leaders again this year, including Cem Kaner, Jerry Weinberg, James Bach, and Jonathan Koomey. It's a conference by testers, for testers. I'll have more to say about some of the speakers in coming weeks, but for now, follow the link and check it out.

Exploratory Testing: Recording and Reporting

2009-04-29T10:11:00.003-05:00

At the QUEST conference in Chicago, April 22 2009, I gave a presentation on recording and reporting for exploratory testers. You can find the presentation notes here. You can also read a more formal paper on the subject, prepared for the 2007 Pacific Northwest Software Quality Conference, here. Both documents include material on notebooks and on Session-Based Test Management, and a bunch of other stuff besides.

Test Coaching and Collaboration Sessions & The Value of Experiential Learning

2009-04-26T13:47:00.007-05:00

I'll be at STAR East Monday, May 4 through May 7 2009. Lots of other colleagues will be there too, including James Bach, Jonathan Kohl, Rob Sabourin, Karen Johnson, and James Lyndsay. I'll be presenting a keynote talk, "What Haven't You Noticed Lately: Building Awareness in Testers" (the title there was cheerfully lifted by me from Mark Federman, who cheerfully lifted it from Terence Gordon, who either lifted or channeled it from Marshall McLuhan, whom Federman explains cogently); an experiential tutorial called "Tester's Clinic: Dealing with Tough Questions and Testing Myths"; and an experiential session called "Insource or Outsource Testing: Understanding Your Context".

At every conference, big or small, I'm now offering coaching and collaboration sessions, based on an idea by (and often in cahoots with) James Bach. They're free, of course; the whole purpose of a conference is conferring (about which more in the next post). Here's how they work: I (or if James is around, we) announce that we're going to be in the hotel lobby bar from 7:30 in the morning, after the regular daytime conference sessions, or any time you'd like to arrange; we bring along a bunch of testing toys, games, and puzzles, some on laptops and some not; and we work on them, engaging in exploratory play, conversation, coaching, critique of the exercises, and maybe exploratory testing of the bar's beer list, for as long as people care to stay. In the fall, at STAR West, some of these sessions went on until 10:00pm before the toys were put away.

At these sessions, everyone learns something. The games and experiential exercises either come from the Rapid Software Testing course or are candidates for it. We already have a set of ideas as to how the puzzles might be relevant, but invariably the people that we're working with discover new angles and give us new epiphanies as to what people can get out of the exercises.

Last week at the QUEST conference in Chicago was a great example of this kind of experience. Xavier Bignon, who attended my one-day exploratory testing workshop, is one of those testers who cannot resist talking about testing, thinking about testing, and solving testing problems. We arranged to meet in the hotel bar on Tuesday evening, after the official end of the conference day. This wasn't an exercise I expected to do; on a whim, I pulled out a deck of cards, showed him a magic trick and asked him to reverse-engineer it. I am not a very good magician, so doing handing him this problem was the equivalent of giving him a program that has lots of interesting bugs and discoverable problems. I watched and coached him as he wrestled with each stage of the exercise. At one point, Xavier posited an explanation as to how I was getting something done; he reckoned that I was memorizing something about the cards, when actually I was performing a quick calculation—a much simpler approach. And that led me to discover a general systems law.

The Eye-Brain Law, (as far as I know identified by Jerry Weinberg in Quality Software Management Vol. 2, First-Order Measurement) says that to a certain extent, mental power can compensate for observational weakness. The Brain-Eye Law (ibid.) says that to a certain extent, observational power can compensate for mental weakness. I'd known about those ones, which provided support for my epiphany. What Xavier's analysis made clear to me were these two:

The Memory-Brain Law says that, to a certain extent, memorization can provide a cheap and effective substitute for calculation. Memorizing your times table allows you to get the answer 56 far faster than adding eight plus eight plus eight plus eight plus eight plus eight plus eight.

The Brain-Memory Law says that, to a certain extent, calculation can provide a cheap and effective substitute for memorization. Calculating 50 x 50 as equal to 5 times five and add two zeroes is a lot faster than memorizing your times table up to 50.

If, in a computer program, you can look up a value in a table, that might be faster than calculating it (in fact, a problem with such a table was the basis for the Pentium bug). Similarly, if you can compute a value quickly, that saves immense time and space over looking it up. The skill in any kind of optimization is to figure out what things to trade, and how to trade them.

Now, it's not like I discovered any of this stuff for the first time, for all humanity; programmers have known about these complementary principles forever. They're two of the founding principles behind many forms of optimization. Matter of fact, people knew about the Memory-Brain Law long before there were computers; it's the idea behind logarithm tables.

But then it struck me that it's also one of the principles behind Rapid Testing itself: learning (ideally memorizing) a handful of lists of guideword heuristics, combined with skill developed through practice, allows testers to respond instantly and expertly to any testing situation. I'd known that intellectually, but chatting about it with Xavier helped me to realize on a deeper level that James and I are in the business of trying to optimize testing and the work that testers do.

Those are the sorts of lessons that experiential learning helps to teach, and that's why we rely on it so heavily in the Rapid Testing course.

For his part, Xavier wrote to me: "I'm really excited about what I've learned with you. If there is a possibility to join a work group or something like that to be involved with your research and/or teaching, I'd love to know more about it." You're welcome any time, Xavier, and I will keep in touch. And everyone else is welcome too.

So... ping me (stareast@developsense.com) any time, and we'll have fun together at STAR East!

A Message from the WAQB

2009-04-08T00:38:00.002-05:00

"Nice.. so Michael want us to buy his book .. maybe that why he have his web adress in his comments :) Michael we did talk to the Ladies, and if you did the same you would know it's fixed. Yes there was a mistake, but it's fixed. If you want adverts for you book pls go to the papers or google adwords. There will come names, faces ect. We have decided that until the end of the Pilot which goes until September we will keep low profile, so we get this structured. We just believe that it will be usefull to work more agile, so we are working on that.. then Michael can do his Rapid Software testing - anybody believes in that ? Steen waqb"

This was posted on LinkedIn by a fellow named Steen Lerche-Jensen (who apparently signed the Agile Manifesto in the week of February 16-22, 2009, along with a link to the International Agile Testing Qualifications Board—a link that spins forever in my browser, as of tonight). Steen's reply landed in my inbox. I'd reply to it on LinkedIn, but the thread has been deleted, so I'll reply here.

1) Sorry to blow your theory, Steen, but I don't have a book. In fact, everything I've written so far is available for free (except for three things: the current issue of Better Software Magazine; the book The Gift of Time, to which I contributed one chapter and for which I receive no royalties; and the Agile Testing book, to which I contributed a sidebar and for which I receive no royalties).

2) That was some kind of "mistake", publishing the table of contents for Crispin and Gregory's book as the course syllabus without attribution and without their consent. But I'm glad you've corrected it.

3) The comments above on keeping a low profile, and the fact of the WAQB approach as it's currently implemented on the Web site, seem to me to be incongruent with the claim on the Web site, "WAQB will use the techniques from Open Source to ensure that the quality of the syllabus is of high quality", and with the nature of the Agile movement itself.

Guest Reply: Rob Bach on Pilots

2009-04-02T22:16:00.005-05:00

A few blog posts back, I tried to emphasize the relative importance of skilled people over documentation by remarking that commercial airlines "tend to have a captain and a first officer in the cockpit, rather than a pilot and a book on how to fly an aircraft". "Tend to" was intended to understate the case; as Rob remarks below, you'll see single pilots only on very small planes (like the seaplane that I took once from Nanaimo to Vancouver—one pilot and six passengers).

gmcrews commented "I don't think you picked a very good analogy. Even though the pilot may get pretty busy, all commercial aircraft can be safely flown by a single person. The most important function that the copilot serves is quality assurance." gmcrews also said, "And regardless of the number of pilots, you will always find checklists actively used in all aircraft." That's true, but my point was that the skilled humans, rather than the checklist, are at the centre of the operation.

I asked Rob Bach, brother of James and Jon and a pilot for a major airline, to respond to that, and he did, although some enthusiastic spam filter appears to have stopped the first attempt. Rob says:

ALL airlines have more than one pilot if there are more than 10 or so passenger seats on the plane. The reason is not for quality assurance, I assure you.

As a pilot for 33 years, a commercial pilot for 22 years, an airline check pilot for a few of all of those years, I can tell you exactly why there are two to FOUR pilots on any given commercial flight.

The Captains are there to keep the First Officers from killing themselves. The First Officers are there to keep the Captains from killing EVERYBODY.

No, seriously:

People make mistakes. Two people make TWICE as many mistakes as a single person, but the likelihood that those mistakes are identical in nature and time are reduced by the way we coordinate our skill sets.

The FO is not there for assurance, but to command the flight if need be, countermand the Captain if need be, learn from the Captain if possible, fly the plane every other trip leg (run the radio gear the other legs), share pre and postflight duties. FLY the plane during emergencies (unless the Cap elects to do so... but it is rare the Cap doesn't run the checklist in an emergency), and on and on.

It physically takes two people...like hanging sheetrock.

The cockpit, the plane, the atmosphere, and the air traffic environment are amazingly complicated places where the room for error is quite small. It takes TWO brains working all sides of a flight from minute to minute to make all the magic happen.

Having flown single-pilot in heavy weather into a busy airport, I can state that I was in over my head and don't relish the thought of going back to that space/time. There's just too much data coming and going through your brain. Like Tetris in some insane hyper-mode where DEATH is the cost of losing the game.

OK...that was a little dramatic.

Two (or three to Europe and four to India) pilots are used to help ease mental and physical fatigue. Imagine performing at peak mental level at a reduced cabin pressure/oxygen level, in a very dry environment, being irradiated by the instrument panel AND the sun, in an uncomfortable chair where you can't stretch your legs easily, where you can't use the bathroom 'CAUSE HEY, WHO'S FLYING THE PLANE! for a 15 hour day back and forth between timezones, sleeping in unfamiliar surroundings, away from family (but still dealing with all the family issues one needs to deal with) , missing graduations, births, weddings, ball games with your kids, for YEARS:

Don't you think the reason we have at least two highly-trained professionals is something more than just quality assurance?

Thanks, Rob!

WAQB: Okay, now it's getting creepy.

2009-04-02T11:26:00.003-05:00

Related to my post about the World Agile Testing Qualifications Board, on March 31, I posted the following discussion on the WAQB LinkedIn list:

Linkedin Groups	March 31, 2009
World Agile Qualifications Board - WAQB
Today's Activity: 1 discussion

Discussions (1)

Does anyone /know/ anything about the World Agile Qualifications Board?	1 comment »
Started by Michael Bolton, Participant in the Workshops on Teaching Software Testing

Don't want to receive email notifications? Adjust your message setting.

Today, when I visit the group (or click on the link above), I see that the discussion is no longer available—evidently removed by a moderator. Why?

A different discussion has started, though, started by Steen Lerche-Jensen, Program Test Manager at StatoilHydro, saying that more people are needed for the review board, and requesting that they apply via the WAQB Web site. There are no replies, as of this writing.

The plot thickens. Nick Malden points out that he has found another Web site, the design of which he finds strikingly similar to the WAQB's: http://www.test4pro.com/home. Some of it is in English, some isn't. Nonetheless, there's lots of interesting information to be obtained. Try comparing it to the WAQB site. Try scrolling down.

Hmmmm.

Of Testing Tours and Dashboards

2009-04-02T09:41:00.012-05:00

Back in the 1980s and 1990s, Quarterdeck Office Systems (later Quarterdeck Corporation)—a company for whom I worked—was in the business of creating multitasking and memory management products to extend and enhance to Microsoft's DOS operating system. The ideas that our programmers developed were so good and so useful that similar ideas were typically adopted by Microsoft and folded into DOS a year or so later. After each new version of the operating system, people would ask us "are you concerned about Microsoft putting more memory management stuff into DOS?" Quaterdeck's reply was always that, as long as Microsoft supported DOS, we would find ways to improve on memory management—and that we were delighted that Microsoft had legitimized the category.

I wasn't lucky enough to attend Dr. James Whittaker's presentation at EuroSTAR 2008, in which he described the concept of touring the software as a way of modeling and approaching exploratory testing. Fortunately, Dr. Whittaker has presented a number of these ideas as part of his recent Webinar "Five Ways to Revolutionize Your QA" on the UTest.com site, which came to my attention on April 1, 2009.

The touring metaphor in testing has been around for a while. I learned about it through James Bach's Rapid Software Testing course, which I started teaching in 2004, and of which I've been a co-author since 2006. In 2004—that's the first version for which I have my own copies of the course notes—Rapid Software Testing included several ideas for tours:

Documentation Tour: Look in the online help or user manual and find some instructions about how to perform some interesting activity. Do those actions. Improvise from them.
Sample Data Tour: Employ any sample data you can, and all that you can. The more complex the better.
Variability Tour: Tour a product looking for anything that is variable and vary it. Vary it as far as possible, in every dimension possible. Exploring variations is part of the basic structure of my testing when I first encounter a product.
Complexity Tour: Tour a product looking for the most complex features and data. Look for nooks & crowds where bugs can hide.
Continuous Use: While testing, do not reset the system. Leave windows and files open. Let disk and memory usage mount. You're hoping that the system ties itself in knots over time.

But the idea had been around before that, too. Tours were also mentioned in the Black Box Software Testing course, co-authored by James and Cem Kaner, which I attended in 2003. They were part of a larger list of test ideas called "Quick Tests", which included other things like interruptions (starting activities and stopping them in the middle; stopping them at awkward times; performing stoppages using cancel buttons, O/S level interrupts, ctrl-alt-delete or task manager, arranging for other programs to interrupt, such as screensavers or virus checkers; suspending an activity and returning later) and continuous use (while testing, avoiding the resetting of the system; leaving windows and files open; letting disk and memory usage mount, hoping that the system ties itself in knots over time).

Note that concept of touring wasn't terribly new in the BBST course notes and appendices either; skilled testers had been using them for a long while before . In 1995, Cem Kaner noted that the user manual is a test planning document; as he said in Liability for Defective Documentation, "It takes you on a tour of the entire program." Elisabeth Hendrickson gave a presentation at STAR East in 2001 called "Bug Hunting: Going on a Software Safari", which gave an overall list of test ideas using the metaphor of a tour. The idea of describing tours of a specific aspect or attribute of the product (namely the menu) appeared in an article by James Bach in the Test Practioner in 2002.

Much more serious work based on the concept of tours happened in 2005. Mike Kelly did some work with James Bach, and blogged some ideas about what they had discussed in an August 2005 blog post. Mike amplified upon that in his more complete list of tours (using the mnemonic FCC CUTS VIDS) in September 2005.

Feature tour: Move through the application and get familiar with all the controls and features you come across.
Complexity tour: Find the five most complex things about the application.
Claims tour: Find all the information in the product that tells you what the product does.
Configuration tour: Attempt to find all the ways you can change settings in the product in a way that the application retains those settings.
User tour: Imagine five users for the product and the information they would want from the product or the major features they would be interested in.
Testability tour: Find all the features you can use as testability features and/or identify tools you have available that you can use to help in your testing.
Scenario tour: Imagine five realistic scenarios for how the users identified in the user tour would use this product.
Variability tour: Look for things you can change in the application - and then you try to change them.
Interoperability tour: What does this application interact with?
Data tour: Identify the major data elements of the application.
Structure tour: Find everything you can about what comprises the physical product (code, interfaces, hardware, files, etc…).

Dr. Whittaker does suggest some interesting notions of his own for tours in the UTest talk:

Money tour: Test the features that users purchase the app for (which is rather like Mike's "user tour" above, I guess)
Rained-out tour: Start and stop tasks, hit cancel, etc. (rather like Kaner's notion of "interruptions" above)
Obsessive compulsive tour: Perform tasks multiple times, perform tasks multiple times, perform tasks multiple times
Back alley tour: Test the least-used features
All-nighter tour: Keep the app open overnight (like Kaner's notion of "continuous use" above)

In his related blog post, "The Touring Test", Dr. Whittaker says, "At Microsoft a group of us test folk from around the division and around the company are experimenting with tour-guided testing." Cool. He also says, at the top of the post, "I couldn’t resist the play on Alan Turing’s famous test when naming this testing metaphor." The idea that he named tours independently is a little surprising, but when we think about the practice of skilled exploratory testing, the "touring" metaphor might be obvious enough to have been arrived at independently. These things happen.

For example, in 1995, James Bach showed me a bunch of test techniques that he called "grokking". (Grokking is a word invented by Robert Heinlein that describes deep, meditative contemplation and comprehension.) I thought "grokking" wasn't the right name for what James was describing, because the tests depended extremely rapid cognition and removing information, the very opposite of reflective contemplation. I was reading Malcolm Gladwell's book Blink at the time, and I suggested that we label the techniques blink testing. Only later, when I was researching the history of similar observational approaches for an article I was writing on blink testing, did I find a reference to astronomer's tool from the 1920s: it was called a Blink Comparator. It was fun to note that discovery, a little sheepishly, in the article. So I can understand how it's easy for people to use the same label for an idea.

But then something else came up.

In the Webinar for uTest.com, Dr. Whittaker also presents the concept of a "low-tech testing dashboard", in which he suggests using a whiteboard and coloured markers to report on project status. This suggestion isn't just a variation on the Big Visible Charts that are recommended in the Agile literature; it's strikingly similar to an idea presented James Bach at STAR East in 1999 in a talk called "A Low-Tech Testing Dashboard", posted on his Web site since around that time, and also part of the Rapid Software Testing course (pages 136-146).

I am delighted that authors as well-respected as Dr. Whittaker and that companies as prominent as uTest and Microsoft are endorsing and helping to spread ideas on tours and dashboards. I think they're worthwhile approaches, and I believe that such endorsement helps in the wider effort to get the ideas accepted. Yet I also believe that it would be a friendly and respectful gesture if Dr. Whittaker's presentation included acknowledgement of prior work in field that it covers. It would be similarly helpful if books like Dr. Whittaker's How To Break Software or Page, Johnson, and Rollison's How We Test Software at Microsoft contained bibliographies so that we could more easily find references to some of the ideas presented.

What does the community think? How important is it to acknowledge earlier work?

World Agile Qualifications Board; God Help Us

2009-03-29T19:34:00.003-05:00

The World Agile Qualifications Board should be seen as an embarrassment even to regular peddlers of certification.

The WAQB has apparently been established recently—but when? By whom? There is a Web site. I hesitate to link to it, because I don't want people to see it... but on the other hand, I do want people to see it, so that they can observe from the outset how these things work.

Note that there is no information to be found on who the World Agile Qualifications Board is. No human's name appears on the Web site. None of the people whom I respect in the field of Agile development (many), nor any of the people that I don't respect (somewhat fewer, but still numerous) appear to be willing to identify themselves with this shadowy organization, either on the site or in other forums. There is a LinkedIn group; see below.

Go to the site and you'll see just how comical this gets. Under the link "Find Training and Certification", those reading as I write (March 29, 2009) will see a graphic that includes the logo for the London Underground (this trademarked logo is doubtless used without the permission of Transport for London) and the words "After London, you deside where to go next" and "Lets hear from you and your team". Then there's a list of countries, preceded by the suggestion "Clik on your chose:" (Yes, everything misspelled above is really spelled that way in the original.)

The WAQB is, apparently, offering a course certificate in Agile Testing. The fee for the course is £990.

What is the course about? The course provides basic knowledge of agile testing. After the course there is the opportunity to sit an examination for the WAQB-T Agile Testing Foundation Certificate. Agile test is gaining recognition as a specialized field. In thedevelopment of systems and software, testing can account for 30-40% of the total cost. It is possible to reduce this cost significantly and still achieve improved quality by adapting agile testing mindset

Who should attend? This is for developers and tester. We mention developer’s upfront, because this is NOT only for traditional testers The course is for anyone involved in agile development. It is worthwhile for project leaders and developers, who need an introduction to agile testing or who test software.

The test above, errors and all, including the missing periods, is cut and pasted from the original. That is, I'm not making any of this up. Here's a little more snipped from the site:

Express courses 2 days* :

Participants who hold another formal certification like: Scrum Master, ISTQB/ISEB Foundation, PMP, PMI, Prince2 or similar a 2 days intensive courses is available.

________________________________________________________________________

The WAQB-T Foundation Certification - WAQB-T Agile Testing Foundation Certificate - statup May 2009

The WAQB-D Foundation Certification - WAQB-D Agile Development Foundation Certificate - statup May 2009

The WAQB-M Foundation Certification - WAQB-M Agile Team Member Foundation Certificate - statup May 2009

As I mentioned above, there is a LinkedIn group. I am the 27th member. No Agilist that I recognize is a member, and very few members of the list identify themselves as a member of other Agile-related LinkedIn groups. No one in my survey of the list members (hey, there are only 26 others, so I looked at most of them) appears to make any claims related to the founding of or involvement with the WAQB.

There is a review board, and you too can join!

WAQB will use the techniques from Open Source to ensure that the quality of the syllabus is of high quality.

You can now become member of the Review Board. As a member of this board you will be asked to review topics in the syllabus. But you will also have the opportunity to suggest new topics, or change of topics.

In this way WAQB are sure that we will get a high quality of the syllabus and course material at any time. The fast changing agile world will force us to be up front and to work in new stuff often.

...but not quite up front enough to identify ourselves from the get-go.

In my opinion, all this shows signs for the WAQB being a scam, and a racket. My opinion is that of an experienced tester, a member of several testing communities, and a teacher of and consultant in testing. I consider the WAQB to be a racket even worse, even more transparent, even more nakedly a way to separate people from their money than the usual certification schemes. Everyone is free to make his own decision, but I believe that one would be a fool to have his (her) pocket picked by these people (or this person). And you're not a fool, right?

In fact, you're an upstanding member of your community, and so you will warn your colleagues and your managers, and everyone else who might innocently or uncritically seek or support certification, agile or otherwise, that isn't skills-based—the kind of certification that is roundly and rightly dismissed by many thoughtful people, including Elisabeth Hendrickson, James Bach, Tom DeMarco, some clever but anonymous person, and the Agile Alliance itself. Oh, and by me.

Requirements Development

2009-03-26T14:52:00.004-05:00

Scott Berkun has an interesting post on requirements, and why they're problemmatic. He says that "data collection" isn't the issue. I agree with that. He also refers to "requirements gathering" (to me, not so good) and "requirements definition" (a little better, perhaps).

I like to think of this whole business not as requirements gathering, but as requirements development. (I credit Ian Heppell for pointing this out to me, but I believe that Karl Wiegers has said something along the same lines.) Some people think and write as though requirements are there to be gathered, like pretty stones or ripe blueberries in more-or-less plain sight. That's often true of desires; they're often visible right on the surface. But many requirements aren't obvious from the start until we've learned more about what we're developing, about what's feasible and what's unachievable. Testing often reveals new information that informs new thinking about what is required. In just about every case, some stakeholders' desires will conflict. That's why requirements must be discovered, revealed, prioritized, negotiated, and decided as the project continues.

Scott suggests giving authority over requirements to one person per major area. Even then, there's a good chance that some requirements will conflict when they involve several major areas. As much as I'm a fan of collaboration, delegation, and self-organizing teams, at a certain point somebody's signing the cheques. For good or ill, that will be the person to decide (or to delegate the decision to some other person) on what is a real requirement and what isn't.

There was another interesting aspect to the post. In the comments, PSJ provides an example of a common problem. He (she?) says "Certainly being given some control and ownership over the requirements document would have made my life easier in the past."

Now, you might think that the problem here is lack of control—and I'd agree, but I see another problem: Note how easily "requirements document" and "requirements" seem to slip into one another.

With respect, I'd suggest that someone (perhaps PSJ him/herself, but certainly the person who owns the project) needs control and ownership over the requirements, not merely the requirements document. The map is not the territory.

On Indispensable People, Documentation, and Skill

2009-03-17T16:03:00.003-05:00

In a blog post on The Test Eye, Martin Jansson has some things to say about the dangers of The Indispensable Worker. The post is worth reading. I commented there, and do a somewhat better job of it here:

Your point about indispensability is well-taken. In workshops that I've attended, Jerry Weinberg has often pointed out the urgency of getting rid of the problem of indispensability. If someone appears to be indispensable, it's a great risk to the organization; it either has become or will become severely maladapted to existence without that person. This isn't to say that people won't be missed when they go; everyone carries tacit knowledge and experience that no one else has. But whether someone will be missed or not, their departure shouldn't destroy the organization. An "indispensable" person will disappear eventually, one way or another. So if you see the problem, get started on addressing it now.

There was a point with which I disagree, though—at least with the emphasis:

As a co-worker you avoid these traps by requiring documentation enough for someone else to perform the task or that you have at least a backup for the critical tasks.

I'd put that in the opposite order. If you've got a backup (in the form of a person who can do that task), then you might not need documentation at all to get the job done; and you might have a better way of performing the tasks that the job requires in the high-pressure moments. This is why commercial airlines tend to have a captain and a first officer in the cockpit, rather than a pilot and a book on how to fly an aircraft.

Note also that there are several relatively high-pressure moments on every flight. Just because something could be done by a single person doesn't mean that it's automatically better policy to have only one person doing it. On the contrary, in most cases.

What's James on about THIS time?

2009-03-09T19:49:00.005-05:00

James Bach reports on a statement allegedly made by Yaron Sinai, the CEO of Elementool, and Joseph Ours comments.

In light of Joseph's comment, I too wonder if Elementool was tested by human testers under the control of their own process. Perhaps it was tested by testers who strictly followed the steps, which Mr. Sinai apparently suggested was all that was necessary.

If the latter, then Joseph's experience can be explained in terms of W. Ross Ashby's general systems law, The Law of Requisite Variety. This law suggests that a system with N states cannot be controlled or understood by a system with fewer than N + 1 states. There's a variation of this law in G.F. Smith's advice that if you want to understand something complicated, you have to complicate yourself (no direct citation available here, but Smith is quoted in Karl Weick's Sensemaking in Organizations, yet another wonderful testing book that isn't about testing). One key to excellence in testing is not only to understand testing itself, but also to explore and investigate widely diversfied interests and skills so that new learnings come to the field.

If Elementool was indeed tested by unskilled human testers, Mr. Sinai's dismissal of the role of interactive testing performed by humans is perhaps explicable. The problem Joseph reports would seem like a pretty difficult bug to pass by skilled testers. (Yet one never knows. Perhaps it was a bug introduced moments before a product manager, under market pressure, decided to ship the product without a review, a unit test, a more general functional test, or a targeted retest of that area. Perhaps there's a platform-based problem, based on an environment that Joseph has and that the Elementools people don't. Perhaps it was a known bug that only happens in the exact case outlined by Joseph, and was deemed insufficiently important to fix. But in such cases, you run the risk of being embarrassed by a tester like Joseph. Or me. On the other hand, maybe you don't embarrass easily.

2) On the page http://www.elementool.com/Services/Common/ControlPanel/EditUsers.aspx, I added a second user to my demo account. I deactivated the user by clicking the associated checkbox and pressing the Update button. Upon returning to that page, I clicked the checkbox to reactivate the user. I got this:

3) Every time I reproduced the error, I got a different error code. This is surprising and interesting; it was true when I reproduced Joseph's problem too. I would have expected a consistent error code per problem. It may be that the error code refers to a specific incident logged in Elementool's tracking system; or it may be that there's no mapping between the error code and anything useful; or there may be some other explanation.

I was curious about the quote in James' blog post, and I wanted to check out Jeff Feinman's other articles on SDTimes' Web site. I did a site search. Ah, here's one:

Clicking on the underlined link calls up the URL listed below the summary. That returns a page of today's top stories, not the article I'm looking for. The URL in my address window is now http://www.sdtimes.com/default.aspx?aspxerrorpath=/static/sdtimes100_07_02.html (my emphasis added there).

So I found another Jeff Feinman article instead. That one, headlined "Serena computerizes agile development" starts "March 6, 2009 — Serena Software is putting agile development into a software-as-a-service model, but the company admitted that transforming such a human-oriented process into a computerized one wasn’t an easy task." For a second time in a few days, Mr. Feinman uses a peculiar form of expression to confuse a process and a tool that aids that process.

The article includes this little gem: "The No. 1 objection that people have against SCM tools in general is that it’s a tool,” said René Bonvanie, senior vice president of marketing for Serena. “Developers hate tools. Anything that forces process or input, they hate. So we needed to build something that was computerized but super simple to use, and very intuitively usable to people who are used to whiteboards."

Well, at least there's some consistency. The marketers for Elementool and Serena provide stupid, sweeping generalizations to offend members of the testing and programming communities equally. It's odd that Mr. Feinman would publish such remarks uncritically, and that no developer, no member of the Agile community has as yet seen fit to object. QiD.

IMVU: The Final Chapter

2009-03-08T11:31:00.004-05:00

Perusing my Blogger page, I suddenly realize that I never posted this final wrap-up to my original observations on IMVU and its 50-deployments-a-day approach, plus the comments here, here, here, and here.

In addition to his Quality is Dead article, James has added his perspective on IMVU specifically.

Is there a problem at IMVU? As an outsider, is this any of my business anyway? I think so, at least to some degree.

I wouldn't use the service myself. That's reasonable; it's not my thing. My stepson might be inclined to use the service at some point, and might be inclined to buy special services or other stuff from them, at which point he'd ask me to pay for it. With my credit card. And then we'd have a problem, because I don't have sufficient trust in IMVU to give them my credit card information, based on everything I've seen so far.

IMVU is a service that seems to appeal mostly to kids. I wonder about the ethics of deploying what seems to be some pretty shoddy software to kids, and thereby teaching them that software doesn't have to work, that their complaints can go unresolved, and that it's routine and tolerable for their accounts to be vulnerable to hacking, as long as the programmers are deploying 50 times a day. As a person involved in developing software, that makes me sad, and I think, in a small way, it tarnishes all of our reputations.

In particular, it stands a good chance of tarnishing the reputation of the Agile community. The Agile Manifesto is a beautiful thing, and I believe in it. But agility, in the dictionary sense, isn't merely about quick movement; it's also about being in control, responding to what's happening in your environment, and keeping your balance. If 50 deployments a day of this stuff is seen as an exemplar of Agile methods, we haven't crossed the chasm; we've dived into it.

IMVU and its issues are one thing, but what really freaks me about this whole business is the oblivion about testing from the respondents to Timothy Fitz's post, and the notion that automation makes testing passé. To me and to my colleagues—people who think seriously about testing, and who study testing—this whole 50-deployments-a-day and a million-automated-tests-a-day thing isn't testing at all. I repeat, from an earlier post: at best, it's checking, and it's checking done almost entirely without inquiry. That, to me, is a corruption of the idea of testing, which is questioning a product in order to evaluate it (Bach), gathering information with the goal of informing a decision (Weinberg), or (deep breath here, but worth it) an empirical technical investigation of a product, done on behalf of stakeholders, with the intention of revealing quality-related information of the kind that they seek (Kaner).

It's IMVU's business; it's their call. I hope it's not contagious, but I fear that it might be, for a while.

Still more IMVU comment followup: The Final Chapter (so far)

2009-03-08T10:31:00.004-05:00

Markus Gärtner commented, in part, Actually what seems to be missing is the pride and responsibility in the software world. From my point of view I can pile up a lot of technical debt, but deliver really fast.

I'm not so sure if pride is missing. Timothy Fitz seemed to be proud of the work that he was doing. Moreover, hubris is a form of pride, and there seems to be no shortage of that in our business.

You raise an important point on technical debt. It's bad enough for us to be running up our credit card bills, but when we don't know how much debt we're running up, that's really dangerous. I see this with every company that makes a conscious decision not to test things that might be important.

You cannot expect to improve your software process by one order of a magnitude. This is unrealistic, stupid and dangerous.

Right. Apropos of that: ever noticed the claims made by the tool vendors?

Ben Simo weighs in: I think that users have become so accustomed to bad software that they don't notice the bugs -- for long. Software users are accustomed to finding workarounds and the quickest way to make errors go away.

Yes, and I think this is amazing. Part of it has to do with the magical aura that we have around computers; any sufficiently advanced technology is indistinguishable from magic, as Arthur C. Clarke said. Magic dazzles us into non-reasoning. Here's an example that I've been using for about 20 years:

You're about to go on a trip. You take your car, which is running fine, to the mechanic for a checkup. Because he leaves work at 6:00pm, and you can't get there until 6:30, you agree that he'll leave your keys with the cashier at the gas bar. At 6:30 you arrive and pick up your keys. You get into the car, put the key in the ignition, and turn. Your car makes a noise (yawp!) and then suddenly goes dead. Now: is your reaction "Aw geez, I'm so lame. I'll never understand these complicated car thingies. It must be something I did." Or do you feel anger and resentment towards the mechanic?

It used to be that users of bad software blamed themselves, but now I think that most people simply don't expect things to work. We've all become like Sam Lowry in Terry Gilliam's Brazil (the relevant scene starts at 3:34 or so in the link above, but if you haven't seen the movie, for heaven's sake, rent it). To top it off, when we call Central Services, we get that wonderful message: "Thank you for calling Central Services. I'm sorry due to temporary staff shortages, Central Services cannot take service calls centrally between 23:00 and 09:00 hours. Have a nice day. This has not been a recording."

Raoul Duke: wow. you 2 are my heroes. completely how i would feel about that kind of thing. it absolutely cracks me up that you were talking to people and asking them if they found bugs when i'm sure they just were 12 year olds looking to cyber. awesome. i want to hire you all but i'm just a nobody.

Raoul, just a reminder: the pseudonym you've taken is one assumed by a man who wrote vigorously, at length, and, above all, clearly.

Adam Goucher: What if they are deploying protocol or backend processes? I'm not sure your testing would have found issues as you appeared to be concentrating on the frontend. Now, the author does mention Selenium so it somewhat implies they deploy frontend stuff as well.

Yes, that occurred to me (as I mentioned before the bug reports in the original post). However, there's a heuristic that I use when testing, which I will now name The Fractal Heuristic: "Every piece of the system reflects some aspect of the entire system." The front end of a system reflects, at least to some degree, the values of the organization that produced it; and the organization that produced it produces the other parts of the system. Further research suggests that IMVU claims 30,000,000 registered users, but the online counters in their chatmatch feature consistently register around 60,000. There are 122,000 posts and 26,000 topics in the online bug reporting system, with headers like "everything in my inventory is missing!" and "Hacked account". In the bug tracking system, there's a report of people not receiving mail from IMVU, to which one member replies "its a known thing that after any imvu work on page code to client that things get changed with out you knowing or doing anything.... ". And when I tried to go to the earliest of the 525 pages of headings in that tracking system, here's what I got all over my screen:

Error:

Database query error

Stack:

/home/webadmin/website.54609/catalog/modules.php: 83
/home/webadmin/website.54609/catalog/includes/modules/phpbb2/viewforum.php: 330 (include)
/home/webadmin/website.54609/catalog/includes/modules/phpbb2/db/mysqlimvu.php: 70 (sql_query)
/home/webadmin/website.54609/catalog/includes/functions/database.php: 148 (tep_db_query)
/home/webadmin/website.54609/catalog/includes/functions/database.php: 163 (tep_db_query_cache)
/home/webadmin/website.54609/catalog/includes/functions/database.php: 339 (tep_db_query2_cache)
/home/webadmin/website.54609/catalog/includes/functions/database.php: 329 (tep_db_query2_cache_shard_uri)
/home/webadmin/website.54609/catalog/includes/functions/database_inner.php: 227 (tep_db_query2_cache_conn)
/home/webadmin/website.54609/catalog/includes/functions/database.php: 123 (tep_db_error)

I hear loud a whistling sound coming from the direction of just past the graveyard.

I wonder how many of us don't fall for the trap Jon mentioned.

I fall into the trap when I forget that my primary mission is to reveal information, rather than to find bugs. The latter is hugely important, but not as important as the former. I learned a long time ago that finding bugs was the tester's mission, and it's taking me a long time to unlearn that. This generation of testers needs to pass that lesson on to the next.

This kinda harkens to James' recent Quality Is Dead article. We've conditioned the average consumer to expect non-perfection. As long as the basic functionality exists and works-ish then they are happy. Well, enough to not go through the hassle of migrating yourself and the rest of your friends to a different service.

I'm not sure who said it, but inertia may be the most powerful force in the Universe. It has to be one of the most powerful forces in the Metaverse.

Anonymous says: Part of the reason IMVU can get away with all their bugs is because they're in the entertainment business, so none of their users rely on them for anything really important.

Sorry, Anonymous, but I disagree. Those logos near the bottom of the page (a screenshot from the IMVU VIP Club subscription page) suggest that IMVU may ask me about things upon which I rely that are very important indeed.

Moreover, look at the Bugs and Testing page, and tell each one of the users complaining that their complaint is not important.

One last wrap-up coming...

More IMVU comment followup: Timothy Fitz's reply

2009-03-07T22:32:00.002-05:00

In response to my post on IMVU, I was delighted to receive a reply from Timothy Fitz, whose original blog entry triggered my investigation.

There are many things to like about Timothy's reply. First of all, it's honest and forthright. Second, he seems not to be taking personally the criticism of the product that he and his company have been working on for years. It's a rare skill not to take things personally. So, thank you, Timothy, for that.

He begins:

I would like to clarify, we do have a Quality Assurance staff. There are numerous quality engineering tasks that only a human can do, exploratory testing and complaining when something is too complex come to mind. They're just not in the "take code and put it into production" process; it can't scale to require QA to look at every change so we don't bother. When working on new features (not fixing bugs, not refactoring, not making performance enhancements, not solving scalability bottlenecks etc), we'll have a controlled deliberate roll out plan that involves manual QE checks along the way, as well as a gradual roll-out and A/B testing.

When someone complains that something too complex, I've been trained by Jerry Weinberg and his work to ask too complex compared to what? In the same way, when someone says that it doesn't scale to have testers look at every change, I ask why it can't scale. Programmers make every change, don't they? After all, "it can't scale" is one of the great myths about the Agile Manifesto. There's a difference between "it's too complex" and "I don't have a more useful model than the one I currently have"; between "it can't scale" and "I don't know how to solve the problem of making it scale."

One approach to solving complexity or scaling problems is to reconsider what testing is and where it happens. Pair programming in which no tests are written is a form of testing (we often call it "continuous review", but review is a form of testing). Behaviour-development, in which we check that each function at least to some degree does what it should as we build it, do is a form of testing. And continuous deployment is a form of testing, too.

One definition of testing is "the gathering of information with the intention of informing a decision" (that's paraphrased from Perfect Software and Other Illusions About Testing, by Jerry Weinberg). The complaint that something is too complex is information. Your testers are telling you that you about the testability of the product, compared to their ability to test it. There are all kinds of details to the story to which we're not privy. Maybe they believe that they have to retest everything in the product every time the programmers make a change; maybe they believe that testing means seeing the visible manifestation of every change from the user's point of view; maybe there are insufficient hooks for the kind of automation they want to apply; maybe they are being mandated to do other things that impinge on their ability to study and grasp the issues that they're facing; or maybe they're the creditors on some of the programmers' technical debt, and the number of bug reports that they have to investigate and report is taking time away from their test design and execution—that is, their test coverage.

There are constraints to every testing assignment, and as Jerry says (quoted in James Bach's article in The Gift of Time), it's the first responsibility of the tester to figure out ways to get around those constraints. But that may not always be possible, given the situation.

Another form of testing is asking questions, like "if you don't involve testers when you fix bugs, make performance enhancements, solve scalability bottlenecks, etc., how do you know that you've fixed, enhanced, or solved?" And others like, "What are your testers doing?" "Are they only testing new features?" "Are you aware of how useful skilled testers can be?" "Do you see any opportunities for adding efficiencies to your model of testing?"

Your point about the sheer number of bugs we have? you're right. Our software has way more bugs than I'd like. It's a byproduct of the choices made when the company was small: to prioritize determining what the customer actually wants at almost any cost. We would absolutely love to have a high quality client, and we're working every day towards that goal.

Continuous Deployment let's you write software *regression free*, it sure doesn't gift you high quality software. As a start-up, we're faced with hard decisions every day about where to make our product higher quality; most of the complaints you have would be immediately ignored by the majority of computer users and so we can't in good faith prioritize them over the things that ARE causing our users trouble.

I'll respond to the things I disagree with in a moment, but I want to draw attention to the most important aspect of Timothy's reply: he highlights that developing software and services and products and systems is a constant set of tradeoffs, and that, just like the rest of us, he and the rest of the IMVU crew are making these decisions all the time. That's important because, as I'd like to emphasize, my notion of IMVU's quality doesn't matter. "Quality is value to some person(s)". When James and I teach Rapid Software Testing, we add something to that: "Quality is value to some person(s) who matter". I'm an outsider. I don't have any interest in using chat illustrated by anime-looking 3D avatars who teleport from place to place. I have no interest in handing IMVU my money for this service. I have no interest in the community this stuff supports. (So why did I even bother to look at the service? I am interested in software development and testing, and I wanted to investigate the relationship between a million test cases a day, a million dollars a month in revenue, and the system being tested thereby.)

I'm going to introduce something perhaps more controversial here. Even if I were working for IMVU, as a tester, I still wouldn't matter. How can I, a tester, say that? It's because my role is not to push my values down the throats of the programmers and business people that I serve. Saying that I don't matter is a simplification; my values don't matter as much as the business values and the customer values do. I do matter, but only precisely to the degree that I deliver those people the information that they value to inform their decisions. I can find defects in the features offered by the product; I can tell them about things that I see as driving risk; I can tell them about things that I've discovered that represent a threat to the value of the product to someone who does matter. And at that point, the decision is up to them.

A couple of other points:

While I agree that continuous deployment doesn't give you high-quality software (in the sense of few bugs), I disagree that it lets you write software regression-free. It performs some tests on the software that might find regression if that regression happens to be covered by one of the tests. That's not a bad thing in itself; that's a good thing. The bad part is that, once again, it provides The Narcotic Comfort of the Green Bar. There's a big difference between "our tests find no regressions" and "there are no regressions".

Second, continuous deployment is Way Cool. As Elisabeth suggested, that IMVU can push out 50 deployments a day is impressive. But it reminds me of a story (I think it was Al Stevens) in which you go to the circus. Last year, they had a bear on roller skates, which impressed you. This year, the bear is on motorized roller skates. And you're dazzled for a moment, until you consider, "Wait a second... do I really want to see a bear on motorized roller skates?" 50 deployments a day is impressive, but 50 deployments of what? For what purpose?

Could it be that the focus on deploying 50 times a day represents opportunity cost against other equally or more desirable goals? Goals like, say, addressing the problem "Our software has way more bugs than I'd like"; or addressing the complaint from the testers that the testing mission is too complex; or investigating the functionality and security problems that the customers seem to be reporting and that might represent a serious threat to the value of the product? Is 50 deployments a day provided business value that can't be delivered any other way? Any other way? Would customers or IMVU itself suffer some loss if they had to wait 30 minutes for a code drop, instead of 15? I repeat: I don't know in this case. I don't know whether five deployments a day would be better, or five hundred a day, or one every five hundred days. I do know that part of testing is noticing the cost of one activity and its potential impact on the value of another.

The vast majority of developers take one look at what they think our product is and don't bother to give it a try; I'm happy to see a thoughtful open-minded dive into IMVU from a developers perspective.

I'm a specific kind of a developer; I'm a tester. As such, it's my particular bent to investigate things, and not to take them on faith, and to report on what I've found. I genuinely appreciate Timothy's thoughtful and open-minded reply, and I thank him for triggering some of the questions and observations above.

More IMVU comment followup: Survivorship Bias

2009-03-07T16:24:00.005-05:00

In the comments to our rapid test of IMVU, Elisabeth Hendrickson said, "Great list of issues. However, I suspect that these issues don't really interfere with the core value to the target users." That might be true. I think it might be more accurate to suggest that the issues don't interfere with the core value to the existing users. That is, IMVU might be hitting the dartboard, but not the target. (I emphasize: I don't know that either way.)

This brings me to a mention of one of the critical thinking errors discussed in The Black Swan: survivorship bias. Nassim Taleb retells a story originally told by Marcus Tullius Cicero.

"One Diagoras, a nonbeliever in the gods, was shown painted tablets bearing the portraits of some worshippers who prayed, then survived a subsequent shipwreck. The implication was that praying protects you from drowning. Diagoras asked, 'Where were the pictures of the people who prayed, then drowned?'

"The drowned worshippers, being dead, would have a lot of trouble advertising their experiences from the bottom of the sea. This can fool the casual observer into believing in miracles.

...

"It is easy to avoid looking at the cemetary while concocting historical theories. But this is not just a problem with history. It is a problem with the way we construct samples and gather evidence in every domain. We shall call this distortion a bias, i.e. the difference between what you see and what is there." (The emphasis there is Taleb's.)

Taleb here is talking about survivorship bias, a form of selection bias. History tends to be a story written by the side that survived the war, and the audience tends to be on the same side. The historians tend to get their stories from the people who survived. Moreover, the people interviewed typically represent a limited subset of the people who could have been interviewed.

Testing informs an ongoing story of the product. When we consider test results, it's easy to be lulled to sleep by the narcotic comfort of the green bar. The green bar provides us with the good news that the tests we have run are all passing. There is great value to that, and I don't want to gainsay it. Yet confirmatory tests are less testing and more checking. If we're going to avoid being fooled, we must also remember to consider what we might learn from the tests that we haven't run. If we're considering the success of our organization, as Peter Drucker suggests, we need to consider the people who aren't our customers, in addition to the people who are.

In the next post, I'll respond to the excellent comment I received from Timothy Fitz, who posted the blog posting that inspired my brief investigation of IMVU.