Last week posted a blog entry on automation and coverage, in which I questioned the usefulness of trying to cover “everything” with automated tests, comparing them to the CCTV cameras that are in use all over the place, but especially in Britain. Despite the limitations of such schemes, there might also be some useful aspects. What might they be?
- For certain areas that we decide to cover with a camera, like public streets or open squares, we won’t necessarily need to monitor all of the images all the time, but if the data were stored, we could review it—especially after trouble had occurred—to try to find out what went wrong and who the instigators were. This is like a continuous logging system for a program under test.
- For places that are hidden or potentially dangerous, like subway underpasses, we might want to set up a motion-activated camera. This is like an event-based logging system, in which we record some aspect of the system state based on some occurrence that we anticipate.
- For high-traffic areas like urban highways, where we want to direct lots of attention about the flow of vehicles, it might be a good idea to set up a number of cameras and monitors at various points on the roadway, and cycle through the images every few seconds. Even if we don’t see the incident as it happens, CCTVs allow us to spot trouble fairly shortly after it happens, since traffic will tend to change its behaviour, typically bunching up behind a blockage. Sophisticated cameras would allow us to pan and zoom, inspecting the specific nature of the blockage, and will help us determine how to respond. This another form of logging, more like polling. We can perhaps combine it with probes or indentifiers on the data to make it easy for us to follow a record or data packet that attracts our interest.
- The same kind of cameras can follow and monitor the (mis)behaviour of drivers, such as those who are blatantly speeding or breaking the rules. This is like a monitor that allows us to track the progress of a particular data set through an application.
- We can set up a particular camera at a particular interface that allows us to stop traffic and alter it or its behaviour in some way before we send it on. I think of tools like Burp Proxy or Fiddler in this way.
- In the last few years, more and more people have obtained portable cameras and camera phones; most of those devices do video, too. See something interesting as you’re testing? Use a literal camera, an inexpensive point-and-shoot model, to record some aspect of your activities as you’re testing, or as you fill up a white board or a set of post-it notes during a meeting.
- Tourists and amateur newshounds, on the spot as an event occurs, can often take pictures that later turn out to be valuable. A tester can take advantage of the data capture tools that the operating system provides. Windows Vista comes with the Snipping Tool, and previous versions of Widows include Print-Screen (to print the screen to the printer), Shift-Print-Screen (to place a copy of the entire screen on the Widows clipboard), and Alt-Print-Screen (to place the topmost window of the screen on the clipboard.) Since these tools are readily available, it’s as though everyone had a camera of his or her own.
- A real video camera, or a screen recording tool like BBST Assistant, can allow you to record the actions of an end-user as he or she works her way through the application.
- Many applications allow you to go forwards and backwards. See if you can get your application into a tangled state by performing activities, backtracking with Ctrl-Z, and then perform the activities again, the same way or with variation. Then backtrack again. Then do it again, perhaps brancing to a different point. Set up a monitor of some kind to help you record and observe what happens. The idea is to get the application to get confused and to tie itself in knots.
- Ask the programmers about the built-in error checking in the program, and ask if the error -checking routine logs its actions, perhaps by a configuration switch.
- A program can be instrumented to provide interfaces to coverage tools, or to requirement tools such as Fitnesse. Not only can you get a shapshot of results from the running program, but you can do it again and again.
- Psychologists often put the patient in a room with a one-way mirror with a camera behind it, maybe hooking some electrodes up to his skull, and interview him or merely watch his behaviour. A debugger is a kind of equivalent of this approach.
- When there’s an important event happening, news stations send television crews to record and report. Sometimes that footage is helpful in determining what’s going on, but note that the film crew is constantly making choices about where to place the cameras and what to observe. That is, every observation is a subjective observation at some level.
- Note that the television show or movie that you see is typically the product of extensive preparation, rehearsal, gobs of technical equipment, and sophisticated editing. If you choose to automate and record everything you will generate a lot of stuff that will end up on the cutting room floor. Who gets to do the editing?
These are just a few examples; I’m sure you’ll come up with many more (and they’re welcome here). Note the kind of automation that I’ve been talking about here involves little to no direct, continuous control or monitoring people, and note that most of the approaches can help to reduce dependence on elaborate prescriptive or retrospective documentation. Not all of these approaches will fit your context, but there are several here that might. The idea is not to adopt them all, but to consider one or more possibilities that might be helpful in your context.