Coding QA Podcast on Exploratory Testing (Part 2)

Here’s Part 2 of my notes on the CodingQA podcast, in which Matthew Osborn and Federico Silva Armas chat with James Bach about the exploratory testing and session-based test management.

Skills of Exploratory Testing

• If you want to develop a list of testing skills, you might find it helpful to start with the Exploratory Skills and Dynamics sheet, by James Bach, Jon Bach, and Michael Bolton.
• One of the core skills of excellent exploratory testing is self-awareness and self-management; managing your attention and your focus.
• Another skill of excellent testing is using your emotions as a test tool. Frustration or confusion about something may affect the user in the same kind of way that it affects you. Treat your emotional reaction as a trigger; start looking into why you might be experiencing that emotion.

Test Cases

• Except in well-defined contexts, counting test cases means absolutely nothing. When someone gives you a number, telling you that they have N test cases, they’re telling you they have N files.
• Test cases are like briefcases; if you want to know something about them, you have to look inside them to know what’s going on.
• What people do when they test is different from what’s in the test case. The test is what the tester does (and thinks –MB), not what the test case says.
• In an automated test, the problem is that there might not be an oracle to observe a bug that’s outside the scope of that test, and there’s no human to say, “That’s odd. I wonder what’s going on there.”
• When managers ask you to count your test cases, they’re asking you to count checks, and that doesn’t get to the essence of what testing is.

Managing Exploratory Testing

• If you’re going to manage and train exploratory testers, it would probably be a good idea to break out a list of constituent skills of excellent testing (see above). Then ask, for each skill: How can we observe it? And how can we teach it?
• If you’re going to manage any highly cognitive work, you’re going to have to participate in the work on a regular basis. If managers aren’t involved in the work themselves, they don’t really know what their people are doing; they know only the story that their people are telling them. The less management is involved in testing, the more likely they are to come up with weird astrology-like methods for managing testing.
• How do you manage testing on a large scale? Like the military, develop a corps of sergeants. Test leads—skilled, trusted testers—are trained on the job, and train other testers and other test leads. This is also consistent with the way that doctors, pilots, scientists, and other cognitively skilled professionals are trained.
• Good doctors, welders, and pilots are not trained by filling out templates.
• The other component to managing exploratory testing is session-based test management: uninterrupted blocks of tester time.

Session-Based Test Management

• A block of uninterrupted time (a session, typically 90 minutes) is accounted in terms of three activities: test design and execution (T-time); bug investigation and reporting (B-time); and setup (S-time). You account from your time as test design and execution, unless it’s interrupted by B-time or S-time.
• T-time has the highest precedence, then B-time, then S-time. If you’re doing more than one thing at once (e.g. investigating a bug while you’re setting up), report it as the higher-precendence activity.
• Accounting for time in this way allows you to identify ways in which more testing can get done. (Here’s the first part of an example, and here’s the second part. Note that the metrics don’t tell you what’s good; they suggest where to look. The purpose of this kind of measurement is not to provide answers, but to provide more focused questions on what we should do next and how we should allocate our time. –MB)
• In a session format, after you’ve gathered a bit of data, a picture emerges of how much you can get done. A middle-level test strategy language evolves: “That part of the product looks like it will take two recon sessions, three analysis sessions, ten deep coverage sessions, and two closure sessions.” (The CodingQA guys have developed their own language—discovery sessions and targeted sessions. By this kind of language, testing sessions turn into calendar time.
• In exploratory testing, testing resources develop: test data, test platforms, test ideas in documents.
• Test leads handle the debriefing of session. Debriefings and formal acceptance of the test notes are essential in developing discipline in note-taking and reporting.
• The SBTM protocol was initially to debrief at the end of every session, but this wasn’t always possible. More recently, the protocol is to debrief in groups; everyone reporting to a given test lead meets at the end of the day for a half-hour debriefing session.
• The overall philosophy should be to tinker with the process to get the bureaucracy out of the way of the thinking tester.
• After two years of success with SBTM, at the place where it was developed, management stopped asking for metrics, but the team kept using the session concept.

For more information on session-based test management, see http://www.satisfice.com/sbtm.

Recording Exploratory Testing

• Every novice’s first set of test notes looks like this: a single page that says, “I DID THE TESTING”. The second set of test notes contains every keystroke and every mouse click. Novices need training and instruction in how to create good notes that strike the right balance between detail and conciseness.
• The fastest way, in the long run, to train novices is to provide them with pairing and personal supervision, showing them how to do the notes.
• Testers are not often challenged—and therefore have not developed the skill—to give expert reports on what they actually did as they were testing. For a while, senior testers who are being trained in this kind of reporting can get annoyed and frustrated.
• The more practice you get in note-taking, the more skill you develop, and the less it interferes with your thought process.
• Watch the episode “Gallileo Was Right” from the HBO series “From the Earth to the Moon”, and look for the segment in which astronauts are trained to become geological observers and reporters. It’s all about rapid testing!
• Federico reports an irony: that on his team, exploratory testing (so often mistaken for “testing without writing anything down”) became subject to the opposite problem: it became all about the test notes, which disrupted testing and learning.
• Test notes in SBTM should be focused not on the literal steps of execution, but on questions, wondering, models, test ideas, coverage ideas, evidence of learning.
• Correct the problem by sitting with testers and test with them.
• In excellent exploratory testing, the testing is paramount. If the paperwork gets in the way, then the paperwork has to change until there’s a reasonable balance between paper and testing.
• Combine the notes with recordings of the session—a videotape; a screen recording system; application and server log files; tools like Burp Proxy or Fiddler. Lean on automation and tools to keep records of the minutiae of what’s going on.
• Beware of systems that have encoded or encrypted communication between server and client. The transactions might be logged, but might be hard to decode or decrypt.
• You don’t look at the data unless you experience a problem; you normally file the video and not worry about it unless you need hard evidence or review.
• With a screen recorder, it might be hard to make sense of what happened. Therefore: keep notes electronically on the same machine that you’re testing on. This causes the notes to pop up on the screen concurrent with the test, which provides insight into what the tester was thinking.
• The notes should be just enough to figure out the highlights of what you were doing.
• Timestamps help to link the test idea with what actually happened and when it happened. This allows you to move quickly, taking notes that are more easily reviewable.