A friend that I’m just getting to know runs a training and skills development program for new testers. Today he said, “My students are now starting a project which includes test design, test techniques, and execution of testing. Do you have any input or advice for them?” Here’s my reply.
Test design, test techniques, and execution of testing are all good things. I’d prefer performing tests to “test execution”. In that preference, I’m trying to emphasize that a test is a performance, by an engaged person who adapts to what he or she is experiencing. “Test execution” sounds more like following a recipe, or a programmed set of instructions.
Of these things, my advice is to perform testing first. But that advice can be a little confusing to people who believe that testing is only operating some (nearly) finished product in a search for coding errors. In Rapid Software Testing, we take a much more expansive view: testing is the process of evaluating a product by learning about it through experiencing, exploring and experimenting, which includes to some degree questioning, studying, modeling, observation, inference, etc.
Testing includes analysis of the product, its domain, the people using it, and risk related to all of those. Testing includes critical thinking and scientific thinking. Testing includes performing experiments—that is, tests—all the way along. But I emphasized the learning part just back there, because testing starts with learning, ends with reporting what we’ve learned, feeds back into more learning, and is about learning every step of the way.
We learn more most powerfully from experiencing, exploring, and experimenting; performing experiments; performing tests. So, my advice to the new tester is to start with performing tests to study the product, without focusing too much on test design and test techniques, at first.
Side note: the “product” that you’ve been asked to test may not be a full, working, running piece of software. It may be a feature or component or function that is a part of a product. It may be a document, or a design drawing, a diagram, or even an idea for a product or feature that you’re being asked to review. In the latter cases, “performing a test” might mean the performance of a thought experiment. That’s not the same as the real-world experience of the running product, hence the quotes around “performing a test”. A thought experiment can be a great and useful thing to help nip bugs in the bud, before bugs in an idea turn into bugs in a product. But if we want to determine the real status of the real product, we’ll need to perform real testing on the real product.
So: learn the product (or feature, or design, or document, or idea), and identify how people might get value from it. Survey the product to identify its functions, features, and interfaces. Explore the product, and gain experience with it by engaging in a kind of purposeful play. Don’t look for bugs, particularly—not right away. Look for benefits. Look for how the product is intended to help people get their work done, to help them to communicate with other people, to help them to get something they want or need, to help them to have fun. Try doing things with the product—accomplishing a task, having a conversation, playing the game.
Record your thoughts and ideas and feelings reasonably thoroughly. Pay attention to things that surprise you, or that trigger your interest, or that prompt curiosity. Note things that you find confusing, and notice when the confusion lifts. If you have been learning the product for a while, and that confusion hasn’t gone away, that’s significant; it means there’s some confusing going on. If you get ideas about potential problems (that is, risks), note those. If you get ideas for designing tests, or applying tools, note those too.
Capture what you’re learning in point form, or in mind maps, or in narratives of what you’re doing. Sketeches and diagrams can help too. Don’t make your notes too formal; formality tends to be expensive, and it’s premature at this stage. It might be a good idea to test with someone else, with one person focusing on interacting with the product, and the other minding the task of taking notes and observations. Or you might choose to narrate and record your survey of the product on video to review later on; or to use like the black boxes on airplanes to figure out what led to problems or crashes.
You’ll probably see some bugs right away. If you do, note them quickly, but don’t investigate them. If you spotted a bug this easily, this early, and you take a quick note about it, you’ll almost certainly be able to see the bug again later. Investigating shallow bugs is not the job at the moment. They job right now is to develop your mental model of the product, so that you become prepared to find bugs that are more subtle, more deeply hidden, and potentially much more important or damaging.
Identify the people who might use the product… and then consider other groups of people you might have forgotten. That would include novice users of the product; expert users of the product; experts in the product domain who are novice users of the product; impatient users; plodding users; users under pressure; disabled users… Consider the product in terms of things that people value: capability, reliability, usability, charisma, security, scalability, compability, performance, installability… (As a new tester, or a tester in training, you might know these as quality criteria.)
You might also want to survey the product from the perspective of people who are not users as such, but who are definitely affected by the product: customer support people; infrastructure and operations people; other testers (like testing toolsmiths, or accessibility specialists); future testers; current developers future developers… Think in terms of what they might value from the product: supportability, testability, maintainability, portability, localizability. (These are quality criteria too, but they’re focused on the internal organization more than on their direct benefit to the end user.)
Refine your notes. Create lists, mind maps, tables, sketches, diagrams, flowcharts, stories… whatever helps you to reflect on your experience.
Share your findings with other people in the test or development (or in this case, study) group. That’s very important. It’s a really good way both to share knowledge and to de-bias ourselves and to reveal things that we might have forgotten, ignored, or dismissed too quickly.
Have these questions in mind as you go: What is this that we’re building? Who are we building it for? How would they get value from it? As time goes by, you’ll start to raise other questions: What could go wrong? How would we know? How could people’s value might be threatened or compromised? How could we test this? How should we test this? Then you’ll be ready to make better choices about test design, and applying test techniques.
Of course, this isn’t just advice for the new tester. It applies to anyone who wants to do serious testing. Testing that starts by reading a document and leaps immediately to creating formal, procedurally scripted test cases will almost certainly be weak testing, uninformed by knowledge of the product and how people will engage with it. Testing that starts with being handed some API documentation and leaps to the creation of automated checks for correct results will miss lots of problems that programmers will encounter—problems that we could discover if we try to experience it the way programmers—especially outside programmers—will.
As we’re developing the product, we’re learning about it. As we’re learning the product, we’re developing ideas about what it is, what it does, how people might use it, and how they might get value from it, and that learning feeds back into more development. As we develop our understanding of the product more deeply, we can be much better prepared to consider how people might try to use it unsuccessfully, how they might misuse it, and how their value might be threatened. That’s why it’s important, I believe, to do test execution perform testing first—to prepare ourselves better for test design and for identifying and applying test techniques—so we can find better bugs.
This post has been greatly influenced by ideas on sympathetic testing that came to me—over a couple of decades—from Jon Bach, James Bach, and Cem Kaner.
Really so valued article, thank you