Yesterday I published the first three premises that underlie the Rapid Software Testing methodology developed and taught by James Bach and me. Today’s two are on the nature of “test” as an activity—a verb, rather than a noun—and the purpose of testing as we see it: understanding the product and imparting that understanding to our clients, with emphasis on problems that threaten the product’s value.
4. A test is an activity; it is a performance, not an artifact. Most testers will casually say that they “write tests” or that they “create test cases.” That’s fine, as far as it goes. That means they have conceived of ideas, data, procedures, and perhaps programs that automate some task or another; and they may have represented those ideas in writing or in program code.
Trouble occurs when any of those things is confused with the ideas they represent, and when the representations become confused with actually testing the product. This is a fallacy called reification, the error of treating abstractions as though they were things. Until some tester engages with the product, observes it and interprets those observations, no testing has occurred.
Even if you write a completely automatic checking process, the results of that process must be reviewed and interpreted by a responsible person.
5. Testing’s purpose is to discover the status of the product and any threats to its value, so that our clients can make informed decisions about it. There are people that have other purposes in mind when they use the word “test.” For some, testing may be a ritual of checking that basic functions appear to work. This is not our view.
We are on the hunt for important problems. We seek a comprehensive understanding of the product. We do this in support of the needs of our clients, whoever they are.
The level of testing necessary to serve our clients will vary. In some cases the testing will be more formal and simple, in other cases, informal and elaborate.
In all cases, testers are suppliers of vital information about the product to those who must make decisions about it. Testers light the way.
I’ll continue with the last three premises of Rapid Software Testing tomorrow.
[…] Premises of Rapid Software Testing, Part 2 (Developsense Blog) […]
[…] Premises of Rapid Software Testing, Part 2 (Developsense Blog) […]
Very nice post. I just stumbled upon your blog
and wanted to say that I have truly enjoyed browsing your blog posts.
In any case I will be subscribing to your rss feed and I hope you write again very soon!
This is good stuff, will you make all premises available on one page?
Michael replies: Yes, we’ll be posting this on both James’ site and on mine.
The noun view seems pretty dominant in a lot of “traditional testing education”, e.g. test plan instead of planning as an activity (that also generates understanding)
A part of the reason for this might be that “nouns” seem easier to teach than “verbs”. E.g. if you show a test plan template it is not difficult to produce something that looks like a test plan. Learning to do the activities for “good” testing is more difficult?
That’s an insightful observation (which is the norm from you, so it seems). Any complex cognitive activity involves a lot of tacit knowledge to perform. Some tacit knowledge can be attained through practice that starts with the study of explicit knowledge (written, spoken, diagrammed, illustrated, and so forth). Other tacit knowledge is transferred through social interactions with skilled people: observation, participation, collaboration, interpretation, feedback, teaching, experiential exercises, conversation (distinct from lecture); working with or even just hanging around with people who are experienced in the activity, or who are in the process of developing it themselves. The social dimension might explain difficulties in some forms of academic learning in some domains where apprenticeship can be more successful.
All this points to a blog post or two or three about the work of Harry Collins. Stay tuned.
[…] Premises of Rapid Software Testing, Part 2 Written by: Michael Bolton […]
Two comments:
1. “A test is an activity; it is a performance, not an artifact.” Given that I have decided to devote my career to create a test case generating tool it is perhaps ironic that I agree with this statement. Even so, I agree with it. It is truly unfortunate that so much of the software testing community doesn’t make this distinction.
Michael replies: It’s not at all ironic to hold that view if you create a test case generating tool that assists skillful test activity. You recognize that the test cases aren’t the tests, and since I’ve known you’ve been actively seeking to understand testing. You have my respect for that. I agree that there’s still lots of work to be done in terms of helping people to notice how much our field depends on skillful, thoughtful practice.
2. I love Rikard’s comments that:
“The noun view seems pretty dominant in a lot of “traditional testing education”, e.g. test plan instead of planning as an activity (that also generates understanding)
A part of the reason for this might be that “nouns” seem easier to teach than “verbs”. E.g. if you show a test plan template it is not difficult to produce something that looks like a test plan. Learning to do the activities for “good” testing is more difficult?”
I’d never thought of things in quite that way but I think Rikard’s comments go to the heart of why Context Driven Testing and RST are so powerful.
Kindly Twitter correspondent Jari Laakso (@JariLaakso) points us to this. And again, stay tuned for more along those lines.
Greetings,
I feel like I’m taking the bait but here goes…
Haven’t we done some testing in order to figure out a plan, design the test cases, write some test code/tool?
Cheers,
Vern
Michael replies: Yes, that’s true (and it’s true to the degree that we should probably polish it eventually). The point is that it’s people configuring, operating, observing, and evaluating some product that constitutes testing. You can do that with anything that people have produced. It’s a bit of a stretch to think about “operating” a specification, but it could mean “trying to make sense or use of it”. So in that case, we’ve been testing the specification, and testing the product in the larger sense; still, we haven’t tested the operating program as such.
Something I would like to share with you.
This morning we had our quarterly general meeting in which our IT business line comes together to learn from business, discuss achievements and lay out plans for the next quarter.
This time one of the talks, by management, was about professionalism. Professionalism, in short, was described as the need to educate yourself through courses, workshops, internet (blogs, twitter), etc. This education should then be leveraged by actively seeking experiences in which it could be applied. As an example of this the manager used the activity of testing. Testing, he said, when performed well uses the expertise of the tester and involves all roles in the act of creating better quality. Not by delivering documents but by its execution and by creating a investigative mindset. Something testers, for instance, have learned last and this year in their (RST, exploratory testing) courses.
Just so you know your efforts have not gone to waste 😉
Michael replies: Ah, that breath of fresh air was sweet. Thank you for telling me about it.
I would like to split ‘the case against the test case’ (ref. James Bach?) or ‘the case against the test as artifact’ into two separate observations. You mention both these observations; abstraction and reification. I think that our quest should be to illuminate the damages caused by both, not just the damages caused by reification.
I believe test cases have their origin in the process methodologies that govern the testing process. Test cases are an abstraction of the work that is done. They are countable, classifiable, quantifiable and controlable units that offer the possibility to manage testing. Tests as artifacts, in essence, are ‘units of control’. Within the traditional ‘command and control’ test management paradigm, control and decision making are thus taken away from the tester. In unfortunate situations the focus on artifacts largely hides the work that is actually being done.
Michael replies: When you refer to test cases as you do above, I think you’re talking about a particular formal sense of “test case”. To be fair, there are other interpretations for “test case”. One definition I find reasonable is this one: “a test case is a specific question that we want to ask of a program.” (I believe that definition is used in the Black Box Software Testing course; I haven’t checked.) As James put it recently, we’re not opposed to test cases so much we’re opposed to the worship of test cases as the centre of testing.
I recently read a nice book by John Seddon (Freedom from Command and Control) on the subject of management by artifacts that are abstracted from the work. I think his theory is right on the money when it comes to test cases.
I haven’t yet read Seddon, and I’d love to, by all accounts. I’m looking forward to meeting him at EuroSTAR this year.
I believe the reification of the test case is something that each and every tester has him- or herself to blame for. There is no compelling reason to stick to the ‘test as artifact’ paradigm when you feel that the actual work is something more, and more valuable, than typing and executing scripts. In any case, our educational programs on software testing, should prevent us from falling into the reification trap.
I agree. That’s certainly one of the goals of Rapid Software Testing.
“For some, testing may be a ritual of checking that basic functions appear to work.”
I think the words “ritual” and “basic” may be perceived as derogatory. ritual might indicate indifferent. I think many testers work very hard and make sure they test *all* functionality. I also think they are implicitly performing other testing.
Michael replies: We chose our words carefully.
Ritual might indicate indifference to you. When I look up “ritual” in the Oxford English Dictionary, I see this: “A religious or solemn ceremony consisting of a series of actions performed according to a prescribed order.” Apart from the religion or solemnity bit, this definition is to me indistinguishable from “scripted testing”. Many testers do work very hard—that’s why we chose the word “some”—but how hard they work is orthogonal to whether they are behaving ritualistically, according to the definition above. Testers who believe that they are testing all functionality are kidding themselves, and there’s more to testing than the functional aspects. People who have studied testing seriously know that. And yes, it’s possible to obtain several kinds of coverage at once, if you are paying attention and not behaving ritualistically.
I don’t think any tester will like to hear those words describing their work.
Again, we chose our words carefully. Note the sentences before the one to which you seem to object: “There are people that have other purposes in mind when they use the word ‘test’.” Those people might be programmers, managers, business analysts, process-model mongers, or testers, and some of those may indeed believe that testing is equivalent to checking. Note the sentence after the one to which you object: “This is not our view.” Whether you’re a tester or not, if you feel the description doesn’t apply to you, no problem‐right? If you feel that it does, you’ve got something to think about.
@nilanjan
“””
I think the words “ritual” and “basic” may be perceived as derogatory. ritual might indicate indifferent. I think many testers work very hard and make sure they test *all* functionality. I also think they are implicitly performing other testing.
“””
The key here is to make the distinction between working very hard and working smart.
My comment does not imply any judgement on people capabilities. Everyone working in this industry is able to work smarter. That path is more demanding imo but more rewarding too.
[…] of Rapid Software Testing part 1 (engagement with software development), part 2 (the nature of testing as an investigative activity) and part 3 (our relationship to our clients) by Michael Bolton – “1) Software projects […]
[…] of Rapid Software Testing by Michael Bolton – Part 1 – Part 2 –Part […]
[…] I am speaking a) authoritatively about how we use terms in Rapid Testing Methodology, b) non-authoritatively of my best knowledge of how testing is thought of more broadly within the […]