In Rapid Software Testing, James Bach, our colleagues, and I advocate an approach that puts the skill set and the mindset of the individual tester—rather than some document or tool or test case or process modelY—at the centre of testing. We advocate an exploratory approach to testing so that we find not only the problems that people have anticipated, but also the problems they didn’t anticipate. We challenge the value of elaborately detailed test scripts that are expensive to create and maintain. We note that inappropriate formality can drive testers into an overly focused mode that undermines their ability to spot important problems in the product. We don’t talk about “automated testing”, because testing requires study, learning, critical thinking, and qualitative evaluation that cannot be automated. We talk instead about automated checking, and we also talk about using tools (especially inexpensive, lightweight tools) to extend our human capabilities.
We advocate stripping documentation back to the leanest possible form that still completely supports and fulfills the mission of testing. We advocate that people take measurement seriously by studying measurement theory and statistics, and by resisting or rejecting metrics that are based on invalid models. We’re here to help our development teams and their managers, not mislead them.
All this appeals to thinking, curious, engaged testers who are serious about helping our clients to identify, evaluate, and stamp out risk. But every now and then, someone objects. “Michael, I’d love to adopt all this stuff. Really I would. But my bosses would never let me apply it. We have to do stuff like counting test cases and defect escape ratios, because in the real world…” And then he doesn’t finish the sentence.
I have at least one explanation for why the sentence dangles: it’s because the speaker is going through cognitive dissonance. The speaker realizes that what he is referring to is not his sense of the real world, but a fantasy world that some people try to construct, often with the goal of avoiding the panic they feel when they confront complex, messy, unstable, human reality.
Maybe my friend shares my view of the world.
That’s what I’m hoping. My world is one in which I have to find important problems quickly for my clients, without wasting my client’s time or money. In my world, I have to develop an understanding of what I’m testing, and I have to do it quickly.
I’ve learned that specifications are rarely reliable, consistent, or up to date, and that the development of the product has often raced ahead of people’s capacity to document it. In my world, I learn most rapidly about products and tools by interacting with them.
I might learn something about the product by reading about it, but I learn more about the product by talking with people about it, asking lots of questions about it, sketching maps of it, trying to describe it. I learn even more about it by testing it, and by describing what I’ve found and how I’ve tested. I don’t structure my testing around test cases, since my focus is on discovering important problems in the product—problems that may not be reflected in scripted procedures that are expensive to prepare and maintain. At best, test cases and documents might help, maybe, but in my world, I find problems mostly because of what I think and what I do with the product.
In my world, it would be fantasy to think that a process model or a document—rather than the tester—is central to excellent testing (just as only in my fantasy world could a document—rather than the manager—be central to excellent management). In my world, people are at the centre of the things that people do.
In my version of a fantasy world, one could believe that conservative confirmatory testing is enough to show that the product fulfills its goals without significant problems for the end user. In my world, we must explore and investigate to discover unanticipated problems and risks.
If I wanted to do fake testing, I would foster the appearance of productivity by creating elaborate, highly-polished documentation that doesn’t help us to do work more effectively and more efficiently. But I don’t want to do that. In my world, doing excellent testing takes precedence over writing about how we intend—or pretend—to do testing.
In my version of a dystopic fantasy world, it would be okay to accept numbers without question or challenge, even if the number had little or no relationship to what supposedly was being measured. In my world, quantitative models allow us to see some things more clearly while concealing other things that might be important. So in my world, it’s a good idea to evaluate our numbers and our models—and our feelings about them—critically to reduce the chance that we’ll mislead ourselves and others.
I could fantasize about a world in which it would be obvious that numbers should drive decisions. In what looks like the real world to me, it’s safer to use numbers to help question our feelings and our reasoning.
Are you studying testing and building your skills? Are you learning about and using approaches that qualitative researchers use to construct and describe their work? If you’re using statistics to describe the product or the project, are you considering the validity of your constructs—and threats to validity?
Are you considering how you know what you know? Are you building and refining a story of the product, the work you’re doing, and the quality of that work? If you’re creating test tools, are you studying programming and using the approaches that expert programmers use? Are you considering the cost and value of your activities and the risks that might affect them? Are you looking only at the functional aspects of the product, or are you learning about how people actually use the product to get real work done?
Real-world people doing real-world jobs—research scientists, statisticians, journalists, philosophers, programmers, managers, subject matter experts—do these things. I believe I can learn to do them, and I’m betting you could learn them too. It’s a big job to learn all this stuff, but learning—for ourselves and for others—is the business that I think we testers are in, in my real world. Really.
Is your testing bringing people closer to what you would consider a real understanding of the real product—especially real problems and real risks in the product—so that they can make informed decisions about it? Or is it helping people to sustain ideas that you would consider fantasies?
Michael,
Thanks for your thoughts. Lots of great questions to consider and cognitive dissonance to wade through. How binary is this view of testing though? I haven’t been to RST yet (though I hope to someday) so I’m speaking without having experienced the training, though I have read others experiences and been to some conference sessions/workshops where ideas or pieces of it have been presented.
Cem Kaner and Rex Black’s debate about the schools of testing at STP Con got me thinking and I just wondered at what point is the methodology (RST) considered by those receiving the training as “the” way to approach testing? Is another way/methodology of testing inherently in opposition to RST merely by existing or is there room for other methodologies that might be equally as effective?
Could the Rapid Software testing methodology be considered an implementation of the principles/ideas in the context-driven school of testing? If so, I’m curious if you are aware of any other implementations/methodologies of those principles?
Brett
Michael replies: How binary is it? I certainly can’t give that a yes or no answer. (See what I did there?)
Our intention is that Rapid Testing is an implementation of the context-driven principles, with some premises and therefore some biases built-in and acknowledged—and, we hope, managed. Note that I said “an implementation”, not “the implementation”. Context-driven thinking doesn’t afford the latter. Any implementation of the principles would by necessity be an implementation. And of course, there are lots of them; as many as there are people who consider and engage the context first.
Rapid Testing comes with its own set of premises and principles, expressed in the Premises of Rapid Testing. The overarching goal of Rapid Testing is to do —or to learn how to do—the fastest, least expensive testing that still completely fulfills the mission we’ve accepted. To the best of our knowledge, none of this is inconsistent with the context-driven principles. If someone were opposed to all of the Rapid Testing premises, and did testing in a way that reflected that opposition, they’d be in opposition to Rapid Testing, and doubtless they would do that because they believed it more effective in their context. Yet there’s no real point in talking about the effectiveness of X without knowing the context in which X happens and the effect that X is intended to achieve.
(We can never be sure that we’ve done done the fastest, least expensive testing we could have, of course. Engineers can never be sure that they’ve built the bridge out of the cheapest, lightest materials they could have. That knowledge is always provisional; you can discover that you haven’t done it, but you can’t be sure that you have.)
We don’t want to teach people how to do specific things, particularly. Instead, we want people to learn a mindset and a skill set on how to approach their work, such that they can make choices that afford good, fast, and inexpensive testing. If people come into the class thinking that it’s the way to do testing, we hope to disabuse them of that. I’m not sure how they could leave thinking it’s the way to do testing, since we try so hard to expose them to many ways of testing during the class. If someone wants to argue with us, we welcome the debate. We state explicitly that your context guides your choices, and both of those evolve over time. Rapid Testing itself has evolved over time, and continues to do so.
[…] In The Real World Written by: Michael Bolton […]
OK, I’m going to play devil’s advocate and hang my head out to get chopped off. As a generalisation:
“In my world, I don’t often have the authority/flexibility to work in such a manner, irrespective if it’s the ‘right’ thing to do.”
Needless to say I’ll never stop trying to persuade/influence people on better ways to do things, just acknowledging it’s often a long a troublesome journey with lots of compromises along the way.
Michael replies: In Rapid Testing, we work on these premises. In the class, we also state an assumption: you have some control over the design and performance of your testing. And all that comes with a tacit premise: no one can control how you think. And, much of the time, people may have a lot more freedom than they believe. In my experience, as soon as I start doing testing work in a way that I prefer—that I own and understand and can describe—and that still completely satisfies the client’s mission, more authority and flexibility is granted to me. Note that I must fulfill my client’s mission, and be seen to be fulfilling it, for this to happen. That can take time, patience, hard work, experimentation, humility, concessions, and a lot of explanation; sometimes a long, troublesome journey, as you suggest.
In this post, I’m not talking about the “right” thing to do in any universal sense. It’s a personal view, evinced by the fact that I keep saying “my world”. But just as I can’t claim ownership of what constitutes the real world, neither can anyone else.
In response to Mark Tolfts:
[In my world, I don’t often have the authority/flexibility to work in such a manner, irrespective if it’s the ‘right’ thing to do]
I still struggle with this. Here is my point-of-view:
Logically, I know that “doing the right thing” is terribly important. And if you try but still cannot “do the right thing”, then “leaving” is ultimately the correct decision.
Michael replies: In this discussion, I recommend using The “A vs. The” Heuristic. So, “a right thing”, “a correct decision”, etc.
I’ve voluntarily left 6 companies (as both a salary and hourly employee) over the past 2 years. The primary reason I left each company was because I could not “affect positive change”. That is, I had no “authority/flexibility to do the right thing”. So, I left.
Do you want to effect positive change, or do you want to do good work? They’re not the same thing, and often you can do them separately. But it’s worth considering: why do you want to effect change in the first place?
But…it is exhausting. Constantly switching jobs drains you in many ways. Intellectually, emotionally, monetarily. I wonder how many more companies must I go through until I find one that allows me the “authority/flexibility to do the right thing”?
I hear a lot of folks saying “Try to do the right thing. If you cannot – leave!” So far, I have heeded this advice. But, to me, the advice smells a bit like a “best practice”. If it is indeed a “best practice”, and there are no such thing as “best practices” (just “good practices in certain contexts”), then should I heed the advice, at all?
What if “leaving” doesn’t make sense in my current context? What if “leaving” preserves my integrity, but jeopardizes other very important aspects of my life? How can I avoid the problem in the first place?
One step: consider that your idea of the right thing might be a right thing. “If you cannot, leave” does sounds like a best practice. Try inserting the words “one option is to” after the comma, and use that to trigger the possibility of other options. You could renegotiate your mission, try some experiments, do it their way for a while, fly below the radar, make some concessions, choose your battles, be more patient, prepare some analyses, report to someone else… Sometimes an organization isn’t a good fit, but represents a learning opportunity. Sometimes an organization isn’t a good fit at first, but becomes a good fit.
As I said, I still struggle with this. And so, I look for support in others points-of-view. Such as:
Adam Knight on “the need to stick to the principle of refusing to do bad work”: http://www.a-sisyphean-task.com/2014/04/knuckling-down.html
A reply to Adam from Huib Schoots: http://www.huibschoots.nl/wordpress/?p=1689
James Bach’s first in a series posts on Integrity: http://www.satisfice.com/blog/archives/1090
Meanwhile, as I struggle, “trying to persuade/influence people on better ways to do things” is a very good temporary solution.
How do you know your way is better? Are you providing help, or inflicting it? Is there a possibility that you could do good work without having to refuse to do bad work?
Michael:
The RST premise “Software projects and products are relationships between people, who are creatures both of emotion and rational thought.” certainly resonates with me. I once had a Project Manager insist one of my testers create a detailed Test Plan document for his project. This was despite the fact we’d already created a concise document outlining the proposed Test Strategy. In line with another RST premise “We commit to performing credible, cost-effective testing, and we will inform our clients of anything that threatens that commitment” I then spent time with the Project Manager to understand why they felt such a document was required and highlighted, among other things, that such a document would add no value and the time would be better spent learning, experimenting and testing the product. In the end I lost this argument and was told by Snr Management; “if it’s ethical and not going to cause harm to human life, just do it”. I argued it wasn’t ethical, but lost that argument, too. So, the document was created, we didn’t use it, and the Project Manager was happy. What did I learn from this? I need to sharpen mine and my team’s skills at defending and justifying our work. And, sometimes compromising on certain things so you can focus on the things in your control isn’t so bad.
Michael replies: Right: that relates to Premise 7:
Notice that we did not say “refuse”; we said “resist”, and followed it with the “At a minimum…” sentence. The client has the absolute right to spend money on whatever he likes, and some things are Just Part of the Mission. One’s level of resistance is a personal thing; and it may turn into a refusal, if performing the requested activity doesn’t fit with your ethics. Of course, you’ll have to deal with the consequences both of refusing and complying. These kinds of decisions are part of the cost of doing business for the self-employed (and everybody is self-employed, although not everybody believes it).
To Damian’s reply:
Some of us aren’t in a position to simply quit for many reasons. Let’s face it, there’s unlikely to be a utopia company to work for. Until I fell I’ve exhausted all avenues and tried my utmost, I’ll keep trying. And, as exhausting as it can be sometimes, the learning and growth you can get along the journey can be equally as rewarding.
Michael,
Thank you for the reply to my reply. I appreciate and enjoy communicating with, and learning from, you.
Below, I think I’ve responded to each of your comments and questions.
First, thanks for the “A vs. The” Heuristic recommendation. I read more about this on your own site (starting at http://www.developsense.com/blog/2012/04/heuristics-for-understanding-heuristics/), and elsewhere. I found it very informative and useful, and will try to be more careful with what words I use, going forward.
However, a few quotes got me thinking:
– a plausible answer that might be right for its context
– Correctness is a human notion, and things are only correct in some context
Those quotes make me think… IF I have carefully considered my context, THEN I should be able to make the right (correct, moral, good, justified, acceptable, appropriate) choice (for those involved, in that place, at that time, in that context). Not “A right choice”, but “The right choice”.
If so, I think that’s what the idiom “Do the right thing” is intended to mean: Considering the context (people, place, time, etc.), choose THE most correct, moral, good, justified, acceptable, appropriate thing to use, do, say, etc.
Michael replies: Right: the right choice, given your level of satisfaction with it; your prioritization of the factors; your awareness of factors that might matter; the inconsequential nature of other factors; some list of alternatives; ignoring counterfactuals; the role of luck in the outcome; so far as we know; at this time… the epistemological equivalent of “works on my machine”. Don’t get me wrong. I’m only trying to remind us to keep our eyes open: there’s are important similarities between “the right thing to do” and “best practice”.
Of course, I want to “do good work”. I also want to “effect positive change”. And “help people”. And “improve things”. I want to…“do the right things” (consider each, individual context, I choose the most correct, moral, good, justified, acceptable, appropriate thing to use, do, say, etc.).
Unfortunately, in my recent positions, I am often asked to “do bad work” (“do the wrong things”). How do I know it is bad? Rather than give a long explanation, I’ll give a single, small example: I’m often asked to report pass vs. fail rates.
Oh. Well, there are other ways around THAT. Let me brainstorm a few.
Unfortunately, in my recent positions, I have not had much (if any) authority. So, I cannot “inflict help”. Instead, I am limited to “trying to persuade/influence people on better ways to do things” in order to “effect positive change”. How do I know “my way” is better? Rather than give a long explanation, I’ll give a single, small example: I have tried to persuade/influence them NOT to report pass vs. fail rates. I have tried to persuade/influence them to “braid the stories” and “deliver the news”.
When you say “them”, whom are you trying to persuade here? Managers or testers? Or clients? If it’s the other testers, they won’t buy in until they’re convinced that managers will buy in. If it’s managers, they won’t buy in until they trust you and the other testers. Are you trying to get them to NOT do something (hard) or to do something that’s more helpful (which you can do in parallel with the unhelpful stuff, until they realize just how unhelpful the unhelpful stuff is)? Do you realize how long it takes to turn a supertanker?
Is it possible that you’re trying to spend trust before you’ve earned it? If it’s six gigs in two years, have you given any one of them enough time to come around? Have you offered them the cited articles (and the What Counts? article) Also: what’s the risk to the company vs. risk to the general public vs. risk to your own sanity vs. risk to your reputation?
(NOTE – I “know” (as much as anyone can know anything) that “this way” (not my way) is better. And, if I could “effect positive change” by getting them to “braid the stories” and “deliver the news”, it would benefit many others, besides just me. That’s why I want to “effect positive change” – to help me, and others, alike.)
Unfortunately, I have failed. But, I did not give up! I then “renegotiated my mission, tried some experiments, did it their way for a while, flew below the radar, made some concessions, chose my battles, was more patient, prepared some analyses, reported to someone else”.
And after all that, I still failed. I was unable to “effect positive change”. I was unable to “do the right things”. And so, I (and everyone else) were still asked to “do bad work”.
So, I am faced with a new decision: stay (and continue to “do bad work”), or leave.
Although “leaving” is often incredibly difficult (and exhausting), I want to “do the right thing”. Since I have tried (and tried and tried), but failed at “effecting positive change”, logically the ultimate decision (for me, in that place, at that time, in that context) is “leave”.
And, I agree – it is vital to learn from each opportunity, whether good or bad.
And last, thanks for correcting my use of affect vs. effect. That’s a tricky one. 🙂
It’s a service I offer. 🙂
[…] Blog: “In The Real World” – Michael Bolton – http://www.developsense.com/blog/2014/04/in-the-real-world/ […]