Cem Kaner recently wrote a reply to my blog post Of Testing Tours and Dashboards. One way to address the best practice issue is to go back to the metaphor and ask “What would be the best tour of London?” That question should give rise to plenty of other questions.
- Are you touring for your own purposes, or in support of someone else’s interests? To what degree are other people interested in what you learn on the tour? Are you working for them? Who are they? Might they be a travel agency? A cultural organization? A newspaper? A food and travel show on TV? The history department of a university? What’s your information objective? Does the client want quick, practical, or deep questions answered? What’s your budget?
- How well do you know London already? How much would you like to leave open the possibility of new discoveries? What maps or books or other documentation do you have to help to guide or structure your tour? Is updating those documents part of your purpose?
- Is someone else guiding your tour? What’s their reputation? To what extent do you know and trust them? Are they going to allow you the opportunity and the time to follow your own lights and explore, or do they have a very strict itinerary for you to follow? What might you see—or miss—as a result?
- Are you traveling with other people? What are they interested in? To what degree do you share your discovery and learning?
- How would you prefer to get around? By Tube, to get around quickly? By a London Taxi (which includes some interesting information from the cabbie? By bus, so you can see things from the top deck? On foot? By tour bus, where someone else is doing all the driving and all the guiding (that’s scripted touring)?
- What do you need to bring with you? Notepad? Computer? Mobile phone? Still camera? Video camera? Umbrella? Sunscreen? (It’s London; you’ll probably need the umbrella.)
- How much time do you have available? An afternoon? A day? A few days? A week? A month?
- What are you (or your clients) interested in? Historical sites? Art galleries? Food? Museums? Architecture? Churches? Shopping? How focused do you want your tour to be? Very specialized, or a little of this and a little of that? What do you consider “in London”, and what’s outside of it?
- How are you going to organize your time? How are you going to account for time spent in active investigation and research versus moving from place to place, breaks, and eating? How are you going to budget time to collect your findings, structure and summarize your experience, and present a report?
- How do you want to record your tour? If you’re working for a client, what kind of report do they want? A conversation? Written descriptions? Pictures? Do they want things in a specific format?
(Note, by the way, that these questions are largely structured around the CIDTESTD guidewords in the Heuristic Test Strategy Model (Customer, Information, Developer Relations, Equipment and Tools, Schedule, Test Item, and Deliverables)—and that there are context-specific questions that we can add as we model and explore the mission space and the testing assignment.)
There is no best tour of London; they have their strengths and weaknesses. Reasonable people who think about it for a moment realize that the “best” tour of London is a) relative to some person; b) relative to that person’s purposes and interests; c) relative to what the person already knows; d) relative to the amount of time available. And such a reasonable person would be able to apply that metaphor to software testing tours too.
What I think is scary is that it’s so obvious that a “one fits all” model doesn’t exists.
But when you speak with alot of colleagues they are completly hung up in a specific test process which they think is the one fit all best practice.
Keep up spreading the words 🙂
Strictly speaking, you’re right.
But if you ask me all these questions as a response to my “What’s the best tour in London”-question, I’ll probably scream after about 5 minutes “Just give me a F*&-ing answer!” (More likely: I swallow the curses and wished I had gone to a Tourist Office.)
Michael replies: I’m not saying that these are questions that you would ask someone else. I’m suggesting that these are questions that you might want to ask yourself, factors that would go into your evaluation of what you might or might not consider “best”. Yes, all these questions would sound awfully noisy compared to no questioning at all. That’s why self-management, rapid evaluation, and heuristics are so important for testers who intend to be of service to their clients: which questions are most important enough to ask in a given context?
So let’s challenge this on 2 points:
A)
A question always has a context that already answers many of your questions. E.g. if I tell you that I’m coming to Toronto for the first time in my life, that it’s for business, but have a few hours to play tourist and ask you, because you’re probably the only one I know in Toronto, what’s the best tour? With that context, you will already know a lot of the answers on the questions above.
Of course. I’m not advocating asking questions for which you already have an answer; I’m saying that “best” has a context for a specific person at a specific time. So, you’re arguing in favour of my point, not challenging it.
One caution: there are important differences between awareness, oblivion, and taking something for granted. Yet from the outside, these all look the same: silent, tacit. As the anthropologist Pierre Bordieu put it, what comes without saying goes without saying.
And the fact that I’m asking you and not the Tourist Office, adds to the context and leads to my second point.
B)
Sometimes the purpose of asking a question is not to get an exact and unambiguous answer, but just to get an opinion.
I value you’re answer and use that as input to decide what tour to take. Don’t be upset if I take a different tour. That’s my decision and you helped me making that decision.
Yes; that’s exactly why the notion of the “best” tour is silly—and why it’s especially silly for me to prescribe to you what the best tour is.
Likewise, I won’t be upset if the tour you advised me to take, is not what I expected of a tour. It is merely a lesson for myself to either give you more input parameters or add more sources of information before making a decision.
You won’t? That sounds like a more carefree attitude than I might expect from, say, someone who had millions of dollars on the line, or someone whose software might kill people. “Live and learn” is one thing; “die and (let someone else) learn” is quite another.
I think it’s the same with “Best Practices”. There is always a context, in my experience there is always a list which implies that there is no single ‘best practice’ which also implies that you may pick the practice that suits ‘best’ to your situation and use it at your own risk.
This makes meaningless the notion of “best practice” or a “best tour”, which is exactly the point that Cem made in his post and that I made in my post. You’re apparently agreeing with that. We haven’t yet examined the role of the tacit in what “best practice” is intended to convey. “Best practice” does not contain some kinds of tacit knowledge, and cannot convey other kinds. Stay tuned.
If you think about it: ‘Best’ by definition is arbitrary. I would define it by ‘most desirable according to some criteria’, but you can always argue about the criteria. Even in sports, which existence is about determining the best, you can always argue if the winner is really the best. “OK, they won, but the opponents played better”.
I perceive that you think you’re thinking my point was different. It’s the same.
My point is that I think everybody knows that “Best Practices” are not generally applicable. So this discussion is a pure theoretical one.
Michael replies: Everybody? What do you mean by everybody? What do you see or hear that helps you to believe that “everybody” knows that “best practices” are not generally applicable? Isn’t the whole idea underlying a “best practice” that it’s not only generally applicable, but universally applicable? Why not call them what they are at best: heuristics? Calling them “heuristics” might trigger a couple of worthwhile results. First, people would ask what “heuristics” really means (they appear not to spend too much time wondering what “best practice” really means. Second, the answer to that question would underscore the idea that heuristics are fundamentally fallible, context-sensitive, and should be applied by those with the judgment and experience to apply them. I don’t see any such caution surrouding “best practices”; the label itself implies unquestioned and unquestionable superiority.
One might say that all the fuzz about this is proving me wrong, but are there really organizations who blindly apply best practices from other organizations? I mean really blindly? Perhaps they deserve to get fired or go bankrupt.
You need look only as far as your nearest ISTQB certification enthusiast to see an example of the best practice trope in action. Another example:
a) http://careers.tescobank.com/job.cfm/reference/676/source/TALEO/
b) http://www.computerworlduk.com/news/it-business/3287653/tesco-bank-issues-customers-with-emergency-guidance/##
Harry Collins, in his book Tacit and Explicit Knowledge, cites Wittgenstein: rules of action don’t contain the rules for their application—”&mdashthe ‘rules regress’. Thus, to apply a rule like ‘do not walk too close to others in the street’, one must know what ‘too close’ means and how it varies from circumstance to circumstance, and how one must know another set of rules to recognize what circumstance it is, and so forth.” Best practice enthusiasts ignore or even reproach that kind of thinking.
As to whether they deserve to get fired or go bankrupt, that’s a very nice moralistic view. Should the customers, shareholders, and employees of the company be required to share that pain? Before answering too quickly, remember the 2008 financial crisis, think well, and choose wisely.
In answer to Arjan’s:
“One might say that all the fuzz about this is proving me wrong, but are there really organizations who blindly apply best practices from other organizations? I mean really blindly? Perhaps they deserve to get fired or go bankrupt.”
I would say that there are a significant number of companies that would love to blindly apply a ‘proven’ set of ‘best practices’ if it would simplify testing (meaning they could hire relatively unskilled, cheap labor to perform the tests) and reduce the amount of time and effort necessary to test an application. The pressures on all companies to reduce their time to market and lower their development costs in general are going to ensure that this issue stays alive long after it has been discredited.
Human nature isn’t going to change. People continually look for simple answers to complex problems. So even if many people consider this argument to be a dead horse, as long as companies saddle it and strut around in jodhpurs and a riding crop we need to continue to beat it and I applaud Michael, Cem and others for their efforts.
Michael replies: Thanks, Chris.
A quick note specifically regarding the URL above for the Tesco Bank “fiasco”. My companies e-commerce site had a similar issue when IE9 was first released. We’re on a slightly older version of that platform, released when IE8 was new. Instead of saddling the users with messing with browser settings (@@@TechSupportNightmare), we addressed it with a simple meta tag fix which forces IE9 to IE8 compatibility. No users were harmed in the implementation of this and most were quite happy that they could now actually place an order on our site using IE9. This also made our customer service folks happy, which is a Nice Thing. All in all it took hours at most to implement, test and release, resulting in:
1. Happy online Customers
2. Happy Customer Service team
3. Happy Execs via orders being placed successfully
And we didn’t get comments from our customers such as:
“so, they put advice as to what to do if you can’t access their website….on their website?! Brilliant! ” (sorry I couldn’t resist the irony of that comment in the article)
But Still, somewhere down the road(map), the real issue of our site’s compatibility still needs to be addressed. I believe we have an Epic written out for that…
😉
Michael replies: A perfect and elegant solution can usually wait.
For more on this kind of issue, I really enjoyed Joel Spolsky’s take, way back when it first came out. The version numbers are different today, but the dilemmas are timeless.
Thanks for the note, Will.
The link to Joel’s article = “Directory Listing Denied” =( which is ok, because now I get to enjoy perusing his site and have already bookmarked a few a articles for some between meetings reading…
Michael replies: Yeah, that’s exactly what I meant you to do. 🙂 Fixed. Thank you.
When I say everybody, I mean every body. OK, every Human body. Well, every Living Human body. See, you can nit-pick about every word. If you don’t want that, better use a programming language or communication protocol (and even then). But (ordinary) languages are not precise and leave room for interpretation (and unfortunately miscommunication).
Michael replies: I have an idea: why not get into the habit of saying something close to what we mean, and recognizing that imprecision rather than trying to sweep it under the rug? That is, instead of saying “best practices”, why not say “good practices in context”? Instead of saying “everybody knows that ‘Best Practices’ are not generally applicable”, why not say, “People who bother to consider the matter thoughtfully should quickly realize that ‘Best Practices’ is a marketing term, and that no practice can be universally considered best?”
Now, it’s not like I can’t think of arguments against speaking and writing precisely. One is laziness. Another is marketing. Yet another is a desire to continue with what seems largely to be the status quo. One small problem with adopting any of these arguments is that they’re each antithetical to excellent testing.
When I first saw the word ‘Heuristic’, don’t remember if it was your blog, James’, or another one, I had to Google the definition. Since I’m not a Linguistic genius nor have a talent of remembering the meaning of difficult words, I had to Google it the 2nd and 3rd time as well. And I will keep doing that until I know it.
Good. “A fallible method for solving a problem or making a decision.” Practice. Repeat it a few times. Think of examples. You can do it.
That is annoying, but doable for blogs, but somehow problematic in live conversations or presentations. So what you see as a benefit that people will ask for the definition of ‘Heuristic’, I perceive as unnecessary use of difficult words and counter-productive in communication.
Far better that we continue not to say what we actually mean, then? What’s the problem? It’s a word that gives a label to a specific kind of idea that is central to testing. The word has been around forever (“Eureka,” Archimedes cry of discovery, is based on exactly the same root; George Polya revived the work in the modern era in the 1940’s; the word is central to Billy Vaughan Koen’s Discourse on the Method, a foundational book on engineering. So what if you’re not familiar with it? You can learn, right? Or is there some kind of lock-in or sell-by date for tester knowledge, after which we admit no more ideas?
In November of 2010, I had a conversation with Stuart Reid. He insisted that “regular testers” were incapable of understanding words like “ontology” (our system of knowledge) “epistemology” (how we know what we know). I’m more optimistic than that. My daughter, at six, was perfectly capable of understanding what “onomatapoeia” means, and was able to compose a knock-knock joke based on it. Perhaps Dr. Reid feels himself incapable of understanding those words, but the claim that regular testers can’t understand them expresses deep contempt for the testers and for the craft, I believe. Learning new jargon and terms of art is an essential skill in our ever-changing craft. Jeez, if you can’t cope with “heuristic”, can you be trusted to deal with any terms new to you as they come up? “Race condition”? “Semaphore”? “Wiki”?
So for me ‘Best Practices’ is the best (!) term for it. As I stated above, you can debate about the criteria what makes it ‘best’ for me.
But isn’t it more important that people share their experiences than how they call it? If they are published as Best Practices, with a good description of the circumstances, then that’s fine. If people use them without thinking, then I encourage you to fight that. I’m just afraid that if you fight ‘Best Practices’ as a term, people are hesitant to share their experiences.
Thank you for the encouragement. It’s part of my life’s work to try to encourage people to say what they mean.
Ask for people being reluctant to share their experiences, why not call them “experiences” instead of “best practices”? Or, once again, is saying what you mean not a “best practice”?
Indeed, “people continually look for simple answers to complex problems”. In principle there is nothing wrong with that, as long as it works. People are naive as well. They bring their hard earned savings to a grocer who promises high interest rates but has only been banking for less than 15 years and expect the same quality as a Triple-A bank. If the site is down for 3 days, or the bank goes bankrupt (like IceSave or DSB in the Netherlands), the world is too small, but in fact they ran the risk by bringing their money there.
It would be interesting to see what regulators and shareholders and customers and lawyers would have to say about your perspective. Are you sure you understand the nature of risk? Of corporate responsibility?
My last sentence, “Perhaps they deserve to get fired or go bankrupt”, was somewhat provocative. But sometimes people learn faster the hard way. Sometimes it seems that it is the only way.
It certainly does seem like the only way as long as people agree to think so carelessly and callously.