In a recent blog post, James Bach discusses the essence of heuristics. A heuristic is a fallible method for solving a problem or making a decision. When used as an adjective, “heuristic” means fallible and conducive to learning. James ends the post by introducing a number of questions in order to test whether someone is teaching you a heuristic effectively. Meeta Prakash, in the comments, remarks “Your questions sound so much like my idea and interpretation of a ‘leader’.”
Yes, Meeta, the asking of those questions sounds like leadership to me too. What is leadership? Leadership is “creating an environment in which everyone is empowered” (that’s Jerry Weinberg’s definition). We believe in that. We believe that excellent testing starts with the skill set and the mindset of the individual tester. Other things might help, but excellence in testing must centre on the tester.
Discussion of the Method is Billy Vaughan Koen’s book on engineering. He describes the engineering method as “the use of heuristics to cause the best change in a poorly understood situation within the available resources”. He notes that the individual engineer, the engineering organization in which he works, and the engineering discipline overall each has a state of the art, which he abbreviates as “sota”. These sotas overlap one another in some places and cover new ground elsewhere. Each leads and lags the others in certain areas. The overall discipline has a sota that is more advanced than that of the organization and the individual in some places. The organization’s sota is aware of things that neither the individual nor the discipline has yet recognized. Each individual’s sota contains some knowledge that is unknown to both the organization and the discipline.
Each sota may advance before that of the other two, and the advances are ongoing. Each sota evolves in a different context. For each stage of evolution, we can’t be certain about what factors might be relevant to success or failure. Thus no practice nor method nor approach can deemed to be best. We don’t know, can’t know, whether some method is best; we don’t know, can’t know, whether someone might invent a better method. We don’t know, can’t know, whether someone has already invented a better method. What we can do, however, is to create environments in which everyone is empowered to discover and apply new heuristics along with those that we already know. That’s important because the discipline and the organization never advance on their own. They advance when someone has the inspiration, the initiative, the courage, and the opportunity to try something new, uncertain as to whether the new approach will work.
Organizations and individuals can foster initiative by empowering people (or at the very least, by leaving them alone). The overall discipline tends not to encourage initiative much, so it seems. With prescriptive, restrictive standards, disciplines often discourage innovation. Why would disciplines do this? One reason is that disciplines are typically led by experts who, as McLuhan said, are heavily invested in their own expertise, and therefore resist change that would threaten that investment. Propaganda and the threat of unemployment are among the crude tools that experts wield. Want evidence for that in testing? You need look no further than the “experts”, the certifiers, and the standards enthusiasts in our craft; their narrow and flawed models of evidence and measurement; their intolerance of uncertainty; and their resistance to acknowledging the exploratory mindset. Yet without exploration, progress in any domain ceases while the rest of the world rushes past.
“All is heuristic,” declares Koen. That’s an absolute statement which appears to declare that there are no absolutes. But rather than denying the paradox, Koen embraces it. All is heuristic, he says, including the heuristic that all is heuristic. The notion that all is heuristic is pretty robust. Koen points out that even algorithms have contexts in which they work and contexts in which they fail, and that algorithms must be chosen by people applying judgement and skill in uncertain conditions. Yet he leaves open the possibility that someone, somewhere, some day, might discover an infallible method for solving a problem.
James, our colleagues, and I deal with a similar paradox when we contend that all testing is heuristic. There’s no test for infallibility! It might be that there are some occasions when there are infallible methods for testing. It’s just that we’ve never seen one, and that we can’t currently imagine a case in which a process or a standard could—without the application of judgement or skill—be guaranteed to solve some testing problem. That’s because skilled testers (let’s call them “we”, without claiming expertise, but asserting that we are students of the craft) have specific advantages over process models, methods, and tools (“they”):
- We have situational awareness (they don’t).
- We work from the assumption that we’re fallible (they don’t).
- We have the capacity to make judgements on questions of cost vs. value (they don’t).
- We have the ability to apply a stopping heuristic at any time (they don’t).
- We have the intelligence to choose which heuristics are applicable and which are not (they don’t).
- We have the opportunity to consult, in real time, with our clients (they don’t).
- We have the inventiveness to work around a problem (they don’t).
- We have the sensory appartus to determine when a heuristic is failing (they don’t).
- We have the humanity to notice something unexpected that might be a problem for people (they don’t).
- We have the capacity to learn (they don’t).
And that’s only a partial list. A couple of notes: every one of these capabilities is itself heuristic; and it should be clear that I’m not talking only about skilled testers, but about any skilled discipline.
It’s not that bodies of knowledge or process models or standards never have anything interesting to say (although, in my experience, most of them do tend to be written in a style that induces immediate and profound slumber). It’s that none of these tools have any intrinsic relevance or value unless and until they are applied by people. If the tools are to be applied effectively, we need to recognize that they are all heuristic. We must test these heurstics and their validity with questions like the ones James poses. We must also recognize the people applying them must have the skills to know how, when, and when not to apply them. To develop those heuristics and those skills, we need to create an environment in which everyone is empowered. That is what we believe.