“The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.”
-Donald Knuth, The Art of Computer Programming
Most of the time testing shouldn’t be too formal — unless you want to miss lots of bugs.
To test something formally means to configure, operate, and observe it in specific ways, obtaining specific evidence to support the idea that a claim about the quality of product is factual.
As developers are writing code, it might make sense to apply automated checks—a very formal kind of testing, since machines require explicit and precise instructions—to support the claim that the low-level functions behave in a way that’s reasonably close to the developers’ intentions.
For testers working closer to the social part of a socio-technological system, our goal is to reveal problems that threaten value. Most of the time, that doesn’t require highly formal testing. That’s good news, because formalizing is expensive and can over-focus the testing work towards demonstration and confirmation.
Excellent testing requires us to experience the product, explore it, and perform experiments on it. Such interaction can be quite informal and quite inexpensive.
In some contexts, testing — or aspects of it — must be highly formalized. Automated checks are extremely formal Machines can’t be given an open mission or guidelines for testing; they don’t have social competence, and must be very specifically told what to do. This isn’t necessarily bad, but creating, maintaining, and interpreting the checks comes with a cost.
In other contexts, integrity and accountability require testers to perform highly formal testing, and to account for it in formalized ways. Even in those contexts, it’s necessary to do a good deal of informal testing first, to learn about cost, value, efficiencies, and risks associated with what to formalize, and when.
As with optimization in programming, premature formalization in testing can be an immense waste of time and effort. This video with James Bach (from the Rapid Software Testing Applied class) provides a story about that.
(Most of this post first appeared on LinkedIn.)
1 reply to “Premature Formalization”