I just spent $3,000 to get my nose fixed, and then I found out it was my tie that was crooked.
There’s a piece of software development mythodology that suggests that it’s always more expensive to fix a problem late in the development process rather than early. Usually the ratios quoted are fantastic; a hundred to one, a thousand to one, ten thousand to one. Let’s put that idea under the microscope for a moment.
The original idea comes from work by Barry Boehm, based on work that he did at TRW, a defense contractor. The idea includes “often”, not always, and a figure of 100 to one.
Here are some stories to contrast with the always/10,000 to one trope.
- This has happened dozens of times for me, and I rather suspect that this has happened at least once for large numbers of testers: just before release, you find a bug, you show it to a developer, and she says, “Oh. Cool. Just a second. [type, type, type]. Fixed.” That didn’t cost 10,000 to one.
- This has also happened for me, I suspect that this has happened for practically everyone as well: you find a bug, you show it to a program manager, and she says, “Oh. Well, it is a problem, I suppose, but we’ve got really important problems to fix and this one isn’t such a big deal. Let’s not bother with fixing it.” After release, no one complains. That didn’t cost 10,000 to one either.
- Similarly, we can spend a lot of time and money trying to work around the persistent and pernicious problems with certain platforms, or we could simply decide to drop support for it and wait for the problem to go away. The recent movement to drop support for Internet Explorer 6.0 is a case in point; IE 6’s market share has been dropping consistently month by month since September 2005 (weirdly, there was an uptick in June 2009), and it’s down to 14.5 per cent now. Dropping support would save Web developers everywhere a good deal of grief, maybe to the temporary displeasure of some customers—who will eventually upgrade anyway. Will that cost 10,000 to one?
- This has happened for some of us: “Gak! Another bug? This feature really isn’t working. We should back this whole feature out.” That might save way more than it costs.
- Another one from my own experience: a programmer (let’s call him Phil) struggled for weeks and weeks with a problem to no avail. He decided to put off fixing it. Shipping time approached, and Phil’s problem list was too long for him to handle on his own. Another programmer (let’s call him Ron), free from having shipped his product, was suddenly available. With the luxury of a clear mind and an absence of preconceptions, he was able to fix the problem within an hour. Had Phil been forced to try to fix the problem earlier, he might have wasted enormous amounts of time. He saved time by delaying.
- We choose to delay shipping the product to fix a problem. In so doing, we miss our shipping schedule, and our company declares a zero-revenue quarter. The drop in the value of our stock puts us out of business. Fixing the problem in that case may well have been unwise.
- Testing reveals that the problem is a bug in someone else’s code. They fix the bug. Our cost to fix the problem is not 10,000 to one; it’s free.
- Many organizations prepare prototypes. There are problems in the prototype. We don’t fix those problems right away; we save them for the Real Thing.
- After deferring the problem for a while, some third party comes up with an update or a library or a toolkit that addresses the problem. Jonathan Bach tells a wonderful story about a bug that was found during testing of a popular commercial software product that he worked on. The developer resisted fixing the problem. The bug languished in development for many months, but was consistently deferred because the development effort (and therefore the cost) for the fix was intolerably high. The team eventually decided that the problem was significant enough to fix. The developer looked into solving the problem, and discovered that a third-party library had been released one week earlier; that library made the effort associated with the fix trivial. Had the developer tried to fix the problem earlier, that effort would have been spent at the cost of not being able to fix other bugs.
- We might decline to fix a trivial bug now because fixing that bug might unblock a bunch of much more serious bugs.
- After wrestling with the problem for a while, and then dropping it, someone comes up with a flash of insight. We develop a new approach to solving it. This takes much less time than our original approach would have taken.
- We choose to release software, having fixed some of the problems and having ignored others, based on our theories of what the customers will like and dislike. We discover that people care very deeply about some of the problems that we didn’t fix, and that they wouldn’t have cared about the ones that we did.
Now: there are plenty of cases in which it does cost vastly less to fix a problem earlier than later. One of the more dramatic cases of this is the 1995 Pentium problem, in which a handful of missing entries from a table caused a number of floating point calculations to be handled incorrectly. That one probably could have been caught with more careful work, and it cost a billion dollars or so to manage the PR debacle. Yet how much would it have cost Intel to make sure that it’s processors were perfect? To this day, Intel doesn’t fix everything that it finds in testing. No one would pay for processors created with that level of effort.
It’s probably the case that it’s often or even usually more expensive to fix a problem later than earlier, based on certain assumptions. Yet I believe that it’s important to make sure that those assumptions aren’t buried under slogans. Software development isn’t assembling Tinkertoys according to a canned plan; it is suffused with questions of context and cost vs. value. It’s not merely assembly; it’s design and learning and discovery and investigation and branching and backtracking and breaking and fixing and learning some more. Yes, it’s important to be a good craftsman, and a good craftsman methodically fixes problems as he finds them… usually. But it’s also important to recognize that craftsmen also make their own decisions about what’s most important right now. As Victor Basili and Forrest Shull say (in a book edited by Boehm)
“It is clear that there are many sources of variation between one development context and another, and it is not clear a priori what specific variables influence the effectiveness of a process in a given context.”
So: Always question simplistic slogans that start with “always”.