Joel Nails It

I’ve just had to buy a new computer, and am now becoming intimate with Windows Vista. “Becoming intimate” is a euphemism that people sometimes use when they mean “being screwed”. The biggest problem I’ve encountered so far is with Vista’s User Interface Feature Concealment module, which took the few remaining things that were reasonably intuitive and accessible in Windows XP and hid them.

Joel Spolsky, he of Joel on Software, is someone that I’ve admired for a long time. In this blog post—the first part of a talk that he gave at Yale on November 28, 2007, along with part two and part three—he identifies a key concept about quality that few people identify: the absence of bugs doth not a quality product make. Technical correctness is a wonderful thing, but there has to be value in that correctness. Quoth Joel (from part one):

The Windows Vista team at Microsoft is a case in point. Apparently—and this is all based on blog rumors and innuendo—Microsoft has had a long term policy of eliminating all software testers who don’t know how to write code, replacing them with what they call SDETs, Software Development Engineers in Test, programmers who write automated testing scripts.

The old testers at Microsoft checked lots of things: they checked if fonts were consistent and legible, they checked that the location of controls on dialog boxes was reasonable and neatly aligned, they checked whether the screen flickered when you did things, they looked at how the UI flowed, they considered how easy the software was to use, how consistent the wording was, they worried about performance, they checked the spelling and grammar of all the error messages, and they spent a lot of time making sure that the user interface was consistent from one part of the product to another, because a consistent user interface is easier to use than an inconsistent one.

None of those things could be checked by automated scripts. And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. The geeky definition of quality won out over the suit’s definition; I’m sure the automated scripts for Windows Vista are running at 100% success right now at Microsoft, but it doesn’t help when just about every tech reviewer is advising people to stick with XP for as long as humanly possible. It turns out that nobody wrote the automated test to check if Vista provided users with a compelling reason to upgrade from XP.

Jerry Weinberg, who has a habit of Getting There First, is one of the people who does talk about this. Here’s an example from Quality Software Management, Vol. 1: Systems Thinking.

Though copious errors guarantee worthlessness, having zero errors guarantees nothing at all about the value of software. Let’s take one example. Would you offer me $100 for a zero defect program to compute the horoscope of Philip Amberly Warblemaxon, who died in 1927 after a 37-year career as a filing clerk in a hat factory in Akron? I doubt it, because to have value, software must be more than perfect. It must be useful to someone.”

Dave Platt also gets it right in the talk that goes with his wonderful book Why Software Sucks: “As surprising as it might be to you, your customers don’t want to use your software. They want to have used your software.”

I have it from a fairly reliable source that Microsoft, as of a couple of months ago, had 650 open positions for teste{backspace}{backspace}{backspace}{backspace}{backspace}SDETs. After my first looks at Vista (over the last couple of weeks), and of Office 2007, I’ll say this: don’t bother filling those open reqs, guys. Instead of finding even more SDETs to write even more scripts to show that the program still works the way some programmer coded it, try hiring more people who are interested in looking at the whole program, not just the API. There are SDETs like Michael Hunter (as of way before 2019-01-15, no longer at Microsoft), but they’re very few and very far between. You have more than enough people with his coding skills; you need to hire more people, way more people, with his thinking skills.