In the testing business, we’ve been wrestling with the measurement problem for quite a while. I think there are two prongs to the problem. The first is the aphorism that “you can’t control what you can’t measure”. The second is the confusion between measurement (which can be either quantitative or qualitative) and metrics, which are mathematical functions of measurements, and therefore fundamentally quantitative, only quantitative.
I don’t know if you can’t control something that you can’t measure, but you can certainly make responsible, defensible choices control things based on non-quantitative measures. For example, I’m hungry right now, and the non-bald parts of my head are a little shaggy. I’m not really comfortable with the keyboard on my new ThinkPad, but I like the display even though the default fonts seem to be a little on the small side for an astigmatic guy approaching his 50s. I can measure and manage all of these things without applying numbers.
I’m going to go grab a bite after I’ve finished this note; I’ll get my wife to give me a haircut before she heads out on the canoe trip, and I’ll trim my beard on my own. I can’t do much about the keyboard, although I can measure it by saying that I liked my old machine’s keys better. And I can grow the fonts in the browser by pressing Ctrl-+ until I’m happy again. In each case, I’m measuring to manage just the effects that I want, even though I’m doing it without quantitative measures. (Thanks to Matt Heusser for pointing out the haircut example to me; and thanks to Cem Kaner for pointing out the significance of the fact that I griped about the keyboard before complimenting the display.)
Apropos of all this, another of my Test Connection columns has been posted on StickyMinds. This one is about measurement and metrics, and the way that people use and confuse them. You can read it by clicking here, or by going to http://www.developsense.com/articles/2009-07-ThreeKindsOfMeasurement.pdf.
I’m grateful for the guidance and compliments given to me by Jerry Weinberg on this one.
I’m also delighted by the appearance of a recent article by Tom DeMarco in IEEE Computer, in which he re-evaluates his thoughts on metrics as expressed in early and influential book, Controlling Software Projects: Management, Measurement, and Estimation (Prentice Hall/Yourdon Press, 1982). He also questions his thoughts on software engineering, as evinced by the title of the piece, “Software Engineering: An Idea Whose Time Has Come and Gone?”. It’s brilliant, and high time that some of Mr. DeMarco’s stature raised these questions. You can read the article here, or by going to http://bit.ly/pRrkd.
The oft-stated insistence that project or career goals must be “SMART” (specific, measurable, attainable, realistic, and time-based) often results in a confusion of metric and measurement, I believe.
I think you could also make an argument that “have a better haircut” is a perfectly reasonable measurement from a goal-setting perspective, without requiring you to take a quantitative survey of your friends.
Thank you for the links to both articles, Michael. They were both personally poignant as I've been working on different ways to talk about metrics lately.
I, too, differentiate between "metrics" and "measurements". I renamed "measurements" to "statistics". Perhaps that is my sports enthusiasm, but I find sports to be a good analogy here.
In North American football there is a metric called Passer Rating. This is based on a formula that takes into account touchdowns, completion percentage, total yards and interceptions. Then a number between 0 and 158.3 is assigned as the rating for a passer.
What is the benefit of this metric? I have no idea, really, but it seems that the media, fantasy football players, and Las Vegas are more interested in it than the players and coaches are. It seems to me that the numbers are not the best way of determining how good a passer is. Through my reading it appears that the NFL teams determine who their passer will be based on first-order measurement. They actually watch the passer in action and decide how well they will fit in their own team, and then let results (usually wins vs. losses) speak for themselves.
Afterall, the statistics for a passer can be influenced by temperature, wind, and the skills of the other players on the field in addition to the passer's skills. I would rather rely on first-order measurement and ignore the statistics or metrics on my test team. I would rather manage by results:
– Increased knowledge of the system.
– Ability of the project team to make smart go-live decisions.
– Well-written bug reports.
– Finding important bugs faster.
One last point – the teams in the NFL invest a lot of money in statisticians, so the stats must be helpful, right? It appears, according to one source, that the reason for the statistics is to market the players. That is the way I’ve seen number used in software testing as well. It is used to make it look like a group/tester is doing a good/bad job.
Thanks again for the great article, Michael.