Test Estimation Is Really Negotiation

Some of this posting is based on a conversation from a little while back on TestRepublic.com.

If anyone has a problem with “test estimation”, here’s a thought experiment:

Your manager (your client) wants to give you an assignment: to evaluate someone’s English skills, with the intention of qualifying him to work with your team. So how long would it take you to figure out whether a Spanish-speaking person spoke English well enough to join your team? Ponder that for a second, and then consider a few different kinds of Spanish-speaking people:

1) The fellow who, in response to every question you ask in English, replies, “Que?”

2) The fellow who speaks very articulately, until you mention the word “array”. And then he says, “Que?”

3) The fellow who spouts all kinds of technical talk perfectly, but when you say, “Let’s go for lunch,” says “Que?”

4) The fellow who speaks perfectly clearly, but every now and then spouts an obscenity.

5) The fellow who speaks English perfectly, but has no technical ability whatsoever.

6) The fellow who has great technical chops and speaks better English than the Queen, but spits tobacco juice in the corner every minute and a half.

How long you need to test a candidate’s capacity to speak English isn’t a question that has a firm answer, since the answer surely depends on

a) the candidate;
b) the extent to which you and the client want to examine them;
c) the mission upon which the candidate will be sent;
d) the information that you discover about the candidate;
e) the demands and schedule of the project for which you’re qualifying candidates;
f) the criteria upon which your client will decide they have enough information;
g) the amount of money and resources that the client is prepared to give you for your evaluation;
h) the amount of time that the client is prepared to give you.

So, yes, you can provide an estimate. Your client will often demand one. Mind you, since (H) is going to constrain your answer every time, you might as well start by asking the client how long you have to test. If the client answers with a date or a time, you don’t have to estimate how long it’s going to take you.

Suppose the client doesn’t provide a date. Do you know anything about the candidate? Before the interview, you find out that he’s only ever been a rickshaw driver; no previous experience with testing; no previous experience with computers. He speaks no English, but has a habit of screaming at the top of his lungs once every twenty minutes. In this case, you probably don’t have to estimate. It would take less time to report to your client that the candidate is likely to be unsuitable than it would to prepare an estimate for how long it will take to evaluate him. Why bother?

So here’s another candidate. This woman has been working at Microsoft for ten years, the first eight as a tester and the last two as a test lead. Her references have all checked out. The mission is to test a text-only Website of three pages, no programmatic features. In this case, you probably won’t have to estimate. It would take less time to report to your client that the candidate is likely to be qualified (or overqualified) than it would to prepare an estimate. Why bother?

The information that you discover in your evaluation of the candidate’s English skills is to a large degree unpredictable. The problem that sinks him might not be related to his English, and you might not discover a crucial problem until after he’s been hired. The problems that you discover might be deemed insufficient to disqualify him from the job, since ultimately it’s the manager who’s going to decide.

So instead of thinking about estimation in testing, think about negotiation. Testing is an open-ended task, and it must respond to development work. The quality of that development work and the problems that we might find are open questions (if they weren’t, we wouldn’t be testing). In addition, the decision to ship the product (which includes a decision to stop testing) is a business decision, not a technical one.

In cases where you don’t know things about the candidate, you can certainly propose a suite of questions and exercises that you’ll put them through, and negotiate that with the client. In case of the first candidate, the very first bit of information that your receive is likely change all of your choices about what to ask them and how you’re going to test them. In the second case, your interview will probably be quick too, but for the opposite reason. It’s in the cases in between, when you’re dealing with uncertainty and want to dispel it, that your testing will take somewhat longer, will require probing and investigation of questions that arise during the interview—and that may require extra time that you may have to negotiate with your client. One thing for sure: you probably don’t want to spend so much time designing the protocol that it has a serious negative impact on your interviewing time, right?

For those who are still interested (or unconvinced) and haven’t seen it, you might like to look at this:

http://www.developsense.com/2007/01/test-project-estimation-rapid-way.html