More IMVU comment followup: Timothy Fitz’s reply

In response to my post on IMVU, I was delighted to receive a reply from Timothy Fitz, whose original blog entry triggered my investigation.

There are many things to like about Timothy’s reply. First of all, it’s honest and forthright. Second, he seems not to be taking personally the criticism of the product that he and his company have been working on for years. It’s a rare skill not to take things personally. So, thank you, Timothy, for that.

He begins:

I would like to clarify, we do have a Quality Assurance staff. There are numerous quality engineering tasks that only a human can do, exploratory testing and complaining when something is too complex come to mind. They’re just not in the “take code and put it into production” process; it can’t scale to require QA to look at every change so we don’t bother. When working on new features (not fixing bugs, not refactoring, not making performance enhancements, not solving scalability bottlenecks etc), we’ll have a controlled deliberate roll out plan that involves manual QE checks along the way, as well as a gradual roll-out and A/B testing.

When someone complains that something too complex, I’ve been trained by Jerry Weinberg and his work to ask too complex compared to what? In the same way, when someone says that it doesn’t scale to have testers look at every change, I ask why it can’t scale. Programmers make every change, don’t they? After all, “it can’t scale” is one of the great myths about the Agile Manifesto. There’s a difference between “it’s too complex” and “I don’t have a more useful model than the one I currently have”; between “it can’t scale” and “I don’t know how to solve the problem of making it scale.”

One approach to solving complexity or scaling problems is to reconsider what testing is and where it happens. Pair programming in which no tests are written is a form of testing (we often call it “continuous review”, but review is a form of testing). Behaviour-development, in which we check that each function at least to some degree does what it should as we build it, do is a form of testing. And continuous deployment is a form of testing, too.

One definition of testing is “the gathering of information with the intention of informing a decision” (that’s paraphrased from Perfect Software and Other Illusions About Testing, by Jerry Weinberg). The complaint that something is too complex is information. Your testers are telling you that you about the testability of the product, compared to their ability to test it. There are all kinds of details to the story to which we’re not privy. Maybe they believe that they have to retest everything in the product every time the programmers make a change; maybe they believe that testing means seeing the visible manifestation of every change from the user’s point of view; maybe there are insufficient hooks for the kind of automation they want to apply; maybe they are being mandated to do other things that impinge on their ability to study and grasp the issues that they’re facing; or maybe they’re the creditors on some of the programmers’ technical debt, and the number of bug reports that they have to investigate and report is taking time away from their test design and execution—that is, their test coverage.

There are constraints to every testing assignment, and as Jerry says (quoted in James Bach‘s article in The Gift of Time), it’s the first responsibility of the tester to figure out ways to get around those constraints. But that may not always be possible, given the situation.

Another form of testing is asking questions, like “if you don’t involve testers when you fix bugs, make performance enhancements, solve scalability bottlenecks, etc., how do you know that you’ve fixed, enhanced, or solved?” And others like, “What are your testers doing?” “Are they only testing new features?” “Are you aware of how useful skilled testers can be?” “Do you see any opportunities for adding efficiencies to your model of testing?”

Your point about the sheer number of bugs we have? you’re right. Our software has way more bugs than I’d like. It’s a byproduct of the choices made when the company was small: to prioritize determining what the customer actually wants at almost any cost. We would absolutely love to have a high quality client, and we’re working every day towards that goal.

Continuous Deployment let’s you write software *regression free*, it sure doesn’t gift you high quality software. As a start-up, we’re faced with hard decisions every day about where to make our product higher quality; most of the complaints you have would be immediately ignored by the majority of computer users and so we can’t in good faith prioritize them over the things that ARE causing our users trouble.

I’ll respond to the things I disagree with in a moment, but I want to draw attention to the most important aspect of Timothy’s reply: he highlights that developing software and services and products and systems is a constant set of tradeoffs, and that, just like the rest of us, he and the rest of the IMVU crew are making these decisions all the time. That’s important because, as I’d like to emphasize, my notion of IMVU’s quality doesn’t matter. “Quality is value to some person(s)”. When James and I teach Rapid Software Testing, we add something to that: “Quality is value to some person(s) who matter“. I’m an outsider. I don’t have any interest in using chat illustrated by anime-looking 3D avatars who teleport from place to place. I have no interest in handing IMVU my money for this service. I have no interest in the community this stuff supports. (So why did I even bother to look at the service? I am interested in software development and testing, and I wanted to investigate the relationship between a million test cases a day, a million dollars a month in revenue, and the system being tested thereby.)

I’m going to introduce something perhaps more controversial here. Even if I were working for IMVU, as a tester, I still wouldn’t matter. How can I, a tester, say that? It’s because my role is not to push my values down the throats of the programmers and business people that I serve. Saying that I don’t matter is a simplification; my values don’t matter as much as the business values and the customer values do. I do matter, but only precisely to the degree that I deliver those people the information that they value to inform their decisions. I can find defects in the features offered by the product; I can tell them about things that I see as driving risk; I can tell them about things that I’ve discovered that represent a threat to the value of the product to someone who does matter. And at that point, the decision is up to them.

A couple of other points:

While I agree that continuous deployment doesn’t give you high-quality software (in the sense of few bugs), I disagree that it lets you write software regression-free. It performs some tests on the software that might find regression if that regression happens to be covered by one of the tests. That’s not a bad thing in itself; that’s a good thing. The bad part is that, once again, it provides The Narcotic Comfort of the Green Bar. There’s a big difference between “our tests find no regressions” and “there are no regressions”.

Second, continuous deployment is Way Cool. As Elisabeth suggested, that IMVU can push out 50 deployments a day is impressive. But it reminds me of a story (I think it was Al Stevens) in which you go to the circus. Last year, they had a bear on roller skates, which impressed you. This year, the bear is on motorized roller skates. And you’re dazzled for a moment, until you consider, “Wait a second… do I really want to see a bear on motorized roller skates?” 50 deployments a day is impressive, but 50 deployments of what? For what purpose?

Could it be that the focus on deploying 50 times a day represents opportunity cost against other equally or more desirable goals? Goals like, say, addressing the problem “Our software has way more bugs than I’d like”; or addressing the complaint from the testers that the testing mission is too complex; or investigating the functionality and security problems that the customers seem to be reporting and that might represent a serious threat to the value of the product? Is 50 deployments a day providing business value that can’t be delivered any other way? Any other way? Would customers or IMVU itself suffer some loss if they had to wait 30 minutes for a code drop, instead of 15? I repeat: I don’t know in this case. I don’t know whether five deployments a day would be better, or five hundred a day, or one every five hundred days. I do know that part of testing is noticing the cost of one activity and its potential impact on the value of another.

The vast majority of developers take one look at what they think our product is and don’t bother to give it a try; I’m happy to see a thoughtful open-minded dive into IMVU from a developers perspective.

I’m a specific kind of a developer; I’m a tester. As such, it’s my particular bent to investigate things, and not to take them on faith, and to report on what I’ve found. I genuinely appreciate Timothy’s thoughtful and open-minded reply, and I thank him for triggering some of the questions and observations above.