In the comments to our rapid test of IMVU, Elisabeth Hendrickson said, “Great list of issues. However, I suspect that these issues don’t really interfere with the core value to the target users.” That might be true. I think it might be more accurate to suggest that the issues don’t interfere with the core value to the existing users. That is, IMVU might be hitting the dartboard, but not the target. (I emphasize: I don’t know that either way.)
“One Diagoras, a nonbeliever in the gods, was shown painted tablets bearing the portraits of some worshippers who prayed, then survived a subsequent shipwreck. The implication was that praying protects you from drowning. Diagoras asked, ‘Where were the pictures of the people who prayed, then drowned?’
“The drowned worshippers, being dead, would have a lot of trouble advertising their experiences from the bottom of the sea. This can fool the casual observer into believing in miracles.
“It is easy to avoid looking at the cemetary while concocting historical theories. But this is not just a problem with history. It is a problem with the way we construct samples and gather evidence in every domain. We shall call this distortion a bias, i.e. the difference between what you see and what is there.” (The emphasis there is Taleb’s.)
Taleb here is talking about survivorship bias, a form of selection bias. History tends to be a story written by the side that survived the war, and the audience tends to be on the same side. The historians tend to get their stories from the people who survived. Moreover, the people interviewed typically represent a limited subset of the people who could have been interviewed.
Testing informs an ongoing story of the product. When we consider test results, it’s easy to be lulled to sleep by the narcotic comfort of the green bar. The green bar provides us with the good news that the tests we have run are all passing. There is great value to that, and I don’t want to gainsay it. Yet confirmatory tests are less testing and more checking. If we’re going to avoid being fooled, we must also remember to consider what we might learn from the tests that we haven’t run. If we’re considering the success of our organization, as Peter Drucker suggests, we need to consider the people who aren’t our customers, in addition to the people who are.
In the next post, I’ll respond to the excellent comment I received from Timothy Fitz, who posted the blog posting that inspired my brief investigation of IMVU.