Many people seem certain about what happened to cause the healthcare.gov fiasco. Stories are starting to trickle out, and eventually they’ll be an ocean of them. To anyone familiar with software development, especially in large organizations, these stories include familiar elements of character and plot. From those, it’s easy to extrapolate and fill in the details based on imagination and experience. We all know what happened.
Well, we don’t. In a project of that size, no one knows what happened. No one can know what happened. Imagine Rashomon scaled up to hundreds of people, each making his own observations and decisions along the way.
As time goes by, I anticipate some people saying that the project will represent a turning point in software development and project management. “Surely,” they will say, “after a project failure of this size and scope, people will finally learn.” Alas, I’m less optimistic. As the first three premises of rapid software testing describe it, software development is a human activity that is surrounded by 1) confusion, 2) complexity, 3) volatility, 4) urgency and… 5) ambition. Increasing ambition causes increases in the other four items too. In our societies, we could help to defend ourselves against future fiascos by restraining our ambitions, but I fear that people will put blindfolds on each other, pass around the keys, and scramble to get back into the driver’s seat of the school bus. How will they do this?
One form of the blindfold is to say “That not going to be a problem here because…”
…failure is not an option.
…we have our best people on it.
…we can’t disappoint the client.
…it doesn’t have to be perfect. (thanks, Joe Miller, @lilshieste)
…we’ll fix it in production.
…no user would ever do that.
…the users will figure it out.
…the users will never notice that.
…THAT bug is in someone else’s code.
…we don’t have to fix that; that’s a new feature request.
…it’s working exactly as designed.
…if there’s no test case for it, it’s not a bug.
…the clients will come to their senses before the ship date.
…we have thousands of automated tests that we run on every build
…this time it will be different.
…we have budget to fix that before we deploy.
…at least the back end is working right.
…if there are performance problems, we’ll just add another few servers.
…we’ve done lots of projects just like this one.
…foreign-language support is something we could cut.
…that list there says that this is a level three threat, not a level one threat.
…the support people can handle whatever problems come up.
…this graph shows that the load will never get that high.
…now is too soon; we’ll tell the clients about the problems after we’ve fixed them.
…we’re thinking positively&mdashthat can-do spirit will see us through.
…we still have plenty of time left to fix that.
…the spec didn’t say anything about having to handle special characters. How are single quotes a big deal?
…the client should have thought of that before.
…seriously, that’s just a cosmetic problem.
…it’s important not to complicate things.
…everybody WILL put in some overtime and we WILL get this thing done.
…well, at least the front end looks good, and people will be happy with that.
…everyone here is committed to making sure this ships on time.
…we’ll just shorten the test cycle.
…if there’s a problem, the other/upstream/downstream team will let us know.
…they can take care of that in training.
…we’ve planned to make sure that nothing unexpected happens.
…we’ve got this fantastic new framework that’ll make things go faster.
…we’ll pull a bunch of people off other projects to work on this one.
I wonder whether these things were said, at one time or another, during the healthcare.gov project. I don’t know if they were. I don’t know what happened. I didn’t work on it. But I’ve heard these things on projects before, I know that I can listen for them, and I know that they’re a sign of trouble ahead. Are they being said on your project?
Is it time that an organisation akin to the national transportation safety bodies is created. With full powers of investigation and ability to require action to be taken when something of this magnitude happens.
Even if all it did was to put software quality on the same level that we want on transport safety. Especially since the knock on costs of these incidents is very large and widespread.
Michael replies: I greatly admire the work of the NTSB. But let’s ask: do we want software quality on the same level as transport safety? With the attendant cost? The inspections and scrutiny? The standardization? In your workplace, too? Who sets the standards? Who guards the guardians? My experience with and observation of the testing standards movement has been that the principal beneficiary of the standardisation movement will be the standardizers—and not the wider community.
I’m all for helping people to produce better software if they want it. Still… be careful what you wish for.
Totally agree with your comments – The road to hell is paved with good intentions. I’ve had this thought in my head for over a year and never really been 100% happy with it. The whole healthcare.gov issue seemed a good lightning rod for a discussion.
Part of my thought is we are in a “you can’t get there from here” situation. How many major govt IT blunders can we allow before something different needs to be tried (can’t believe I’m advocating a something must be done / this is something position).
Having some form of neutral body would at the very least reduce the partisan noise / fear that goes with government IT.
Finally you’ve got to admit it would be great to watch James Bach destroy a congressional committee hearing on his appointment to the new body 🙂
Michael replies: What makes you believe that the problem lies with government IT? (Answers: 1) Since government is a public institution, its failures are public. Private organizations don’t have to open the baggage for inspection. 2) I suggest that this kind of failure is closely correlated with size and politics. The bigger they come, the harder they fall. Although there are big organizations in the world, few are bigger and none more political than governments.)
I’m sure I’ve heard at least half of those quotes over the years.
“That not going to be a problem here because…”
Take a step back and look at all the predictions from critics following the passage of the ACA: that it would be expensive, that it would not reduce the budget deficit, that people would lose their current health insurance, that health insurance would cost much more due to all the new requirements, that the medical device tax would lose jobs, that employers would reduce working hours for many to under 30 hours a week, that employers would stop hiring, that employers would be forced to drop subsidized health insurance plans as a benefit for employees, and so on.
All those people who predicted such doom and gloom have been, so far, correct.
“That not going to be a problem here because…” we have to pass the bill in order to find out what’s in it. Remember when Nancy Pelosi said that? Suspend whatever side you cheer for (and I cheer for none) and chew on that a while. You need to imagine Nancy Pelosi as the ultimate pointy-headed executive; she’s the utterly clueless wealthy executive who has cashed in her stock options already and who doesn’t care what you think. Same with Obama, who signed the bill into law without having read it.
“That not going to be a problem here because…” the IRS is going to enforce the law. Didn’t the US create an unworkable law-of-the-land back in 1919? That’s right, alcohol prohibition, aka the Volstead Act. And who enforced that law? A small division within the Treasury Department. They couldn’t do it. Look at all the wonderful byproducts we got out of it: powerful organized crime, murders, deathly bathtub gin, and many people turned into criminals just trying to fulfill a want.
Alcohol prohibition was eventually repealed.
Michael replies: Brian, I strongly considered bouncing this comment because I couldn’t get the sound of grindstones on axes out of my ears. This blog is about software development and testing, and not about politics in the large. However, there’s one thing that I can salvage from your comment: systems that are developed under preposterous pressure from supporters and opponents alike stand no chance of success.
That said, all future comments on the overall politics of America’s health care will simply be bounced. I hold Canadian and American citizenship, I have lived under both systems, and I have friends on both sides of the border who have had terrible health problems. Some people end up okay, but only friends from one side of the border have been effectively bankrupted by those problems. Pretty much any discussion of America’s approach to health care quickly disgusts me. Selah.
[…] Can You Hear The Alarm Bells? Written by: Michael Bolton […]
Michael, according to http://www.nytimes.com/2013/11/23/us/politics/tension-and-woes-before-health-website-crash.html?_r=2&pagewanted=all& it was understood at least several months in advance that the project was in trouble.
So it looks that “failure is not an option” (We must launch on October 1st!) was the main culprit. Probably a failure to communicate has also contributed to this fiasco: http://yurymak.blogspot.ca/2013/11/healthcaregov-fiasco-last-flight-of.html.
[…] Blog: Can You Hear The Alarm Bells? – Michael Bolton – http://www.developsense.com/blog/2013/11/canyouhearthealarmbells/ […]
Great post, Michael. While the exact history of the project may be unknown, I do think that it might be possible to do enough investigative journalism to Satisfice. Then again, there are enough /conflicting/ reports of Healthcare.gov that your comment is merited. 🙂
I especially appreciate your list of warning statements – almost ‘trigger phrases’ to ‘not go there’, when, in fact, the investigation probably does need to go there.
Sometimes, the more things change, the more they stay the same. Thank you.
Michael replies: Thanks, Matt. At the possible cost of a perception of log-rolling here, I’d like to acknowledge your article on healthcare.gov, with a special nod to your point #2, “The System Produced the Outcome, Not the Lack of Testing”.
If there’s one constant throughout history, it is that we don’t learn from it. There have been bigger blunders than this one and the game is still being played. This is not an IT problem it’s about money, power and influence. What get’s ignored is ethics. Projects this size ex definition cannot be run ethically. Once you hang the $$$$ carrot out there it’s all about getting the carrot and not delivery.
Generally I believe anything over $5m in IT fails. Failure being one of over budget, reduced functionality or delayed. They are just too big a carrot AND I think that is about the limit a group of humans can loosely define what they want for it in return.
I’ve seen so many failures sold as successes (and successes actually being sold as failures too, which is really sad, when it happens!) that it is no wonder we can’t cope. How do you fix something that isn’t perceived as broken? It is our inability to live and deal with failure that is keeping us from actually getting better at what we do. All the sentences you listed are there to hide ethical issues, where we have to actually admit and deal with failure.
Therefore I’m always HAPPY when projects fail because it actually lets us talk over and deal with things we should be (of course not happy about any personal consequences for individuals that is!). Healthcare.gov won’t be the failure to end all failures (like WWI wasn’t the war to end all wars). We will advance a bit and have to wait for it to take hold, then we have to wait for the next one to do the next step.
I’ve heard more than 95% of your list above and every time it was ethically wrong, hiding a well known fact nobody dared to mention, would get you fired if you did (“the mortgage problem”) and certainly stopped you from actually fixing what needed to be fixed/ask for help and get helped. I also note that each project of the kind used at least a third of them on a regular basis. Nobody ever just uses one of these statements.
There is also a list of sentences these projects use in interviews to “entice” you to start there. I’ve only started thinking about this when I read the article but yes, they are definitely there. And I do know some and try and stay away from those projects where I can. My favourite one is “This is a mega project worth $xx,000,000.00….”. I tend not to hear what comes after that. I just ask some verifying questions on where they are in the project spiral of death and then go and look elsewhere….that is, if I have an option.
(btw, happily in project right atm and took me a long time to find)