Government tech vs Valley tech

The ongoing slow-motion disaster of the HealthCare.gov exchanges has provided vast amounts of entertainment for software engineers, and not a little of "if only they'd used this (product/process/language/company) they'd have been fine." There is much talk of a tech "surge" to get highly-skilled engineers who actually know what they're doing to help with fixing the site, but that runs into problems as Jessica Myers points out in Politico:

"The skill that is needed most for someone to come in is the knowledge of how the system works," said Eric Ries, a Silicon Valley startup founder and creator of the popular "lean startup" philosophy. "Even if you got Google up to speed on the crazy architecture that makes no sense, [...] it's like if you have a predigital clock and you want to hire a hotshot. You need someone who knows how an antique clock works."

It's well known maxim - indeed, known in the trade as Brook's Law that adding more manpower to a late project makes it later. HealthCare.gov is no exception. You'll spend ages getting your new guys up to speed on the system, architecture and tools in use - and that education process has to be conducted by the best people you already have, taking them away from their current troubleshooting. That's not to say that it's necessarily the wrong choice at this time, but it's certainly not going to bring the project in early.

One of the strategic problems faced by the developers was the very nature of government IT:

Government IT comprises a network of systems that have developed over the past half-century, said Mike Hettinger, the Software & Information Industry Association's director of public sector innovation. In some cases, thousands of homegrown networks feed into one payroll or financial system. Whereas a scrappy Silicon Valley startup could wipe out a project that doesn't work, a much larger government agency doesn't have that luxury.

This is not a problem peculiar to government IT - payroll systems in particular in private companies are notorious legacy systems that quickly become too complex and full of undocumented behavior to replace without large amounts of pain. However, in private industry there's usually a point at which the cost of supporting and working around the legacy system becomes annoying enough that people are willing to put up with the temporary pain of replacement. Sometimes all it takes is someone hired from outside to come in, set their sights on replacing the legacy system as their first big project in the firm, and it will happen - the original system developer has probably moved on to another firm by now, and so no-one cares about it. Maybe the new finance director is fed up of paying IBM squillions of dollars a year to keep the system running. Whatever, the presence of a legacy system is unstable - very few people have a vested interest in keeping it around.

Government IT, by contrast, can grow a whole ecosystem around this one legacy system, in charge of its care and feeding, providing manual work-arounds for activities the system doesn't support or automates poorly. A government departmental budget is there for spending, so a system that is awkward to use is actually more likely to get budget because the manager can demonstrate a need: "we are up to 50 man-days of work a month to issue invoices, and our two full-time accounting assistants can't cope." The empire grows, and more people have a vested interest - their jobs, in some cases - in maintaining the status quo. As such, government departments are a near-ideal environment for these systems to flourish, rather than withering in the metaphorical dog-poop corner of the departmental garden as they should. The only business environments which can provide a similar level of support are very large firms (IBM, Microsoft, big banks etc.) where a growing budget and headcount is a mark of success to be funded, not failure to be squashed.

The reason that Silicon Valley startups and successful established businesses wipe out projects that don't work well, as opposed to keeping them around to work around their idiosyncratic ways, is because they realise that sooner or later they will be forced to wipe them out anyway - eventually the system will grind to a halt, or everyone who knows how to fix it will have left, or the hardware it depends on will fail with no supplier remaining to provide the necessary parts, or a new regulation will be passed forcing the system to behave in a new way which it cannot possibly do, or the client traffic will grow past the system's performance limit... you get the picture. If you have a mad dog in your garden, you don't wait until it's bitten one of the children - it's a mad dog, everyone knows it's mad and that bitten children are inevitable, which is why you pull your Mossberg 535 from the gun cabinet and let the dog have it.

Back to how this whole mess got started:

"At the end of the day, Washington and how we procure technology for the federal government is just different," Hettinger said.

Yes, it certainly is. One wonders why anyone would think this "different" to be synonymous with "better", when "insane" seems a better fit. Unless, of course, producing a working system is a very secondary consideration to the people in procurement and the Washington-friendly contractors (IBM, Oracle and friends).