Belated realization of what works

I've previously blogged about the contrast between the technically sophisticated Obama re-election campaign and the dog's breakfast that is Healthcare.gov. Go take a quick look to refresh your memories.

Now it turns out that at least one of the team being "tech-surged" to worked on the highly successful tech of Obama's re-election campaign:

One of two surge team members named by the agency was Michael Dickerson, which [sic, who taught CNN subs grammar?] CMS said was on leave from Google.
"He has expertise in diving into any layer of the tech stack ... in order to deliver some of the world's most reliable online services," CMS spokeswoman Julie Bataille said.

Dickerson is a site reliability engineer at Google and worked on some of the key performance-critical systems for the Obama team, as per his CV:

Designed and implemented, with Chris Jones and Yair Ghitza, the 2012 realtime election day monitoring and modeling (based on "Gordon" or vanpollwatcher.com).
Also: Wrote a tool for computing walkability of potential contacts, used by several states to prioritize GOTV contacts. Helped create the algorithm for targeting national TV cable ads to party preference and behavior, and wrote the tool that was used to do it. Prepared disaster recovery for all of OFA's mysql databases before Hurricane Sandy. Conducted various scalability and reliability assessments for many teams in OFA Tech and Analytics.

Finally the federal government is getting smart about how to fix the healthcare.gov problems - find people who a) have an interest in seeing this effort not fall on its arse, b) have the technical chops to know about the issues involved in a near-realtime distributed DB-backed system, and c) are willing and able to kick ass, then hand them a stick with a nail in the end and give them an open-ended mandate to pull the HHS chestnuts out of the fire.

Too late? Maybe. The government has committed to having things working by the end of the month. Without knowing specifics, and assuming a virtually unlimited budget, I think they are finally getting the right kind of people in to sort out their problems. The question is how many reputations and careers of the incumbent project managers and developers they are willing to sacrifice. I suspect at this point the answer is "all".

The curse of experts

Megan McArdle, who has been all over the HealthCare.gov and the Affordable Care Act rollout like a rash, has a superb piece at Bloomberg on the reason that the implications of the ACA came as a surprise to most people:

"We all knew" that preventive care doesn't save money, electronic medical records don’t save money, reducing uncompensated care saves very little money, and "reining in the abusive practices" of insurance companies was likely to raise premiums, [my italics] not lower them, because those "abuses" mostly consist of refusing to cover very sick people. But that information did not get communicated very well to the public.

This is, profoundly, what dooms any number of projects. For instance any software engineer or technical manager worth their salt will implicitly believe that a) testing a system with something like real traffic is the only way to detect and mitigate launch problems, and b) if you're only planning on testing one week before a hard deadline then You're Going To Have A Bad Time. Yet, if the project is being managed elsewhere and the project managers are not really asking the engineers about their opinions, just handing down features and deadlines, then the facts that "all the experts know" never get presented to the project manager in a way that makes them understand.

This reminded me of the testimony of CMS head Marilyn Tavenner about the awesome project fuck-up that was the HealthCare.gov launch and her part in it as the official directly responsible for its launch:

During the Tuesday hearing, Tavenner rejected the allegation that the CMS mishandled the health-care project, adding that the agency has successfully managed other big initiatives. She said the site and its components underwent continuous testing but erred in underestimating the crush of people who would try to get onto the site in its early days.
"In retrospect, we could have done more about load testing," she said.

You see what I mean? All the experts "know" that load testing a site that's going to be heavily used is not optional and not to be left to the last moment.

Reassuringly, Tavenner did demonstrate some skills in her area of competence: blame-shifting.

Under questioning, Tavenner pointed the finger at CGI Federal, saying the company sometimes missed deadlines. "We've had some issues with timing of delivery," she said.

I'm sure that's right. I'm equally sure that it's the project manager's job to anticipate, plan for and adjust schedules to handle late (or even early) deliveries - and CMS was the project manager. You'll note from her bio that Tavenner is a life-long health administrator - I'd bet her early career as a nurse lasted just long enough to get her into admin - and has as much business leading a complicated software development project as I do running an emergency room. Less, probably, because at least I know that air goes in and out, blood goes round and round, and any variation on this is a bad thing.

Ironically the Chief Technology Officer of Health and Human Services (HHS being the parent department of the CMS) whose bio indicates reasonable technical chops wasn't actually involved much in the project:

...an employee of Amazon Web Services Inc (AWS) emailed two HHS officials on October 7 saying, "I hear there are some challenges with Healthcare.gov. Is there anything we can do to help?"
HHS' Chief Technology Officer Bryan Sivak replied to Amazon by email on October 8: "I wish there was. I actually wish there was something I could do to help. [my emphasis]"

The Chief Information Officer by contrast is an ex-IBM marketeer and strategizer, and is putting his strategizing skills to good use making clear his distance from the smoking wreck of the project:

HHS' Chief Information Officer Frank Baitman replied to Amazon on October 7, "Thanks for the offer! Unfortunately, as you know, I haven't been involved with Healthcare.gov. I'm still trying to figure out how I can help, and may very well reach out for assistance should the opportunity present itself."

Nice one, Frank. Of course, Sivak is the one who comes across as actually human.

It looks like Tavenner's CMS wanted all the glory and kudos from the HealthCare.gov launch, but instead has become the focus the frustrations and hate of millions of Americans. The lessons here: be careful what you wish for, and if you want to know what the "experts know" then you really need to ask them.

For some needs, the government comes through

There's a lot of anger in America currently about the general incompetence of the federal government, but it's encouraging to see that at least one government agency is actually good at what it's paid to do:

The National Security Agency has secretly broken into the main communications links that connect Yahoo and Google data centers around the world, according to documents obtained from former NSA contractor Edward Snowden and interviews with knowledgeable officials.

Privacy concerns aside, you've got to admire the NSA for actually conducting some good modern communications interception. Someone probably deserves a substantial bonus; he won't get it, of course, because it's a government payroll - he'll no doubt defect to the private sector eventually, or maybe the SVR will make him the proverbial un-refusable offer.

It would be fascinating to know whether the NSA is just tapping links external to the USA (presumably including links with no more than one node in the USA) or have general access to intra-USA traffic. It's also interesting to speculate on the connection between this eavesdropping and Google's move back in September to encrypt the traffic that the NSA seems to have been intercepting. Yahoo still seems to be open, based on a rather inadequate denial from their PR:

At Yahoo, a spokeswoman said: "We have strict controls in place to protect the security of our data centers, and we have not given access to our data centers to the NSA or to any other government agency."

and one has to wonder about Facebook, Apple, Amazon etc.

So congratulations, citizens of the USA - you have a productive and competent government agency! Perhaps you should have put the NSA in charge of healthcare...

Reliability through the expectation of failure

A nice presentation by Pat Helland from Salesforce (and before that Amazon Web Services) on how they built a very reliable service: they build it out of second-rate hardware:

"The ideal design approach is 'web scale and I want to build it out of shit'."
Salesforce's Keystone system takes data from Oracle and then layers it on top of a set of cheap infrastructure running on commodity servers

Inituitively this may seem crazy. If you want (and are willing to pay for) high reliability, don't you want the most reliable hardware possible?

If you want a somewhat-reliable service then sure, this may make sense at some price and reliability points. You certainly don't want hard drives which fail every 30 days or memory that laces your data with parity errors like pomegranate seeds in a salad. The problems come when you start to get to demand more reliability - say, four nines (99.99% uptime, about 50 minutes downtime per month) and scaling to support tens if not hundreds of concurrent users across the globe. Your system may consist of several different components, from your user-facing web server via a business rules system to a globally-replicating database. When one of your hard drives locks up, or the PC it's on catches fire, you need to be several steps ahead:

1. you already know that hard drives are prone to failure, so you're monitoring read/write error rates and speeds and as soon as they cross below an acceptable level you stop using that PC;
2. because you can lose a hard drive at any time, you're writing the same data on two or three hard drives in different PCs at once;
3. because the first time you know a drive is dead may be when you are reading from it, your client software knows to back off and look for data on an alternate drive if it can't access the local one;
4. because your PCs are in a data centre, and data centres are vulnerable to power outages / network cables break / cooling failures / regular maintenance, you have two or three data centres and an easy way to route traffic away from the one that's down;

You get the picture. Trust No One, and certainly No Hardware. At every stage of your request flow, expect the worst.

This extends to software too, by the way. Suppose you have a business rules service that lots of different clients use. You don't have any reason to trust the clients, so make sure you are resilient:

1. rate-limit connections from each client or location so that if you get an unexpected volume of requests from one direction then you start rejecting the new ones, protecting all your other clients;
2. load-test your service so that you know the maximum number of concurrent clients it can support, and reject new connections from anywhere once you're over that limit;
3. evaluate how long a client connection should take at maximum, and time out and close clients going over that limit to prevent them clogging up your system;
4. for all the limits you set, have an automated alert that fires at (say) 80% of the limit so you know you're getting into hot water, and have single monitoring page that shows you all the key stats plotted against your known maximums;
5. make it easy to push a change that rejects all traffic matching certain characteristics (client, location, type of query) to stop something like a Query of Death from killing all your backends.

Isolate, contain, be resilient, recover quickly. Expect the unexpected, and have a plan to deal with it that is practically automatic.

Helland wants us to build our software to fail:

...because if you design it in a monolithic, interlinked manner, then a simple hardware brownout can ripple through the entire system and take you offline.
"If everything in the system can break it's more robust if it does break. If you run around and nobody knows what happens when it breaks then you don't have a robust system," he says.

He's spot on, and it's a lesson that the implementors of certain large-scale IT systems recently delivered to the world would do well to learn.

Government tech vs Valley tech

The ongoing slow-motion disaster of the HealthCare.gov exchanges has provided vast amounts of entertainment for software engineers, and not a little of "if only they'd used this (product/process/language/company) they'd have been fine." There is much talk of a tech "surge" to get highly-skilled engineers who actually know what they're doing to help with fixing the site, but that runs into problems as Jessica Myers points out in Politico:

"The skill that is needed most for someone to come in is the knowledge of how the system works," said Eric Ries, a Silicon Valley startup founder and creator of the popular "lean startup" philosophy. "Even if you got Google up to speed on the crazy architecture that makes no sense, [...] it's like if you have a predigital clock and you want to hire a hotshot. You need someone who knows how an antique clock works."

It's well known maxim - indeed, known in the trade as Brook's Law that adding more manpower to a late project makes it later. HealthCare.gov is no exception. You'll spend ages getting your new guys up to speed on the system, architecture and tools in use - and that education process has to be conducted by the best people you already have, taking them away from their current troubleshooting. That's not to say that it's necessarily the wrong choice at this time, but it's certainly not going to bring the project in early.

One of the strategic problems faced by the developers was the very nature of government IT:

Government IT comprises a network of systems that have developed over the past half-century, said Mike Hettinger, the Software & Information Industry Association's director of public sector innovation. In some cases, thousands of homegrown networks feed into one payroll or financial system. Whereas a scrappy Silicon Valley startup could wipe out a project that doesn't work, a much larger government agency doesn't have that luxury.

This is not a problem peculiar to government IT - payroll systems in particular in private companies are notorious legacy systems that quickly become too complex and full of undocumented behavior to replace without large amounts of pain. However, in private industry there's usually a point at which the cost of supporting and working around the legacy system becomes annoying enough that people are willing to put up with the temporary pain of replacement. Sometimes all it takes is someone hired from outside to come in, set their sights on replacing the legacy system as their first big project in the firm, and it will happen - the original system developer has probably moved on to another firm by now, and so no-one cares about it. Maybe the new finance director is fed up of paying IBM squillions of dollars a year to keep the system running. Whatever, the presence of a legacy system is unstable - very few people have a vested interest in keeping it around.

Government IT, by contrast, can grow a whole ecosystem around this one legacy system, in charge of its care and feeding, providing manual work-arounds for activities the system doesn't support or automates poorly. A government departmental budget is there for spending, so a system that is awkward to use is actually more likely to get budget because the manager can demonstrate a need: "we are up to 50 man-days of work a month to issue invoices, and our two full-time accounting assistants can't cope." The empire grows, and more people have a vested interest - their jobs, in some cases - in maintaining the status quo. As such, government departments are a near-ideal environment for these systems to flourish, rather than withering in the metaphorical dog-poop corner of the departmental garden as they should. The only business environments which can provide a similar level of support are very large firms (IBM, Microsoft, big banks etc.) where a growing budget and headcount is a mark of success to be funded, not failure to be squashed.

The reason that Silicon Valley startups and successful established businesses wipe out projects that don't work well, as opposed to keeping them around to work around their idiosyncratic ways, is because they realise that sooner or later they will be forced to wipe them out anyway - eventually the system will grind to a halt, or everyone who knows how to fix it will have left, or the hardware it depends on will fail with no supplier remaining to provide the necessary parts, or a new regulation will be passed forcing the system to behave in a new way which it cannot possibly do, or the client traffic will grow past the system's performance limit... you get the picture. If you have a mad dog in your garden, you don't wait until it's bitten one of the children - it's a mad dog, everyone knows it's mad and that bitten children are inevitable, which is why you pull your Mossberg 535 from the gun cabinet and let the dog have it.

Back to how this whole mess got started:

"At the end of the day, Washington and how we procure technology for the federal government is just different," Hettinger said.

Yes, it certainly is. One wonders why anyone would think this "different" to be synonymous with "better", when "insane" seems a better fit. Unless, of course, producing a working system is a very secondary consideration to the people in procurement and the Washington-friendly contractors (IBM, Oracle and friends).

HHS doesn't understand the problem so won't produce a solution

I apologise for turning this into the HealthCare.Gov train-wreck site, but it's such a material-rich environment that I can't help myself.

Today the US government Health and Human Services department issued a statement on what they are doing to fix HealthCare.Gov:

To ensure that we make swift progress, and that the consumer experience continues to improve, our team has called in additional help to solve some of the more complex technical issues we are encountering.
Our team is bringing in some of the best and brightest from both inside and outside government to scrub in with the team and help improve HealthCare.gov.

Interesting. I wonder in particular who from inside government is going to lend their expertise to this disaster-in-motion of software mis-engineering?

We are also defining new test processes to prevent new issues from cropping up as we improve the overall service and deploying fixes to the site during off-peak hours on a regular basis.

I really hope that this is a PR writer mis-understanding what she was told. You can't generally prevent new issues from cropping up from your code changes, because you don't know what those issues might be. You can however make a good stab at preventing old issues by setting up regression tests, running cases based on past errors to verify that the errors do not re-occur. Perhaps that's not forward-looking enough for HHS, but the sad fact is that crystal balls have very limited utility in software engineering. You're far better to improve your existing monitoring and logging so that at least you can identify and characterise errors that are occurring now.

I liked Republican Senator John McCain's suggestion for how to fix things:

"Send Air Force One out to Silicon Valley, load it up with some smart people, bring them back to Washington, and fix this problem. It's ridiculous. And everybody knows that."

The irony is that this is more or less what the Obama campaign did for the 2012 election campaign and it worked spectacularly well. If they'd done something similar for HealthCare.Gov, recruiting interested and motivated tech people from Silicon Valley (notoriously Democrat-heavy) to design and oversee the healthcare exchange, then quite possibly it would not have gone horrendously wrong. The problem now is that they are stuck with their existing design and implementation, and any redesign would necessarily trash most of their existing code and tests and require months of work to produce anything.

I'm reminded of the tourist in Ireland who asks a local how to get to Kilkenny, and the local responds "Ah well, if I wanted to get to get to Kilkenny, I wouldn't start from here."

Federal IT project comparisons

Stewart Baker at the esteemed Volokh Conspiracy argues that not all big Federal IT projects are disasters:

... it isn't impossible, even with stiff political opposition, to manage big public-facing federal IT projects successfully. I can think of three fairly complex IT projects that my old department delivered despite substantial public/Congressional opposition in the second half of George W. Bush's administration. They weren't quite as hard as the healthcare problem, but they were pretty hard and the time pressure was often just as great.

He quotes three examples:

1. ESTA: international visa waiver, serving 20M foreign customers per year and serving results to US border ports;
2. E-verify: US employers checking entitlement to work, about 0.5M transactions per year
3. US-VISIT: electronic fingerprint checks at US borders, about 45M queries per year

ESTA is a pretty good comparison to the health exchange: the user creates something like an account, uploads their identity information for offline consideration and conducts a financial transaction (paying for the visa). 20 million visitors per year sounds a lot, but it's spread fairly evenly across the day, week and year as the traffic source is world-wide. You're actually looking at an average of well under 1 user per second, and there are only a couple of pages on the site so average queries per second is in single figures. You could serve this with about 6 reasonably-specced PCs in three physically separate locations so that you always have at least two locations active and at least one PC in each location active even allowing for planned and unplanned outages. This is a couple of orders of magnitude less than the health exchange traffic - it's not a bad system to evaluate in preparing for implementation of the health exchange, but you can't expect to just translate across the systems and code. The unofficial rule of thumb is that if you design a system for traffic level X, it should (if well designed) scale fine to 10X traffic, but by the time you approach 100X you need a completely different system. The serving to border checks is a similar scale - most visitors with an ESTA visit the US about once per year, so you expect about 20M border checks per year and so around 1 query per second.

E-verify can be dismissed immediately as not comparable: it's an extremely lightweight check and has very low traffic levels.

US-VISIT is more interesting: although it's only a couple of queries per second, fingerprint matching is well known to be computationally intensive. Fortunately it's very easy to scale. You "shard" the fingerprint database by easily identified characteristics, breaking it into (possibly overlapping) subgroups; say, everyone with a clockwise whorl on their right thumb and anticlockwise spiral on their left index finger goes into subgroup 1. That your frontend receiving a fingerprint set can identify an appropriate subgroup and query one of a pool of machines which has all fingerprint sets matching that characteristic. You have a few machines in each pool in three separate sites, as above.

These are interesting applications, and I agree that they are reasonable examples of federal IT projects that work. But they are relatively simple to design and build, and they did not have the huge publicity and politically imposed deadlines that the health exchanges have. If any lesson comes from these projects, it's that well defined scopes, low traffic levels and relaxed performance requirements seem to be key to keep federal IT projects under control.

How to build and launch a federal health care exchange

Since the US government has made a pig's ear, dog's breakfast and sundry other animal preparations of its health care exchange HealthCare.Gov, I thought I'd exercise some 20/20 hindsight and explain how it should (or at least could) have been done in a way that would not cost hundreds of millions of dollars and would not lead to egg all over the face of Very Important People. I don't feel guilty exercising hindsight, since the architects of this appalling mess didn't seem to worry about exercising any foresight.

A brief summary of the problem first. You want to provide a web-based solution to allow American citizens to comparison-shop health insurance plans. You are working with a number of insurers who will provide you with a small set of plans they offer and the rules to determine what premium and deductible they will sell the plan at depending on purchaser stats (age, family status, residential area etc.) You'll provide a daily or maybe even hourly feed to insurers with the data on the purchasers who have agreed to sign up for their plans. You're not quite sure how many states will use you as their health care exchange rather than building your own, but it sounds like it could be many tens of states including the big ones (California, Texas). We expect site use to have definite peaks over the year, usually in October/November/early December as people sign up in preparation for the new insurance year on Jan 1st. You want it to be accessible to anyone with a web browser that is not completely Stone Age, so specify IE7 or better and don't rely on any JavaScript that doesn't work in IE7, Firefox, Safari, Chrome and Opera. You don't work too hard to support mobile browsers for now, but Safari for iPad and iPhone 4 onwards should be checked.

Now we crunch the numbers. We expect to be offering this to tens of millions of Americans eventually, maybe up to 100M people in this incarnation. We also know that there is very keen interest in this system, and so many other people could be browsing the site or comparison-shopping with their existing insurance plans even if they don't intend to buy. Let's say that we could expect a total of 50M individual people visiting the site in its first full week of operation. The average number of hits per individual: let's say, 20. We assume 12 hours of usage per day given that it spans America (and ignore Hawaii). 1bn hits per week divided by 302400 seconds yields an average hit rate of about 3300 hits per second. You can expect peaks of twice that, and spikes of maybe five times that during e.g. news broadcasts about the system. So you have to handle a peak of 15000 hits per second. That's quite a lot, so let's think about managing it.

The first thing I think here is "I don't want to be worrying about hardware scaling issues that other people have already solved." I'm already thinking about running most of this, at least the user-facing portion, on hosted services like Amazon's EC2 or Google's App Engine. Maybe even Microsoft's Azure, if you particularly enjoy pain. All three of these behemoths have a staggering numbers of computers. You pay for the computers you use; they let you keep requesting capacity and they keep giving it to you. This is ideal for our model of very variable query rates. If we need about one CPU and 1GB of RAM to handle three queries per second of traffic, you'll want to provision about 5000 CPUs (say, 2500 machines) during your first week to handle the spikes, but maybe no more than 500 CPUs during much of the rest of the year.

The next thought I have is "comparison shopping is hard and expensive, let's restrict it to users whom we know are eligible". I'd make account creation very simple; sign up with your name, address and email address plus a simple password. Once you've signed up, your account is put in a "pending" state. We then mail you a letter a) confirming the sign-up but masking out some of your email address and b) providing you with a numeric code. You make your account active and able to see plans by logging in and entering your numeric code. If you forget your password in the interim, we send you a recovery link. This is all well-trodden practice. The upshot is that we know - at least, at a reasonable level of assurance - that every user with an active account is a) within our covered area and b) is not just a casual browser.

As a result, we can design the main frontend to be very light-weight - simple, cacheable images and JavaScript, user-friendly. This reduces the load on our servers and hence makes it cheaper to serve. We can then establish a second part of the site to handle logged-in users and do the hard comparison work. This site will check for a logged-in cookie on any new request, and immediately bounce users missing cookies to a login page. Successful login will create a cookie with nonce, user ID and login time signed by our site's private key with (say) a 12 hour expiry. We make missing-cookie users as cheap as possible to redirect. Invalid (forged or expired) cookies can be handled as required, since they occur at much lower rates.

There's not much you can do about the business rules evaluation to determine plan costs: it's going to be expensive in computation. I'd personally be instrumenting the heck out of this code to spot any quick wins in reducing computation effort. But we've already filtered out the looky-loos to improve the "quality" (likelihood of actually wanting to buy insurance) of users looking at the plans, which helps. Checking the feeds to insurers is also important; put your best testing, integration and QA people on this, since you're dealing with a bunch of foreign systems that will not work as you expect and you need to be seriously defensive.

Now we think about launch. We realise that our website and backends are going to have bugs, and the most likely place for these bugs is in the rules evaluation and feeds to insurers. As such, we want to detect and nail these bugs before they cause widespread problems. What I'd do is, at least 1 month in advance of our planned country-wide launch, launch this site for one of the smaller states - say, Wyoming or Vermont which have populations around 500K - and announce that we will apply a one-off credit of $100 per individual or $200 per family to users from this state purchasing insurance. Ballpark guess: these credits will cost around $10M which is incredibly cheap for a live test. We provision the crap out of our system and wait for the flood of applications, expect things to break, and measure our actual load and resources consumed. We are careful about user account creation - we warn users to expect their account creation letters within 10 days, and deliberately stagger sending them so we have a gradual trickle of users onto the site. We have a natural limit of users on the site due to our address validation. Obviously, we find bugs - we fix them as best we can, and ensure we have a solid suite of regression testing that will catch the bugs if they re-occur in future. The rule is "demonstrate, make a test that fails, fix, ensure the test passes."

Once we're happy that we've found all the bugs we can, we open it to another, larger, state and repeat, though this time not offering the credit. We onboard more and more states, each time waiting for the initial surge of users to subside before opening to the next one. The current state-by-state invitation list is prominent on the home page of our site. Our rule of thumb is that we never invite more users than we already have (as a proportion of state population), so we can do no more than approximately double our traffic each time.

This is not a "big bang" launch approach. This is because I don't want to create a large crater with the launch.

For the benefit of anyone trying to do something like this, feel free to redistribute and share, even for commercial use.

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Update: also very worth reading Luke Chung's take on this application, which comes from a slightly different perspective but comes up with many similar conclusions on the design, and also makes the excellent usability point:

The primary mistake the designers of the system made was assuming that people would visit the web site, step through the process, see their subsidy, review the options, and select "buy" a policy. That is NOT how the buying process works. It's not the way people use Amazon.com, a bank mortgage site, or other insurance pricing sites for life, auto or homeowner policies. People want to know their options and prices before making a purchase decision, often want to discuss it with others, and take days to be comfortable making a decision. Especially when the deadline is months away. What's the rush?

Project management: harder than one might think

One of the most startling revelations in the continuing slow-motion carnage of the US federal health exchanges is that the government's Center for Medicare and Medicaid Services (CMS) decided to manage the whole affair themselves:

The people I spoke with did all confirm the importance of one other detail in the Times story: that CMS did not hire a general contractor to manage the exchange project but handled that overall technical management task itself. None of the people I spoke with wanted to get into how this decision was made or at what level, but all of them agreed that it was a very bad idea and was at the core of the disaster they have so far experienced.

This is, I believe, the inevitable result of government agencies (UK and USA specifically, but I'm sure other countries are equally guilty) of hiring "generalists", who tend to have liberal arts degrees. Because the subject of these degrees (English, Geography, History, PPE etc.) is unlikely to be directly useful in their owner's regular government work, the story told is that the general communication, analysis and critical thinking skills absorbed are what makes that graduate more valuable in the workplace than (say) someone with pre-university qualifications.

This analysis more or less works for government work which involves reporting and planning, and even for some low-level management. Unfortunately, it fails comprehensively when hard technical issues come up. I still remember the expression on the face of a 25 year old Civil Service fast stream grad (Oxford, PPE) as my grizzled engineering boss tried to explain to her the main engineering issues of the project she was allegedly managing. Picture a dog being taught Greek and you won't be far off. She was so far out of her depth that James Cameron could have been exploring below her. To be fair, you'd get a similar effect by putting an engineering grad in charge of a biochemistry research project, or a chemistry grad in charge of a micro-lending organisation - but at least they'd both be numerate enough to spot errors in the finances.

I note that anyone who proposed that the UK Border Agency head honchos oversee and project-manage the construction of a major bridge or power system would rightly be excoriated in public. "How the hell could they even know where to start? What do they know about compressive strength / high voltage transmission?" Why, then, do we assume that IT projects are any easier for non-experts to manage? I suspect the answer lies in a combination of the infinite malleability of software, and the superficial familiarity that most people have with using web interfaces (and even tweaking HTML themselves). After all, it's just words and funny characters, how hard could it be?

Allow me to link to my favourite XKCD cartoon ever:

Back to the exchanges: there's about as much reason to believe that the CMS has expertise in project management as there is to believe that I'm capable of designing a line of clothes to rival the products of Chanel and DvF. The fact that I can draw something that might be recognisable as a dress (if you squint a little) has absolutely no relevance to being able to design something that millions of people would want to wear - and that can be made for a reasonable sum of money while being resilient to the huge range of stresses and strains imposed on clothing by its wearers. What appalls me is that, given the quote above, no-one stopped the CMS from taking on the project management role despite the fact that everyone seemed to know that it was a terrible idea. Either this was a dastardly covert Tea Party guerilla plot to sabotage the exchanges, or there was a serious break-down in communication. Health and Human Services secretary Kathleen Sebelius is ultimately on the hook for the failure of the health exchanges; did she just not care that they were doomed to fail, or was there someone in the upper chain of reporting who knew what happens to the bearer of bad news and hence decided that discretion was preferable to being unceremoniously fired?

Sebelius, incidentally, is the first daughter of a governor to be elected governor in American history. She has a liberal arts BA and a master's in Public Administration. The CMS Chief Operating Officer is Michelle Snyder who holds advanced degrees in Clinical Psychology and Legal Studies and Administration. She has been a manager in the HHS budget office and had assignments with the Office of Management and Budget, Congress, the Social Security Administration, and as a management consultant in the private sector.

I'm sure liberal arts majors and management consultants have an important role to play in modern society. That role does not, apparently, include being in charge of a major IT project. Not only are they incompetent to run it, it seems that they are incompetent to appoint someone competent to run it. Personally, I'd have started with Richard Granger,, ex-head of the UK NHS Connecting for Health program that pissed £10-15 billion down the drain for no result. Yes, his track record is beyond absymal - on the other hand, a) he now knows first-hand all the mistakes you shouldn't make and b) when you announce his appointment the expectations on your project will plunge so low that even delivering a badly-working underperforming system will impress people.

Drop dead dates

I had the educational privilege, a few years ago, to watch a team in my workplace try to roll out a new business system to replace an existing system which had worked well for a while but grown gnarled, unmaintainable and no longer scaled to likely future demands. Well aware of the Second System Effect they made the new system feature-for-feature compatible, and even had a good stab at bug-for-bug. However, it was a complex problem and they spent many months spinning up a prototype system.

Eventually their manager decided that they needed to run something in production, so they picked a slice of the traffic on the existing business system that was representative but not critical, and set a target deadline of a week hence to launch it. The developers were privately rather twitchy about the prospect, but recognised the pressure that their manager was under and were willing to give it a shot. Come switchover day the new system was enabled - and promptly fell on its face. The developers found the underlying bugs, fixed them and restarted. It ran a little longer this time, but within a few hours fell over again. They fixed that problem, but within 12 hours it became clear that performance was steadily degrading with time...

The developers had a miserable time during the subsequent week. I got in pretty early as a rule, but the dev team was always in (and slurping coffee) by the time I arrived, and never left before I got home. The bugs posted in their area steadily accumulated, the system repeatedly fell down and was restarted with fixes. The team were living on the ragged edge, trying to keep the system up at the same time as triaging the bugs, adding tests and monitoring to detect the bugs, and trying to measure and improve the performance. This was analogous to changing the wheels on Sebastian Vettel's F1 car mid-lap - one hiccup and either you lose a limb or the car embeds itself in a track barrier. It became clear that the team's testing system had huge gaps, and their monitoring system couldn't generally detect failures happening - you could more or less infer what had caused the failure by checking the logs, but someone had to mail the team saying "hey, this job didn't work" for the team to look at the logs in question.

After a fortnight of this, with the team having pulled an average of 80-90 hour weeks, their manager sensibly realised that this approach was not sustainable. He announced the switch back from the new system to the old system effective next day, and immediately shaped expectations by announcing that they would not be switching back to the new system before three months had passed. The team breathed a sigh of relief, took a few days off, and re-scheduled themselves.

Once the system was pulled offline, the developers made reasonably rapid progress. They'd accumulated a host of bug reports, both in functionality and performance, and (more importantly) had identified crucial gaps in testing and monitoring. For each functional and performance bug, they first verified that they could reproduce it in their testing system - which was where they spent the bulk of their development time for several weeks after turndown - and that the monitoring would detect the condition and alert them appropriately. They triaged the bug reports, worked their way through them in priority order, built load tests that replicated the system load from normal operation and added metrics and monitoring on system latency. The time spent running in production had provided a wealth of logs and load information which gave them a yardstick against which they could measure performance.

After a few months they felt ready to try again, so they spun up the fixed system and loaded in the current data. This went much more smoothly. There were still occasional crashes, but their monitoring alerted them almost instantly so they could stop the system, spend time precisely characterising the problem, fix it, test the fix, deploy the fix and restart. The average time between crashes got longer and longer, the impact of failures got smaller and smaller, and after 6 months or so the system achieved its stated goal of greater scale and performance than its predecessor. However, all this was only possible because of the decision to roll back its initial roll-out.

I was reminded of this today when I saw that informed insiders were estimating the US federal healthcare exchanges as "only 70% complete" and needing "2 weeks to 2 months more work" to be ready. Since there are several tens of millions of potential users who need to register before January 1st, this looks to be a precarious situation. It's doubly precarious when you realise that "70% complete" in a software project is code for "I have no idea when we're going to be done." My personal rule of thumb is that "90% complete" means that you take the number of weeks spent in development so far, and expect the same again until the system is working with the specified reliability.

Megan McArdle, whose coverage of the health care exchanges has been consistently superb, makes a compelling case that Obamacare needs to set a deadline date for a working system, and delay the whole project a year if it's not met:

...given that they didn't even announce that they were taking the system down for more fixes this weekend, I'm also guessing that it's pretty bad. Bad enough that it's time to start talking about a drop-dead date: At what point do we admit that the system just isn't working well enough, roll it back and delay the whole thing for a year?

She's right. If the system is this screwed up at this point, with an unmoveable deadline of January 1st to enroll a large number of people, any sane project manager would move heaven and earth to defer the rollout. In the next 6-9 months they could address all the problems that the first roll-out has revealed, taking the time to test both functionality and performance against the traffic levels that they now know. There's no practical compulsion to run the exchanges now - the American healthcare system has been screwed up for several decades, the population is used to it, waiting another year won't make a great difference to most voters.

Chance of this happening? Essentially zero. The Democrats have nailed their colours to the mast of the good ship Affordable Care Act, and it's going out this year if it kills them. If they hold it over until next year then the full pain of the ACA's premium hikes will hit just before the mid-term elections, and they will get pummelled. They're hoping that if they launch now then the populace will be acclimatised to the costs by next November. As such, launching this year is a politically non-negotiable constraint. Politics, hard deadlines and under-performing software - a better recipe for Schadenfreude I can't imagine.