NHS Track+Trace - what went wrong

By now, you've presumably seen how Public Health England screwed up spectacularly in their testing-to-identification pipeline, such that they dropped thousands of cases - because they hit an internal row limit in Excel.

Oops.

Still, how could anyone have predicted that Public Health England - who were founded in 2013 with responsibility for public health in England - could have screwed up so badly? Well, anyone with any experience of government IT in the past... 40 years, let's say. Or anyone who observed that the single most important job of a public health agency is to prepare for pandemics, which roll around every 10 years or so - remember SaRS 2003? H1N1? And that duty, as illustrated in their 2020 performance, is one that PHE could not have failed at any more badly if they'd put their best minds to it.

Simply, there's no incentive for them to be any good at what they do.

It's tempting to simply roll out the PHE leadership and have them hung from the nearest lamp post - or at least, claw back all they payments they received as a result of being associated with Public Health England. For reference, the latest page shows this list as:

• Duncan Selbie
• Prof Dr Julia Goodfellow
• Sir Derek Myers
• George Griffin
• Sian Griffiths
• Paul Cosford
• Yvonne Doyle
• Richard Gleave
• Donald Shepherd
• Rashmi Shukla

However, this misses the point; there's plenty more where they came from. Many of these people are actually smart, or at least cunning. None of them actively wanted tens of thousands of people in the UK to die, or the UK's coronavirus response to become an absolute laughing-stock. Yet, here we are.

When you set up a data processing pipeline like this, your working assumptions should be that:

1. The data you ingest is often crap in accuracy, completeness and even syntax;
2. At every stage of processing, you're going to lose some of it;
3. Your computations are probably incorrect in several infrequent but crucial circumstances; and
4. When you spit out your end result, the system you send it to will be frequently partially down, so drop or reject some or all of the (hopefully) valid data you're sending to it.

Given all these risks, one is tempted to give up managing data pipelines for a living and change to an easier mode of life such as a career civil servant in the Department for Education where nothing you do will have the slightest effect, yet you'll still get pay and pension. Still, there's a way forward for intrepid souls.

The insight you need is that you accept that your pipeline is going to be decrepit, leaky and contaminate your data. That's OK as long as you know when it's happening, and approximately how bad it is.

Let's look at the original problem. From the BBC article:

The issue was caused by the way the agency brought together logs produced by commercial firms paid to analyse swab tests of the public, to discover who has the virus. They filed their results in the form of text-based lists - known as CSV files - without issue.

We want to have a good estimate, for each agency, whether all the records have been received. Therefore we supplement the list of records with some of our own - which have characteristics which we expect to survive through processing. Assuming each record is a list of numerical values (say, number of virus particles per mL - IDK, I'm not a biologist) a simple way to do this is to make one or more fields in our artificial records have values that are 100x higher or lower than practically feasible. Then for a list of N records, you add one artifical record to the start, one at the end and one in the middle, so you ship N+3 records to central processing. For extra style, change the invalidity characteristic of each of these records - so e.g. you know that an excessively high viral load signals the start of a records list, and excessively low load signals the end.

The next stage:

PHE had set up an automatic process to pull this data together into Excel templates so that it could then be uploaded to a central system and made available to the NHS Test and Trace team, as well as other government computer dashboards.

First check: this is not a lot of data. Really, it isn't. Every record represents the test of a human, there's a very finite testing capacity (humans per day), and the amount of core data produced should easily fit in 1KB - 100 or more double-precision floating point numbers. It's not like they're uploading e.g. digital images of mammograms.

So the first step, if you're competent, is for Firm A to read-back the data from PHE:

• Firm A has records R1 ... R10. It computes a checksum for each record - a number which is a "summary" of the record, rather like feeding the record through a sausage machine and taking a picture of the sausage it produces.
• Firm A stores checksums C1, C2, ..., C10 corresponding to each record.
• Firm A sends records R1, R2, ..., R10 to PHE, tagged with origin 'Firm A' and date '2020-10-06'
• Firm A asks PHE to send it checksums of all records tagged 'Firm A', '2020-10-06'
• PHE reads its internal records, identifies 10 records, sends checksums D1, D2, ... D10
• Firm A checks that the number of checksums match, and each checksum is the same: if there's a discrepancy, it loudly flags this to a human.

This at least assures Firm A that its data has been received, is complete, and is safely stored.

If PHE wants to be really cunning then one time in 50 it will deliberately omit a checksum in its response, or change one bit of a checksum, and expect the firm to flag an error. If no error is raised, we know that Firm A isn't doing read-backs properly.

Now, PHE wants to aggregate its records. It has (say) 40 firms supplying data to it. So it does processing over all the records and for each record produces a result: one of "Y" (positive test), "N" (negative test), "E" (record invalid), "I" (record implausible). Because of our fake record injection, if 40 firms send 1000 records in total, we should expect zero "E" results, 120 "I" results, and the total of "Y" and "N" results should equal 880. If we calculate anything different, the system should complain loudly, and we send a human to figure out what went wrong.

The system isn't perfect - the aggregation function might accidentally skip 1 in 100 results, for instance, and through bad luck it might not skip an erroneous record. But it's still a good start.

I just pulled this process out of my posterior, and I guarantee it's more robust than what PHE had in place. So why are we paying the Test+Trace system £12 billion or more to implement a system that isn't even as good as a compsci grad would put in place in return for free home gigabit Ethernet, with an incentive scheme based around Xena tapes and Hot Pockets?

Nobody really cared if the system worked well. They just wanted to get it out of the door. No-one - at least, at the higher levels of project management - was going to be held accountable for even a failure such as this. "Lessons will be learned" platitudes will be trotted out, the company will find one or two individuals at the lower level and fire them for negligence, but any project manager not actually asleep on the job would have known this was coming. And they know it will happen again, and again, as long as the organisation implementing systems like this has no direct incentive for it to work. Indeed, the client (UK Government) probably didn't even define what "work" actually meant in terms of effective processing - and how they would measure it.

Hillary doesn't deserve to be President

I've just finished watching the #2 US Presidential Debate, chaired by Anderson Cooper - for whom I have a reasonable amount of respect as a more-fair-than-average interviewer - and Martha Raddatz, who was hopelessly out of her depth and showing awful bias. Coming out of the debate, I have one question for Hillary: how, with all the advantages you had two hours ago, did you manage to lose?

Going into this debate, Hillary had Donald cornered by the media after his not terribly edifying 2005 remarks about pussy-grabbing opportunities in showbiz were reported. Near-universal media agreement was that The Donald was fatally holed beneath the waterline. Even Trump's own Vice President pick, Pence, was publicly disapproving of Trump's comments. Republican senators and Congress critters were denouncing Trump and saying they wouldn't vote for it. In golf, this would be like being 2 inches away from the hole when your opponent is 200 yards away in a bunker, and it has just started to rain.

And yet... Hillary missed the putt, kept missing it, and Donald chipped his ball onto the green and snuck it into the hole before Hillary found her game.

Trump is not a great public speaker. His train of thought wanders as he speaks, and he assumes technical and factual knowledge in the audience rather than explaining as he goes along. These traits were in full display this evening. A great example was in the "birther" issue where Hillary accused Trump of asking the "racist" question about whether President Obama had actually been born in the USA. Trump (accurately) pointed out that this issue had first been raised by Hillary's consiglieri Sidney Blumenthal, but he did it in such an indirect way that anyone not substantially familiar with the people concerned would have had no idea what he was talking about and how it was tied to Hillary.

Still, somehow he did a better job of debating than Hillary herself. Tonight's debate format seemed to work better for him, because he's comfortable doing spontaneous exposition on topics. Hillary is awful at this, visibly working her way through pre-prepared points on each topic rather than going with the flow of the question and debate. Trump was prone to wander off the thread to include the attacks he wanted to make on Hillary (Bill's disbarring, Russia, black poverty, Syria, tax policy and of course her email server) but seemed to make most of it stick and force Hillary to respond.

Raddatz did her best to cover for Hillary's poor quality responses - Cooper, to his credit, did not - but it seemed clear to me that Trump had managed to bring up nearly all the Hillary dirty laundry that he had avoided in the first debate. Hillary did a variable job in responding to these points, but looked really weak on Russia/Syria, and her responses on the email server were strong but - frankly - flat-out lies. If Donald could learn to speak with more clarity and focus, he'd crucify her. As it was, this was a win on points only, but compared to expectations Donald killed it tonight.

Why was the pussy-grab tape such a non-event in this debate? I think it was because of the apology. Trump apologised for what he said on the tape a few hours after it was publicised, and did so again in the debate as soon as it was brought up. Once he'd done that, it was much harder for Hillary to use it as leverage. "He said these horrible things!" "I've apologised for that, you heard me." Where do you go from there? You can try "this shows what he thinks about women!" but Trump was willing to go on the offense about Bill Clinton and his bimbo eruptions - perhaps the lack of challenge in this area is a sign of how vulnerable Hillary thinks she is here.

By contrast, Hillary's mea culpa for the email server still had a whiff of "I'm sorry I got caught" - her assertions around "no evidence that anyone hacked the server" were incredibly weaselly. A responsible candidate would have agreed that it was quite likely that unfriendly nations had got at least some access to that server, and taken personal responsibility for any consequences arising from their decision to use it.

Conclusion? It's still game on for November 8th. Somehow Donald has mitigated the worst of the impact of the pussy-grab, and is challenging Hillary on the issues again. What other gotchas for him has she got left to leak? Are they good enough to be game-ending, or are they just "the same again"?

Unexpected consequences of Obamacare and immigration amnesties

I'm not sure why this hasn't generated more outrage yet: the Washington Times has spotted that President Obama's plan to legalize employment for illegal immigrants might screw over American workers even more than initially suspected:

President Obama's temporary amnesty, which lasts three years, declares up to 5 million illegal immigrants to be lawfully in the country and eligible for work permits, but it still deems them ineligible for public benefits such as buying insurance on Obamacare's health exchanges.

Seems sensible enough, although the amnesty beneficiaries might well be eligible for the Earned Income Tax Credit if they have kids. But there's a consequence for the lack of health exchange rights:

Under the Affordable Care Act, that means businesses who hire them won't have to pay a penalty for not providing them health coverage [my emphasis] — making them $3,000 more attractive than a similar native-born worker, whom the business by law would have to cover.

Oopsie. Since the immigrants will tend to participate in the lower-paid end of the employment spectrum, that means the $3000 delta will be a huge fraction of the wage. That's quite the competitive advantage. Sure, it means in practice that they won't have ACA-compliant health care - and in fact I'd expect many employers to pay their amnestied workers a higher headline wage to compensate for this lack of employer-supported healthcare. Nevertheless, once it's legal to employ these workers openly, the wage differential makes them look very attractive.

This won't affect unionized jobs where wages can't easily be varied, but in the private sector the medium-sized businesses who have more than 50 employees will start sucking up all the amnestied labor they can and will stop hiring the locals. Small businesses which have pushed workers into part-time slots to avoid the ACA can now replace two part-time workers with a full-time amnestied worker.

This is what happens when you create a baroque, complicated legal framework for employment and health insurance. When you subsequently make changes, you will find that they have unexpected effects.

Don't ask for your emails to be deleted

Darrell Issa, Republican congressman from California (yes, amazingly they exist) releases the oversight report on the initial rollout of Healthcare.gov and it wasn't pretty. The bulk of the report was based off emails that they managed to retrieve from Health + Human Services and their CMS subsidiary, and the report authors did a nice job of excerpting the damning snippets from the emails that confirmed everyone's suspicions about the rollout: the grunts implementing and testing the site knew darned well that it wasn't ready, but they were overridden.

I don't find any particular reason in the report to believe that the President knew the site wasn't ready; it looks very much like he and his advisors were assured that everything was in hand, and he had no particular reason to disbelieve it. The problems occurred lower down in the hierarchy:

Mr. Sivak showed Mr. Baitman emails that were made public by Congress in the wake of Healthcare.gov's disastrous launch. In these emails, dated September 27, 2013 [launch date was Oct 1st], a CMS official working on the FFM development, wrote "the facts are that we have not successfully handled more than 500 concurrent users filling out applications in an environment that is similarly in size to Day 1 production." In response, Mr. Baitman wrote "Frankly, it’s worse than I imagined!" Mr. Sivak replied, "Anyone who has any software experience at all would read that and immediately ask what the fuck you were thinking by launching."

Indeed, we were asking almost exactly that question. And there was no naivety about motivations:

How did one week Henry Chao tell us there was no way Account Transfer would be ready, then a meeting at the White House and a week later, oh, yeah, everything is back on track, we’ll meet the dates? That’s what I mean by WTF. You could definitely see the CYA moves coming a mile away

Doublethink is clearly very important for project managers. Henry Chao was one of the prime Healthcare.gov project managers and it appears he knew that the site was heading to disaster, but for some reason he couldn't or wouldn't articulate this to the administration.

Issa, of course, has plenty of partisan reasons to bash the administration and the Healthcare.gov backers, but it's hard to conclude anything other than that this launch was destined to crash and burn spectacularly, that this was known well in advance, and that it was egregiously mis-managed. That Mikey Dickerson and his crew managed to retrieve some semblance of success from this state was amazing, but not something that should be relied on by any future project manager.

Once again, the maxim "Do not write anything in an email that you do not want to see on the front page of a major newspaper" is confirmed. The usual wisdom around this is a combination of a) mail is transferred in the clear between servers on the public internet, although this is changing, and b) the risk of including the wrong person on your To: or Cc: lines. This report highlights a third option: the risk that your email will be retrieved during a legal discovery process. If you send your email from a company email system it'll be archived there and prone to later legal discovery even if you and the recipient delete it. This also applies if any of your recipients use a company or government email address.

The Verge provides a nice summary of the highlights in the report if you don't have the stomach to read the whole thing.

Formalising success in a bureaucracy

It's only natural, when you've managed to get out of a hole against all odds, that you want to re-use the people and/or planning that made the difference. You'd be wasteful if you didn't, to be honest. Following this line of thinking, and after a small team of digital fixers managed to save the flagship Healthcare.Gov federal healthcare exchange from near-certain doom, the White House is trying to do just that.

Today they announced the launch of the new U.S. Digital Service which aims to replicates the lessons of the (relative) success in saving Healthcare.Gov with other troubled US federal government IT projects. Heaven knows that there's no shortage of potential targets for USDS to help with. The question of the moment is: can this new government team actually succeed? If so, what does success look like?

US CIO Steve van Roekel outlined the USDS role:

"This isn't going to be a group that we parachute in to write code," as Van Roekel put it in a call earlier this summer, and with perhaps the Department of Health and Human's experience with HealthCare.gov on the brain, "This isn't decending a group of developers onto the scene." Rather, the focus is going to be on helping agencies figure out where their weak points are and how to fix them.

Note that therefore the role of USDS staff isn't actually the same as the Healthcare.Gov fixers, but that might be OK as the fixing itself wouldn't scale; if you want to solve the key IT problems of more than one government agency at at time then you can't have most your staff embedded in one project, and there's no reason to think that the government can recruit multiples of the motivated team that fixed Healthcare.gov. They're going to have to strike a balance, though. They won't be able to determine the principal IT problems of an agency without spending time working with and talking to the agency's tech team. The more time they spend there, the more trust they'll gain and the better the quality of information they'll gather - but then they won't be able to help as many agencies.

The danger with any new government agency is that after a time it accumulates bureaucrats whose primary purpose is propagating their own employment and importance. Van Roekel seems to be aware of this and planning to bring in people for 2-4 year rotations. With placements of 3-6 months this may be about right; long enough for the new people to spend a placement or two with the veterans and absorb the institutional knowledge, do a couple more placements as peers while encouraging their friends to join up, then lead new recruits in placements as the veterans leave.

What's going to be interesting is to see how the USDS embeds are treated in the troubled agencies. Are they going to have the influence and effective power to remove obstructions - such as long-term barnacle workers who hoard knowledge and obstruct progress? If not, they're unlikely to be able to change much. If so, the agency's workers are going to hunker down and be terrified of being fired or reassigned. It's going to be quite a challenge for tech sector workers to get their heads around the government worker mindset sufficiently to influence those workers into getting things fixed.

Incidentally, www.usds.gov was not resolving as of posting time; I actually consider that a potential sign of success as the new team is focusing on getting operational before getting any marketing/PR in place; still, they're going to need a portfolio of some form after a few months in order to attract their new short-term hires.

The importance of words

CiF poster Scott "the most" Lemieux is aggrieved at today's ruling in D.C. that puts something of a crimp in the Affordable Care Act (aka Obamacare):

Up first: an outrageous two-to-one decision by a panel of the Court of Appeals for the DC Circuit ruling against sensible subsidies that real people need, based on what we can charitably called the "reasoning" of the two Republican nominees on the three-judge panel – the opinion was written by an appointee of George HW Bush, along with a judge nominated by his son.

I do like the "play the man, not the ball" approach here, by the way. Mr. Lemieux is appalled that in Halbig vs Burwell the D.C. Circuit judges have thrown a major spanner in the works of the Obama administration's flagship Act. Since Mr. Lemieux is a professor of political science at a college in New York, you may safely assume that he knows how the legal process works and has the proper perspective to come to such a judgement.

What is this horrendous decision which has so appalled Mr. Lemieux? Let us consult the blogging lawyers at the Volokh Conspiracy:

In a 2-1 opinion, the Court held that the Internal Revenue Service regulation authorizing tax credits in federal exchanges was invalid. Judge Griffith, writing for the court, concluded, "the ACA unambiguously restricts the section 36B subsidy to insurance purchased on Exchanges 'established by the State.'" In other words, the court reaffirmed the principle that the law is what Congress enacts — the text of the statute itself — and not the unexpressed intentions or hopes of legislators or a bill's proponents.

What made the Affordable Care Act affordable for many people was that for low-to-medium incomes you could get tax credits to subsidise the (fairly expensive) policies available on the exchanges. Now the original idea was for most states to run their own exchanges, but more and more of them have used the shared federal exchange since it turns out that developing and running an exchange is fairly hard. Unfortunately, the ACA itself only allowed tax credits for insurance purchased on exchanges established by the State, which was the point of contention in this case - should the IRS be allowed to issue tax credits to people buying insurance on federal-run exchanges, which is the case in more than half of the states. The D.C. Circuit said "no, you can't apply the law as you wish it was written, you have to apply the law as it is." Apparently this approach is too radical and subversive for Mr. Lemieux and he wishes to blame the D.C. Circuit rather than (say) the original drafters of the ACA.

From the actual court decision:

Appellants argue that if taxpayers can receive credits only for plans enrolled in “through an Exchange established by the State under section 1311 of the [ACA],” then the IRS clearly cannot give credits to taxpayers who purchased insurance on an Exchange established by the federal government. After all, the federal government is not a “State,” see 42 U.S.C. § 18024(d) (defining “State” to “mean[] each of the 50 States and the District of Columbia”), and its authority to establish Exchanges appears in section 1321 rather than section 1311, see id. § 18041(c)(1).

There was a lot of controversy at the time the ACA was passed due to the very short time between it being presented and being rammed through Congress and the Senate. Democratic senator Nancy Pelosi told us not to worry about the contents of the bill at the time:

But we have to pass the bill so that you can find out what is in it – away from the fog of the controversy.

Well, now we've all found out. Perhaps a little more scrutiny at the time of passing would have been in order so that problems like the tax credits language could have been spotted before being signed into law. This is why complex laws are bad - they cause problems for everyone including those that they were intended to help.

Healthcare.gov - it's not over yet

After the well-publicised last-minute save of the Healthcare.gov federal health insurance exchange in 2013, there was a general sighing of relief from the Affordable Care Act's supporters. The months of suspense and error codes had been very painful, but finally the federal exchange was working more or less, and millions were able to enroll in time for the deadline (even if they couldn't then find a doctor who would accept their plan).

It seems that such confidence may have been misplaced; the federal site was fixed but a number of states decided to implement their own exchanges and it hasn't turned out too well for many of them, and people with badges are looking at Oregon's exchange quite hard:

The various state exchanges were funded by the U.S. Center for Medicare and Medicaid Services. As of March 28, it had awarded $4.7 billion in grants nationally. Oregon has been awarded $303 million in five increments.
Typically in federally funded projects, states must show evidence they've met certain milestones before they get additional grants. In Oregon's case, the Oregon Health Authority at first, and later Cover Oregon, had several "gate reviews" where federal officials reviewed progress on the project.
Documents show Oregon may have presented a misleading picture to the federal government.

Oregon has a population of about 4 million people. Since ACA sign-up is about 2.5% of the US population, let's be generous and say 3% for Oregon; that's $303 million for 120K people, or about $2500 per sign-up - just for the website. They've now dropped the idea entirely and fallen back to use the federal site which at least kind of works.

Most of the money went to Covered Oregon's contractor Oracle, who probably won't be promoting this as a triumph because the website has not managed to enroll a single private customer:

After discovering a series of technical problems before the Oct. 1 launch -- including inaccurate calculations of the tax subsidies for which consumers would be eligible -- Cover Oregon decided to scratch its plans to go live along with the rest of the country and never managed to get online enrollment started.

This hasn't come from out of the blue: back in November Oracle were being grilled about missed deadlines. Their VP Tom Budnar reported that they were bringing in an Oracle "SWAT team" but it would appear their efforts were in vain. Oracle's stock price has steadily risen from $39.5 to $42 in the past month, so at least Mr. Budnar doesn't have to worry about loss of public confidence in the company. Mind you, a search of Oracle's website doesn't show any results with his surname, so maybe they're already planning to cut him loose - if they've not done so already. He's going to have a difficult round of interviews if that's the case; I'd advise him to take a year of sabbatical and hope most people have forgotten about Oregon...

Oregon is egregiously bad, but unfortunately not too far out of line with other states' exchanges. As NPR reports this week, costs per enrollee of the exchanges have made the federal effort look like a model project:

Even Covered California, the most efficient of the state-run exchanges at $758 per enrollee, still spent more than the average for the federal exchange. And California was the only state-run exchange with a per-person average under $1,000.

Okay, so why did this cost so much? We can look at a comparable solution in the private sector in the USA: TurboTax (web edition) which is delivered by the firm Intuit. Filing federal and state taxes in the USA is a substantial undertaking; most people either go to a tax preparer (H+R Block is a popular one for lower-income taxpayers) or use some tax prep software. Filling in Form 1099 and its many supplemental forms by hand is a great way to lose money, get audited, develop a drinking habit and hate of humanity, or all three. TurboTax is a popular web system for filing taxes; it does automatic imports of data from a number of sources (major employees and brokerages), has a user-friendly interface, lets you submit forms electronically, and generally makes tax preparation a breeze. My expat colleagues who have used TurboTax sing its praises as being slightly easier to use than the short-form UK tax form, performing substantially more complex calculations and optimizations.

US tax season is not dissimilar to healthcare signups - a short period (1-2 months) after you get the tax information from your employer and brokerages and before the April 15th deadline, usage spiking in evenings and on weekends. So lots (millions, I'd estimate) of users crowd onto the TurboTax site in a short time, with downtime or even slowness being commercially unacceptable - there are several alternatives to TurboTax. Turbotax covers its costs and generates income by serving basic users for free, simple uses for $30 per user and complex uses for $50 per user (state filing is an additional $20 or so). You can even cover your costs if you use Amazon heavily.

It's not an exaggeration to say that if the states and federal goverment had contracted TurboTax's Intuit to build and operate the site for $70 per user - one tenth of the current cost in reality - then they could probably have made it work. They already have the storage and serving systems capable of handling that level of traffic, the marginal cost is probably less than developing from scratch. So why didn't the states and feds do this? Why did Oregon contract Oracle? Because Oracle and other consulting firms are very good at bidding for federal and state contracts. They're pretty awful at making them work, but this doesn't matter - strange to relate, contracts are not issued and paid based on whether the delivered solution actually works. The incentives in government contracting have generated an ecosystem where actual ability to build working systems is one of the less important characteristics for which the environment selects.

The defence of the federal contracting system (instead of assigning the contract to a company who might actually be able to make the contract work) starts with "we have to protect the taxpayer from being exploited in no-bid contracts." Except that it seems that the taxpayer is already being exploited. If Intuit charged $400 per user, made out like a bandit, gave bonuses through the company right down to the mailroom boy's cat and sent the execs off on 1 year fact-finding trip to the Bahamas, the taxpayer would still come out ahead, and would have the bonus of a website that actually worked well and was user-friendly.

Dodgy assertions from CASH's head medic

The salt-haters have been praising the reduction in dietary salt for an important role in the 42% fewer stroke fatalities and 40% drop in those dying from coronary heart disease:

The researchers, who include Britain's leading campaigner against added salt in food, claim that diminishing levels of salt was "an important contributor" to falls in blood pressure over the eight-year period. "As a result, the decrease in salt intake would have played an important role in the reduction of stroke and ischaemic heart disease mortality during this period," say the authors.

"Would have played"? That's a funny way of saying "was shown at a 95% confidence level to have played"... Co-author Graham MacGregor is the chair of CASH; his daytime job is Professor of cardiovascular medicine at the Wolfson Institute of Preventive Medicine at Queen Mary. So surely we can expect a rigorous and impartial analysis of the data from him.

If I'd been looking to prove or disprove this assertion, I'd have looked at stroke and heart disease rates in a range of patients over this time frame, where I had some objective measure of salt in their diet (urine samples), and looked to see whether patients with lower salt levels (in a group of patients with otherwise similar exercise, age, gender, racial stats) were correlated with lower stroke and heart disease rates. Is this what they did?

Patrick Wolfe, professor of statistics at University College London, took issue with the authors for assuming that the improved blood pressure seen in the 2003-2011 was largely the result of reduced salt intake. "Plausibility of assumption does not equal evidence," he said.

Oh. Apparently not, then. That's a piss-poor basis for the claims CASH (and international co-conspirator WASH) have been touting around about salt reduction. As commentor ID4968047 notes this reduction in strokes and heart disease could equally have come from the reduction in smoking in the past 10 years - the obligation is on Prof. MacGregor to show otherwise. Looking at CASH's writeup of the paper (the link to the paper isn't available yet, looks like) they say:

Confounding factors that were looked at include age, gender, ethnicity, education, incomes, alcohol consumption, fruit and vegetable intake and BMI.

Exercise and smoking are not mentioned. Nor do they reference the increase in statin use - and indeed Aseem Malhotra from Action on Sugar claims that statins are harmful and don't reduce mortality which is interesting as they seem to be a prime competitor to CASH/Action on Sugar's crusades against sugar and salt. Malhotra's claims got panned for lack of evidence by Prof. Rory Collins from Oxford.

It seems that others in the medical stats community have doubts too:

David Spiegelhalter, professor of the public understanding of risk at Cambridge university, cited the researchers' admission that the fall over that time in systolic blood pressure would be expected to reduce strokes by just 11% and heart attacks by 6%, small amounts of the total falls. [my emphasis] Reduced blood pressure did not represent the authors' claimed "substantial contribution" to the reduced death rates.

This is not to say that Graham MacGregor is obviously wrong in his claims. They might be true but it is a real reach to claim that this study supports them. And if this is the best he can do, I'd suggest the Marcela Trust / OMC Investments crowd who are backing CASH find someone with a better stats background to organise their crusade against salt and sugar.

Update: just managed to dig up the link to the full text in BMJ Open. From a quick look the focus was on linking salt reduction with BP reduction but not explicitly with stroke/CVD reduction.

The authors themselves admit:

It is likely that several factors, that is, the fall in BP, total cholesterol and smoking prevalence, the reduction in salt intake and the increase in the consumption of fruit and vegetables, along with improvements in the treatments of BP, cholesterol and CVD, contributed to the decrease in stroke and IHD mortality.

They have a stab at isolating the effect of salt by casting tea leaves:

it was estimated that a 2.7 mm Hg reduction in systolic BP that occurred with salt reduction would be predicted to reduce stroke by approximately 11% and IHD by 6%.

but even then the 2.7mmm Hg reduction figure they quote is the net over 8 years including factors such as decrease in smoking and increase in statins, so to attribute it to just salt reduction is "optimistic". They appeal to studies in Japan and Finland in the late 60's / early 70's but the huge gaps in time, diet and environment between now and then render the comparison unconvincing. If that's the best argument they've got to offer, I'd hate to see the ones that didn't get selected for use in the paper.

The conclusions are what kill the paper for me:

The reduction in salt intake is likely to be an important contributor to the falls in BP in England from 2003 to 2011. As a result, the decrease in salt intake would have played an important role in the reduction in stroke and IHD mortality during this period. [my emphasis]

That's a terribly weak conclusion even to my relatively untrained eyes. If they could state this more strongly, they would. Instead, they reserve their strength for polemic:

... the mean salt intake in England (8.1 g/day in 2011) was still 35% higher than the recommended level of 6 g/day, and 70% of the adult population (80% men and 58% women) had a daily salt intake above the recommended level.[14] Therefore, continuing and much greater efforts are needed to achieve further reductions in salt intake to prevent the maximum number of stroke and IHD deaths.

Reference 14 doesn't justify the 6g/day level, it's just a measurement of sodium levels. The authors don't make any reference I can see to why the recommended level should be 6g/day and not (say) 10g/day or 3g/day. If you're appealing to magic figures in your conclusion it doesn't give great confidence in the rest of your article.

Obamacare is going to have a doctor problem

An aspect of the American Affordable Care Act (aka Obamacare) which I hadn't appreciated until this week was the degree to which it may be scuppered by primary care physicians (PCP, the USA equivalent of a GP). Chatting with a middle-aged friend who is moving to a small-ish American town and changing jobs, she opened my eyes to the mysteriously unpublicised problems which the implementation of the ACA is about to cause everyone.

She was visiting her new town last week, sorting out her house and meeting her new workmates in preparation for moving there for good in June. One item on her list was sorting out a new PCP. She had the details for her new insurance plan with one of the state providers, so dropped by a doctor's office near her house to register. No joy, that doctor wasn't accepting new patients. OK, so go back to the insurance provider website to identify a wider shortlist of providers in town.

No joy at the first doctor. Or the second. Or the third. This was getting ridiculous. She had a friend already working in her new workplace, so asked him if he could help. His doctor wasn't accepting new patients either, but the friend had a good relationship with the doctor, so asked him directly. Still no luck. Eventually my friend managed to find a large medical center open to new patients - over 60 miles away. She signed up, because that was the best offer there was.

There was one doctor in a nearby town open to new patients, but after a quick check on his reviews it became rapidly clear why. It sounds like he hadn't progressed much past the "trial by ordeal" approach to illness, and was still reading pre-Pasteur literature on infection control.

Why this insanity of unavailability? It turns out that the coming avalanche of patients signed up on ACA-compliant plans is not making doctors sleep well at night. Not only are these patients much more likely to be sick than their current patients, the main concern of practices is that they're going to lose money on treating these patients. The remuneration rate for doctors for ACA patients is - at least in some states, and I suspect all - based on 80% of the standard practice fees from 3 years ago. That's not great, but much worse is the experience with Medicare (federal medical coverage for the elderly). My friend used to work in a doctor's office, and they had a profitably employed office employee whose sole job was to push back against government agencies spuriously denying payment for Medicare claims. If there were any abnormalities at all in a claim, Medicare would deny it in the hope that only solidly valid claims would be retried. A denial is, after all, cheap.

I digress, but bear with me. Do you older readers remember when The Simpsons started and they introduced us to Marge's sisters Patty and Selma, stalwart misanthropic employees of the local Department of Motor Vehicles? There's a reason why the American viewers of the show laughed, and it wasn't to do with absurdity. There's no-one more un-fireable than a federal or state bureaucrat, and their attitude to their customers is exactly what you'd expect - entirely down to their general disposition to humanity. That's what the doctors' billing teams are going to be dealing with. Only by being better acquainted with the rulebook will they be able to get their due payments out of the ACA system, and even then those payments will be small and paid late.

Having a new government agency and new set of rules means that small practices are extremely worried about their costs and ability to maintain existing patients. Because (I believe, from chatting to the front desks of a few offices) you can't be selectively open to new patients based on their particular insurer, the easiest option for small providers is to batten down the hatches until the picture is clearer. Only the very large healthcare centers can take advantage of their economy of scale to accept the new stream of ACA-covered patients - and with them the standard employer-covered and easier to manage new patients.

I wonder whether this problem with finding doctors is behind the proposed rise in rates for ACA-compliant plans? Perhaps you will be able to look at the rate rise in the next few years and extrapolate the "marginal cost of doing business with government". As it stands, the best they can hope for is to break even:

Cigna, which is selling Obamacare plans in five states and is considering whether to expand that further in 2015, has said it won't make money on the business this year. It expects to have signed up as many as 100,000 new members under the program for this year.

So for the Affordable Care Act, if you're sick then you may be able to get coverage where you couldn't before, but sure as heck if you're outside a major city then you're going to find it a challenge to get a primary care physician to see you and get your care started. Now I'm wondering what the ER departments of ACA-plan-accepting hospitals are going to experience in the next year.

Fixing Healthcare.gov - the inside story

The new Time covers in depth the work of the team who fixed Healthcare.gov. It's a fantastic read, with good access to the small but extremely competent team who drove the fix - go absorb the whole thing.

The data coming out of the story confirms a lot of what I suspected about what was wrong and how it needed to be fixed. Breaking down by before-and-after the hit team arrived:

Before

1. By October 17 the President was seriously contemplating scrapping the site and starting over.
2. Before this intervention, the existing site's teams weren't actually improving it at all except by chance; the site was in a death spiral.
3. No one in CMS (or above) was actually checking whether the site would work before launch.
4. The engineers (not companies) who built the site actually wanted to fix it, but their bosses weren't able to give them the direction to do it.
5. There was no dashboard (a single view) showing the overall health of the site.
6. The key problem the site had was being opened up to everyone at once rather than growing steadily in usage.
7. The site wasn't caching the data it needed in any sensible way, maximising the cost of each user's action; just introducing a simple cache improved the site's capacity by a factor of 4.

I refer the reader in particular to my blogpost The Curse of Experts where CMS head Marilyn Tavenner was trying to dodge blame.

During the Tuesday hearing, Tavenner rejected the allegation that the CMS mishandled the health-care project, adding that the agency has successfully managed other big initiatives. She said the site and its components underwent continuous testing but erred in underestimating the crush of people who would try to get onto the site in its early days. "In retrospect, we could have done more about load testing," she said.

As the Time article shows, this was anything but the truth about what was actually wrong.

After

1. There wasn't any real government coordination of the rescue - it was managed by the team itself, with general direction but not specific guidance from the White House CTO (Todd Park)
2. The rescue squad was a scratch team who hadn't worked together before but was completely aligned in that they really wanted to make the site work, and had the technical chops to know how to make this happen if it was possible.
3. Fixing the website was never an insurmountable technical problem: as Dickerson noted "It's just a website. We're not going to the moon." It was just that no-one who knew how to fix it had been in a position to fix it.
4. The actual fixes were complete in about 6 weeks.
5. One of the most important parts in improving the speed of fixing was to avoid completely the allocation of blame for mistakes.
6. Managers should, in general, shut up during technical discussions: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it."
7. The team refused to commit to artificial deadlines: they would fix it as fast as they could but would not make promises about when the fixes would be done, refusing to play the predictions game.
8. Having simple metrics (like error rate, concurrent users on the site) gave the team a good proxy for how they were doing.
9. Targeted hardware upgrades made a dramatic difference to capacity - the team had measured the bottlenecks and knew what they needed to upgrade and in what order.
10. Not all problems were fixed: the back-end communications to insurance companies still weren't working, but that was less visible so lower priority.

The overall payoff for these six weeks of work was astonishing; on Monday 23rd December the traffic surged in anticipation of a sign-up deadline:

"We'd been experiencing extraordinary traffic in December, but this was a whole new level of extraordinary ... By 9 o'clock traffic was the same as the peak traffic we'd seen in the middle of a busy December day. Then from 9 to 11, the traffic astoundingly doubled. If you looked at the graphs, it looked like a rocket ship." Traffic rose to 65,000 simultaneous users, then to 83,000, the day's high point. The result: 129,000 enrollments on Dec. 23, about five times as many in a single day as what the site had handled in all of October.

Despite this tremendous fix, however, President Obama didn't visit the team to thank them. Perhaps the political fallout from the Healthcare.gov farce was too painful for him.

The best quote that every single government on the planet should read:

[...] one lesson of the fall and rise of HealthCare.gov has to be that the practice of awarding high-tech, high-stakes contracts to companies whose primary skill seems to be getting those contracts rather than delivering on them has to change. "It was only when they were desperate that they turned to us," says Dickerson. "I have no history in government contracting and no future in it ... I don't wear a suit and tie ... They have no use for someone who looks and dresses like me. Maybe this will be a lesson for them. Maybe that will change."

The team who pulled President Obama's chestnuts out of the fire didn't even think they were going to be paid for their work initially; it looks like they did eventually get some money, but nowhere near even standard contracting rates. And yet, money wasn't the motivator for them - they deeply wanted to make Healthcare.gov work. As a result they did an extraordinary job and more or less saved the site from oblivion. This matches my experience from government IT developments: it's reasonable to assume that the government don't care about whether the project works at all, because if they did then they'd run it completely differently. Though if I were President I'd be firing Marilyn Tavenner, cashing in her retirement package and using it to pay bonuses to the team who'd saved my ass.

If you have a terribly important problem to solve, the most reliable way to solve it is to find competent people who will solve it for free because they want it to work. Of course, it's usually quite hard to find these people - and if you can't find them at all, maybe your problem shouldn't be solved in the first place.

Moobs for Windows

I loved this. From Orange County, CA Craigslist:

Busy and techy plastic surgery office in Newport Beach is looking for an IT person to service the office per need basis. The doctor specializes in gynecomastia surgery in addition to performing traditional cosmetic procedures. If this is something you are interested in having done, this would be a great way to trade service for service.

[Note that Craigslist expires ads, so this link won't work forever.]

If you're wondering "Huh?", you either need to know the definition of gynecomastia or you've never encountered an IT geek.

I'm now expecting a wave of part-time IT Craiglist ads with optometrists, dieticians, sartorialists, hairdressers and fitness instructors offering to trade professional services for improved IT.

The world is getting better, not worse

That's the conclusion of billionaire philanthropist Bill Gates. In his annual Gates Foundation letter he explains (and justifies) why three myths about world poverty are untrue - and why they block progress for the poor:

1. POOR COUNTRIES ARE DOOMED TO STAY POOR
2. FOREIGN AID IS A BIG WASTE
3. SAVING LIVES LEADS TO OVERPOPULATION

Now I consider Windows Vista a crime against humanity, but can't fault Mr. and Mrs. Gates for their commitment to making the world a better place with their own money. Bear in mind that Bill Gates doesn't seem to have any vested interest in these matters - he's richer than Croesus, he doesn't seem to care about political power, he doesn't have people to pay off; if he makes these claims, it's most likely to be because he believes them. And he's not dumb or politically naive by any stretch of the imagination. Bill Gates - and it pains me to say this after trying to use Word 6.0 - has more authority in this area than pretty much any government, business or NGO.

Just to give an example of his claims:

Income per person has in fact risen in sub-Saharan Africa over that time, and quite a bit in a few countries. After plummeting during the debt crisis of the 1980s, it has climbed by two thirds since 1998, to nearly $2,200 from just over $1,300. Today, more and more countries are turning toward strong sustained development, and more will follow. Seven of the 10 fastest-growing economies of the past half-decade are in Africa.
Africa has also made big strides in health and education. Since 1960, the life span for women in sub-Saharan Africa has gone up from 41 to 57 years, despite the HIV epidemic. Without HIV it would be 61 years. The percentage of children in school has gone from the low 40s to over 75 percent since 1970. Fewer people are hungry, and more people have good nutrition. If getting enough to eat, going to school, and living longer are measures of a good life, then life is definitely getting better there.

Sure, capitalism has its faults. But the implications of these figures are pretty hard to dispute. You can claim that if we had centrally-planned ethical socialism in the Western world, life in Africa would be better, but given the above data the onus is on you to show that your claims are true. Life in Africa is getting much better; if you're going to change the world, how will you prove that your changes won't reverse this progress?

Gaming the system - ambulance response times

It turns out that if emergency services try to chase response times then the public can get screwed over, in a very real and non-reversibly fatal sense:

Emergency services were called at 23:15 GMT and a paramedic arrived within 15 minutes. The paramedic contacted the control room three times asking for the ambulance to arrive sooner, but it did not arrive until 01:00 GMT the next day, Mr Nelson's family said.

Presumably this was a motorcycle paramedic, who will carry some fluids though probably not blood and certainly not more than a few pints of them. The unfortunate 26 year old Mr. Nelson is described as suffering from haemorrhaging, which was almost certainly internal and hence could not be successfully treated without surgery; all the paramedic could do was buy time pending transfer of Mr. Nelson to a hospital with an on-call surgery team, so that Mr. Nelson could a) receive whole blood in volume to replace his loss and b) be opened up so that the surgery team could clamp the offending major blood vessel to stop the loss. Unfortunately it seems that the required ambulance took another 90 minutes to arrive, which was way too late.

So why did the ambulance take so long? We can reasonably assume that the paramedic made a diagnosis of internal bleeding and called in for an urgent transport, so the available ambulances must have been elsewhere:

He added: "It seems that if they meet the target for the whole of the east of England, it satisfies the government target but the danger is they focus on urban areas where they can easily hit the target and rural areas get neglected.

Bingo! Why is this? Here's one possible explanation.

Suppose you have a reasonable-sized city (e.g. Reading, Oxford) surrounded by a fairly large rural area. Your ambulance, fire and police stations are somewhere in the city. At regular times you have a small number (say 2-4) of available ambulances, waiting to respond to calls. Most of your calls will come from within the city as not only do you have most of your people there but they are in an environment more likely to cause accidents (heavy traffic, concentrated drinking etc.) Anticipating this, you station most if not all of your ambulances around the city ring road and near major junctions so that they can either head straight in to the city, head straight out to the rural towns in their sector, or drive around the ring road to access a different sector. Your hospital will be within the city so your vehicles will go "green" (available) there; you can direct them to go straight to the next call or send them to one of your vacant ring-road sectors.

Blakeney, the home of Mr. Nelson, is 80 minutes from Great Yarmouth and 50 minutes from Kings Lynn (the nearest major towns). Without wanting to second-guess Norfolk ambulance control I'd imagine that they might have had an ambulance stationing point near Cromer or Swaffham, but someone else called first and that ambulance was taken; once they received the priority call from the paramedic, the ambulance would have nearly an hour of driving just to reach Blakeney. Because the incident happened on a Thursday night they probably had fewer ambulances available than on the busier Friday or Saturday nights, and because it happened around 11pm it was during the busiest period.

If the East of England Ambulance Trust wanted to reduce the incidence of long waits for ambulances in rural towns, it would have to position more ambulances way out from its major urban centres. The problem is that this would increase response times for the bulk of incidents during busy times when the remote-stationed ambulances were required near the cities. For the sake of significantly improving response times in relatively rare scenarios (multiple incidents away from the cities) you're going to be significantly impinging on your common-or-garden city incidents.

So what's the ambulance response time target?

Immediately life threatening – An emergency response will reach 75% of these calls within eight minutes. Where onward transport is required, 95% of life-threatening calls will receive an ambulance vehicle capable of transporting the patient safely within 19 minutes of the request for transport being made.

The NHS has at least addressed tail latency here ("95% within 19 minutes") but the problem is that this is a national target. It's much easier to meet in the densely-populated southeast than the more sparsely populated areas of the country. In the latter case, an ambulance trust's best bet is to concentrate resources around towns as discussed above, since they won't have a prayer of meeting "75% within 8 minutes" otherwise. It also allows wildly increasing times for 1/20th of the patients - if you can't get an ambulance to them in 20 minutes, there's no additional penalty for taking 90 minutes to reach them despite the fact you're identified these patients as needing onwards transport.

The dominant problem here is a national service (the NHS) requiring national targets for regional services, not making any allowance for the wildly different demographic distribution across the country. There's nothing conceptually wrong with the form of the target, but they need to vary the numbers as populations become less dense. You'd expect the tail latency requirement to remain fairly constant, but the initial response time to increase as population density decreases, and you should also add a 99% latency requirement (say, 30 minutes) to reduce the long waits for needy rural patients. Your response targets may no longer fit within a soundbite, but at least they are now aimed at saving lives across the country.

States vs territories and the unintended effects of Obamacare

For all those arguing that smart people in government can solve healthcare problems, a case for you to consider. As well as the 50 states that make up the USA, there are various territories which are overseen directly by the Federal Government but which are not themselves states; Guam, the Northern Marianas Islands, Puerto Rico and the US Virgin Islands ("organized" territories with a degree of self-rule) and American Samoa, Midway Islands and a bunch of small atolls and islands ("unorganized" territories). Federal government rulings apply to these territories in the same way that they do the states.

It turns out that the implications of recent Affordable Care Act were not entirely thought through with respect to these territories:

While the Affordable Care Act requires health insurers in the territories to accept all shoppers no matter how sick, it does not mandate that all territorial residents buy plans nor does it provide subsidies to make coverage more affordable--as it does in the 50 states and the District of Columbia.

The big win for poor people in the ACA was that they would receive subsidies to purchase the (rather expensive) health coverage that the ACA mandated they buy, and the big win for sick people was that they could not be refused insurance or be priced out of the market due to pre-existing conditions. The way the finances balanced was a mandate to purchase insurance under penalty of fines. But those subsidies aren't provided to residents of these territories, so ACA plans are extremely expensive; and the mandate does not apply in the territories. Result: most people aren't buying ACA plans because a) they are expensive and b) they don't have to. The only people buying ACA plans are the really sick people for whom even unsubsidized ACA plans are far better than their alternatives.

So the insurance companies in those territories are stuck having to accept really sick people without any ability to dilute the effect on their returns by including a large pool of healthy people:

The administration has offered technical assistance to alleviate the problem alongside potential policy work-arounds. One solution Health and Human Services has suggested is having the territories pass their own individual mandates, just as Massachusetts did back in 2006. But the regulators say that won't work either, because they don't have enough money to subsidize the purchase of insurance coverage for their citizens.

In other words: if Guam mandates purchase of insurance by Guam citizens, they'll have to pay full price for the ACA-compliant plans and they'll march on 155 Hesler Place with torches, pitchforks and lengths of rope.

It appears that no-one drafting the Affordable Care Act asked "hey, how does this affect the non-state territories?" As a result, they've made a horrific mess of healthcare in those areas. Oopsie. Next time someone proposes that the government step in to fix something, remember how badly they got this wrong.

Next step in health care - rationing by availability

Now that the enrolments of US citizens under the Affordable Care Act are finally rising (albeit slowly) it seems that the next challenge for participants once they can afford the payments will be finding a doctor who accepts their insurance:

Independent insurance brokers who work with both insurance companies and doctor networks estimate that about 70 percent of California's 104,000 licensed doctors are boycotting the exchange.
Mazer, a past president of the San Diego County Medical Society, agreed, saying, "I cannot find anybody in my specialty in the area that has signed a contract directly with any of these plans."

It seems as if the way that insurers on the California exchange managed to make premiums as (relatively) low as they were was by dropping reimbursement rates; consequently, a large number of doctors aren't going to be playing. They already have plenty of business with customers via employer plans that reimburse at acceptable rates, why should they drop their rates for other customers?

By the middle of next year the effect of the Affordable Care Act plans on regular customers should be clearer. It'll certainly be an improvement for people with preexisting conditions who couldn't get insurance, but it seems that an awful lot of people forced onto the exchanges will be paying more, for plans with higher deductibles, and yet will struggle to find a nearby doctor who will accept them...

Eventually if there's enough of a market I'd expect more doctors to come in and open up large treatment centres to make economies of scale and provide OK-if-not-great care at lower rates, but this rather depends whether the hassle of dealing with ACA regulations and insurers is going to make it worth their while...

(The next "logical" step if this turns out to be a problem is for the government to force doctors to accept ACA exchange insurance rates as a condition of practice...)

The difference between government and private industry

The US insurance industry has indicated 50,000 sign-ups for the Affordable Care Act insurance so far. This is less than 10% of what they were aiming for by this time. That's bad enough, but more instructive is how the US Government will officially count enrollments:

When the Obama administration releases health law enrollment figures later this week, though, it will use a more expansive definition. It will count people who have purchased a plan as well as those who have a plan sitting in their online shopping cart but have not yet paid.

Holy crap. Someone must have signed off on this definition, and I'd love to know how they kept a straight face doing it.

David Burge (aka IowaHawk) nails this:

If Amazon included uncompleted shopping baskets as "sales," Jeff Bezos would go to prison http://t.co/inFR4NeEK3

— David Burge (@iowahawkblog) November 12, 2013

The inevitable conclusion to be drawn: the HealthCare.gov enrollment figures are so dreadful that it's preferable to focus attention on a ridiculous definition of "enrollment" than on the number of actual paid-up enrollees.

Belated realization of what works

I've previously blogged about the contrast between the technically sophisticated Obama re-election campaign and the dog's breakfast that is Healthcare.gov. Go take a quick look to refresh your memories.

Now it turns out that at least one of the team being "tech-surged" to worked on the highly successful tech of Obama's re-election campaign:

One of two surge team members named by the agency was Michael Dickerson, which [sic, who taught CNN subs grammar?] CMS said was on leave from Google.
"He has expertise in diving into any layer of the tech stack ... in order to deliver some of the world's most reliable online services," CMS spokeswoman Julie Bataille said.

Dickerson is a site reliability engineer at Google and worked on some of the key performance-critical systems for the Obama team, as per his CV:

Designed and implemented, with Chris Jones and Yair Ghitza, the 2012 realtime election day monitoring and modeling (based on "Gordon" or vanpollwatcher.com).
Also: Wrote a tool for computing walkability of potential contacts, used by several states to prioritize GOTV contacts. Helped create the algorithm for targeting national TV cable ads to party preference and behavior, and wrote the tool that was used to do it. Prepared disaster recovery for all of OFA's mysql databases before Hurricane Sandy. Conducted various scalability and reliability assessments for many teams in OFA Tech and Analytics.

Finally the federal government is getting smart about how to fix the healthcare.gov problems - find people who a) have an interest in seeing this effort not fall on its arse, b) have the technical chops to know about the issues involved in a near-realtime distributed DB-backed system, and c) are willing and able to kick ass, then hand them a stick with a nail in the end and give them an open-ended mandate to pull the HHS chestnuts out of the fire.

Too late? Maybe. The government has committed to having things working by the end of the month. Without knowing specifics, and assuming a virtually unlimited budget, I think they are finally getting the right kind of people in to sort out their problems. The question is how many reputations and careers of the incumbent project managers and developers they are willing to sacrifice. I suspect at this point the answer is "all".

The curse of experts

Megan McArdle, who has been all over the HealthCare.gov and the Affordable Care Act rollout like a rash, has a superb piece at Bloomberg on the reason that the implications of the ACA came as a surprise to most people:

"We all knew" that preventive care doesn't save money, electronic medical records don’t save money, reducing uncompensated care saves very little money, and "reining in the abusive practices" of insurance companies was likely to raise premiums, [my italics] not lower them, because those "abuses" mostly consist of refusing to cover very sick people. But that information did not get communicated very well to the public.

This is, profoundly, what dooms any number of projects. For instance any software engineer or technical manager worth their salt will implicitly believe that a) testing a system with something like real traffic is the only way to detect and mitigate launch problems, and b) if you're only planning on testing one week before a hard deadline then You're Going To Have A Bad Time. Yet, if the project is being managed elsewhere and the project managers are not really asking the engineers about their opinions, just handing down features and deadlines, then the facts that "all the experts know" never get presented to the project manager in a way that makes them understand.

This reminded me of the testimony of CMS head Marilyn Tavenner about the awesome project fuck-up that was the HealthCare.gov launch and her part in it as the official directly responsible for its launch:

During the Tuesday hearing, Tavenner rejected the allegation that the CMS mishandled the health-care project, adding that the agency has successfully managed other big initiatives. She said the site and its components underwent continuous testing but erred in underestimating the crush of people who would try to get onto the site in its early days.
"In retrospect, we could have done more about load testing," she said.

You see what I mean? All the experts "know" that load testing a site that's going to be heavily used is not optional and not to be left to the last moment.

Reassuringly, Tavenner did demonstrate some skills in her area of competence: blame-shifting.

Under questioning, Tavenner pointed the finger at CGI Federal, saying the company sometimes missed deadlines. "We've had some issues with timing of delivery," she said.

I'm sure that's right. I'm equally sure that it's the project manager's job to anticipate, plan for and adjust schedules to handle late (or even early) deliveries - and CMS was the project manager. You'll note from her bio that Tavenner is a life-long health administrator - I'd bet her early career as a nurse lasted just long enough to get her into admin - and has as much business leading a complicated software development project as I do running an emergency room. Less, probably, because at least I know that air goes in and out, blood goes round and round, and any variation on this is a bad thing.

Ironically the Chief Technology Officer of Health and Human Services (HHS being the parent department of the CMS) whose bio indicates reasonable technical chops wasn't actually involved much in the project:

...an employee of Amazon Web Services Inc (AWS) emailed two HHS officials on October 7 saying, "I hear there are some challenges with Healthcare.gov. Is there anything we can do to help?"
HHS' Chief Technology Officer Bryan Sivak replied to Amazon by email on October 8: "I wish there was. I actually wish there was something I could do to help. [my emphasis]"

The Chief Information Officer by contrast is an ex-IBM marketeer and strategizer, and is putting his strategizing skills to good use making clear his distance from the smoking wreck of the project:

HHS' Chief Information Officer Frank Baitman replied to Amazon on October 7, "Thanks for the offer! Unfortunately, as you know, I haven't been involved with Healthcare.gov. I'm still trying to figure out how I can help, and may very well reach out for assistance should the opportunity present itself."

Nice one, Frank. Of course, Sivak is the one who comes across as actually human.

It looks like Tavenner's CMS wanted all the glory and kudos from the HealthCare.gov launch, but instead has become the focus the frustrations and hate of millions of Americans. The lessons here: be careful what you wish for, and if you want to know what the "experts know" then you really need to ask them.

Government tech vs Valley tech

The ongoing slow-motion disaster of the HealthCare.gov exchanges has provided vast amounts of entertainment for software engineers, and not a little of "if only they'd used this (product/process/language/company) they'd have been fine." There is much talk of a tech "surge" to get highly-skilled engineers who actually know what they're doing to help with fixing the site, but that runs into problems as Jessica Myers points out in Politico:

"The skill that is needed most for someone to come in is the knowledge of how the system works," said Eric Ries, a Silicon Valley startup founder and creator of the popular "lean startup" philosophy. "Even if you got Google up to speed on the crazy architecture that makes no sense, [...] it's like if you have a predigital clock and you want to hire a hotshot. You need someone who knows how an antique clock works."

It's well known maxim - indeed, known in the trade as Brook's Law that adding more manpower to a late project makes it later. HealthCare.gov is no exception. You'll spend ages getting your new guys up to speed on the system, architecture and tools in use - and that education process has to be conducted by the best people you already have, taking them away from their current troubleshooting. That's not to say that it's necessarily the wrong choice at this time, but it's certainly not going to bring the project in early.

One of the strategic problems faced by the developers was the very nature of government IT:

Government IT comprises a network of systems that have developed over the past half-century, said Mike Hettinger, the Software & Information Industry Association's director of public sector innovation. In some cases, thousands of homegrown networks feed into one payroll or financial system. Whereas a scrappy Silicon Valley startup could wipe out a project that doesn't work, a much larger government agency doesn't have that luxury.

This is not a problem peculiar to government IT - payroll systems in particular in private companies are notorious legacy systems that quickly become too complex and full of undocumented behavior to replace without large amounts of pain. However, in private industry there's usually a point at which the cost of supporting and working around the legacy system becomes annoying enough that people are willing to put up with the temporary pain of replacement. Sometimes all it takes is someone hired from outside to come in, set their sights on replacing the legacy system as their first big project in the firm, and it will happen - the original system developer has probably moved on to another firm by now, and so no-one cares about it. Maybe the new finance director is fed up of paying IBM squillions of dollars a year to keep the system running. Whatever, the presence of a legacy system is unstable - very few people have a vested interest in keeping it around.

Government IT, by contrast, can grow a whole ecosystem around this one legacy system, in charge of its care and feeding, providing manual work-arounds for activities the system doesn't support or automates poorly. A government departmental budget is there for spending, so a system that is awkward to use is actually more likely to get budget because the manager can demonstrate a need: "we are up to 50 man-days of work a month to issue invoices, and our two full-time accounting assistants can't cope." The empire grows, and more people have a vested interest - their jobs, in some cases - in maintaining the status quo. As such, government departments are a near-ideal environment for these systems to flourish, rather than withering in the metaphorical dog-poop corner of the departmental garden as they should. The only business environments which can provide a similar level of support are very large firms (IBM, Microsoft, big banks etc.) where a growing budget and headcount is a mark of success to be funded, not failure to be squashed.

The reason that Silicon Valley startups and successful established businesses wipe out projects that don't work well, as opposed to keeping them around to work around their idiosyncratic ways, is because they realise that sooner or later they will be forced to wipe them out anyway - eventually the system will grind to a halt, or everyone who knows how to fix it will have left, or the hardware it depends on will fail with no supplier remaining to provide the necessary parts, or a new regulation will be passed forcing the system to behave in a new way which it cannot possibly do, or the client traffic will grow past the system's performance limit... you get the picture. If you have a mad dog in your garden, you don't wait until it's bitten one of the children - it's a mad dog, everyone knows it's mad and that bitten children are inevitable, which is why you pull your Mossberg 535 from the gun cabinet and let the dog have it.

Back to how this whole mess got started:

"At the end of the day, Washington and how we procure technology for the federal government is just different," Hettinger said.

Yes, it certainly is. One wonders why anyone would think this "different" to be synonymous with "better", when "insane" seems a better fit. Unless, of course, producing a working system is a very secondary consideration to the people in procurement and the Washington-friendly contractors (IBM, Oracle and friends).

HHS doesn't understand the problem so won't produce a solution

I apologise for turning this into the HealthCare.Gov train-wreck site, but it's such a material-rich environment that I can't help myself.

Today the US government Health and Human Services department issued a statement on what they are doing to fix HealthCare.Gov:

To ensure that we make swift progress, and that the consumer experience continues to improve, our team has called in additional help to solve some of the more complex technical issues we are encountering.
Our team is bringing in some of the best and brightest from both inside and outside government to scrub in with the team and help improve HealthCare.gov.

Interesting. I wonder in particular who from inside government is going to lend their expertise to this disaster-in-motion of software mis-engineering?

We are also defining new test processes to prevent new issues from cropping up as we improve the overall service and deploying fixes to the site during off-peak hours on a regular basis.

I really hope that this is a PR writer mis-understanding what she was told. You can't generally prevent new issues from cropping up from your code changes, because you don't know what those issues might be. You can however make a good stab at preventing old issues by setting up regression tests, running cases based on past errors to verify that the errors do not re-occur. Perhaps that's not forward-looking enough for HHS, but the sad fact is that crystal balls have very limited utility in software engineering. You're far better to improve your existing monitoring and logging so that at least you can identify and characterise errors that are occurring now.

I liked Republican Senator John McCain's suggestion for how to fix things:

"Send Air Force One out to Silicon Valley, load it up with some smart people, bring them back to Washington, and fix this problem. It's ridiculous. And everybody knows that."

The irony is that this is more or less what the Obama campaign did for the 2012 election campaign and it worked spectacularly well. If they'd done something similar for HealthCare.Gov, recruiting interested and motivated tech people from Silicon Valley (notoriously Democrat-heavy) to design and oversee the healthcare exchange, then quite possibly it would not have gone horrendously wrong. The problem now is that they are stuck with their existing design and implementation, and any redesign would necessarily trash most of their existing code and tests and require months of work to produce anything.

I'm reminded of the tourist in Ireland who asks a local how to get to Kilkenny, and the local responds "Ah well, if I wanted to get to get to Kilkenny, I wouldn't start from here."