How to build and launch a federal health care exchange

Since the US government has made a pig's ear, dog's breakfast and sundry other animal preparations of its health care exchange HealthCare.Gov, I thought I'd exercise some 20/20 hindsight and explain how it should (or at least could) have been done in a way that would not cost hundreds of millions of dollars and would not lead to egg all over the face of Very Important People. I don't feel guilty exercising hindsight, since the architects of this appalling mess didn't seem to worry about exercising any foresight.

A brief summary of the problem first. You want to provide a web-based solution to allow American citizens to comparison-shop health insurance plans. You are working with a number of insurers who will provide you with a small set of plans they offer and the rules to determine what premium and deductible they will sell the plan at depending on purchaser stats (age, family status, residential area etc.) You'll provide a daily or maybe even hourly feed to insurers with the data on the purchasers who have agreed to sign up for their plans. You're not quite sure how many states will use you as their health care exchange rather than building your own, but it sounds like it could be many tens of states including the big ones (California, Texas). We expect site use to have definite peaks over the year, usually in October/November/early December as people sign up in preparation for the new insurance year on Jan 1st. You want it to be accessible to anyone with a web browser that is not completely Stone Age, so specify IE7 or better and don't rely on any JavaScript that doesn't work in IE7, Firefox, Safari, Chrome and Opera. You don't work too hard to support mobile browsers for now, but Safari for iPad and iPhone 4 onwards should be checked.

Now we crunch the numbers. We expect to be offering this to tens of millions of Americans eventually, maybe up to 100M people in this incarnation. We also know that there is very keen interest in this system, and so many other people could be browsing the site or comparison-shopping with their existing insurance plans even if they don't intend to buy. Let's say that we could expect a total of 50M individual people visiting the site in its first full week of operation. The average number of hits per individual: let's say, 20. We assume 12 hours of usage per day given that it spans America (and ignore Hawaii). 1bn hits per week divided by 302400 seconds yields an average hit rate of about 3300 hits per second. You can expect peaks of twice that, and spikes of maybe five times that during e.g. news broadcasts about the system. So you have to handle a peak of 15000 hits per second. That's quite a lot, so let's think about managing it.

The first thing I think here is "I don't want to be worrying about hardware scaling issues that other people have already solved." I'm already thinking about running most of this, at least the user-facing portion, on hosted services like Amazon's EC2 or Google's App Engine. Maybe even Microsoft's Azure, if you particularly enjoy pain. All three of these behemoths have a staggering numbers of computers. You pay for the computers you use; they let you keep requesting capacity and they keep giving it to you. This is ideal for our model of very variable query rates. If we need about one CPU and 1GB of RAM to handle three queries per second of traffic, you'll want to provision about 5000 CPUs (say, 2500 machines) during your first week to handle the spikes, but maybe no more than 500 CPUs during much of the rest of the year.

The next thought I have is "comparison shopping is hard and expensive, let's restrict it to users whom we know are eligible". I'd make account creation very simple; sign up with your name, address and email address plus a simple password. Once you've signed up, your account is put in a "pending" state. We then mail you a letter a) confirming the sign-up but masking out some of your email address and b) providing you with a numeric code. You make your account active and able to see plans by logging in and entering your numeric code. If you forget your password in the interim, we send you a recovery link. This is all well-trodden practice. The upshot is that we know - at least, at a reasonable level of assurance - that every user with an active account is a) within our covered area and b) is not just a casual browser.

As a result, we can design the main frontend to be very light-weight - simple, cacheable images and JavaScript, user-friendly. This reduces the load on our servers and hence makes it cheaper to serve. We can then establish a second part of the site to handle logged-in users and do the hard comparison work. This site will check for a logged-in cookie on any new request, and immediately bounce users missing cookies to a login page. Successful login will create a cookie with nonce, user ID and login time signed by our site's private key with (say) a 12 hour expiry. We make missing-cookie users as cheap as possible to redirect. Invalid (forged or expired) cookies can be handled as required, since they occur at much lower rates.

There's not much you can do about the business rules evaluation to determine plan costs: it's going to be expensive in computation. I'd personally be instrumenting the heck out of this code to spot any quick wins in reducing computation effort. But we've already filtered out the looky-loos to improve the "quality" (likelihood of actually wanting to buy insurance) of users looking at the plans, which helps. Checking the feeds to insurers is also important; put your best testing, integration and QA people on this, since you're dealing with a bunch of foreign systems that will not work as you expect and you need to be seriously defensive.

Now we think about launch. We realise that our website and backends are going to have bugs, and the most likely place for these bugs is in the rules evaluation and feeds to insurers. As such, we want to detect and nail these bugs before they cause widespread problems. What I'd do is, at least 1 month in advance of our planned country-wide launch, launch this site for one of the smaller states - say, Wyoming or Vermont which have populations around 500K - and announce that we will apply a one-off credit of $100 per individual or $200 per family to users from this state purchasing insurance. Ballpark guess: these credits will cost around $10M which is incredibly cheap for a live test. We provision the crap out of our system and wait for the flood of applications, expect things to break, and measure our actual load and resources consumed. We are careful about user account creation - we warn users to expect their account creation letters within 10 days, and deliberately stagger sending them so we have a gradual trickle of users onto the site. We have a natural limit of users on the site due to our address validation. Obviously, we find bugs - we fix them as best we can, and ensure we have a solid suite of regression testing that will catch the bugs if they re-occur in future. The rule is "demonstrate, make a test that fails, fix, ensure the test passes."

Once we're happy that we've found all the bugs we can, we open it to another, larger, state and repeat, though this time not offering the credit. We onboard more and more states, each time waiting for the initial surge of users to subside before opening to the next one. The current state-by-state invitation list is prominent on the home page of our site. Our rule of thumb is that we never invite more users than we already have (as a proportion of state population), so we can do no more than approximately double our traffic each time.

This is not a "big bang" launch approach. This is because I don't want to create a large crater with the launch.

For the benefit of anyone trying to do something like this, feel free to redistribute and share, even for commercial use.

This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.

Update: also very worth reading Luke Chung's take on this application, which comes from a slightly different perspective but comes up with many similar conclusions on the design, and also makes the excellent usability point:

The primary mistake the designers of the system made was assuming that people would visit the web site, step through the process, see their subsidy, review the options, and select "buy" a policy. That is NOT how the buying process works. It's not the way people use Amazon.com, a bank mortgage site, or other insurance pricing sites for life, auto or homeowner policies. People want to know their options and prices before making a purchase decision, often want to discuss it with others, and take days to be comfortable making a decision. Especially when the deadline is months away. What's the rush?

Project management: harder than one might think

One of the most startling revelations in the continuing slow-motion carnage of the US federal health exchanges is that the government's Center for Medicare and Medicaid Services (CMS) decided to manage the whole affair themselves:

The people I spoke with did all confirm the importance of one other detail in the Times story: that CMS did not hire a general contractor to manage the exchange project but handled that overall technical management task itself. None of the people I spoke with wanted to get into how this decision was made or at what level, but all of them agreed that it was a very bad idea and was at the core of the disaster they have so far experienced.

This is, I believe, the inevitable result of government agencies (UK and USA specifically, but I'm sure other countries are equally guilty) of hiring "generalists", who tend to have liberal arts degrees. Because the subject of these degrees (English, Geography, History, PPE etc.) is unlikely to be directly useful in their owner's regular government work, the story told is that the general communication, analysis and critical thinking skills absorbed are what makes that graduate more valuable in the workplace than (say) someone with pre-university qualifications.

This analysis more or less works for government work which involves reporting and planning, and even for some low-level management. Unfortunately, it fails comprehensively when hard technical issues come up. I still remember the expression on the face of a 25 year old Civil Service fast stream grad (Oxford, PPE) as my grizzled engineering boss tried to explain to her the main engineering issues of the project she was allegedly managing. Picture a dog being taught Greek and you won't be far off. She was so far out of her depth that James Cameron could have been exploring below her. To be fair, you'd get a similar effect by putting an engineering grad in charge of a biochemistry research project, or a chemistry grad in charge of a micro-lending organisation - but at least they'd both be numerate enough to spot errors in the finances.

I note that anyone who proposed that the UK Border Agency head honchos oversee and project-manage the construction of a major bridge or power system would rightly be excoriated in public. "How the hell could they even know where to start? What do they know about compressive strength / high voltage transmission?" Why, then, do we assume that IT projects are any easier for non-experts to manage? I suspect the answer lies in a combination of the infinite malleability of software, and the superficial familiarity that most people have with using web interfaces (and even tweaking HTML themselves). After all, it's just words and funny characters, how hard could it be?

Allow me to link to my favourite XKCD cartoon ever:

Back to the exchanges: there's about as much reason to believe that the CMS has expertise in project management as there is to believe that I'm capable of designing a line of clothes to rival the products of Chanel and DvF. The fact that I can draw something that might be recognisable as a dress (if you squint a little) has absolutely no relevance to being able to design something that millions of people would want to wear - and that can be made for a reasonable sum of money while being resilient to the huge range of stresses and strains imposed on clothing by its wearers. What appalls me is that, given the quote above, no-one stopped the CMS from taking on the project management role despite the fact that everyone seemed to know that it was a terrible idea. Either this was a dastardly covert Tea Party guerilla plot to sabotage the exchanges, or there was a serious break-down in communication. Health and Human Services secretary Kathleen Sebelius is ultimately on the hook for the failure of the health exchanges; did she just not care that they were doomed to fail, or was there someone in the upper chain of reporting who knew what happens to the bearer of bad news and hence decided that discretion was preferable to being unceremoniously fired?

Sebelius, incidentally, is the first daughter of a governor to be elected governor in American history. She has a liberal arts BA and a master's in Public Administration. The CMS Chief Operating Officer is Michelle Snyder who holds advanced degrees in Clinical Psychology and Legal Studies and Administration. She has been a manager in the HHS budget office and had assignments with the Office of Management and Budget, Congress, the Social Security Administration, and as a management consultant in the private sector.

I'm sure liberal arts majors and management consultants have an important role to play in modern society. That role does not, apparently, include being in charge of a major IT project. Not only are they incompetent to run it, it seems that they are incompetent to appoint someone competent to run it. Personally, I'd have started with Richard Granger,, ex-head of the UK NHS Connecting for Health program that pissed £10-15 billion down the drain for no result. Yes, his track record is beyond absymal - on the other hand, a) he now knows first-hand all the mistakes you shouldn't make and b) when you announce his appointment the expectations on your project will plunge so low that even delivering a badly-working underperforming system will impress people.

Drop dead dates

I had the educational privilege, a few years ago, to watch a team in my workplace try to roll out a new business system to replace an existing system which had worked well for a while but grown gnarled, unmaintainable and no longer scaled to likely future demands. Well aware of the Second System Effect they made the new system feature-for-feature compatible, and even had a good stab at bug-for-bug. However, it was a complex problem and they spent many months spinning up a prototype system.

Eventually their manager decided that they needed to run something in production, so they picked a slice of the traffic on the existing business system that was representative but not critical, and set a target deadline of a week hence to launch it. The developers were privately rather twitchy about the prospect, but recognised the pressure that their manager was under and were willing to give it a shot. Come switchover day the new system was enabled - and promptly fell on its face. The developers found the underlying bugs, fixed them and restarted. It ran a little longer this time, but within a few hours fell over again. They fixed that problem, but within 12 hours it became clear that performance was steadily degrading with time...

The developers had a miserable time during the subsequent week. I got in pretty early as a rule, but the dev team was always in (and slurping coffee) by the time I arrived, and never left before I got home. The bugs posted in their area steadily accumulated, the system repeatedly fell down and was restarted with fixes. The team were living on the ragged edge, trying to keep the system up at the same time as triaging the bugs, adding tests and monitoring to detect the bugs, and trying to measure and improve the performance. This was analogous to changing the wheels on Sebastian Vettel's F1 car mid-lap - one hiccup and either you lose a limb or the car embeds itself in a track barrier. It became clear that the team's testing system had huge gaps, and their monitoring system couldn't generally detect failures happening - you could more or less infer what had caused the failure by checking the logs, but someone had to mail the team saying "hey, this job didn't work" for the team to look at the logs in question.

After a fortnight of this, with the team having pulled an average of 80-90 hour weeks, their manager sensibly realised that this approach was not sustainable. He announced the switch back from the new system to the old system effective next day, and immediately shaped expectations by announcing that they would not be switching back to the new system before three months had passed. The team breathed a sigh of relief, took a few days off, and re-scheduled themselves.

Once the system was pulled offline, the developers made reasonably rapid progress. They'd accumulated a host of bug reports, both in functionality and performance, and (more importantly) had identified crucial gaps in testing and monitoring. For each functional and performance bug, they first verified that they could reproduce it in their testing system - which was where they spent the bulk of their development time for several weeks after turndown - and that the monitoring would detect the condition and alert them appropriately. They triaged the bug reports, worked their way through them in priority order, built load tests that replicated the system load from normal operation and added metrics and monitoring on system latency. The time spent running in production had provided a wealth of logs and load information which gave them a yardstick against which they could measure performance.

After a few months they felt ready to try again, so they spun up the fixed system and loaded in the current data. This went much more smoothly. There were still occasional crashes, but their monitoring alerted them almost instantly so they could stop the system, spend time precisely characterising the problem, fix it, test the fix, deploy the fix and restart. The average time between crashes got longer and longer, the impact of failures got smaller and smaller, and after 6 months or so the system achieved its stated goal of greater scale and performance than its predecessor. However, all this was only possible because of the decision to roll back its initial roll-out.

I was reminded of this today when I saw that informed insiders were estimating the US federal healthcare exchanges as "only 70% complete" and needing "2 weeks to 2 months more work" to be ready. Since there are several tens of millions of potential users who need to register before January 1st, this looks to be a precarious situation. It's doubly precarious when you realise that "70% complete" in a software project is code for "I have no idea when we're going to be done." My personal rule of thumb is that "90% complete" means that you take the number of weeks spent in development so far, and expect the same again until the system is working with the specified reliability.

Megan McArdle, whose coverage of the health care exchanges has been consistently superb, makes a compelling case that Obamacare needs to set a deadline date for a working system, and delay the whole project a year if it's not met:

...given that they didn't even announce that they were taking the system down for more fixes this weekend, I'm also guessing that it's pretty bad. Bad enough that it's time to start talking about a drop-dead date: At what point do we admit that the system just isn't working well enough, roll it back and delay the whole thing for a year?

She's right. If the system is this screwed up at this point, with an unmoveable deadline of January 1st to enroll a large number of people, any sane project manager would move heaven and earth to defer the rollout. In the next 6-9 months they could address all the problems that the first roll-out has revealed, taking the time to test both functionality and performance against the traffic levels that they now know. There's no practical compulsion to run the exchanges now - the American healthcare system has been screwed up for several decades, the population is used to it, waiting another year won't make a great difference to most voters.

Chance of this happening? Essentially zero. The Democrats have nailed their colours to the mast of the good ship Affordable Care Act, and it's going out this year if it kills them. If they hold it over until next year then the full pain of the ACA's premium hikes will hit just before the mid-term elections, and they will get pummelled. They're hoping that if they launch now then the populace will be acclimatised to the costs by next November. As such, launching this year is a politically non-negotiable constraint. Politics, hard deadlines and under-performing software - a better recipe for Schadenfreude I can't imagine.

American habits the UK should adopt - jailing politicians

And I'm not talking about a few months in chokey for falsifying tens of thousands of pounds of expenses, or eight months for perjury. Ex-mayor of Detroit Kwame Kilpatrick is looking at twenty eight years in the slammer for running a criminal enterprise through the mayor's office:

Kilpatrick used his power as mayor … to steer an astounding amount of business to Ferguson. There was a pattern of threats and pressure from the pair.
This wasn't to protect minority contracts. In fact, they ran some of them out of work.
He was larger than life. He lived the high life. He hosted lavish parties. He accepted cash tributes. He loaded the city payroll with family and friends.
He had an affair with his chief of staff, lied about it, and went to jail for perjury.

Note: he's already done time for perjury. The criminal enterprise sentence is on top of this...

I'd personally add a year to the sentence for membership of Mayors Against Illegal Guns which is rampant posturing if I've ever seen it... Still, if we had decade-long jail sentences for criminal financial malfeasance in a public office, I wonder if it would put a brake on trough-wallowing politicos? Or do they inevitably believe "it can't possibly happen to me"?

Caveat emptor

The Chinese are sternly warning the Americans not to default on their debt:

Mr Zhu said that China and the US are "inseparable". Beijing is a huge investor in US Treasury bonds.
"The executive branch of the US government has to take decisive and credible steps to avoid a default on its Treasury bonds," he said.

Google found me the major foreign holders of US debt as of July 2013:

1. China: $1.3 trillion
2. Japan: $1.2 trillion
3. Caribbean banking centers: $300 billion
4. Oil exporters: $260 billion
5. Brazil: $260 billion

I'm reminded of the maxim: "Borrow $1000 and the bank owns you; borrow $1 million and you own the bank." China's GDP is about $8 trillion, so US debt that it owns is about 12% of GDP. Japan's GDP is about $6 trillion so US debt that it owns is 20% of its GDP. Is China seriously concerned that the US might default on its debt? If Japan is similarly concerned, it seems to be keeping very quiet.

I expect that the problem arises from the Chinese banks relentlessly trying to get out of yuan before the Chinese economic bubble starts to pop. There are huge flows of money out of China to buy dollar-denominated assets; million-dollar houses all over Silicon Valley are being bought up for cash by Chinese buyers. As a data point, friends of mine who just put a $800K townhouse on the market in the South Bay were almost immediately given a cash offer by a Chinese couple wanting to buy a house for their daughter to live in when she goes to college in late 2014. If the US were to even threaten default, the dollar would drop significantly in value - in the past three months alone, the pound has risen from $1.50 to $1.60 due to the concern about the US political situation. If Chinese banks have leveraged investments in dollar-denominated assets, the shockwaves from even a technical US default could land them in very hot water.

Glenn Greenwald - weasel

Watching Glenn Greenwald being interviewed on BBC Newsnight by Kirsty Wark it struck me that he's remarkably blasé about US and UK secrets leaking out to foreign intelligence services. Up to now I've given him the benefit of the doubt that he thought he was doing the right thing, but this interview made it painfully clear what an arrogant little weasel Greenwald actually is.

Wark did a pretty good job pressing him on his motivations and the implications of the leaked data, not to mention the safety of the remaining encrypted data. Greenwald asserted that he and the Guardian had protected the data with "extremely advanced methods of encryption" and he is completely sure that the data is secure. Well, that's fortunate. No danger of anyone having surreptitiously planted a keylogger in either software or hardware on the relevant Guardian computers? No danger of one of the Guardian journalists with access having been compromised by a domestic or foreign security service? Greenwald seems remarkably sure about things he can't practically know about. Perhaps he just doesn't give a crap.

Wark was curious (as am I) about Greenwald's recent contacts with Snowden and Snowden's current welfare. Greenwald claimed that Edward Snowden has protected the data has with "extreme levels of encryption", proof against cracking by the NSA and the "lesser Russian intelligence agencies". Russia being a country where math prodigies are ten a penny, I fear Greenwald may be underestimating their cryptography-fu. Asserting that Snowden didn't spend his life fighting surveillance just to go to Russia and help them surveil, Greenwald stated that the evidence we know makes it "ludicrous" to believe that the Russians or Chinese had access to Snowdon's data.

Hmm. Glenn, I suggest you Google rubber hose cryptanalysis. If I were the Russian FSB, given that they have effectively complete access to and control over Snowden, I'd be extremely tempted to "lean" on him until he gave up the keys that decrypted his stash of data. Heck, why wouldn't they? They'd be practically negligent not to do so. Nor are they likely to shout from the rooftops if they have done so; they're far more likely to exploit the data quietly and effectively while conveniently being able to blame Greenwald and co. for any leaks.

I invite you to contrast this with Greenwald's note that the UK Government "very thuggishly ran roughshod over press freedoms, running criminal investigations and detaining my partner." Detaining David Miranda for nine hours was not necessarily a good plan by the UK, but he was a foreign national and was not a journalist as far as I (and the Guardian) am aware. So Greenwald's reference to press freedom is a little disingenous. As far as "running roughshod" goes, Greenwald can only pray that he doesn't end up in the hands of the FSB... as Guardian journalist Luke Harding could tell him:

Luke Harding, the Moscow correspondent for The Guardian from to 2007 to 2011 and a fierce critic of Russia, alleges that the FSB subjected him to continual psychological harassment, with the aim of either coercing him into practicing self-censorship in his reporting, or to leave the country entirely. He says that FSB used techniques known as Zersetzung (literally "corrosion" or "undermining") which were perfected by the East German Stasi.

The Russian affairs expert Streetwise Professor has been following the Snowden saga with a critical eye for a while now, believing that he's being made to dance to Putin's tune. Most recently he noted that we have no recent statements known to come from Snowdon; even his most recent statement to the UN was read out on his behalf, there's no proof that the statement came from Snowdon himself and indeed the text suggests Greenwald and other Snowden "colleagues" had a hand in his text. If the Russians are treating Snowden well, why isn't he a regular appearance on TV or YouTube?

It must be nice to be as arrogantly cocksure as Greenwald. I bet Snowden for one would be happy to change places with him right now.

Some brief questions for the architects of healthcare.gov

There are various theories bouncing around about why the HealthCare.gov website is having such problems with capacity:

One possible cause of the problems is that hitting "apply" on HealthCare.gov causes 92 separate files, plug-ins and other mammoth swarms of data to stream between the user's computer and the servers powering the government website, said Matthew Hancock, an independent expert in website design. He was able to track the files being requested through a feature in the Firefox browser.

That's not very promising. You'd somewhat hope that one of the pivot actions of a website was as lightweight as possible from the client side; there are going to be enough problems with the server persisting everything into its database. The page from which you apply should have as much of the JavaScript already loaded (and appropriately marked as cacheable so it's not re-loaded when the page changes). Loading everything in the time-critical period when you're trying to lock a table (and I bet they're not row-level locking) in the database to write the user application is asking for trouble.

Anyway, that's just speculation. But here are the questions for the HealthCare.gov folks:

1. How many concurrent users did you determine you could support on the system with the resources at launch?
2. What were the system bottlenecks that imposed that limit?
3. What were your contingency plans to add resources to temporarily raise system capacity as required at launch?
4. What was your estimate of visitor traffic to the site on launch week, based on the anticipated heavy country-wide publicity?
5. What performance degradation did you anticipate when 2x and 10x the maximum users attempted to use the site?
6. What plan did you have to divert users by geo-located IP and / or run the system in a degraded mode in the event of overload, to focus the available performance on people who might actually want to buy healthcare?
7. If you hadn't done the performance and load testing to determine the above numbers and evolve the above plans before launch, would you care to explain why not and what you thought you were being paid to do?

In short; did you determine in advance when and how the system was going to fall over with the traffic you were going to get and how to get it back on its feet, and if you didn't then why not?

If the Senate is looking for questions to ask the HealthCare.gov architects when the inevitable reprisals come around, they could do worse than start with these.

Sneak peak: Facebook's Q3 results

I'm a moderately active and engaged Facebook user (to my eternal shame), so in the run-up to the Facebook Q3 results later this month I thought I'd see what ads Facebook are offering me.

On desktop: zip. None. Nada. Niente. Wala. Nichts. Plenty of recommended pages, but nothing that looks like it would give anyone even a sniff of revenue. It took several reloads before I triggered a single sponsored link: "Online savings account" from GE Capital Bank. Not even remotely relevant. Note that now and again I click particularly irrelevant ads to hide them with appropriate feedback (irrelevant / duplicate / against my views etc) but this month FB seems to have run out of anything to show me.

On mobile: I scrolled down several days in my stream and there was nothing. No ads at all. So, this vaunted Facebook mobile advertising...

Facebook is currently priced at $51.07 a share, up from sub-$30 3 months ago. What. The Fuck? I wait with keen interest to see what they show this quarter in mobile revenue. Either they're making a heck of a lot from selling social graph information to clients for advertising elsewhere, or they're about to get their pants pulled down and rogered with the traditional curare-tipped iron fencepost.

Dancing around the Great Firewall of China

It seems a little unfair to give Apple heat over its China policies, given how much employment it creates in China, but apparently Apple have censored a Chinese firewall avoiding-app:

Chinese web users have criticised Apple after the company pulled an iPhone app which enabled users to bypass firewalls and access restricted internet sites. The developers of the free app, OpenDoor, reportedly wrote to Apple protesting against the move. [...] Apple asks iPhone app developers to ensure that their apps "comply with all legal requirements in any location where they are made available to users".

Aha. But the problem here is: China does not acknowledge the existence of the Great Firewall of China (GFW). In fact, any mention of it in a blog post or other social media is enough to get that posting censored. China certainly has strong legal requirements about being able to identify the real person behind an Internet identity on a China-hosted service and foreign firms having to "partner" with a local firm for Internet "compliance", and it freely blocks traffic going outside China (via the GFW) which could retrieve user-generated content relating to sensitive topics, but from a legal perspective the GFW itself cannot be the subject of a legal violation since the GFW does not officially exist because you can't talk about it (and the GFW will censor your traffic if you try to do this across the border). Is your head hurting yet?

This, by the way, is perfectly pragmatic behaviour from Apple. They like being able to do business in China, so it's not enough to satisfy the letter of the law - they want to keep the Chinese government happy. As such, dropping GFW-circumventing apps from the App Store makes perfect business sense. It is, however, particularly weasel-like for them to hide behind "legal requirements", or avoid the topic all together. If they want to play ball with the Chinese government for commercial reasons - and it's their fiduciary duty to improve their commercial prospects - why can't they just say so? (Yes, this is a rhetorical question.)

The OpenDoor app developers purport to be bemused:

"It is unclear to us how a simple browser app could include illegal contents, since it's the user's own choosing of what websites to view," the email says.
"Using the same definition, wouldn't all browser apps, including Apple's own Safari and Google's Chrome, include illegal contents?"

Yes they could, in theory. But browsers use well-known protocols: HTTP, which is clear text, and which the GFW can scan for illegal content like "T1ANANM3N []"; HTTPS, which is secure but can be blocked either based on destination IP or just universally. OpenDoor probably (I haven't looked) does something sneaky to make its traffic look like regular HTTP with innocuous content. The GFW could, with some work, drop OpenDoor traffic based on its characteristics and/or destinations, but they would always be playing catch-up. Instead, Apple "voluntarily" (we don't know if any Chinese government pressure was formally applied) drops it from the App Store in China. Everyone's happy! No-one gets any distressing news about human rights abuses in China, and gatherings of subversives are prevented.

Apple are bending over to help the Chinese government, and that's perfectly acceptable in a capitalist society - let's just be clear that it's voluntary, and in search of profits.

Try again later for healthcare

The much-anticipated switch-on of the Affordable Care Act ("Obamacare") health exchanges happened today, and as anticipated, it was not exactly a smooth ride:

And those going to the federal exchange site, which is handling enrollment for 36 states that didn't fully establish their own exchanges are greeted with this message: "Health Insurance Marketplace: Please wait. We have a lot of visitors on our site right now and we're working to make your experience here better. Please wait here until we send you to the login page. Thanks for your patience!"

California also had quite the surge of users:

The agency that runs the [California State] exchange, Covered California, said it received 1 million hits on the website during the first 90 minutes after the exchange opened. By 3 p.m., the site had received 5 million hits and the two service centers had received 17,000 phone calls.

There are 5400 seconds in 90 minutes, so that's about 200 hits per second. A respectable number, to be sure; many of those will be people just looking at the (static, small, hand-written, easily cached) front page, which is cheap to serve. It would be more interesting to see a rate of user sign-ups and quote serving, with the fraction of errors served in each case, but I guess that's too much to hope for. (For those interested, Covered California is using Microsoft IIS/7.5 and ASP for at least its front page.) I was amused to see the range of languages covered: Spanish, Tagalog, Vietnamese and Chinese as you'd expect, but no Indian languages I recognised. Arabic, Farsi, Russian and Armenian but not French, Portuguese or German. Korean, Hmong and Khmer but not Thai. It seems fairly clear which immigrants learn the language and which don't...

The "please try again later" screen is actually entirely the right approach. The current traffic spike is a short-lived phenomenon - the widespread publicity has led a tidal wave of real users to the exchange websites in a short time window, along with plenty of other "browsers" who are interested in seeing what kind of plan quotes are available. The right thing to do is to tell users to try again later. Real exchange users aren't going to go elsewhere for their plans, and they're not really time-limited - they will try again later in the evening, or tomorrow, or next weekend. The browsers will be put off and either not come back at all (preferred) or come back much later (manageable). In the meantime the exchange operators will have discovered the bottlenecks in their system, and can make tactical fixes for quick wins and start planning for the more involved capacity fixes that may take weeks or months to implement.

Based on the above I'd say that the ACA exchange rollout has gone roughly as well as could be expected. As the user load flattens out over the next few days and weeks the exchanges should work better, and downtime will be less frequent and shorter.

Information Week has a nice dissection from Oregon of why operating a health exchange is hard:

Presenting a product -- an insurance policy -- isn't the hard part. The hard part is figuring out which federal and state programs and tax credits a person or family is eligible for. Getting that part right takes creating an extremely complex rules engine.
About 1,700 individual rules affect eligibility for health insurance subsidies in Oregon.

For each user you have to gather their circumstances - family size, age, income, location - which is fairly straight forward. You then have to write this into a distributed data base, handling de-duplication where required. Then you hand off the data to the rules matching engine - probably the computational workhorse and main memory and disk hog of the system - and wait for it to come back with an offer. There will be separate links to the insurance providers which will receive feeds of what they are prepared to bid for customers, and publish notifications of what bids they made, won and lost.

The real technical challenge for the exchanges will come in the next few months, and it won't be the performance/load issue which has been making headlines today. There will be the inevitable security breaches, usually by insiders trying to sell on personal data. There will be discrepancies between the insurance bid record held by the exchanges and that recorded at the insurers' backends - and hot disputes about which is correct. There will be IT failures on both sides leading to users getting too-high or stupidly-low offers for insurance, and insurers being pressured to honour too-good-to-be-true offers which they (or the exchange on their behalf) made and a wave of consumers snapped up.

(Update: those security breaches seem to be coming earlier than I thought:

One exchange staffer’s simple mistake gave insurance broker Jim Koester access to an Excel document of Social Security numbers, names, addresses and other personal data for whole a list of insurance agents.

Oopsie.)

The policy problems for the exchanges and insurers will start to become more visible next year. Insurers aren't allowed to discriminate on the basis of pre-existing conditions; as such, the first wave of insurance buyers will be heavily seeded with sick or disabled people with no current insurance. The insurers will be hoping to pull in as many healthy people as possible (due to the "individual mandate" making people buy ACA-compliant insurance or face a fine. However, that fine is relatively small for the first year (the greater of $95 per adult and $47.50 per child, or 1 percent of family income), giving healthy adults a good reason to avoid the exchanges for at least a year. The fine will gradually rise, and it will be interesting to see the fine level required to make most people get onto the exchange. In the interim, the insurers are going to be stuck with a wave of very expensive people requiring treatment. There is a natural throttle though - the number of doctors and hospitals willing to accept the reimbursement rates of those insurers is going to go down steadily, so the insurers will have some limit on treatment rates. But then there are going to be widespread complaints that people with insurance from the exchanges can't get timely treatment - a complaint that the insurers can only relieve by raising their reimbursement rates, which isn't going to happen.

Hmm, healthcare rationed by medical provider availability and payer-authorised treatments. This is starting to sound awfully familiar... Think of the NHS but without its effective monopoly on health care purchases to keep down prices.

The $1bn question is whether the insurers are willing to stay on the exchange and absorb the costs of the treatments of their disproportionately sick initial customers, in anticipation of future revenue from young, healthy adults compelled to buy overpriced (for them) and needlessly comprehensive "insurance" - in effect, an inefficient payment plan for all medical costs. This is going to hinge on what they think the federal government is going to give them in subsidies (tax breaks for poorer payers) and how much purchasing leverage they have on the existing health system. Since bureaucracy is one of the biggest problems in the American health care system, and Washington politics is notoriously dysfunctional and lobby-driven, I don't expect this to work out too well.

I should add that there will be beneficiaries of this system - if you are poor and sick, but have enough money to cover your year's deductible, then it's a potentially life-saving bonanza. I fear, though, that help for these people - who undoubtedly need help - is going to come at a high price for everyone else.