When perfection is less desirable than excellence

An interesting view into the trade-offs of large-scale computing from LISA 2013:

[Google's] engineers aim to make its products as reliable as possible, but that's not their sole task. If a product is too reliable — which is to say, beyond the five 9s of reliability (99.999 percent) — then that service is "wasting money" in the company’s eyes.
"The point is not to achieve 100 percent availability. The point is to achieve the target availability — 99.999 percent—while moving as fast as you can. If you massively exceed that threshold you are wasting money," Underwood said.

It's interesting that "five nines" seems to be viewed as the desirable limit of reliability. Recall that this means 10 minutes of downtime per year; it seems reasonable that it's unlikely for anyone to notice this level of downtime unless it's a 24x7x365 service with hundreds of millions of global users (Gmail, Facebook etc.). If we assume 50M daily users distributed evenly across the planet, and an average of 5 minutes of daily engagement (times when they'd notice the service failing to respond) then that's about 150,000 users who would notice and maybe 1% who would publicly complain (via twitter etc) so 1500 tweets - that's around the margins of a detectable level of complaint. Certainly from recorded Gmail outages it seems to be about right. If you have 10% of this number of daily users, you could have a four-nines reliability for the same level of complaint.

The really interesting (and no doubt intentionally controversial) comment was on the end of the age of the BOFH:

Underwood, who has a flair for the dramatic, stated: "I think system administration is over, and I think we should stop doing it. It's mostly a bad idea that was necessary for a long time but I think it has become a crutch."

It's not yet obvious that small companies are going to shed BOFHs in order to outsource their system maintenance to "the cloud" no matter how apparently economically appealing this is; I suspect that having a person physically on-site that you can shout at when things go wrong is going to be sufficiently psychologically helpful that BOFHs (or at least PFYs) will be with the SME for a while yet. There's also the practical matter of selecting the correct combination of storage, network bandwidth and peak vs average processing power for the business - you have to hire someone who knows how to make this choice, and you can't easily fire them once they've made this choice for you. Perhaps cloud computing can let CTOs scale back their IT departments, but I'd be surprised if they can be completely eliminated.

The difference between government and private industry

The US insurance industry has indicated 50,000 sign-ups for the Affordable Care Act insurance so far. This is less than 10% of what they were aiming for by this time. That's bad enough, but more instructive is how the US Government will officially count enrollments:

When the Obama administration releases health law enrollment figures later this week, though, it will use a more expansive definition. It will count people who have purchased a plan as well as those who have a plan sitting in their online shopping cart but have not yet paid.

Holy crap. Someone must have signed off on this definition, and I'd love to know how they kept a straight face doing it.

David Burge (aka IowaHawk) nails this:

If Amazon included uncompleted shopping baskets as "sales," Jeff Bezos would go to prison http://t.co/inFR4NeEK3

— David Burge (@iowahawkblog) November 12, 2013

The inevitable conclusion to be drawn: the HealthCare.gov enrollment figures are so dreadful that it's preferable to focus attention on a ridiculous definition of "enrollment" than on the number of actual paid-up enrollees.

Blowing up the minimum wage in a confined space

An excellent little experiment for economists and minimum-wage advocates is about to kick off in Seattle:

Workers are optmistic that the SeaTac "Good Jobs Initiative" will pass after jumping out to an early lead in the election. And with the latest ballot count on Wednesday night, with 3,942 votes counted, that optimism reigns with a tally of 53% to 47% supporting the initiative.
The initiative seeks to raise the minimum wage to $15 an hour for workers in Seattle-Tacoma International Airport and at airport-related businesses. [my italics]

$15/hour? Bloody hell. Even in ultra-expensive San Francisco they're only going up to $10.74 an hour. So what's going to happen here? If we look within the confines of SeaTac, it seems plain that travellers are captive users - they don't have any ability to change to a better-value business - so we'd expect a certain fraction of travellers with non-discretionary spending (stuck in airport during transit, business-funded) to grit their teeth and pay the higher prices inevitable as a result of wage increases of up 50% (and service / rental charges to businesses rising as a result of cleaning, security and catering staff wage hikes). Other travellers such as gift shoppers will be more reluctant to pay the higher prices at the margins, leading to lower sales overall and especially in staff-intensive low-margin businesses. If I were the manager of the SeaTac McDonald's, I'd definitely be trialling automated ordering points. Overall I'd expect SeaTac revenue to be approximately neutral, so if they're going to be spending more per worker then they'll likely have to be employing fewer workers.

The real fun is going to come in the definition of "airport-related businesses", of course. The Washington worker unions will be pushing to stretch this definition as far as possible, so that any business which supplies anything of note to the airport will be subject to the law. As a result, non-aviation businesses will decline airport custom as fast as they can. That reduces supply to the airport, so pushes up prices. As noted above, an airport has a very limited captive customer base. Personally I don't buy anything more than coffee and a Brad Thor novel in an airport unless it's on someone else's dime.

Now I've travelled through SeaTac a fair bit and had the (dubious) benefits of service at several of its establishments. With the notable exception of the high-priced but excellent Vino Volo, I was generally made to feel as if I was intruding on the personal time of the transport, security and retail staff there. The TSA was particularly slack-jawed, idle and incompetent - and that's a pretty low bar to crawl under. I'm not sure how raising the minimum wage is going to help this situation; it's possible that the attractions of a $15/hr job would make the worker work harder to keep the job, but beyond that why try harder for an undoubtedly negligible salary increase?

Of course, this is a slippery slope:

Organizers expect their message to spread beyond SeaTac workers. This summer in Seattle, fast food workers also rallied to raise the minimum wage to $15 an hour and both Mayor Mike McGinn and Ed Murray supported the idea, and we’re told the city council may take up the issue as soon as this week.

"Hey, my neighbor gets $15/hr mandated, why shouldn't I?" Expect a rapid rise in wage demands across Washington state, at which point it becomes crystal-clear a) which areas have politicians funded by labour, b) which by business and c) that customers will travel from a) to b) if at all feasible when they want to buy something labour-intensive.

Check back in a year and see what SeaTac looks like. If I go through there towards the end of 2014, I'll report back.

Sebelius on websites

Vast amusement from Tennessee Senator Kelsey presenting HealthCare.gov and HHS boss Kathleen Sebelius with "Websites for Dummies". Ooh, burn!

Of course, he should actually have given her Schlossnagle's "Scalable Internet Architecture":

As a developer, you are aware of the increasing concern amongst developers and site architects that websites be able to handle the vast number of visitors that flood the Internet on a daily basis. Scalable Internet Architectures addresses these concerns by teaching you both good and bad design methodologies for building new sites and how to scale existing websites to robust, high-availability websites. Primarily example-based, the book discusses major topics in web architectural design, presenting existing solutions and how they work. Technology budget tight? This book will work for you, too, as it introduces new and innovative concepts to solving traditionally expensive problems without a large technology budget.

For a mere $34, this could have saved Sebelius from an awful lot of career-ending heartache. The technology it describes is dated by a few years, but compared to the fetid mess that government IT has produced it's state of the art.

Politicians on crack

Entertainingly rampant speculation today about claims that Toronto Mayor Rob Ford is on video smoking from a crack pipe:

Allegations that the mayor of Canada's largest city had been caught on video smoking crack cocaine surfaced in May. Two reporters with the Toronto Star and one from the U.S. website Gawker said they saw the video but did not obtain a copy. Police Chief Bill Blair told a news conference Thursday he was "disappointed" in Ford but said the video did not provide grounds to press charges against him.

Note in passing that Toronto is not Canada's capital city - that's Ottawa. Toronto is however by far the most populous Canadian city, with nearly 7M people in the Greater Toronto Area and 2.5M people in Toronto itself, nearly twice that of nearest rival Montreal. Mr. Ford is the most visible mayor in Canada, so this is why the story is getting so much airtime. As to the veracity of the story I have no idea, but I suspect Chief Blair is correct - even if it clearly shows someone who is indisputably Rob Ford smoking from something very much like a crack pipe, how can you show the smoke originated from an illegal substance?

I do wonder, though, whether this chasing of Ford is a door that other politicians really want to open. There are two arguments you could make against a politician taking illegal drugs; one is that they should obey the law as to do otherwise is to set a bad example to their citizens. The other is that imbibing such substances could impair their judgement and effectiveness in their roles. However, if this is really a concern, we should not wait for videos of our politicians toking or smoking to arise; instead, we should be conducting random drug tests. This is what we do for people conducting safety-related work (train operators, railway maintenance workers etc.) Why should politicians be immune? Turn up at City Hall with drug and alcohol testing kits, randomly select politicos and publish a list of detections - allowing the politicians a second test if they fail first time around. Of course, since impairment is a concern, we should cover alcohol in the tests - no more politico boozing before important votes.

I used to hang around with people who subsequently went into politics, and they were some of the most blatant weed consumers I've known. Most of them ended up on the red side of the benches. If politicians are to be kicked out of politics based on drug consumption, we'll have many fewer politicians. We should probably also start to look at those who report on politics, since they determine how political actions appear to the rest of us. Is this really what the people wailing about Rob Ford want to happen?

Belated realization of what works

I've previously blogged about the contrast between the technically sophisticated Obama re-election campaign and the dog's breakfast that is Healthcare.gov. Go take a quick look to refresh your memories.

Now it turns out that at least one of the team being "tech-surged" to worked on the highly successful tech of Obama's re-election campaign:

One of two surge team members named by the agency was Michael Dickerson, which [sic, who taught CNN subs grammar?] CMS said was on leave from Google.
"He has expertise in diving into any layer of the tech stack ... in order to deliver some of the world's most reliable online services," CMS spokeswoman Julie Bataille said.

Dickerson is a site reliability engineer at Google and worked on some of the key performance-critical systems for the Obama team, as per his CV:

Designed and implemented, with Chris Jones and Yair Ghitza, the 2012 realtime election day monitoring and modeling (based on "Gordon" or vanpollwatcher.com).
Also: Wrote a tool for computing walkability of potential contacts, used by several states to prioritize GOTV contacts. Helped create the algorithm for targeting national TV cable ads to party preference and behavior, and wrote the tool that was used to do it. Prepared disaster recovery for all of OFA's mysql databases before Hurricane Sandy. Conducted various scalability and reliability assessments for many teams in OFA Tech and Analytics.

Finally the federal government is getting smart about how to fix the healthcare.gov problems - find people who a) have an interest in seeing this effort not fall on its arse, b) have the technical chops to know about the issues involved in a near-realtime distributed DB-backed system, and c) are willing and able to kick ass, then hand them a stick with a nail in the end and give them an open-ended mandate to pull the HHS chestnuts out of the fire.

Too late? Maybe. The government has committed to having things working by the end of the month. Without knowing specifics, and assuming a virtually unlimited budget, I think they are finally getting the right kind of people in to sort out their problems. The question is how many reputations and careers of the incumbent project managers and developers they are willing to sacrifice. I suspect at this point the answer is "all".

The curse of experts

Megan McArdle, who has been all over the HealthCare.gov and the Affordable Care Act rollout like a rash, has a superb piece at Bloomberg on the reason that the implications of the ACA came as a surprise to most people:

"We all knew" that preventive care doesn't save money, electronic medical records don’t save money, reducing uncompensated care saves very little money, and "reining in the abusive practices" of insurance companies was likely to raise premiums, [my italics] not lower them, because those "abuses" mostly consist of refusing to cover very sick people. But that information did not get communicated very well to the public.

This is, profoundly, what dooms any number of projects. For instance any software engineer or technical manager worth their salt will implicitly believe that a) testing a system with something like real traffic is the only way to detect and mitigate launch problems, and b) if you're only planning on testing one week before a hard deadline then You're Going To Have A Bad Time. Yet, if the project is being managed elsewhere and the project managers are not really asking the engineers about their opinions, just handing down features and deadlines, then the facts that "all the experts know" never get presented to the project manager in a way that makes them understand.

This reminded me of the testimony of CMS head Marilyn Tavenner about the awesome project fuck-up that was the HealthCare.gov launch and her part in it as the official directly responsible for its launch:

During the Tuesday hearing, Tavenner rejected the allegation that the CMS mishandled the health-care project, adding that the agency has successfully managed other big initiatives. She said the site and its components underwent continuous testing but erred in underestimating the crush of people who would try to get onto the site in its early days.
"In retrospect, we could have done more about load testing," she said.

You see what I mean? All the experts "know" that load testing a site that's going to be heavily used is not optional and not to be left to the last moment.

Reassuringly, Tavenner did demonstrate some skills in her area of competence: blame-shifting.

Under questioning, Tavenner pointed the finger at CGI Federal, saying the company sometimes missed deadlines. "We've had some issues with timing of delivery," she said.

I'm sure that's right. I'm equally sure that it's the project manager's job to anticipate, plan for and adjust schedules to handle late (or even early) deliveries - and CMS was the project manager. You'll note from her bio that Tavenner is a life-long health administrator - I'd bet her early career as a nurse lasted just long enough to get her into admin - and has as much business leading a complicated software development project as I do running an emergency room. Less, probably, because at least I know that air goes in and out, blood goes round and round, and any variation on this is a bad thing.

Ironically the Chief Technology Officer of Health and Human Services (HHS being the parent department of the CMS) whose bio indicates reasonable technical chops wasn't actually involved much in the project:

...an employee of Amazon Web Services Inc (AWS) emailed two HHS officials on October 7 saying, "I hear there are some challenges with Healthcare.gov. Is there anything we can do to help?"
HHS' Chief Technology Officer Bryan Sivak replied to Amazon by email on October 8: "I wish there was. I actually wish there was something I could do to help. [my emphasis]"

The Chief Information Officer by contrast is an ex-IBM marketeer and strategizer, and is putting his strategizing skills to good use making clear his distance from the smoking wreck of the project:

HHS' Chief Information Officer Frank Baitman replied to Amazon on October 7, "Thanks for the offer! Unfortunately, as you know, I haven't been involved with Healthcare.gov. I'm still trying to figure out how I can help, and may very well reach out for assistance should the opportunity present itself."

Nice one, Frank. Of course, Sivak is the one who comes across as actually human.

It looks like Tavenner's CMS wanted all the glory and kudos from the HealthCare.gov launch, but instead has become the focus the frustrations and hate of millions of Americans. The lessons here: be careful what you wish for, and if you want to know what the "experts know" then you really need to ask them.

For some needs, the government comes through

There's a lot of anger in America currently about the general incompetence of the federal government, but it's encouraging to see that at least one government agency is actually good at what it's paid to do:

The National Security Agency has secretly broken into the main communications links that connect Yahoo and Google data centers around the world, according to documents obtained from former NSA contractor Edward Snowden and interviews with knowledgeable officials.

Privacy concerns aside, you've got to admire the NSA for actually conducting some good modern communications interception. Someone probably deserves a substantial bonus; he won't get it, of course, because it's a government payroll - he'll no doubt defect to the private sector eventually, or maybe the SVR will make him the proverbial un-refusable offer.

It would be fascinating to know whether the NSA is just tapping links external to the USA (presumably including links with no more than one node in the USA) or have general access to intra-USA traffic. It's also interesting to speculate on the connection between this eavesdropping and Google's move back in September to encrypt the traffic that the NSA seems to have been intercepting. Yahoo still seems to be open, based on a rather inadequate denial from their PR:

At Yahoo, a spokeswoman said: "We have strict controls in place to protect the security of our data centers, and we have not given access to our data centers to the NSA or to any other government agency."

and one has to wonder about Facebook, Apple, Amazon etc.

So congratulations, citizens of the USA - you have a productive and competent government agency! Perhaps you should have put the NSA in charge of healthcare...

Reliability through the expectation of failure

A nice presentation by Pat Helland from Salesforce (and before that Amazon Web Services) on how they built a very reliable service: they build it out of second-rate hardware:

"The ideal design approach is 'web scale and I want to build it out of shit'."
Salesforce's Keystone system takes data from Oracle and then layers it on top of a set of cheap infrastructure running on commodity servers

Inituitively this may seem crazy. If you want (and are willing to pay for) high reliability, don't you want the most reliable hardware possible?

If you want a somewhat-reliable service then sure, this may make sense at some price and reliability points. You certainly don't want hard drives which fail every 30 days or memory that laces your data with parity errors like pomegranate seeds in a salad. The problems come when you start to get to demand more reliability - say, four nines (99.99% uptime, about 50 minutes downtime per month) and scaling to support tens if not hundreds of concurrent users across the globe. Your system may consist of several different components, from your user-facing web server via a business rules system to a globally-replicating database. When one of your hard drives locks up, or the PC it's on catches fire, you need to be several steps ahead:

1. you already know that hard drives are prone to failure, so you're monitoring read/write error rates and speeds and as soon as they cross below an acceptable level you stop using that PC;
2. because you can lose a hard drive at any time, you're writing the same data on two or three hard drives in different PCs at once;
3. because the first time you know a drive is dead may be when you are reading from it, your client software knows to back off and look for data on an alternate drive if it can't access the local one;
4. because your PCs are in a data centre, and data centres are vulnerable to power outages / network cables break / cooling failures / regular maintenance, you have two or three data centres and an easy way to route traffic away from the one that's down;

You get the picture. Trust No One, and certainly No Hardware. At every stage of your request flow, expect the worst.

This extends to software too, by the way. Suppose you have a business rules service that lots of different clients use. You don't have any reason to trust the clients, so make sure you are resilient:

1. rate-limit connections from each client or location so that if you get an unexpected volume of requests from one direction then you start rejecting the new ones, protecting all your other clients;
2. load-test your service so that you know the maximum number of concurrent clients it can support, and reject new connections from anywhere once you're over that limit;
3. evaluate how long a client connection should take at maximum, and time out and close clients going over that limit to prevent them clogging up your system;
4. for all the limits you set, have an automated alert that fires at (say) 80% of the limit so you know you're getting into hot water, and have single monitoring page that shows you all the key stats plotted against your known maximums;
5. make it easy to push a change that rejects all traffic matching certain characteristics (client, location, type of query) to stop something like a Query of Death from killing all your backends.

Isolate, contain, be resilient, recover quickly. Expect the unexpected, and have a plan to deal with it that is practically automatic.

Helland wants us to build our software to fail:

...because if you design it in a monolithic, interlinked manner, then a simple hardware brownout can ripple through the entire system and take you offline.
"If everything in the system can break it's more robust if it does break. If you run around and nobody knows what happens when it breaks then you don't have a robust system," he says.

He's spot on, and it's a lesson that the implementors of certain large-scale IT systems recently delivered to the world would do well to learn.

Government tech vs Valley tech

The ongoing slow-motion disaster of the HealthCare.gov exchanges has provided vast amounts of entertainment for software engineers, and not a little of "if only they'd used this (product/process/language/company) they'd have been fine." There is much talk of a tech "surge" to get highly-skilled engineers who actually know what they're doing to help with fixing the site, but that runs into problems as Jessica Myers points out in Politico:

"The skill that is needed most for someone to come in is the knowledge of how the system works," said Eric Ries, a Silicon Valley startup founder and creator of the popular "lean startup" philosophy. "Even if you got Google up to speed on the crazy architecture that makes no sense, [...] it's like if you have a predigital clock and you want to hire a hotshot. You need someone who knows how an antique clock works."

It's well known maxim - indeed, known in the trade as Brook's Law that adding more manpower to a late project makes it later. HealthCare.gov is no exception. You'll spend ages getting your new guys up to speed on the system, architecture and tools in use - and that education process has to be conducted by the best people you already have, taking them away from their current troubleshooting. That's not to say that it's necessarily the wrong choice at this time, but it's certainly not going to bring the project in early.

One of the strategic problems faced by the developers was the very nature of government IT:

Government IT comprises a network of systems that have developed over the past half-century, said Mike Hettinger, the Software & Information Industry Association's director of public sector innovation. In some cases, thousands of homegrown networks feed into one payroll or financial system. Whereas a scrappy Silicon Valley startup could wipe out a project that doesn't work, a much larger government agency doesn't have that luxury.

This is not a problem peculiar to government IT - payroll systems in particular in private companies are notorious legacy systems that quickly become too complex and full of undocumented behavior to replace without large amounts of pain. However, in private industry there's usually a point at which the cost of supporting and working around the legacy system becomes annoying enough that people are willing to put up with the temporary pain of replacement. Sometimes all it takes is someone hired from outside to come in, set their sights on replacing the legacy system as their first big project in the firm, and it will happen - the original system developer has probably moved on to another firm by now, and so no-one cares about it. Maybe the new finance director is fed up of paying IBM squillions of dollars a year to keep the system running. Whatever, the presence of a legacy system is unstable - very few people have a vested interest in keeping it around.

Government IT, by contrast, can grow a whole ecosystem around this one legacy system, in charge of its care and feeding, providing manual work-arounds for activities the system doesn't support or automates poorly. A government departmental budget is there for spending, so a system that is awkward to use is actually more likely to get budget because the manager can demonstrate a need: "we are up to 50 man-days of work a month to issue invoices, and our two full-time accounting assistants can't cope." The empire grows, and more people have a vested interest - their jobs, in some cases - in maintaining the status quo. As such, government departments are a near-ideal environment for these systems to flourish, rather than withering in the metaphorical dog-poop corner of the departmental garden as they should. The only business environments which can provide a similar level of support are very large firms (IBM, Microsoft, big banks etc.) where a growing budget and headcount is a mark of success to be funded, not failure to be squashed.

The reason that Silicon Valley startups and successful established businesses wipe out projects that don't work well, as opposed to keeping them around to work around their idiosyncratic ways, is because they realise that sooner or later they will be forced to wipe them out anyway - eventually the system will grind to a halt, or everyone who knows how to fix it will have left, or the hardware it depends on will fail with no supplier remaining to provide the necessary parts, or a new regulation will be passed forcing the system to behave in a new way which it cannot possibly do, or the client traffic will grow past the system's performance limit... you get the picture. If you have a mad dog in your garden, you don't wait until it's bitten one of the children - it's a mad dog, everyone knows it's mad and that bitten children are inevitable, which is why you pull your Mossberg 535 from the gun cabinet and let the dog have it.

Back to how this whole mess got started:

"At the end of the day, Washington and how we procure technology for the federal government is just different," Hettinger said.

Yes, it certainly is. One wonders why anyone would think this "different" to be synonymous with "better", when "insane" seems a better fit. Unless, of course, producing a working system is a very secondary consideration to the people in procurement and the Washington-friendly contractors (IBM, Oracle and friends).