When perfection is less desirable than excellence

An interesting view into the trade-offs of large-scale computing from LISA 2013:

[Google's] engineers aim to make its products as reliable as possible, but that's not their sole task. If a product is too reliable — which is to say, beyond the five 9s of reliability (99.999 percent) — then that service is "wasting money" in the company’s eyes.
"The point is not to achieve 100 percent availability. The point is to achieve the target availability — 99.999 percent—while moving as fast as you can. If you massively exceed that threshold you are wasting money," Underwood said.

It's interesting that "five nines" seems to be viewed as the desirable limit of reliability. Recall that this means 10 minutes of downtime per year; it seems reasonable that it's unlikely for anyone to notice this level of downtime unless it's a 24x7x365 service with hundreds of millions of global users (Gmail, Facebook etc.). If we assume 50M daily users distributed evenly across the planet, and an average of 5 minutes of daily engagement (times when they'd notice the service failing to respond) then that's about 150,000 users who would notice and maybe 1% who would publicly complain (via twitter etc) so 1500 tweets - that's around the margins of a detectable level of complaint. Certainly from recorded Gmail outages it seems to be about right. If you have 10% of this number of daily users, you could have a four-nines reliability for the same level of complaint.

The really interesting (and no doubt intentionally controversial) comment was on the end of the age of the BOFH:

Underwood, who has a flair for the dramatic, stated: "I think system administration is over, and I think we should stop doing it. It's mostly a bad idea that was necessary for a long time but I think it has become a crutch."

It's not yet obvious that small companies are going to shed BOFHs in order to outsource their system maintenance to "the cloud" no matter how apparently economically appealing this is; I suspect that having a person physically on-site that you can shout at when things go wrong is going to be sufficiently psychologically helpful that BOFHs (or at least PFYs) will be with the SME for a while yet. There's also the practical matter of selecting the correct combination of storage, network bandwidth and peak vs average processing power for the business - you have to hire someone who knows how to make this choice, and you can't easily fire them once they've made this choice for you. Perhaps cloud computing can let CTOs scale back their IT departments, but I'd be surprised if they can be completely eliminated.

The difference between government and private industry

The US insurance industry has indicated 50,000 sign-ups for the Affordable Care Act insurance so far. This is less than 10% of what they were aiming for by this time. That's bad enough, but more instructive is how the US Government will officially count enrollments:

When the Obama administration releases health law enrollment figures later this week, though, it will use a more expansive definition. It will count people who have purchased a plan as well as those who have a plan sitting in their online shopping cart but have not yet paid.

Holy crap. Someone must have signed off on this definition, and I'd love to know how they kept a straight face doing it.

David Burge (aka IowaHawk) nails this:

If Amazon included uncompleted shopping baskets as "sales," Jeff Bezos would go to prison http://t.co/inFR4NeEK3

— David Burge (@iowahawkblog) November 12, 2013

The inevitable conclusion to be drawn: the HealthCare.gov enrollment figures are so dreadful that it's preferable to focus attention on a ridiculous definition of "enrollment" than on the number of actual paid-up enrollees.

Blowing up the minimum wage in a confined space

An excellent little experiment for economists and minimum-wage advocates is about to kick off in Seattle:

Workers are optmistic that the SeaTac "Good Jobs Initiative" will pass after jumping out to an early lead in the election. And with the latest ballot count on Wednesday night, with 3,942 votes counted, that optimism reigns with a tally of 53% to 47% supporting the initiative.
The initiative seeks to raise the minimum wage to $15 an hour for workers in Seattle-Tacoma International Airport and at airport-related businesses. [my italics]

$15/hour? Bloody hell. Even in ultra-expensive San Francisco they're only going up to $10.74 an hour. So what's going to happen here? If we look within the confines of SeaTac, it seems plain that travellers are captive users - they don't have any ability to change to a better-value business - so we'd expect a certain fraction of travellers with non-discretionary spending (stuck in airport during transit, business-funded) to grit their teeth and pay the higher prices inevitable as a result of wage increases of up 50% (and service / rental charges to businesses rising as a result of cleaning, security and catering staff wage hikes). Other travellers such as gift shoppers will be more reluctant to pay the higher prices at the margins, leading to lower sales overall and especially in staff-intensive low-margin businesses. If I were the manager of the SeaTac McDonald's, I'd definitely be trialling automated ordering points. Overall I'd expect SeaTac revenue to be approximately neutral, so if they're going to be spending more per worker then they'll likely have to be employing fewer workers.

The real fun is going to come in the definition of "airport-related businesses", of course. The Washington worker unions will be pushing to stretch this definition as far as possible, so that any business which supplies anything of note to the airport will be subject to the law. As a result, non-aviation businesses will decline airport custom as fast as they can. That reduces supply to the airport, so pushes up prices. As noted above, an airport has a very limited captive customer base. Personally I don't buy anything more than coffee and a Brad Thor novel in an airport unless it's on someone else's dime.

Now I've travelled through SeaTac a fair bit and had the (dubious) benefits of service at several of its establishments. With the notable exception of the high-priced but excellent Vino Volo, I was generally made to feel as if I was intruding on the personal time of the transport, security and retail staff there. The TSA was particularly slack-jawed, idle and incompetent - and that's a pretty low bar to crawl under. I'm not sure how raising the minimum wage is going to help this situation; it's possible that the attractions of a $15/hr job would make the worker work harder to keep the job, but beyond that why try harder for an undoubtedly negligible salary increase?

Of course, this is a slippery slope:

Organizers expect their message to spread beyond SeaTac workers. This summer in Seattle, fast food workers also rallied to raise the minimum wage to $15 an hour and both Mayor Mike McGinn and Ed Murray supported the idea, and we’re told the city council may take up the issue as soon as this week.

"Hey, my neighbor gets $15/hr mandated, why shouldn't I?" Expect a rapid rise in wage demands across Washington state, at which point it becomes crystal-clear a) which areas have politicians funded by labour, b) which by business and c) that customers will travel from a) to b) if at all feasible when they want to buy something labour-intensive.

Check back in a year and see what SeaTac looks like. If I go through there towards the end of 2014, I'll report back.

Sebelius on websites

Vast amusement from Tennessee Senator Kelsey presenting HealthCare.gov and HHS boss Kathleen Sebelius with "Websites for Dummies". Ooh, burn!

Of course, he should actually have given her Schlossnagle's "Scalable Internet Architecture":

As a developer, you are aware of the increasing concern amongst developers and site architects that websites be able to handle the vast number of visitors that flood the Internet on a daily basis. Scalable Internet Architectures addresses these concerns by teaching you both good and bad design methodologies for building new sites and how to scale existing websites to robust, high-availability websites. Primarily example-based, the book discusses major topics in web architectural design, presenting existing solutions and how they work. Technology budget tight? This book will work for you, too, as it introduces new and innovative concepts to solving traditionally expensive problems without a large technology budget.

For a mere $34, this could have saved Sebelius from an awful lot of career-ending heartache. The technology it describes is dated by a few years, but compared to the fetid mess that government IT has produced it's state of the art.

Politicians on crack

Entertainingly rampant speculation today about claims that Toronto Mayor Rob Ford is on video smoking from a crack pipe:

Allegations that the mayor of Canada's largest city had been caught on video smoking crack cocaine surfaced in May. Two reporters with the Toronto Star and one from the U.S. website Gawker said they saw the video but did not obtain a copy. Police Chief Bill Blair told a news conference Thursday he was "disappointed" in Ford but said the video did not provide grounds to press charges against him.

Note in passing that Toronto is not Canada's capital city - that's Ottawa. Toronto is however by far the most populous Canadian city, with nearly 7M people in the Greater Toronto Area and 2.5M people in Toronto itself, nearly twice that of nearest rival Montreal. Mr. Ford is the most visible mayor in Canada, so this is why the story is getting so much airtime. As to the veracity of the story I have no idea, but I suspect Chief Blair is correct - even if it clearly shows someone who is indisputably Rob Ford smoking from something very much like a crack pipe, how can you show the smoke originated from an illegal substance?

I do wonder, though, whether this chasing of Ford is a door that other politicians really want to open. There are two arguments you could make against a politician taking illegal drugs; one is that they should obey the law as to do otherwise is to set a bad example to their citizens. The other is that imbibing such substances could impair their judgement and effectiveness in their roles. However, if this is really a concern, we should not wait for videos of our politicians toking or smoking to arise; instead, we should be conducting random drug tests. This is what we do for people conducting safety-related work (train operators, railway maintenance workers etc.) Why should politicians be immune? Turn up at City Hall with drug and alcohol testing kits, randomly select politicos and publish a list of detections - allowing the politicians a second test if they fail first time around. Of course, since impairment is a concern, we should cover alcohol in the tests - no more politico boozing before important votes.

I used to hang around with people who subsequently went into politics, and they were some of the most blatant weed consumers I've known. Most of them ended up on the red side of the benches. If politicians are to be kicked out of politics based on drug consumption, we'll have many fewer politicians. We should probably also start to look at those who report on politics, since they determine how political actions appear to the rest of us. Is this really what the people wailing about Rob Ford want to happen?