When protecting minorities screws them over

Fascinating. I came across this Dave Winer story of tech hiring and firing in 1985's Silicon Valley via the money quote:

...every time a company hires someone who is not a young male, they run the risk that the new hire isn't there to work, rather is there to scam you.

since from that quote I wondered "what the hell planet is this author on?"

Then I read the story. And blow me down if I didn't end up at least partially agreeing with the author. Go read the whole thing.

Commenter sep332 clarifies that the problem described (an older tech worker using his age to file a discrimination claim after being fired) isn't actually about age, sex or anything in particular:

The laws about protected classes are not about classes of people. Anyone can claim that they were discriminated against for gender reasons, not just women. White people can claim they were targeted because of their race, etc. So the people who couldn't realistically claim discrimination are the people who are most like the rest of the company. [my emphasis] I mean, if women are the majority of your company, then it would be hard for a woman to claim gender discrimination.

In the tech sector, the majority of your company are likely to be young, male, and (in Silicon Valley) a mix of white, Chinese and Indian in race; probably also straight although I have known a couple of small firms where that was not the case. If you're middle-aged, female, transgender, black, Inuit, Pacific Islander or Hispanic then you're almost certainly in a clear minority and hence possibly a "protected class".

As Winer notes, in a small struggling tech firm if someone comes at you with a discrimination lawsuit then you haven't the money or time to fight it. Unless it's a complete no-brainer (an 18 year old white male alleging discrimination on the grounds of inability to get out of bed) your best option is to pay up and move on. So what do you do when you have the risk of recruiting people who can sit back and do nothing while being practically un-fireable? Simply minimise the risk of recruiting them, by avoiding anyone who is in a good position to do this to you.

An ironic "well done" to everyone who has pushed through these employment laws, and a special raspberry to everyone who has filed (or backed) an abusive lawsuit exploiting these laws. You've screwed over everyone in the tech sector who's not a young male.

Storing in the cloud

So this is interesting. Google is dropping its cloud storage rates to $10 per month per TB (though 100GB costs $2, twice that rate). Amazon S3 storage is currently $85 per TB per month and Microsoft Azure is $64 per TB per month for their cheapest option (Locally Redundant Storage). I'd expect these prices to be dropping fairly soon in response to Google's move.

How much does this actually cost to provide? Let's look at the cost of storing and accessing 1 TB of data. An internal SATA 1TB hard drive costs about $60 on Amazon - but a 2TB costs $85, and a 4TB costs $160 (retail). So we can figure on about $40 per TB of storage. How long will this drive last? Mean time between failures of hard drives is between 18 months and 3 years depending on manufacturer and usage; let's split the difference and say 2 years. Buying 1TB of storage over 2 years will cost the supplier about $40 in capital costs. Isn't this a rip-off?

Well, having a hard drive is one thing - being able to access it is another. You've got to get data into that hard drive, and probably you want to get it out again. Assuming that the entire volume of that drive is written once and read twice per 2 years (probably a lowball estimate) at a rate of about 5 Mbits/s, that means that in 1 day (86400 seconds) you could read (86400 * 5 / 8) MB or about 54 GB per day so it would take up about 10 days per year per user, and so you could support about 36 users on a 5 Mbit connection. Let's say we're using 4 TB drives so you need a 5 Mbit connection for each 9 computers in your storage.

It's not quite that straight forward though. Cloud storage is supposed to be reliable, and hard drives are manifestly not - they die all the time. Therefore you want at least a second copy of your data on a separate hard drive, and ideally you want that second copy to be in at least a separate building in case of a physical disaster (flooding, fire, tornado). Generally the further away the better, at least up to 100 miles or so, though distance tends to increase the expense of hosting because for every write to the data you need to send the write to your remote facility as well. Azure gives you the explicit option of how physically separate you want your data to be; Locally Redundant Storage vs Geographically Redundant Storage.

There's also the distinction between data loss and data unavailability; if the primary copy of the user's data is unavailable (e.g. because the data center has a planned or unplanned availability outage) cloud providers may give their customers the option of reading data from (or writing data to) the backup copy of the data. Customers can buy this kind of read access from Azure as an additional option (Read Access GRS).

If you as the cloud storage operator rack up 3600 users, then, you'll need 900 computers with a total of 3600 TB of storage in each of 2 sites. You'll need 500 Mbit/s of bandwidth on each site if you want to offer read redundancy, and about 170 Mbit/s of bandwidth between sites to replicate writes. You don't need customised hardware for this amount of traffic, but you do need to buy the bandwidth to get the data to and from the user. Azure quotes $120 for egressing 1TB of data; if we estimate that it actually costs them $100 then each user will cost you $200 (reading their data twice), so you will have $720,000 of bandwidth cost.

If the computers last about as long as the hard drives you'd expect them to cost you about $300 plus the storage ($160), say about $500 once you take into account rack and network switch hardware. I suspect power won't be too much of an issue since storage isn't a CPU intensive operation and user access is intermittent - idling computers without a display consume about 20W. So each site will cost you 900 x $500 over two years, and consume a steady 18 KW of power. Electricity costs about $100 per MWh to make in the cheaper parts of the USA, so power will cost you $2 per hour per site. So power is only about 7% of your equipment costs for a 2 year lifetime, and you end up paying about $1.8M in total in hardware, power and bandwidth to provide 1TB of cloud storage to 3600 users over 2 years. Each user pays $240 over that time, or $860K in total. So it would seem that $10 per hour is a massively losing proposition for the provider even before we take into account the human costs of designing, building and operating the system.

The real picture is more nuanced. We implicitly assumed that every user would use all the storage (and bandwidth) they paid for. In practice, they could conceivably be consuming only half of what they've paid for. As long as we can dynamically provision for users (having a small amount of storage headroom, and adding on more computers and drives as that headroom is threatened) we could provide maybe 60% of the maximum hardware needed, so instead of $1.8M our costs would be down to $1M or so. Still something of a loss.

I think the way cloud companies can make money - or at least avoid a loss - on this is to make use of the fact that the computers providing access to the drives are seldom even slightly busy. Instead of buying $300 of low power computing hardware to support each 4TB drive, just chain a few 4TB drives onto an existing computer that you're using for something else (say, Bing search, Google maps, Amazon website serving). When a user starts to access their data, temporarily reserve a core or two on the machines holding that data to serve it. That way you save nearly 60% of your hardware costs and might just be bringing your operation into slight profit.

You'll still need to pay the design and implementation costs for your system, not to mention the usual marketing and business operations, but these don't scale in proportion to your number of users. The more users you have, the better your business looks.

$10 per month per TB is a bit of a game changer. Suddenly storing in the cloud isn't massively more expensive than storing on your hard drive. I wonder what the next couple of years will bring?

Piling on Piers Morgan

I was initially surprised that the news of Piers Morgan's fall from grace resulting in the cancellation of his CNN prime time talk show had been news in the UK, but I guess the prospect of Morgan returning to Blightly was sufficiently generally appalling that the Brits were quite concerned about the prospect.

Yesterday Piers had comedienne Chelsea Handler on his show. Noted for brutal honesty, such as discussing personal abortion and DUI stories, Chelsea didn't disappoint but possibly not in the way Piers expected. The discussion following the show's commercial break was quite enlightening:

Chelsea: I mean, in the middle of the commercial break – I want your viewers to know; they must know, because they're probably following you on Twitter. I mean you can't even pay attention for 60 seconds. You're a terrible interviewer.
Piers: Well, you just weren't keeping my attention. It's more of an issue with you than me.
Chelsea: That's not my problem. This is your show, you have to pay attention to the guest that you invited on your show.
Piers: If they're interesting enough ...[cut off]
Chelsea: Listen, it doesn't matter how interesting I am. You signed up for this job. [..] Well, maybe that's why your job is coming to an end.

Piers has - or had - the 9pm-10pm slot on CNN, which is prime time. (West Coast viewers, 3 hours behind the East Coast times quoted, can usually see the show live at 6pm Pacific or repeated 3 hours later). There's a constant battle for viewers between CNN and Fox in the evening, and Fox announced in August that newscaster Megyn [yes, really] Kelly would be taking over the Fox 9pm slot from previous incumbent Sean Hannity. Piers seemed keen on the challenge, tweeting "Bring it on, Megyn Kelly". Kelly duly brought it on, starting in early October and two months later was beating Piers 5 to 1 in viewers, up 10%-20% from Hannity.

Why did CNN viewers turn away from Piers in droves? Kelly is clearly easier on the eyes than Piers, but that can't be the whole story. I think Tim Stanley in the Telegraph gets closest to it:

But he acted as though no one had ever thought to discuss the subject[gun control] before. Like, ever. He tried to make gun control his own personal crusade, to "school" the Americans on law and order. And he displayed a crass insensitivity towards issues such as the importance of the Constitution or the American tradition of self-reliance. The scale of his ego was extraordinary. No US liberal has ever managed to challenge their country's fundamental respect for gun ownership. Why did he imagine that a guy with an English accent – the accent of George III no less – would succeed where Bill Clinton, Teddy Kennedy or Barack Obama had failed?
[...]
The embarrassing thing about the whole Piers Morgan affair is that it has turned the tables on us Brits. We always insisted that we were the courteous ones and the Americans were the boobs. In this case, it's been the other way around.

Americans really don't like being told, imperiously, what to do. Piers did this all the time, and worse wouldn't admit when he'd been beaten. He invited young Republican Ben Shapiro to debate guns with him after the Sandy Hook shooting, and had his clock unexpectedly cleaned - Shapiro tried to load the game by going for a "town hall" format which would have stuffed the audience with shooting victims and Shapiro wisely refused, telling him 1-on-1 or nothing. So, nothing.

Looking at interview styles, probably the best comparison on Fox with Piers is Bill O'Reilly who hosts the 8pm-9pm slot. O'Reilly has a similar format, inviting guests to discuss contentious issues 1-on-1, or sometimes 2-on-1, with him. One key difference, I think, is that O'Reilly doesn't try to lay traps - he relies on his (formidable) preparation and debating skills. Unlike Piers he has regular guests from a range of political views - liberal Juan Williams is one of the favourites - and the debate can get quite shouty at times but the guests generally know O'Reilly, know the position he'll take, and have wrestled with him often enough to give a good showing. More important, O'Reilly doesn't belittle them. He tries to reason, and even if you don't agree with his (often reactionary) position you can see that he's playing the man rather than the ball. Piers by contrast is a classic bully, trying to belittle opponents.

CNN's Anderson Cooper is a marked contrast to Piers; I've seen Anderson do interviews, and he can be tough - Greg Smith who resigned publicly from Goldman Sachs did an Anderson Cooper interview, and Cooper didn't go easy on him at all - but he's fair. You feel as if he's trying to make the interviewee explain what he's hiding from the audience, rather than browbeating the interviewee from a position of power just to establish a pecking order like a bully does. It would be hard to overstate how much most Americans look down on bullies. The War of Independence was essentially a reaction against perceived bullying (imposition by means of might) of the American colony by King George III. Piers is a classic bully, and Americans simply don't like that kind of person.

Interestingly it seems that even his own staff weren't keen on him:

"The makeup girls suffered the worst — he was rude and belligerent," says our source. "The general feeling is Morgan didn't show any respect to anyone working under him — the people who were trying to make him look good."

It would be uncharitable to note how hard a job that is. But this is Piers Morgan, so screw charity.

The next step in the gentrification wars?

I'm going out on a limb here and saying that the massive fire in a new apartment building in San Francisco is not unrelated to the past year's increasingly violent struggle between long-term residents and new arrivals from the tech sector. According to local TV station KVTU:

Another Strata resident, 25-year-old Hisham Bajwa, said he could see the fire start to burn outside his window shortly before 5 p.m.
"There were two main points of fire, one on the left and one on the right," Bajwa said. "It got pretty big pretty fast."

An interesting observation; it would be surprising for an accidental conflagration to manifest in two separate points. Now eyewitness reports are famously inaccurate, and the fire could have spread internally before being visible in two external points, but it does make you wonder...

You can see the building under construction to the west of the intersection between 4th Street and China Basin Street. The implication of the size of the construction is that it was a new apartments block - opposite it is Strada Apartments which had to be evacuated, and just up the street is Channel Mission Bay Apartments. So why does a huge new apartment building start to burn down? A welding accident? A gas leak igniting? Or something more deliberate?

The Mission District in San Francisco next to where these apartments are located has been Ground Zero for the protests against the influx of tech and biotech workers from Apple, Google, Facebook, Genentech and others. Assaulting Google Glass wearers in bars, blockading shuttle buses or just generally protesting tech nerds has become an increasingly popular sport in central San Francisco. Since it's nearly impossible to increase rents significantly in San Francisco apartments or evict a renter who doesn't want to leave - even if their contract is at an end they can require the landlord to renew it at substantially the same rent and terms - the only way that most landlords can improve their rent income is to invoke the Ellis Act to "go out of business" and sell their properties to another company, which then changes the use of the building (often via a drastic knock-down and rebuild).

The organised protests to date have mostly been focused around the shuttle buses which take the tech workers to Cupertino, Mountain View, Sunnyvale and other places across the South Bay; they are a very visible and concentrated target. Perhaps now the anti-tech movement is changing tactics: instead of impeding the transport, make building luxury apartments a much more expensive and chancy business. If they cause enough disruption maybe they'll be able to slow the influx of tech money and keep their existing apartments.

The investigation into this (hundred millions of dollars?) fire could be very interesting.

Update: KTVU confirms a $220M+ project with 360 apartments and "Arson investigators were on the scene Tuesday and will return Wednesday morning. " I bet they will.

Fixing Healthcare.gov - the inside story

The new Time covers in depth the work of the team who fixed Healthcare.gov. It's a fantastic read, with good access to the small but extremely competent team who drove the fix - go absorb the whole thing.

The data coming out of the story confirms a lot of what I suspected about what was wrong and how it needed to be fixed. Breaking down by before-and-after the hit team arrived:

Before

1. By October 17 the President was seriously contemplating scrapping the site and starting over.
2. Before this intervention, the existing site's teams weren't actually improving it at all except by chance; the site was in a death spiral.
3. No one in CMS (or above) was actually checking whether the site would work before launch.
4. The engineers (not companies) who built the site actually wanted to fix it, but their bosses weren't able to give them the direction to do it.
5. There was no dashboard (a single view) showing the overall health of the site.
6. The key problem the site had was being opened up to everyone at once rather than growing steadily in usage.
7. The site wasn't caching the data it needed in any sensible way, maximising the cost of each user's action; just introducing a simple cache improved the site's capacity by a factor of 4.

I refer the reader in particular to my blogpost The Curse of Experts where CMS head Marilyn Tavenner was trying to dodge blame.

During the Tuesday hearing, Tavenner rejected the allegation that the CMS mishandled the health-care project, adding that the agency has successfully managed other big initiatives. She said the site and its components underwent continuous testing but erred in underestimating the crush of people who would try to get onto the site in its early days. "In retrospect, we could have done more about load testing," she said.

As the Time article shows, this was anything but the truth about what was actually wrong.

After

1. There wasn't any real government coordination of the rescue - it was managed by the team itself, with general direction but not specific guidance from the White House CTO (Todd Park)
2. The rescue squad was a scratch team who hadn't worked together before but was completely aligned in that they really wanted to make the site work, and had the technical chops to know how to make this happen if it was possible.
3. Fixing the website was never an insurmountable technical problem: as Dickerson noted "It's just a website. We're not going to the moon." It was just that no-one who knew how to fix it had been in a position to fix it.
4. The actual fixes were complete in about 6 weeks.
5. One of the most important parts in improving the speed of fixing was to avoid completely the allocation of blame for mistakes.
6. Managers should, in general, shut up during technical discussions: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it."
7. The team refused to commit to artificial deadlines: they would fix it as fast as they could but would not make promises about when the fixes would be done, refusing to play the predictions game.
8. Having simple metrics (like error rate, concurrent users on the site) gave the team a good proxy for how they were doing.
9. Targeted hardware upgrades made a dramatic difference to capacity - the team had measured the bottlenecks and knew what they needed to upgrade and in what order.
10. Not all problems were fixed: the back-end communications to insurance companies still weren't working, but that was less visible so lower priority.

The overall payoff for these six weeks of work was astonishing; on Monday 23rd December the traffic surged in anticipation of a sign-up deadline:

"We'd been experiencing extraordinary traffic in December, but this was a whole new level of extraordinary ... By 9 o'clock traffic was the same as the peak traffic we'd seen in the middle of a busy December day. Then from 9 to 11, the traffic astoundingly doubled. If you looked at the graphs, it looked like a rocket ship." Traffic rose to 65,000 simultaneous users, then to 83,000, the day's high point. The result: 129,000 enrollments on Dec. 23, about five times as many in a single day as what the site had handled in all of October.

Despite this tremendous fix, however, President Obama didn't visit the team to thank them. Perhaps the political fallout from the Healthcare.gov farce was too painful for him.

The best quote that every single government on the planet should read:

[...] one lesson of the fall and rise of HealthCare.gov has to be that the practice of awarding high-tech, high-stakes contracts to companies whose primary skill seems to be getting those contracts rather than delivering on them has to change. "It was only when they were desperate that they turned to us," says Dickerson. "I have no history in government contracting and no future in it ... I don't wear a suit and tie ... They have no use for someone who looks and dresses like me. Maybe this will be a lesson for them. Maybe that will change."

The team who pulled President Obama's chestnuts out of the fire didn't even think they were going to be paid for their work initially; it looks like they did eventually get some money, but nowhere near even standard contracting rates. And yet, money wasn't the motivator for them - they deeply wanted to make Healthcare.gov work. As a result they did an extraordinary job and more or less saved the site from oblivion. This matches my experience from government IT developments: it's reasonable to assume that the government don't care about whether the project works at all, because if they did then they'd run it completely differently. Though if I were President I'd be firing Marilyn Tavenner, cashing in her retirement package and using it to pay bonuses to the team who'd saved my ass.

If you have a terribly important problem to solve, the most reliable way to solve it is to find competent people who will solve it for free because they want it to work. Of course, it's usually quite hard to find these people - and if you can't find them at all, maybe your problem shouldn't be solved in the first place.

Don't give the guests power over the residents

Top legal blog "The Volokh Conspiracy", now at the Washington Post, analyses the recent California 9th Circuit decision that wearing American flag shirts at high school can legally be prohibited. Eugene Volokh notes that the actions of the principal (Mr. Rodriguez) in banning wearing of American flag clothing in fear of it causing violent disruption may be constitutional but not at all a good idea:

This is a classic "heckler's veto" — thugs threatening to attack the speaker, and government officials suppressing the speech to prevent such violence. "Heckler's vetoes" are generally not allowed under First Amendment law; the government should generally protect the speaker and threaten to arrest the thugs, not suppress the speaker’s speech. But under Tinker's "forecast substantial disruption" test, such a heckler's veto is indeed allowed.

I have to confess sympathy for Mr. Rodriguez in his predicament - his job is to ensure order and prevent disruption at his school, and students who wear the American flag did seem very prone to be correlated with disruption:

At least one party to this appeal, student M.D., wore American flag clothing to school on Cinco de Mayo 2009. M.D. was approached by a male student who, in the words of the district court, "shoved a Mexican flag at him and said something in Spanish expressing anger at [M.D.;s] clothing."

Now there are plenty of legal Mexican immigrants in the USA, so we shouldn't assume anything about the angry student's immigration status, but if (for instance) a student of Scottish heritage took offense to a Live Oak student wearing an American flag on St. Andrew's Day and threatened him "I'll cae the pins o' ye!" I can't imagine Mr. Rodriguez reacting the same way. The principal does seem to be bowing to the opinions of a category of "guest" students in preference to those who are citizens of the country. (Irish students aren't likely to cause problems on St Patrick's Day because it's celebrated in the USA as much if not more than in Ireland).

If you want to know more about what Live Oak High School is like, take a look at the California department of education stats for the school. It's about 50-50 demographic split between white and Hispanic/Latino students. The standardised scoring indicates that white, Asian and black students improved significantly in the past academic year but the Hispanic/Latino students went backwards. It seems that pandering to them isn't doing them any favours academically. Incidentally, I'm dubious about the "Two or more races" stats - only 1 student of mixed heritage out of 858? My arse.

I can do no better than quote Volokh's takeaway:

The school taught its students a simple lesson: If you dislike speech and want it suppressed, then you can get what you want by threatening violence against the speakers. The school will cave in, the speakers will be shut up, and you and your ideology will win. When thuggery pays, the result is more thuggery. Is that the education we want our students to be getting?

If Live Oak High School really wants to help its Hispanic/Latino students, it should insist that they meet the standards expected of all other students.

Ukraine and Clancy's prescience

At Christmas I read the newest (and final, given the author's passing) Tom Clancy novel Command Authority where hero and president Jack Ryan is battling to keep Russia from taking over Ukraine with military force, using agents provocateur in conjunction with the substantial number of Russian passport holders in east Ukraine as a pretext for involvement and invasion.

All is passing as Clancy has forseen:

Russia's large landing ship Nikolai Filchenkov has arrived near the Russia Black Sea Fleet's base at Sevastopol, which Russia has leased from Ukraine since the fall of the Soviet Union in 1991.
The ship is reported to be carrying as many as 200 soldiers and has joined four additional ship carrying an unknown amount of Special Forces troops. Flot.com also reported over the weekend that personnel from the 45th Airborne Special Forces unit and additional divisions had been airlifted into Anapa, a city on Russia's Black Sea coastline. In addition, it is believed that Russia's Sevastopol base contains as many as 26,000 troops already, according to the German Institute for International And Security Affairs.

Clancy will never be remembered as a giant of literature, but you have to give the man his due in anticipating world events. Russia is no doubt terrified of Ukraine pivoting westwards, especially given the billions of dollars invested there:

Moody’s cited Putin as saying Ukrainian borrowers owed Russia about $28 billion, according to a report this month, before the $15 billion of bond purchases. Ratings agencies will "eventually" have to look at this exposure, Grafe said.

With fracking and Canadian oil exerting downwards pressure on fuel prices, one wonders where the Russian siloviki are going to find the money to maintain their lifestyles. For sure V. V. Putin can turn off the taps of Ukraine's gas supply, but that's only likely to make the citizens more angry at Russia.

I'd hope that rolling tanks across the border would be a step too cheeky even for Putin, but who knows what internal pressures are being exerted?

Apple's SSL bug - better code reviews required

There's a great technical discussion by Adam Langley at Imperial Violet on the inadvertent security hole that Apple introduced to iOS 7 and later versions of OS X. They've released a patch for iOS (which is how people noticed) but are still working on the OS X fix. My sympathies are with Apple despite them being panned for the delay - the fix is straight forward, but building, qualifying, canarying and distributing the desktop fix inevitably takes a while, and if you try to speed up this process then you have a high risk of making things much, much worse.

The effect of the bug is that it allows a certain kind of attack ("man in the middle") which intercepts secure web connections, say from a user on an Apple laptop to their online banking system. An attacker with sufficient access and resources can pretend to the user to be their online banking server, and the user will have no practical way to detect this. However in practice it is very difficult to exploit, and is only really a concern for users who believe that they may be targeted by government agencies or well-funded and persistent private parties; it's unlikely that it will be widely exploited. Modern iOS and Safari users are not a large fraction of internet traffic, even if you only look at HTTPS traffic.

The bug itself is probably only interesting to code nerds such as your humble correspondent, but how it came about is quite telling about how software development works at Apple.

Here's a cut-down version of the offending function:

static OSStatus
SSLVerifySignedServerKeyExchange(SSLContext *ctx, bool isRsa, SSLBuffer signedParams,
                                 uint8_t *signature, UInt16 signatureLen)
{
	OSStatus        err;
	[...]
	if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
		goto fail;
	if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
		goto fail;
		goto fail;
	if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
		goto fail;
	[...]

fail:
	SSLFreeBuffer(&signedHashes);
	SSLFreeBuffer(&hashCtx);
	return err;
}

See that third "goto fail;" line in the middle? That's the error. Almost certainly it was the result of a fat-finger in a code editor, it's very unlikely to be a deliberate change. For tedious reasons related to how code blocks work in the C programming language, the effect of the third "goto fail;" is very different to the first two. It isn't tied to a condition, so if the program manages to get past the first two "if" statements successfully (the initial secure connection checks) then it never carries out the third check. When it reaches the end of the code, the result in the variable "err" actually represents whether the first two checks completed successfully, not (as required) whether all three checks completed successfully.

The reason this interests me is that this change made it into an official Apple release without being detected. I claim that if this code change was reviewed by a human being (as it definitely should have been) then anyone paying the slightest attention would have seen the duplicate "goto fail;" line which would have made absolutely no sense. I can fully understand this error not being picked up by automated testing - it's not straight forward to build a test which could cause this particular case to fail - but this is another indicator that Apple are not paying nearly enough attention to developing software reliably. Getting another person to review your code changes is a basic part of the software development process. If it's not being done, or only being conducted in a cursory fashion, you're going to leave your code riddled with bugs. There is no shortage of bad actors waiting for you to make a mistake like this.

I'm really curious about how this got noticed. My money is on someone browsing the code to make an unrelated change, and being drawn to the duplicate line, but that's only speculation.

I've given Apple heat for their sloppy approach to security in the past and I'm concerned that they're not reacting to the clear signs that they have a problem in this area. If code changes to a key security library are not going through human review, they're going to continue to have problems.

Unionisation is in trouble when data is being fudged

Keen observers of the progress of unionisation in the USA have been following with interest the story of VW's Tennessee workers rejecting unionisation despite management neutrality on the issue. The Union of Auto Workers (UAW) - whose stranglehold in Detroit has worked out so well for that city - tried and failed to persuade Tennessee that their interests were aligned, despite VM management being studiously neutral on unionisation.

Now CBS staff writer Sadhbh Walshe desperately tries to put a pro-union spin on these events. (For those curious about the name, it's pronounced Sive).

In Germany, for instance, auto workers at VW plants get paid an average of $67.14 an hour. That's more than double the average hourly rate for an established unionized worker in Detroit, and it's more than three times what the non unionised workers in Chattanooga can hope to earn.

Wow. Because, of course, the cost of living in German cities is totally comparable to the cost of living in both Detroit and Tennessee. I wonder why Ms. Walshe didn't bring up this point in the article? Amazingly, she doubles down on it in the comments:

I'm not sure what the rates are in other countries, but I do know that the cost of living in the US is comparable to Germany so salaries ought to be comparable too.

Ah, "the US". Because 330 million people spread across 3.8 million square miles and 51 different states is just one entity for comparison purposes, despite Germany being 80 million people and less than 5% of the area. Heck, even in the single state of California the differnce in cost of living between the Bay Area (San Francisco to San Jose) and Fresno is huge. A $40K annual salary would be pretty comfortable in Fresno, but practically poverty-level in San Francisco. Contrast Chattanooga, with a median home price of $114K, with Kassel in Germany (home of a VW plant) where you'd be lucky to get a garden shed for that price.

That German earnings figure is extremely suspect even in isolation; as commenter GregUS notes:

Are you seriously claiming that assembly line auto workers make $67.14/hour? This works out to $140,000 per year, which is about the average salary of a primary care physician in Germany (€110,000/year).

German workers are also nearly twice as productive as USA workers, so even if they are paid higher it would seem reasonable for VW to require USA productivity to rise before paying US workers more.

I particularly like Ms. Walshe's spin on right to work states:

It's true that with Tennessee being a so called "right to work" state, where wages are generally low and poverty is high, $19.50 an hour probably seems like a pretty good salary.

"Right to work" means (in essence) that you are not obliged to join a union or pay union dues if you join a unionized workplace. This would seem a no-brainer to most UK workers, but in fact state law bans this practice in about half the states. The non-right-to-work states are CA, OR, WA, MT, CO, NM, MN, WI, MO, IL, KY, OH, WV, PA, NY, VT, MD, NJ, NY, NH, ME, MA, NH, DE, CT, RI.
Let's compare this with the top 20 states with highest unemployment, in descending order. States in bold are non-right-to-work:

RI, NV, IL, MI (just changed to Right to Work), CA, DC, MS, KY, TN, AZ, GA, CT, AR, NJ, OH, NY, OR, MA, PA.

See a pattern here? Of course, I can't claim causation with this superficial evidence - but at the very least, it seems that states without the right-to-work law seem to be more likely to suffer from unemployment.

Sadhbh Walshe is desperate to have us believe that the only thing standing between Tennessee auto workers and prosperity is them needing to join the UAW, but it seems that even compulsory unionization is more likely to cause unemployment than raise wages.

A tip on tipping

Oh, for fuck's sake. Arun Sethi in CiF complains about the poverty of tipped employees in the USA:

Across the country, advocates are organizing, rallying, and speaking out in support of raising the US tipped minimum wage of $2.13.
Yes, you read that right. Tipped workers include parking lot attendants, bellhops, baggage porters, manicurists, and barbers. They also include many people in the restaurant industry – waiters, waitresses, and food deliverers. They haven't seen a raise since 1991.

That's a rise in their employer-paid minimum wage. Why aren't they deserting these roles in droves for a (federal, hence minimum across all states) minimum wage of $7.25/hr, say at the local Taco Bell? Well it might just be because retail prices are rising rapidly, and the tip % is rising independently. All that employer minimum wage covers for salary is slack hours when you're doing nothing; as soon as you get a customer, tips come in to play.

If you haven't lived in America, here's a primer on tipping. A reasonable tip for most roles is 15%-20% of the purchase price - and in the past 10 years this has moved closer to 20% than 15%. For bartenders it's $1-$2 per drink (glass of wine, shot of regular spirits, pint or bottle of beer). If you go out to a restaurant for a family meal, say 4 people at a table for 1 hour and a bill of $50-$80 then you'll pay a tip of $10-16. If the waitress only serves one table per hour she'd be way over minimum wage. Realistically she's covering more like 5-10 tables - she'll do really well out of this; not as well as the figures suggest since she has to "tip out" (pay a % of tips) to the busboys and kitchen staff, but still a very comfortable wage. She certainly won't get the high rates during the 2pm-7pm slot, since the customers drop off, but even two occupied tables per hour should keep her quids in.

Attendants, porters and bellhops get $1 per bag (ballpark); I don't know much about their baggage carrying rate, but they should clear minimum wage without any problems and should be able to make $12/hour (one customer with one bag per 5 mins) just fine during moderately busy periods. Manicurist charges are about $20/30 mins, so 20% yields $8/hour; this is by no means extravagant, and unlike waitressing is capped at one customer per 30 mins or so, but is still above minimum wage. They get screwed over by slack periods of course. But compare "service" at a restaurant in the UK versus one in Manhattan, Dallas or Minneapolis and you'll see just how well financial incentives work.

If by "parking lot attendants" Sethi means "valet" - you don't tip the person in the booth at the city parking lot - then they do staggeringly well. At a hotel with valet you pay $10-$20 per day and tip $1 min, $5 max per fetch (5 mins). Valet 8 cars per hour with minimal tip and you're still ahead of the game. I've typically tipped in the $2-$3 region, and the valets I've seen are busy for most of the day.

Note that fast food staff aren't covered by this since you don't normally tip at McDonalds, Five Guys, In-N-Out, Wendys, Taco Bell; they get regular minimum wage.

You don't tip gas station attendants; postal workers and gardeners get a one-off Christmas tip. Taxi drivers get 15%-20% and do just fine - try leaving a 10% tip for a Manhattan cabbie and learn inventive new words. And only assholes don't tip at all; I've left a sub-15% tip on maybe 5% of occasions, and only for service that was well below "adequate" and tending to "derisory".

Who is this mendacious clown?

Arjun Sethi is an attorney in Washington, DC, and a frequent commentator on civil rights and social justice related issues. He is a member of the Minimum Wage Coalition, a consortium of more than 40 groups, nonprofits, and faith-based organizations, working to increase the wage.

Wow, certainly no conflict of interest there. I note that he doesn't add "and conspiring to reduce employment" to that description, despite that being the net effect of his lobbying; highly-qualified and privileged asshole that he is.