Observations on boiling frogs

I've had the recent opportunity to observe a post-COVID concerted attempt by tech leadership to reduce its company's costs, and it has been quite eye-opening. A common theme in employee chat spaces when discussing the changes is the whole boiling frog meme, but I think that a lot of people miss the deeper implications of what's actually going on.

Q: Why do you boil frogs in the first place?
A: You're making frog soup.

When you make frog soup, you can add in lots of other ingredients to change / disguise the taste, but inevitably you will need to have a certain amount of frog to attract customers. Yes, these customers will probably be French, and you'll have to deal with everything that comes along with that, but that is your customer base. Frog soup eaters, who pay you for the soup, want frogs in their soup.

Ah, but how many frogs will there be? You fill the tureen with lukewarm water, drop (say) 100 frogs into it, and turn on the gas cooker. Will you get a 100-frog soup? No!

• Some frogs will jump out of the tureen, just because they're interested in the wider world - the water temperature means nothing to them.
• Some frogs are sensitive to heat, and at the first warming sensation they'll try to escape.
• Once the activity becomes noticeable to the broader frog population, there will be general concern in the tureen, and some frogs will try to jump out just because they notice other frogs jumping out. Generally, you lose the healthiest frogs at this point. The old, sick frogs are stuck.
• You might try to drop additional frogs into the tureen to replace those lost. Unfortunately, word gets around the frog community fairly quickly, and the larger frogs will squirm out of your hands. You're left with the young frogs who don't know any better.

Eventually, the soup comes to the boil, and you're left with... substandard frog soup. Bon appetit!

The Twitter Whistleblower report - how bad was Twitter, really?

Prompted by a post by everyone's favourite Portugal-based squirrel-torturing blogger, Tim Worstall, I thought I'd dive into the practical implications of all the (frankly, horrendous) technical, security and privacy problems that Twitter was identified as having before Elon Musk rocked up as owner and CEO.

Usual disclaimer: I'm going by the reports. Reality might be different. I cite where I can.

For background: both USA and European authorities take a dim view of corporate access to, and usage of, individual user data. Remember the European "ePrivacy Directive"? Also known as the "'f+ck these annoying cookie pop-ups' law"... Governments in both Europe and the USA are keenly interested in companies tracking individual users' activities, though my personal opinion is that they're just jealous; they'd like to do it too, but they're just not competent. Anyway, a company doing individual tracking at large scale for profit - Twitter, Google, YouTube, Meta, Amazon - attracts their attention, and their laws.

Security

Let's talk about security - and, more importantly, access to secure data. A fundamental principle of security is "least privilege" - everyone should have the smallest set of access privileges to be able to do their job. You could argue that 5000+ people in Twitter "need" to be able to change things in production at some point to do their jobs, but they certainly don't "need" to have always-on, cross-production access. Not least, because someone running a command they found on an internal playbook as an experiment, could easily break a large chunk of the service. But don't rely on me, ask their job candidates:

Twitter's practice was a huge red flag for job candidates, who universally expressed disbelief. One Vice President of Information Technology [his current role, not the target role] considered withdrawing his application on the (accurate) rationale that Twitter's lack of basic engineering hygiene in their arrangement presaged major headaches.

Hire that guy.

Certainly, every company is far from perfect in this area, but those with regulators are continually seeking to narrow the number of people with access, and the scope of access those people have. Twitter pre-Musk clearly did not give a crap about the count and scope of access. One can only imagine why; were they, for instance, relying on a large base of pre-approved employees to intercept and downgrade/block opinions outside the mainstream? How would we tell if this were not the case? Can Twitter show that they were engaged in a systematic reduction of number and scope of access to production? If not, who will be held to account?

Auditing

Control is one thing - but at least, if a human performs an action in the production environment (change, or query), that action should at least be logged, so future audit can see what happened. This is not a high bar, but was apparently too high for pre-2022 Twitter:

There was no logging of who went into the production environment or what they did.

FFS
To make clear the implications: in general, there was no way of finding out who queried (for their own purposes) or changed (deleted posts, down-rated users, etc) the production environment at any particular time. "Why did [event] happen?" "Beats the hell out of me, someone probably changed something." "Who? When?" "No idea."

This is particularly interesting because Twitter's Chief Information Security Officer - who resigned post-Musk - was also their former head of privacy engineering, and before that, apparently, global lead of privacy technology at Google. One could only imagine what that implies.

Control

There is also a wide range of engineering issues. Data integrity (not losing user-entered data) was obviously a critical issue, but Twitter had been aware for a while that they teetered on the edge of a catastrophic production data loss:

even a temporary but overlapping outage of a small number of datacenters would likely [my italics] result in the service going offline for weeks, months, or permanently.

This is not quite as bad as it first seems. After a year or so in operation, companies have a fairly good idea what happens with a datacenter outage - because they're more frequent than you imagine. Say, Henry the intern accidently leans against the Big Red Button on the datacenter floor, that cuts power to everywhere. Or you do a generator test, only to discover that a family of endangered hawks have made their nest in the generator housing for Floor 2... So you get used to (relatively) small-scale interruptions.

If you want to run a global service, though, you need to be able to tolerate single site outages as routine, and multiple site outages (which turn out to be inevitable) have to be managed within the general bounds of your service's promised availability - and latency, and data availability. Even if all your physical locations are very separate, there will inevitably be common cause failures - not least, when you're pushing binary or config changes to them. So, don't wait for these events to sneak up on you - rather, anticipate them.

This means that you have to plan for, and practice these events. If you're not doing so, than a) it will be obvious to anyone asking questions in this area, and b) when things inevitably do run off the rails, there will be bits of burning infrastructure scattered everywhere, around the highly-paid morons who are busy writing memos to cover their asses: "how could we have foreseen this particular event? Clearly, it wasn't our fault, but pay us 20% extra and we might catch or mitigate the next such event."

Go looking for those people. Fire them, and throw them into a den of hungry pigs.

Leaving the doors open

By far the most horrific aspect, however, was the general relaxed attitude about government agencies - and heaven only knows what other NGOs, cabals, and individuals - having under-the-table access to Twitter's data. Just the tolerance of user-installed spyware on privileged devices would be enough for any sane security engineer to be tearing out their hair, but actually letting in individuals known to be employed by foreign - and even domestic - governments for the purposes of obtaining intelligence information, and potentially affecting the flow of information to their and other countries... one is lost for words.

At some stage, Twitter had to either grow up, or close down. Under Dorsey's crew, the latter was inevitable - and likely not far away. It's still too early to tell if Musk can get them to option 1, but there's still hope.

"Chaos Monkeys" and how it got Antonio Martinez fired from Apple

Dedicated readers of this blog (all 1 of them) may recall last month's post about author Antonio Garcia Martinez being fired by Apple because a bunch of neurotic employees didn't like what he'd written in a book five years ago. I promised a review of that book: "Chaos Monkeys - Obscene Fortune and Random Failure in Silicon Valley" and, dear readers, this is that review.

It's a great book. Is it the "Liar's Poker of Silicon Valley"? Maybe, maybe not, but they have a lot in common. Martinez takes you through his career at Goldman Sachs in New York, joining a dying Silicon Valley startup (Adchemy), fleeing to do his own startup (AdGrok), dancing through lawsuits and VC funding, and finally playing Twitter for an acquisition before skipping to join Facebook as an ads product manager. His stint in Silicon Valley is 2008 to 2016 and, to the best of my knowledge, accurately represents the people, companies and society there at the time.

Most importantly, like Michael Lewis of "Liar's Poker", Martinez is a compelling writer. He is opinionated, informed, funny and - unlike Lewis - cheerfully portrays himself as an averagely terrible human being. He fathers two children out of wedlock, kind of screws over his startup partners - though there's a twist at the end - gets away with drunk driving and outrageous speeding, has a torrid all-over grope with a busty fellow product manager in a Facebook janitor's cupboard, and plays off Twitter against Facebook with misleading information to boost the acquisition value of his start-up. He's contemptuous of the CEO Murthy Nukala, although to be fair Mr Nukala does not sound like a pleasant human being himself, and of the ass-kissing divisional leadership of Facebook.

Martinez is a really interesting and colorful guy. I would totally buy him lunch to hear a few of his stories. I would probably not want him dating my girlfriends though.

The full list of grievances of the Apple employees is given in the petition that was leaked to The Verge. Zoe Schiffer's byline there is no surprise, she is the leak-destination-of-choice for Big Tech. The top grievance was of course about Martinez's portrayal of women in Silicon Valley:

Most women in the Bay Area are soft and weak, cosseted and naive despite their claims of worldliness, and generally full of shit. They have their self-regarding entitlement feminism, and ceaselessly vaunt their independence, but the reality is, come the epidemic plague [my emphasis] or foreign invasion, they’d become precisely the sort of useless baggage you’d trade for a box of shotgun shells or a jerry can of diesel.

I've encountered a good number of Silicon Valley women in tech in the past year, and I have to say that Martinez hits the nail on the head here. The pandemic has demonstrated in spades the neuroticism of many of these women. There has been very little get-up-and-go demonstrated, instead just a whinyness and cowering that makes one despair for the future of the human race. If there is any criticism of Martinez here, it's that he omitted that many men in tech exhibit the same characteristics, which is maybe even less excusable. Interestingly, you don't see the same weakness in most of the Bay Area natives, nor in tech immigrants from Central / Latin America or the former Soviet states - India too, to a lesser degree. It is mostly a white-women-in-and-around-tech thing.

It's notable that Martinez contrasts this with the self-determinism of a number of interesting women he encountered and dated / screwed during his time in the Valley. He clearly isn't a misogynist in this respect - he just doesn't like a bunch of people.

The aforementioned janitor cupboard fondling also upset the Apple whiners, especially the description of his facilitator:

PMMess, as we’ll call her, was composed of alternating Bézier curves from top to bottom: convex, then concave, and then convex again, in a vertical undulation you couldn’t take your eyes off of. Unlike most women at Facebook (or in the Bay Area, really) she knew how to dress; forties-style, form-fitting dresses from neck to knee were her mainstay.
...[and later, when he's about to be let go]...
There were few women one would call conventionally attractive at Facebook. The few there were rarely if ever dressed for work with their femininity on display in the form of dresses and heels. A fully turned out member of the deuxième sexe in a conference room was as clear an angel of death as a short-barreled .38 Special revolver. Gokul [the manager firing Martinez] gave an awkward smile, and bolted out the door the moment I sat down. I looked across the table. If her look was supposed to disarm me, she needed either more cleavage or more charm.

Two things about this stand out: a) boy, Martinez knows how to write, and b) he is an astute observer of the unsayable. Techies have never been famed for their dress sense, and most women (and men) in a tech role do not really try to dress up. There's a principled thing here where they want to let their work speak for itself and not be judged by conventional metrics of attractiveness - but you can't then turn around and get annoyed when someone observes, correctly, that you aren't attractive. I'd imagine that it would be a very different experience in banking where how you dress can be the line between success and mockery.

He also observes:

It occurred to me that perhaps this most recent experiment in fertility—and the first—had been planned on British Trader’s part, her back up against the menopause wall, a professional woman with every means at her disposal except a willing male partner—in which case I had been snookered into fatherhood via warm smiles and pliant thighs, the oldest tricks in the book.

Would the Apple employees like to content that this is not, in fact, one of the oldest tricks in the book? Is it unacceptable to say precisely because it is the truth?

Go and read "Chaos Monkeys". It is a highly enjoyable book, it gives great if biased insights into Silicon Valley for both startups and Big Tech, and more importantly does so for the companies, the technology, and the humans involved. You won't regret it. And despite being fired because of it, I expect Martinez does not regret writing it.

As for the prissy Apple employees who signed the petition: I'd hire one Martinez over ten of them, any day.

Apple used to say "Think different." I guess those days are long gone.

Update: Garcia himself speaks without specificity on the firing. I'm guessing he got paid very well for signing that non-disclosure agreement, unlike the one he was offered (and declined) at Facebook. If it was less than six figures, I'd be very surprised.

A really ballsy move would be for Google, Oracle or even Twitter to hire him, to stick two fingers up at the pusillanimous HR skirts at Apple - and at their own self-important neurotic engineers. Won't happen, of course, but if you happened to have a division that you wanted shot of, and it was infested by this kind of person, hiring Martinez into it - and standing behind him - would be nothing short of hilarious.

Testing for determinism

Apropos of nothing^[1], here's a view on testing a complicated system for deterministic behaviour. The late, great John Conway proposed the rules for "Game of Life", an environment on an arbitrary-sized "chess board" where each square could be either alive or dead, and potentially change at every "tick" of a clock according to the following rules.

1. Any live cell with two or three live neighbours survives.
2. Any dead cell with three live neighbours becomes a live cell.
3. All other live cells die in the next generation. Similarly, all other dead cells stay dead.

You'd think that this would be a very boring game, given such simple rules - but it in fact generates some very interesting behaviour. You find eternally iterating structures ("oscillators"), evolving structures that travel steadily across the board ("spaceships"), and even "glider guns" that fire a repeated sequence of spaceships.

Building a simulation of Conway's Game of Life is something of a rite of passage for programmers - doing it in a coding language new to the programmer generally shows that they have figured out the language enough to do interesting things. But how do they know that they have got it right? This is where "unit testing" comes into play.

Unit testing is a practice where you take one function F in your code, figure out what it should be doing, and write a test function that repeatedly calls F with specific inputs, and checks in each case that the output is what's expected. Simple, no? If F computes multiplication, you check that F(4,5)=20, F(0,10)=0, F(45,1)=45 etc.

Here's a unit test script. It's written in Go, for nerds, ^[2] but should be understandable based on function names to most people with some exposure to programming. First, you need to check the function that you've written to see whether two Life boards are equivalent, so you create empty 4x4, 4x5, 5x4 boards and see if your comparison function thinks they're the same.
(In Go, read "!" as "not", and "//" marks a comment which the computer will ignore but programmers can, and should, read)

  b1 := life.NewBoard(4,4)
  b2 := life.NewBoard(4,4)
  // These should be equivalent
  if ! life.AreEqual(b1,b2) {
     t.Error("blank 4x4 boards aren't the same")
  }
  b3 := life.NewBoard(5,4)
  b4 := life.NewBoard(4,5)
  if life.AreEqual(b1,b3) {
    t.Error("different size boards are the same")
  }

That's easy, but you also need to check that adding a live cell to a board makes it materially different:

  // Add in a block to b1 and compare with b2
  life.AddBlock(0,0,b1)
  if life.AreEqual(b1,b2) {
    t.Error("one board has a block, blank board is equivalent")
  }
  // Add the same block to b2 in same place, they should be equal
  life.AddBlock(0,0,b2)
  if ! life.AreEqual(b1,b2) {
    t.Error("2 boards, same block, unequal")
  }

This is helpful, but we still don't know whether that "block" (live cell) was added in the right place. What if a new block is always added at (2,3) rather than the coordinates specified? Our test above would still pass. How do we check for this failure case?

One of the spaceships in Life, termed a glider, exists in a 3x3 grid and moves (in this case) one row down and one column across every 4 generations. Because we understand this fundamental but fairly complex behaviour, we can build a more complicated test. Set up a 5x5 board, create a glider, and see if

1. the board is different from its start state at time T+1;
2. the board does not return to its start state at time T+2 through T+19; and
3. the board does return to its start start at time T+20.

Code to do this:

  b5 := life.NewBoard(5,5)
  life.AddGlider(0, 0, b5, life.DownRight)
  b6 := life.CopyBoard(b5)
  if ! life.AreEqual(b5,b6) {
    t.Error("Copied boards aren't the same")
  }
  // A glider takes 4 cycles to move 1 block down and 1 block across.
  // On a 5x5 board, it will take 5 x 4 cycles to completely cycle
  for i := 0 ; i< 19 ; i++ {
    life.Cycle(b5)
    if life.AreEqual(b5,b6) {
      t.Error(fmt.Sprintf("Glider cycle %d has looped, should not", i))
  }
  life.Cycle(b5)
  if ! life.AreEqual(b5,b6) {
    t.Error("Glider on 5x5 board did not cycle with period 20")
  }

Now, even if you assume AreEqual(), NewBoard(), CopyBoard() work fine, you could certainly construct functions AddGlider(), Cycle() which pass this test. However you'd have to try pretty hard to get them right enough to pass, but still wrong. This is the essence of unit testing - you make it progressively harder, though not impossible, for a function to do the wrong thing. One plausible failure scenario is to make the adjacent-cells locator in Cycle() incorrect such that the glider goes up-and-across rather than down-and-across. To fix that, you could add some code to turn-on a critical cell at (say) time 8, such that that cell would be live in the expected motion, so no effect, but empty in the other motion.

Clearly, for unit testing to work, you want a unit tester who is at least as ingenious (and motivated) as the coder. In most cases, the coder is the unit tester, so "soft" unit tests are unfortunately common - still, at least they're a basis to argue that the code meets some kind of spec. And if the client isn't happy with the tests, they're free to add their own.

Why am I so mad at Neil Ferguson? He's free to make whatever epidemiological assumptions that he wants, but he usurped the "authority" of computer modelling to assert that his model should be trusted, without actually undertaking the necessary and fundamental computer science practices - not least, unit testing.

[1] Lies: Neil Ferguson, take note
[2] Object-oriented model avoided for clarity to readers

Taking advice from Greta Thunberg

Suppose we were looking to build a bridge, say across Avon Gorge, to give us substantially more traffic capacity than the existing Clifton Suspension Bridge has. (The Dear Reader may insert their favourite joke about needing much more capacity for traffic leaving Bristol than for entering it).

It wouldn't be surprising that a lot of people would have strong opinions on what kind of bridge we should build. Imagine, however, that a 16 year old high school student was championing a bridge structure that comprised a sequence of road segments chained together and suspended from helium balloons. Imagine that such a proposal was lauded by at least 30% of the people involved as bold, innovative, and a wonderful example of youthful thinking, despite the fact that a first year engineering student could shoot the proposal as full as holes as a particularly perforated Swiss cheese.

That, ladies and gentlemen, is where we find ourselves with young Greta Thunberg.

Young Greta is clearly sincere , and cares deeply about the environment. Unfortunately, "sincerity" is as useful a factor in planning a 21st century industrial strategy as it is in building a bridge. If a bridge builder tells me that she "sincerely" believes it will support the expected peak weight of traffic in peak adverse conditions, and be durable for a lifetime of 50+ years, I will smile and nod; if anyone I care about will be traversing the bridge, I will then ask pointed questions about stress calculations, FEMs analysis, safety engineering analysis, and all the inconvenient hard science that lets us calculate at least a ballpark probability of the bridge suddenly failing and casting a few hundred people into the abyss. Nearly anyone can be sincere. To be correct requires actual maths, materials knowledge, ability to program R / Matlab / other mathematical tool of choice and produce a verifiable assertion, given generally accepted axioms, that the bridge will meet specs.

Somehow, I don't see this level of mathematical / physical / engineering rigour coming from young Ms Thunberg. Or her singer/actor parents, for that matter.

The correct response to Greta Thunberg and her parasitic (in every sense of the word) hangers-on is as follows:

• Give us a practical - by which we mean can-be-implemented-with-existing-technologies - 20 year plan for reducing carbon emissions world-wide by X%.
• Cover the top 10 current CO2 polluters; either assume they continue on current trend, or argue why they will change.
• You cannot assume any existing technology improves by more than 4% per year for cost/efficiency.
• Include the expected economic impact on the top 10 world economies.

Greta would (quite rightly) say: "I'm 16 years old, how could you possibly expect me to answer this?"

Greta käraste, if you can't be expected to answer the hard questions, why should we listen to your easy answers?

Blacklist your master, and whitelist your slaves - Silicon Valley word police

Working in Silicon Valley ("putting the crazy into California!") is always an education; there seems to be a Shepard tone of neuroticism in and out of the workplace. Every time you think you've seen the craziest thing you can imagine, something nuttier comes along shortly afterwards.

In the world of global-scale computing, big services like Facebook, Twitter and Gmail are very strongly interested in what happens when a machine in their service infrastructure fails. (This is relevant, I promise.) If only one machine knows how to handle data from user Joe, then Joe is going to be very upset when that machine reboots for an OS upgrade (5-15 minutes downtime), or worse becomes permanently unavailable because a data center technician accidentally bridged the rack bus bar onto the hard drive with her ^[1] screwdriver because she was paying too much attention to the shapely arse of the technician fixing the next rack over.

The natural solution is that you have multiple machines - maybe in multiple datacentres - which know how to handle data from Joe, and there's some kind of load-balancing across them which knows which of those machines are healthy, and which aren't. But out of all of those machines, you need to have at one which has the canonical state of Joe's data, and which all other machines agree to take data from. Otherwise you end up in the state where there are two or more different views of Joe's data, and can't tell which is valid. In that case, the machine with canonical state is known as the "master", and the other machines receiving state from it are known as "slaves".

I think you can see why this terminology has started to become "controversial" to the Usual Suspects:

Problem:
The term Master in Master Components is potentially offensive to people of color and women, and I suggest we use a more inclusive synonym.
Proposed Solution:
Suggest renaming to "Primary Components" or "Leader Components"

(By contrast, when the failure occurs at a higher level in the software, you end up writing garbage to all copies of the data - on both masters and slaves. If you've overwritten previous data, your only hope is to bring it back from an earlier system state snapshot - witness this Gmail inbox wipe-out from 2011.)

That was silly enough, but now the common terminology of "whitelist" (allow these items, but not others) and "blacklist" (allow all items except these) has come under attack:

Per https://twitter.com/dhh/status/1032050325513940992, [Tweet by Ruby-on-Rails founder] I'd like for Rails to set a good example and tone by using better terminology when we can. An easy fix would be to replace our use of whitelist with allowlist and blacklist with denylist.
We can even just use them as verbs directly, as we do with the former terms. So something is allowlisted or denylisted.

Obviously the narrative here is that "black" is associated with negative connotations ("block") and "white" associated with positive connotations ("allow"). So I'd be fascinated to know why they continue to allow Code Pink to seize a positive affirmation space for people of the predominant Western European ethnicity, and refuse to attack the use of "yellow" for cowardice.

It's not just limited to colour of skin - there are a long-term crusades to stop people using "guys" as a generic term for a group of familiar people, "handicapped" for people who are disabled, and "innumerate" to describe Diane Abbott.

It's clear that this is a concerted effort to control the use of language in order to shape ideas - if you're forced to use an approved (restricted) vocabulary, you can't easily express concepts that are regarded as unacceptable by the vocab approvers. And if you think it's going to stop here, I have a bridge to sell you.

I don't have any intrinsic objection to using alternative terminology for master/slave, or for blacklist/whitelist. But I've scrutinised the people calling for this change, and I'm going to keep using the original terminology because civilised people should not yield an inch to these totalitarian fuckers.

If I were tired of employment, I'd be tempted to make a traditional English dish and bring it to my next group potluck. "Oooh, these are tasty, what do you call them?" "Faggots." It would be worth it just to hear the sharp intakes of breath and see the (put-on) outrage. I could even double down: "Are you saying my cultural heritage is offensive?" although of course I'd lose badly by the rules of intersectionality and Victimhood Poker.

[Complete tangent - traditional English terminology for the testicles of an animal is "fries", so you can have "lamb fries", "pig fries" etc. Therefore when someone from an older generation asks you "do you want fries with that?" you might get more than you bargained for.]

[1] All the recent training examples I've seen have had women take a dominant role as problem-solvers, and men nearly exclusively doing the stupid / illegal / morally dubious actions. In the spirit of gender equality, this is me trying to redress the balance.

Prospects for unionizing in Silicon Valley

A topic I've heard increasing buzz about at parties^[1] is the idea that Silicon Valley tech workers should be unionizing. The New York Times was discussing unionization in digital media a month ago:

Daniel Marans, a reporter at HuffPost, said the treatment of employees at digital media companies should not remain stuck in a time when websites were small and scrappy, staffed by younger workers who were happy to see their names in pixels.
"That comes to things like transparency on pay, having a decent pay scale that allows a ladder of sustainability where you can support yourself on such an income, and having due process and a guarantee of severance in the case of layoffs," Mr. Marans said.

Ooh, that looks like a great slate of demands, straight out of the union playbook. Let's unpack it.

The union demands

Transparency on pay

Know what everyone else is paid based on level - no practical scope for varying pay based on the positive or negative impact to the company. Any perceptible skew by race, gender or other minority status gets jumped on. This ties in to the next point very well.

Ladder of sustainability

a.k.a. "pay by seniority". The longer you work here, the more pay you get. No concept of "you haven't materially contributed more - or even as much - this year than you did last year, no rise for you." Per the above point, if you're a mother who's been working short hours to match with your daycare needs then you should be paid as much as a single man who's been employed for the same duration as you but has put in twice the hours. (Also as much as a single woman in the same situation as the man, which is even more invidious, but for some reason the law doesn't care about this situation.) And if you've spent 75% of your working day on Twitter supporting the Resistance Against Trump, or endorsing Chelsea Manning for Senate, that is a perfectly appropriate component of your day job.

Due process

Several states in the USA - including California, home of Silicon Valley - follow employment at will where a company can fire a worker just because they don't like them. They don't have to conduct a specific act of misconduct, it's just "it's not working out between us, goodbye!" There are carefully crafted exceptions in each state's laws, but the basic principle holds true for most employees. This violates one of the fundamental tenets of union laws worldwide - employees should not be fireable except in the most egregious circumstances.

Where you can support yourself on such an income

This refers to the lower-level employees - in practice, contractors - and the minimum wage. The more money union employees earn, the higher the dues that the union can ask for. "You're getting $15/hour? We Fought For Fifteen!" Of course, the employees who lost their jobs because their labor wasn't worth $15/hour don't really benefit from this. But screw them, right?

Guarantee of severance in the case of layoffs

As noted above, unions don't really believe in layoffs unless you're irretrievably conservative or Republican - in which case, fuck you. But if severance is unavoidable, you may be out of luck. I was surprised to learn that even in California, severance pay is not required although in practice it's present in most contracts.

Where is this coming from?

My personal opinion - which you should take with a whole bag of salt - is that this drive is a reaction to the past year's tepid (by Social Justice Warrior standards) reaction by Silicon Valley engineer peons to the cases of "hate speech" by such luminaries as Googler James Damore. The 2014 ousting of Mozilla's Brendon Eich seems to have been a misleading catalyst for social justice organizing: the perception was that the relatively small number of social justice crusaders had disproportionate power to influence media opinions and drive online lynch mobs.

The carefully union-unaffiliated Tech Workers Coalition has been pushing this line for a while:

The Tech Workers Coalition is a home for progressives in tech in the Bay Area. We’re an all-volunteer community organization. Our active participants include workers in the tech industry, members from labor union locals, community organizers, and friends.

"Labor union locals", huh? Why am I not surprised?

And now unions are concerned about the possibility of a nationwide “right-to-work” law which would effectively gut their funding. Tech workers need to stand with service workers in these fights.

Translation: we need tech money to fight the union-gutting right-to-work law. California in particular is not a right to work state - if you want to be a public school teacher, for instance, you're going to pay union dues.

Certain things are safer than others, and safer for different people. An undocumented contract worker is in a very different situation than a salaried citizen worker.

Well, there's the teeny tiny issue that the contract company is clearly breaking the law of the nation, so yes...

For tech, it’d be cool to see the strike weapon on the table. History shows us the tactics that will change the world for the better — the tactics that will not only get rid of Trump, but change the conditions that we’re all forced to live and work under.

Oh, that'll be an interesting one. Tech workers striking - "Facebook will go dark for 24 hours unless FB guarantees contractors the right to employ undocumented workers". How exactly do you expect the tech company leadership to react to this existential threat?

You should also give careful scrutiny to Coworker.org who has been publicly allying with union-oriented Silicon Valley employees. It looks to be funded principally by New Venture Fund (a $315M turnover organization whose turnover doubled from 2014 to 2015, and whose 2016 and 2017 turnover I'd be extremely interested to see. In turn they get "advised" by Arabella Advisors who have a very interesting management team with cited connections to e.g. Barack Obama's secretary of commerce, a company focus on regional food and divestment from fossil fuels.

Will it work?

What do I think? Twitter, Facebook and Google offices in the USA are going to be hit with unionization efforts in the next 12 months, initially as a trial in the most favorable locations but if they succeed then this will be ramped up quickly nationwide. This will be framed as a push to align the companies to approved socially just policies - which their boards mostly favor already - but will be used to leapfrog the activist employees into union-endorsed and -funded positions of influence. That approach neatly nullifies the increasing concern about their lack of material contribution to the company as they spend more time on Twitter and producing social justice memes than actually writing code and making the applications work better.

I wonder, though. The bulk of Silicon Valley engineering employees - who are still the majority of the company - are white, Indian and Chinese males. They are used to ruthless meritocracy from the age of, oh, eight or so. The prospect that some slacker [foreign epithet] could supplant them in promotion or pay just by unfireably hanging around the company while they sweat blood, or block them from a union-favored sinecure by dint of being black / female / transgender / identifying as a dragon is unlikely to be something they'd lie down and accept. I fear that the social justice crusaders are mistaking silence for acceptance, and the settling of accounts after the unionization effort will be (metaphorically) bloody indeed.

I doubt this will get off the ground with Apple. They are notoriously controlling and will both detect and ruthlessly act on any twitches of unionization.

For Amazon, of course, it's much more simple. Any Amazon employee pushing unionization will be deniably but publically killed by an Amazon warehouse robot. I can't imagine Jeff Bezos taking such a challenge to his authority lying down.

TL;DR - there will be a big unionization push for Silicon Valley companies in 2018, and it will go horribly wrong.

[1] You almost certainly don't want to go to the kind of parties I go to. There are no kegs, vol-au-vents, or mini sausage rolls. There's organic Chardonnay, sushi of dubious provenance, and acceptably ethnic cuisine like Vietnamese bánh cuốn and Mexican chilaquiles. I happen to like bánh cuốn, but am under no illusion that the food and beverages are based on what the guests find appealing.

Since we can't challenge diversity policy, how to prevent mistakes?

The James Damore affair at Google has made it very clear that discussion of companies' diversity policy is completely off the table. When I say "discussion" here, I mean "anything other than adulation". I've seen plenty of the latter in the past week. The recent 'letter from Larry Page' in The Economist was a classic example. It was in desperate need of someone tagging it with a number of [citation needed] starting from paragraph 4:

You’re wrong. Your memo was a great example of what’s called “motivated reasoning” — seeking out only the information that supports what you already believe. It was derogatory to women in our industry and elsewhere [CN]. Despite your stated support for diversity and fairness, it demonstrated profound prejudice[CN]. Your chain of reasoning had so many missing links[CN] that it hardly mattered what you based your argument on. We try to hire people who are willing to follow where the facts lead, whatever their preconceptions [CN]. In your case we clearly got it wrong.

Let's accept, for the sake of argument, that random company employees questioning diversity policy is off the table. This is not an obviously unreasonable constraint, given the firestorm from Damore's manifesto. Then here's a question for Silicon Valley diversity (and leadership) types: since we've removed the possibility of employee criticism from your diversity policy, what is your alternative mechanism for de-risking it?

In all other aspects of engineering, we allow - nay, encourage - ideas and implementations to be tested by disinterested parties. As an example, the software engineering design review pits the software design lead against senior engineers from other development and operational teams who have no vested interest in the new software launching, but a very definite interest in the software not being a scaling or operational disaster. They will challenge the design lead with "what if..." and "how have you determined capacity for metric X..." questions, and expect robust answers backed by data. If the design lead's answers fall short, the new software will not progress to implementation without the reviewer concerns being addressed.

Testing is often an adversarial relationship: the testing team tries to figure out ways that new software might break, and craft tests to exploit those avenues. When the test reveals shortcomings in the software, the developer is not expected to say "well, that probably won't happen, we shouldn't worry about it" and blow off the test. Instead they either discuss the requirements with the tester and amend the test if appropriate, or fix their code to handle the test condition.

Netflix's Chaos Monkey subjects a software service to adverse operational conditions. The software designer might assert that the service is "robust" but if Chaos Monkey creates a reasonably foreseeable environment problem (e.g. killing 10% of backend tasks) and the service starts to throw errors at 60% of its queries, it's not Chaos Monkey which is viewed as the problem.

Even checking-in code - an activity as integral to an engineer's day as operating the coffee machine - is adversarial. For any code that hits production, the developer will have to make the code pass a barrage of pre-existing functional and syntax checks, and then still be subject to review by a human who is generally the owner of that section of code. That human expects new check-ins to improve the operational and syntactic quality of the codebase, and will challenge a check-in that falls short. If the contributing engineer asserts something like "you don't appreciate the beauty of the data structure" in reply, they're unlikely to get check-in approval.

Given all this, why should diversity plans and implementations - as a critical component of a software company - be immune to challenge? If we have decided that engineer-authored manifestos are not an appropriate way to critically analyse a company's diversity system then what is the appropriate way?

Please note that there's a good reason why the testing and development teams are different, why representatives from completely different teams are mandatory attendees of design reviews, and why the reviewer of new code should in general not be someone who reports to the person checking in the code. The diversity team - or their policy implementors - should not be the sole responders to challenges about the efficacy of their own systems.

"PC considered harmful" - hand grenade thrown into Valley tech

Wow. I've not seen this amount of heat, light, sound and fury directed towards a minority group since a fat man broke wind loudly over Nagasaki. [I've heard of good taste, and want no part of it.]

Anyone in Silicon Valley tech industry who hasn't been living under a rock has seen the frothing rage on Twitter about a Google employee penning an internal-shared personal doc about their perspective on the company's hiring and training priorities relating to women and "minorities" (which in Silicon Valley almost always refers to Black and 'Latinx' - apparently, very few "woke" people are really interested in the experiences of Native Americans, Koreans, Filipinos or South Americans.) My Twitter tech timeline has exploded in the past 24 hours, almost universally with people demanding the author's head - mostly metaphorically.

Tech site Gizmodo today obtained the text of the document in question. I've read through it, and assuming it's an accurate representation of the original, I can understand the furore - but it has been flagrantly misrepresented. A summary of the author's points is:

1. Google is big on removing unconscious bias, but a lot of Google has a strong leftwards political bias;
2. Left and right political leanings have their own biases; neither are correct, you need both to make a company work well;
3. If you're not a leftist, expressing your opinions at work can be a severely career-limiting move;
4. On average, men and women have behavioural differences which are (list); but these are only averages and don't tell you squat about an individual person;
5. Given those average women's interest, you're going to struggle to get a 50% representation of women in tech, particularly in the higher career and stress levels because of (reasons based on the above list)
6. Doing arbitrary social engineering to achieve this 50% as an end in itself is a bad idea;
7. Google does various things to improve gender and race representation, some of which I think aren't appropriate and might lower the bar [Ed: this was the point I thought least well argued in this doc]
8. Overcoming inbuilt biases is hard; this applies to both sides of the spectrum;
9. The internal climate alienates and suppresses viewpoints of people of a conservative political nature, and this is a bad thing;
10. We should have a more open discussion about what our diversity programs achieve and what do they cost (in a wide sense); make it less uncomfortable to hold and express opinions against the orthodoxy;
11. Indoctrinating people who determine promotion about bias might not have unalloyed benefit for the firm's long-term interests.

Very little of this seems, on the face of it, obviously incorrect or sociopathic. I think the author strayed into moderately unjustified territory on point 7, but otherwise they seemed to be quite reasonable in their arguments and moderate in their conclusions.

I've particularly enjoyed reading tweets and posts from tech woman flaming the original poster for blatant sexism. Really ladies, you should read the post more carefully. He described a contrast of the average male and female behaviors, and took particular pains to point out that this did not say anything about any particular woman's (or man's) effectiveness in a tech role. The behavior biases he described seemed bang on in my experience - and I've met women matching the male biases, and men matching the female biases, but on average the skew is as he has described.

It's almost as if many of the women responding to his post have more bias towards describing their feelings about the ideas, rather than ideas themselves; looking at the "big picture" rather than carefully analysing the detail of what he said. Perish the thought that this reflects the gender biases he described...

Of course, if you challenge the Silicon Valley orthodoxy like this - even if you originally intended for it to be for an internal-only debate - you can expect a certain amount of kick-back. And oh boy, did they get it. I've seen public calls for them to be fired and beaten up, and that was from people using social media accounts associated with their real names. The prevailing theme seemed to be that anyone expressing - or even holding - opinions like this in Silicon Valley was inherently poisonous to the work environment and should be fired forthwith. For goodness' sake, this was one person's opinion, quite mildly expressed. Alphabet (Google's parent company) has 75,000 people. You'd think that an isolated instance of crimethink would not be a big deal, but apparently you'd be very wrong.

Google has just acquired a new Head of Diversity, Danielle Brown from Intel. I don't know if they had one previously, or if this is a new slot, but my goodness this is quite the baptism of fire. She's posted an internal memo which has, inevitably, leaked:

Part of building an open, inclusive environment means fostering a culture in which those with alternative views, including different political views, feel safe sharing their opinions.
But that discourse needs to work alongside the principles of equal employment found in our Code of Conduct, policies, and anti-discrimination laws.

This probably wasn't a bad holding action - it would piss off the conservatives defending every point that the original poster made (because it was hinted as contradictory to equal employment), and it would piss off the outraged mob because it wasn't along the lines of "we threw this person out of the company so fast that his backside made scorch marks along Amphitheater Parkway". You could reasonably call it even-handed. The difference is that the conservatives within Google won't be calling publicly for Ms Brown to reconsider her approach or risk riots in the streets.

I asked a San Francisco based Google engineer buddy what he thought about this. "Are you [censored] kidding me? I wouldn't touch this with a ten foot pole" was a reasonable summary of his reaction. He did note that the author's name was widely known internally and that he viewed it as inevitable that their name would leak, but he'd be damned if he was going to be the one to leak it.

It's also not a little ironic that this comes on the heels of the US Department of Labor accusing Google of discriminating by gender in salaries. If the original author's claims are taken at face value - which is a big "if", to be fair - Google is actually trying to discriminate in favour of women.

For extra points, it's instructive to note the reaction to this in conjunction with President Trump's proposed ban on transgendered troops serving in the military. [Bear with me, I have a point I promise.] One of the grounds for this ban was transgender people having a much higher rate of mental instability (depression, self-harm, suicide attempts) which is not what you want in a front-line military unit where there are plenty of intrinsic causes of instability. We see one bloke in Google writes a document, and every trans blogger I know of explodes in a frenzy of rage and demands for his head - despite the fact that he didn't mention transgender issues at all in the manifesto. One can only imagine what would happen if the author had drawn attention to the relatively high proportion of male-to-female trans people among the female engineering population and ask what it meant...

The modern day lynch mob is alive and well, and it seems to be driven by dyed-in-the-wool Democratic voters against anyone daring to express an opinion contrary to today's right-think on gender and racial issues. Plus ça change, plus la même chose.

Scentrics finds that security is hard

Two years ago I wrote about Scentrics and their "Key Man" security proposal. I wondered idly what had happened there so did some Googling. Turns out that I'm the top two hits for [scentrics key man] which is heart-warming for me but suggests that their world-beating security patent might have sunk like a stone...

I went to their website www.scentrics.com and noted that it didn't redirect to https. I tried https://www.scentrics.com and lo! Chrome's Red "Not secure" Warning of Death appears. Seems that Scentrics can't even secure their website, which is not a little ironic when their home page trumpets "Secure with Scentrics".

All the pages on the site - even "Overview and Vision" and "Careers" - are hidden behind a sign-on box, declaring the website "invitation only" and inviting you to contact "admin@scentrics.com" if you'd like access. You can view headers, but that's about it. You wonder why they would be so sensitive about exposing information like that.

The 2016 news included a nugget from the Daily Telegraph in June:

Scentrics is poised to seek new funding that would value the company at more than $1 billion as it prepares to rollout its infrastructure for the first time.

"Poised", huh? I like that. I read that as "not yet ready". I also like the uncritical write-up of the company's pitch:

Individual messages and documents sent over the internet can be unlocked without compromising the overall security of the network, according to Scentrics's pitch to operators and governments.

Remember that this essentially involved encrypting one copy of a message with the recipient's public key, and another with a government/agency public key, and storing the latter to give the agency access on demand. The government and security agencies involved might not think that this "compromises" the overall security of the network, but as a consumer of the network's function I can assure them that I'd feel very differently. And of course for this to be effective all network users would have to use a very small ecosystem of only approved apps / browsers which implemented this dual encryption, and maintained the central repository of government-friendly encrypted messages. I'm sure there's no risk of systematic system compromise there by insiders at all.

Companies House shows three officers plus a secretarial company including our old friend Guruparan "Paran" Chandrasekaran. Looks like Sir Francis Mackay, David Rapoport and Dr. Thaksin Shinawatra resigned since 2014, which is interesting because the latter gent used to be the Prime Minister of Thailand, and Scentrics trumpted his role in the Telegraph piece, but as of 1 month ago he's out of his company role.

According to their June 2015 accounts they have about GBP4.2M in net assets, looks like they had an infusion of about GBP4.5M during the year. Going from this to a $1bn valuation seems... optimistic.

Update: Looks like Scentrics are diving into Singapore with advertisements for Project Manager and Devops roles there. This seems to be part of the Singapore government's "Smart Nation" project for a unified network in Singapore:

• A Smart Nation is one where people are empowered by technology to lead meaningful and fulfilled lives.
• A Smart Nation harnesses the power of networks, data and info-comm technologies to improve living, create economic opportunity and build a closer community.
• A Smart Nation is built not by Government, but by all of us - citizens, companies, agencies. This website chronicles some of our endeavours and future directions.

Cutting through the marketing speak, Singaporeans will be using a government-provided network for all services including personal and business communication. With Scentrics playing a role, the benevolent semi-dictatorship of Singapore will be able to snoop on all its citizens' internal communications at will.

Scentrics seems to be very comfortable enabling a government's surveillance on its citizens. I wonder how this is going to work out for them long-term given the distinctly libertarian tilt of most software engineers.

[Disclaimer: no share position in Scentrics. Financially I don't care if they live or die. Personally, I'd incline towards the latter.]

Don't blame the tech industry for its "lack of diversity"

Tekla S. Perry, who's experienced enough in the technology world to know better, wrote a provocative piece in IEEE Spectrum this week titled "Why Isn't the Tech Industry Doing Better on Diversity? It's Google's and Facebook's Fault". This sprang from a discussion at "Inclusion In Silicon Valley" where Leslie Miley, Slack's director of engineering, excoriated Bay Area tech companies for their alleged lack of inclusion:

You come to Silicon Valley and you don't see people that look like me in positions of power [Miley is black]. If that's not hostile, what is?

You don't see Chinese Americans or Indian Americans in positions of power in the Federal government, despite 8 years of a black president. If that's not hostile to Chinese and Indian Americans, what is?

Leslie Miley is a mendacious asshole. There are many legitimate points to make about the disproportionately small number of black software engineers, and the horrendous educational and societal failings behind that - and let's be clear, prejudice against academically successful black engineers is a real thing from both the black and white communities - but Leslie's point is not one of those. He is jumping from "X is not happening" (observation) to "X must be being blocked by Y" (assumption). You'd think that a competent engineer would be better acquainted with logical reasoning. But looking at Miley's LinkedIn profile he's only spent a series of 2-3 year stints at a list of major tech companies (Google, Apple, Twitter) in engineering management roles; since you spend 3-6 months coming up to speed with a job like that, and assume you draw down effort in the 3 months looking for a replacement job before you leave, his actual engineering experience doesn't seem that great, and you wonder why he kept leaving each firm before his stock options started to vest in quantity... (This is of course the "play the man, not the ball" approach to argument, which is intellectually facile but no less well founded that Miley's approach to argument.)

I've said this before but let's say it again. The main reason that people of Afro-Caribbean descent are under-represented in the software engineering industry is because the dominant education requirement for that industry is a bachelor's degree in a numerical subject (STEM), and such people are correspondingly under-represented in that qualification bucket. Such under-representation is a major issue that needs fixing, but it's happening way before the Silicon Valley and other engineering companies get involved. There's a secondary issue that engineering companies in general should get better at finding bright numerate non-STEM-degree holders who will do well in software engineering with a small investment of training, but that's another blog post entirely - and in any case, Silicon Valley big firms do spend time and money looking in that general area.

It's not just Miley who's making dumb remarks at this diversity love-fest, of course:

The lack of diversity stems from hidden and systemic bias, believes Monique Woodard, a partner in 500 startups. "If you turned off the imported talent, would you look to Oakland and Atlanta? I'm not sure people would," she said.

This is bollocks on stilts, but not just for the reasons you think. Oakland is stuffed full of Bay Area tech workers, especially junior engineers. They live there because it is relatively cheap compared to San Francisco, Palo Alto, San Jose, Milpitas etc. Tech companies recruit people from Oakland all the gosh-darn time. What Monique Woodard means is that she doesn't believe that tech companies will go looking for the black talent in Oakland and Atlanta. Why isn't she saying this explicitly? You be the judge.

"Changing the practices that perpetuate the overwhelmingly white and male character of the Silicon Valley workforce are not going to be easy"

Male: yep. White: nope. In Silicon Valley, Caucasians are actually under-represented per the general population; Chinese and Indians are significantly overrepresented. In my experience, people who openly identify as gay or transgender are also markedly over-represented. By many reasonable measures, Silicon Valley is one of the most diverse environments there is - there is a huge population of people whose national original is not the USA, and they aren't just Indians and Chinese: there are substantial Russian, Korean, Polish, Filipino, Vietnamese and other nationalities.

What Ms. Woodard is actually saying is: "there aren't enough engineers with dark skin - excluding Indians - in Silicon Valley." Well, Ms. Woodard, why is that? Is there a peculiar conspiracy in hiring where the recruiters and hiring deciders are wide open to all sorts of people except those who are of Afro-Caribbean extraction? Is that what you are saying, or is it such a ridiculous notion that you have to resort to camouflaging it behind the umbrella of "diversity"?

Behind Miley's comments, at least, there's a nugget of good sense. The competition for engineers in Silicon Valley and its environs, and to some extent other places like Seattle (Microsoft/Amazon) and New York (Big Finance) is intense. If big firms want to find a cheaper source of good engineers then they should look at other major cities, such as Atlanta, Dallas, Austin. This is something of a risk though: you need to start a new engineering office, which means recruiting many tens of new engineers in addition to migrating some of your existing senior engineers down there to help build and train the teams, reinforce company culture and keep strong communication with the root offices. Up until now, this has been more of a risk than just upping the game in recruiting from the Bay: I suspect soon the numbers will cross a threshold that makes new engineering offices sufficiently financially attractive to be worth a try.

Bringing in new engineers from Republican states such as Texas and Georgia is also excellent for increasing diversity in the heavily Democratic (and worse, Californian) engineering cohorts of Silicon Valley. Yet, why is it that I suspect that Miley, Woodard et al don't regard that kind of diversity as desirable?

neveragain.tech virtue signalling

In the past couple of days I've seen all manner of prompts to add my name to the petition at neveragain.tech, solemnly swearing to:

1. refuse to participate in the creation of databases of identifying information for the United States government to target individuals based on race, religion, or national origin.
2. advocate within our organizations:
• to minimize the collection and retention of data that would facilitate ethnic or religious targeting.
• to scale back existing datasets with unnecessary racial, ethnic, and national origin data.
to responsibly destroy high-risk datasets and backups.
• to implement security and privacy best practices, in particular, for end-to-end encryption to be the default wherever possible. to demand appropriate legal process should the government request that we turn over user data collected by our organization, even in small amounts.
3. if I discover misuse of data that I consider illegal or unethical in my organizations:
• I will work with our colleagues and leaders to correct it.
• If we cannot stop these practices, we will exercise our rights and responsibilities to speak out publicly and engage in responsible whistleblowing without endangering users.
• If we have the authority to do so, we will use all available legal defenses to stop these practices.
• If we do not have such authority, and our organizations force us to engage in such misuse, we will resign from our positions rather than comply.
4. raise awareness and ask critical questions about the responsible and fair use of data and algorithms beyond my organization and our industry.

The more perceptive readers will be surprised at how closely this declaration follows the election of Donald Trump as President of the USA, and wonder why - following the past 8 years of progressive weaponization of the Federal government - the tech industry has suddenly decided that unlimited government power is A Bad Thing to be strenuously resisted.

OK, maybe it's not much of a mystery.

Seriously though, one has to wonder why so many tecchies - who are, on average, very intelligent and somewhat resistant to regular bullshit - are signing this petition. The classic excuse comes from the role of IBM's equipment in the Holocaust, used by the Nazis to process the data around selection and slaughter of Jews in Europe. IBM itself acknowledges its role:

It has been known for decades that the Nazis used Hollerith equipment and that IBM's German subsidiary during the 1930s -- Deutsche Hollerith Maschinen GmbH (Dehomag) -- supplied Hollerith equipment. As with hundreds of foreign-owned companies that did business in Germany at that time, Dehomag came under the control of Nazi authorities prior to and during World War II. It is also widely known that Thomas J. Watson, Sr., received and subsequently repudiated and returned a medal presented to him by the German government for his role in global economic relations.

It's a bit unfair to single out IBM here. The premise is that equipment from an IBM-owned subsidiary was instrumental to the Nazis being able to kill Jews more efficiently. Nowadays, how would we feel if Syria's Bashar Assad used an Excel spreadsheet or two to organise slaughter of non-Alawite citizens? I'm fairly sure that Microsoft's Excel developers couldn't realistically be held accountable for this. Even if a Microsoft sales rep sold a 1000-seat Excel license to the Syrian regime, it would be a bit of a stretch to blame them for any resulting massacre. After all, the regime could always use OpenOffice for a free-as-in-beer-and-freedom solution to programmatic pogrom.

As you might expect from a Silicon Valley initiative, this is primarily intended as strenuous virtue-signalling. "Look at me, how right-thinking I am and how willing to prevent persecution of minorities!" Really though, it will have zero effect. The US Government does not contract out to random Silicon Valley firms for immigration and related database work. They have their own information systems for this, developed at horrific expense and timescales by the Beltway Bandit consulting firms and government IT workers. The US Citizenship and Immigration Services department isn't going to ask Twitter or a San Francisco start-up to develop a new immigrant tracking system - even though I suspect they'd get one with 10% of the downtime and 20% of the cost of the one that the Bandits will develop for them.

The most plausible concern of the signatories is the existing social graph and personally identifiable information in systems like Facebook and Twitter. Religion and national origin isn't stored systematically, and visa status isn't stored at all, but from analysis of posts and relationship activities I can imagine that you could fairly reliably infer areas of the relationship graph that are likely to be e.g. Guatemalan in origin and using Latin American Spanish as their primary language, working in low-wage industries, and physically located in Southern California (checking in from IPs known to be in LA and its environment). If you wanted to identify a pool of likely illegal immigrants, that would be a good place to start. Since Facebook already has this data, and sells access to parts of their information to advertisers, I wonder what these signatories are going to do about it?

$20 says "not a damn thing." They like their jobs and status too much. They won't find other companies as accepting of their social activism and public posturing. They won't take on new jobs targeting minorities, but then no-one sane is going to ask them to take on that kind of work because the D.C. consulting firms want the money instead and have lobbyists ensuring that they'll get it.

Expensive integer overflows, part N+1

Now the European Space Agency has published its preliminary report into what happened with the Schiaparelli lander, it confirms what many had suspected:

As Schiaparelli descended under its parachute, its radar Doppler altimeter functioned correctly and the measurements were included in the guidance, navigation and control system. However, saturation – maximum measurement – of the Inertial Measurement Unit (IMU) had occurred shortly after the parachute deployment. The IMU measures the rotation rates of the vehicle. Its output was generally as predicted except for this event, which persisted for about one second – longer than would be expected. [My italics]

This is a classic software mistake - of which more later - where a stored value becomes too large for its storage slot. The lander was spinning faster than its programmers had estimated, and the measured rotation speed exceeded the maximum value which the control software was designed to store and process.

When merged into the navigation system, the erroneous information generated an estimated altitude that was negative – that is, below ground level.

The stream of estimated altitude reading would have looked something like "4.0km... 3.9km... 3.8km... -200km". Since the most recent value was below the "cut off parachute, you're about to land" altitude, the lander obligingly cut off its parachute, gave a brief fire of the braking thrusters, and completed the rest of its descent under Mars' gravitational acceleration of 3.8m/s^2. That's a lot weaker than Earth's, but 3.7km of freefall gave the lander plenty of time to accelerate; a back-of-the-envelope calculation (v^2 = 2as) suggests a terminal velocity of 167 m/s, minus effects of drag.

Well, there goes $250M down the drain. How did the excessive rotation speed cause all this to happen?

When dealing with signed integers, if - for instance - you are using 16 bits to store a value then the classic two's-complement representation can store values between -32768 and +32767 in those bits. If you add 1 to the stored value 32767 then the effect is that the stored value "wraps around" to -32768; sometimes this is what you actually want to happen, but most of the time it isn't. As a result, everyone writing software knows about integer overflow, and is supposed to take account of it while writing code. Some programming languages (e.g. C, Java, Go) require you to manually check that this won't happen; code for this might look like:

/* Will not work if b is negative */
if (INT16_MAX - b >= a) {
   /* a + b will fit */
   result = a + b
} else {
   /* a + b will overflow, return the biggest
    * positive value we can
    */
   result = INT16_MAX
}

Other languages (e.g. Ada) allow you to trap this in a run-time exception, such as Constraint_Error. When this exception arises, you know you've hit an overflow and can have some additional logic to handle it appropriately. The key point is that you need to consider that this situation may arise, and plan to detect it and handle it appropriately. Simply hoping that the situation won't arise is not enough.

This is why the "longer than would be expected" line in the ESA report particularly annoys me - the software authors shouldn't have been "expecting" anything, they should have had an actual plan to handle out-of-expected-value sensors. They could have capped the value at its expected max, they could have rejected the use of that particular sensor and used a less accurate calculation omitting that sensor's value, they could have bounded the calculation's result based on the last known good altitude and velocity - there are many options. But they should have done something.

Reading the technical specs of the Schiaparelli Mars Lander, the interesting bit is the Guidance, Navigation and Control system (GNC). There are several instruments used to collect navigational data: inertial navigation systems, accelerometers and a radar altimeter. The signals from these instruments are collected, processed through analogue-to-digital conversion and then sent to the spacecraft. The spec proudly announces:

Overall, EDM's GNC system achieves an altitude error of under 0.7 meters

Apparently, the altitude error margin is a teeny bit larger than that if you don't process the data robustly.

What's particularly tragic is that arithmetic overflow has been well established as a failure mode for ESA space flight for more than 20 years. The canonical example is the Ariane 5 failure of 4th June 1996 where ESA's new Ariane 5 rocket went out of control shortly after launch and had to be destroyed, sending $500M of rocket and payload up in smoke. The root cause was an overflow while converting a 64 bit floating point number to a 16 bit integer. In that case, the software authors had actually explicitly identified the risk of overflow in 7 places of the code, but for some reason only added error handling code for 4 of them. One of the remaining cases was triggered, and "foom!"

It's always easy in hindsight to criticise a software design after an accident, but in the case of Schiaparelli it seems reasonable to have expected a certain amount of foresight from the developers.

ESA's David Parker notes "...we will have learned much from Schiaparelli that will directly contribute to the second ExoMars mission being developed with our international partners for launch in 2020." I hope that's true, because they don't seem to have learned very much from Ariane 5.

Silicon Valley in the Time of Trump

The past few days have given me a great view into how the famously liberal population of the Bay Area has taken the election of Donald Trump. "Not well" is fair, but a yuuuuge understatement.

Do you know what California's principal export is? Whine.

The Bay Area is probably the most pro-Clinton anti-Trump group outside the island of Manhattan, and the residents were never going to be entirely happy with a Trump victory. I predicted butthurt-ness, and was I ever right. However even I, with my jaundiced view of human nature, never expected the level of rage and opprobrium directed at Trump and his voting enablers. So far I've seen - not heard but actually seen written on group emails and forums - the following:

• claims of suicidal feelings, particularly from trans and gender-fluid folks;
• assertions that anyone voting for Trump needs to publicly denounce Trump's perceived opinions about Black Lives Matter, Hispanics, gays (wut?) and immigrants;
• statements that anyone voting for Trump needs to go work for another company;
• room-sized group hugs to support each other post-election; and
• claims that Trump and Pence wanted to electrocute people who were gay or trans.

Thank goodness Trump has elephant-thick skin, because there's probably enough libel in every Bay Area tech company's emails to pay for the building of another Trump Tower.

The straw that broke the camel's back for me was a bundle of complaints around the theme:

"I was hoping to teach my girls that, if you work hard and dream big, you can be anything you want to be. I would like to thank 2016 for putting me right."

It seems that a large number of people were going to use "Hillary as first woman president" as the totem for their children to show that the glass ceiling had been shattered. While I'm all in favour of showing children role models, is Hillary really the model you want to use?

I actually found it inspiring, in a way. The lesson I took from the election was that if you are a woman, even if you are a revolting and corrupt human being, you can make it to within a gnat's chuff of being the President of the United States, and your party organisation will happily screw over men to help you get its nomination. It wouldn't have taken much of a vote change in one or two swing states for Hillary to be elected, at which point I guarantee that no-one on the Dems side would be talking about upsetting the electoral college applecart.

Hillary is (of course) not happy and blames FBI Director Comey for her narrow defeat:

But our analysis is that [FBI Director James B.] Comey's letter raising doubts that were groundless, baseless, proven to be, stopped our momentum,” she said. “We dropped, and we had to keep really pushing ahead to regain our advantage — which going into the last weekend, we had."

She's right, of course. Comey's letter was quite possibly enough to cause Hillary voters in key states to stay home on polling day.

On the other hand, there were many other what-ifs, any one of which was probably enough to get her elected:

• what if she had actually achieved something of note as Secretary of State?
• what if she and Bill hadn't gone around the world soliciting hundreds of millions of dollars from various dubious countries and individuals?
• what if she were actually personally likeable?
• what if she'd not blown her chance to land a kill-shot on The Donald in the debates?
• what if she'd insisted that the DNC not put its thumb on the scales, and instead beat Bernie fairly in the nomination?

All these were in her control, so to blame solely Comey for her loss seems rather obtuse.

And on the flip side, what if Comey had taken the - apparently quite reasonable - step to indict her for her recklessness in running her own email server and exposing any amount of State classified material to any intelligence service worth its name? Isn't she grateful to him for not doing that, at least?

Analysing the blue-red hat problem in the face of user error

Everyone knows computers are getting smarter - unless they're being programmed by a major corporation for a government contract - but there has recently been another leap in the level of smart. DeepMind (now part of Google) has built an AI that has successfully deduced the optimal solution to the hat problem:

100 prisoners stand in line, one in front of the other. Each wears either a red hat or a blue hat. Every prisoner can see the hats of the people in front – but not their own hat, or the hats worn by anyone behind. Starting at the back of the line, a prison guard asks each prisoner the colour of their hat. If they answer correctly, they will be pardoned [and if not, executed]. Before lining up, the prisoners confer on a strategy to help them. What should they do?

Tricky, n'est ce pas?

The obvious part first: the first prisoner to answer, whom we'll designate number 1, has no information about his hat colour. Assuming blue and red hats are assigned with equal probability, he can answer either "red" or "blue" with a 50% chance of success and 50% chance of getting executed; he has no better strategy for self-survival. What about the other prisoners?

Applying information theory, our system has 100 binary bits of state - 100 people, each with 1 bit of state relating to whether their hat is blue or not. We generate 99 bits of knowledge about that state as the hat-wearers give answers. So the maximum we can expect to discover reliably is 99/100 hat values. How can we get close to this?

If everyone just guesses their own hat colour randomly, or everyone says "blue", or everyone says "red", then on average 50% of people survive. How to do better? We need to communicate information to people further down their line about their hat colour.

Let's get the first 50 people in line to tell the next 50 people in line about their hat colour. Person 1 announces the hat colour of person 51, person 2 of person 52 and so on. So the last 50 people are guaranteed to survive because they have been told their hat colour. The first 50 people each have a 50-50 chance of survival because the colour they "guess" has no necessary relation to the colour of their hat. On average 25 of them survive, giving an average survival of 75% of people.

The DeepMind algorithm relies on an insight based on the concept of parity: an 0/1 value encapsulating critical state, in this case the number of blue hats seen and guessed, modulo 2. The first user counts the number of blue hats seen and says "blue" if that number is even, and "red" if odd. He still has a 50-50 chance of survival because he has no information about his hat. The second user counts the number of blue hats. If even, and the first person said "blue", then he and the first person both saw the same number of blue hats - so his hat must be red. If even, and the first person said "red", his hat must be blue because it changed the number of blue hats seen between the first person and him. Similar reasoning on the odd case means that he can announce his hat colour with full confidence.

What about person 3? He has to listen to person 1 and person 2, and observe the hat colours in front of him, to deduce whether his hat is blue; his strategy, which works for all others after him too, is to add the parity values (0 for blue, 1 for red) for heard and seen hats modulo 2, and if 0 then announce "blue", if 1 then announce "red". Follow this down the line, and persons 2 through 100 are guaranteed survival while person 1 has a 50-50 chance, for an average 99.5% survival rate.

Of course, this is a fairly complicated algorithm. What if someone mis-counts - what effect does it have? We don't want a fragile algorithm where one person's error can mess up everyone else's calculations, such as with "Chinese whispers." Luckily, a bit of thought (confirmed by experiment) shows us that both the future-casting and parity approaches are resilient to individual error. For future-casting, if one of the first 50 people makes an error then it makes no difference to their chance of survival but their correspondent in the second half of the line is doomed. If one of the second 50 people makes an error then they are doomed unless their correspondent also makes a mistake - generally unlikely, a 10% chance. So if 10% of users make errors then the approximate number of survivors is (75 - 10) + 1, i.e. 66%.

Surprisingly, the parity approach is also robust. It turns out that if user N makes a mistake then they doom themselves, and also doom user N+1 who relies on user N's calculation. But because both user N and N+1 make erroneous guesses, this brings the parity value back in line for user N+2, whose guess will be correct (absent any other errors). So the approximate number of survivors given a 10% error rate is 99.5 - 10*2 = 79.5%

Here's Python code to test the various algorithms: save it as "hats.py" and run it (e.g. "chmod 0755 hats.py ; ./hats.py" on OS X or Linux). It runs 10 trials of 100 people and reports the average number of survivors, based on a 10% error rate in hat wearers following their strategy. Default strategy is the parity algorithm.

#!/usr/bin/python

import random

person_count = 100
half_person_count = person_count / 2
# Hat choices
hat_choices = ['r','b']
hat_opposite = {'b':'r', 'r':'b'}
# 10% error rate in guesses
error_rate = 0.1

def guess_constant(heard_guesses, seen_hats):
    return 'b'

def guess_random(heard_guesses, seen_hats):
    return random.choice(hat_choices)

def guess_future(heard_guesses, seen_hats):
    """ First half of list calls out hat of correspondent in second half of list """
    full_list = heard_guesses + ['x'] + seen_hats
    my_index = len(heard_guesses)
    if my_index < half_person_count:
        # Call out the hat of the person in the second half of the list, hope same as mine
        return full_list[my_index+half_person_count]
    else:
        # Remember what was called out by my corresponding person in first half of list
        return heard_guesses[my_index - half_person_count]

def guess_parity(heard_guesses, seen_hats):
    """ Measure heard and seen parity of blue hats, call out blue for even, red for odd."""
    heard_blue_count = len([g for g in heard_guesses if g == 'b'])
    seen_blue_count = len([s for s in seen_hats if s == 'b'])
    if (heard_blue_count + seen_blue_count) % 2 == 0:
        return 'b'
    else:
        return 'r'

def run_test(guess_fun):
    hat_list = [ random.choice(hat_choices) for i in range(0, person_count) ]
    print "Actual: " + "".join(hat_list)
    answer_list = []
    score_list = []
    error_list = []
    correct = 0
    for i in range(0, person_count):
        guess = guess_fun(answer_list, hat_list[i+1:])
        if random.random() < error_rate:
            guess = hat_opposite[guess]
            error_list.append('X')
        else:
            error_list.append('-')
        answer_list.append(guess)
        if guess == hat_list[i]:
            correct += 1
            score_list.append('-')
        else:
            score_list.append('X')
    print "Called: " + "".join(answer_list)
    print "Score:  " + "".join(score_list)
    print "Errors: " + "".join(error_list)
    print "%d correct" % correct
    return correct

if __name__ == "__main__":
    trial_count = 10
    correct_total = 0
    for i in range(0, trial_count):
        print "\nTrial %d" % (i+1)
        correct_total += run_test(guess_parity)
    print "\nAverage correct: %d" % (correct_total / trial_count)

You can change the "guess_parity" value in the run_test() invocation on the penultimate line to "guess_future" for the "warn the second half" strategy, or "guess_random" for the random choice.

This is a lousy problem for use in software engineering job interviews, by the way. It's a famous problem, so candidates who have heard it are at a major advantage to those who haven't. It relies on a key and non-obvious insight. A candidate who hasn't encountered the problem before and solves it gives a very strong "hire" signal, but a candidate who fails to find the optimal solution could still be a valid hire. The least worst way to assess candidates based on this problem is whether they can write code to evaluate these algorithms, once the algorithms are described to them.

Bringing the diversity of car manufacturers to Silicon Valley

I should start this blog by warning the reader of my prejudice towards Jesse Jackson. I think he's a fairly despicable human being; a race hustler who is standing on the shoulders of the giants of the US Civil Rights Movement (Parks, MLK et al) to further his own petty shakedown rackets and attempts to gain political power.

That said, let's examine his latest crusade: bringing the focus of the US Equal Employment Opportunity Commission onto the diversity disaster area that is Silicon Valley.

"The government has a role to play" in ensuring that women and minorities are fairly represented in the tech workforce, Jackson told a USA TODAY editorial board meeting. He said the U.S. Equal Employment Opportunity Commission needs to examine Silicon Valley's employment contracts.

The trigger for this appears to be Twitter's release of workforce diversity statistics (select the Twitter tab, the default is Yahoo). They show a global 70% male workforce with 50% white, 29% Asian, 3% Hispanic, 2% black, 3% mixed and 4% other. Jackson claims that this is proof that the EEOC needs to step in. Because what could possibly go wrong with that?

The gaping hole in USA Today's argument:

Of Twitter's U.S. employees, only 3% are Hispanic and 5% black, but those groups along with Asian Americans account for 41% of its U.S. users.

Wow, talk about a misleading stat. I assume "mixed" is rolled in with "black" to make the 5%, using the Halle Berry "one drop of blood" theory, but note that if you add Asian Americans in it becomes:

Of Twitter's U.S. employees, only 3% are Hispanic and 5% black plus 29% Asian making 37% total, but those groups account for 41% of its U.S. users.

Hmm, that's a little bit different, no?

Since Silicon Valley is in focus, let's look at the demographics in the Bay Area from the 2010 census:

• 52.5% White including white Hispanic
• 6.7% non-Hispanic African American
• 23.3% Asian (7.9% Chinese, 5.1% Filipino, 3.3% Indian, 2.5% Vietnamese, 1.0% Korean, 0.9% Japanese plus rounding errors for others)
• 23.5% Hispanic or Latino of any race (17.9% Mexican, 1.3% Salvadoran)
• 5.4% from two or more races
• 10.8% from "other race"

The categories aren't an exact overlap, but you'll note that whites are almost exactly represented in Twitter as in the Bay Area population. Asians are over-represented in Twitter (29% vs 23%), African Americans under-represented (7% vs 5%) but the real under-representation is Hispanic (24% vs 3%). Why is that? Hispanics in California are disproportionately over-represented in the menial jobs currently. This is starting to change a little with the new generation of America-born Hispanic kids but their parents can't generally afford top-tier universities for engineering or CS courses so it'll be at least one more generation before they start to appear in the engineering/CS student pool for recruitment.

The really disgusting thing about Jackson is when you realize what he is actually implying - that Silicon Valley engineers systematically discriminate in hiring against black and Hispanic engineers just on the basis of their skin colour. Yet somehow they discriminate in favour of Chinese and Indian engineers on the same basis - so they're racist, but very narrowly so. What Jackson fails to point out - because it wrecks his entire thesis - is that the real demographic problem is in the pool of engineers eligible for these jobs. African-American and Hispanic students are massively under-represented here. This isn't Twitter's fault, or Google's fault, or Facebook, Apple, or IBM. The problem starts at the awful public (state) schools which poor American students attend and which completely fail to give them any reasonable preparation for university courses with objective (numeric) subjects - maths, computer science, physics - that are the grounding for computer science careers. But delving into those facts might take an enquiry into unionised teaching and teacher tenure rules, and I'd bet Jesse's union buddies wouldn't like that.

The engineers I know who conduct interviews for computing firms day in, day out, are overwhelmingly thoughtful and fair individuals who strive to give any new candidate a fair go at getting hired. Even the occasional monster among them is uniformly brutal - white, Chinese and Indian candidates have as brutually intellectual an interview as Hispanic and black candidates. If Jackson were to appear before those engineers and accuse them explicitly of bad-faith prejudice against black and Hispanic candidates, they'd probably punch him.

The real problem in Silicon Valley demographics is the male vs female disparity in engineering. There are plenty of good, smart, talented women - they're just not going into engineering. Until we figure out why, we're missing out on a heck of a lot of talent. But Jackson is not pushing this angle - perhaps he's figured out that he has nothing to say on the subject and so there's no money in it for him and his cronies.

I can do no better than conclude with Jackson's own words:

The former two-time Democratic presidential candidate said he'll continue pushing the issue and has no plans to retire. "The struggle for emancipation is my life," he said in an interview. "It's my calling."

Well it's your revenue stream, at least. God, that man gets on my wick.