The Twitter Whistleblower report - how bad was Twitter, really?

Prompted by a post by everyone's favourite Portugal-based squirrel-torturing blogger, Tim Worstall, I thought I'd dive into the practical implications of all the (frankly, horrendous) technical, security and privacy problems that Twitter was identified as having before Elon Musk rocked up as owner and CEO.

Usual disclaimer: I'm going by the reports. Reality might be different. I cite where I can.

For background: both USA and European authorities take a dim view of corporate access to, and usage of, individual user data. Remember the European "ePrivacy Directive"? Also known as the "'f+ck these annoying cookie pop-ups' law"... Governments in both Europe and the USA are keenly interested in companies tracking individual users' activities, though my personal opinion is that they're just jealous; they'd like to do it too, but they're just not competent. Anyway, a company doing individual tracking at large scale for profit - Twitter, Google, YouTube, Meta, Amazon - attracts their attention, and their laws.

Security

Let's talk about security - and, more importantly, access to secure data. A fundamental principle of security is "least privilege" - everyone should have the smallest set of access privileges to be able to do their job. You could argue that 5000+ people in Twitter "need" to be able to change things in production at some point to do their jobs, but they certainly don't "need" to have always-on, cross-production access. Not least, because someone running a command they found on an internal playbook as an experiment, could easily break a large chunk of the service. But don't rely on me, ask their job candidates:

Twitter's practice was a huge red flag for job candidates, who universally expressed disbelief. One Vice President of Information Technology [his current role, not the target role] considered withdrawing his application on the (accurate) rationale that Twitter's lack of basic engineering hygiene in their arrangement presaged major headaches.

Hire that guy.

Certainly, every company is far from perfect in this area, but those with regulators are continually seeking to narrow the number of people with access, and the scope of access those people have. Twitter pre-Musk clearly did not give a crap about the count and scope of access. One can only imagine why; were they, for instance, relying on a large base of pre-approved employees to intercept and downgrade/block opinions outside the mainstream? How would we tell if this were not the case? Can Twitter show that they were engaged in a systematic reduction of number and scope of access to production? If not, who will be held to account?

Auditing

Control is one thing - but at least, if a human performs an action in the production environment (change, or query), that action should at least be logged, so future audit can see what happened. This is not a high bar, but was apparently too high for pre-2022 Twitter:

There was no logging of who went into the production environment or what they did.

FFS
To make clear the implications: in general, there was no way of finding out who queried (for their own purposes) or changed (deleted posts, down-rated users, etc) the production environment at any particular time. "Why did [event] happen?" "Beats the hell out of me, someone probably changed something." "Who? When?" "No idea."

This is particularly interesting because Twitter's Chief Information Security Officer - who resigned post-Musk - was also their former head of privacy engineering, and before that, apparently, global lead of privacy technology at Google. One could only imagine what that implies.

Control

There is also a wide range of engineering issues. Data integrity (not losing user-entered data) was obviously a critical issue, but Twitter had been aware for a while that they teetered on the edge of a catastrophic production data loss:

even a temporary but overlapping outage of a small number of datacenters would likely [my italics] result in the service going offline for weeks, months, or permanently.

This is not quite as bad as it first seems. After a year or so in operation, companies have a fairly good idea what happens with a datacenter outage - because they're more frequent than you imagine. Say, Henry the intern accidently leans against the Big Red Button on the datacenter floor, that cuts power to everywhere. Or you do a generator test, only to discover that a family of endangered hawks have made their nest in the generator housing for Floor 2... So you get used to (relatively) small-scale interruptions.

If you want to run a global service, though, you need to be able to tolerate single site outages as routine, and multiple site outages (which turn out to be inevitable) have to be managed within the general bounds of your service's promised availability - and latency, and data availability. Even if all your physical locations are very separate, there will inevitably be common cause failures - not least, when you're pushing binary or config changes to them. So, don't wait for these events to sneak up on you - rather, anticipate them.

This means that you have to plan for, and practice these events. If you're not doing so, than a) it will be obvious to anyone asking questions in this area, and b) when things inevitably do run off the rails, there will be bits of burning infrastructure scattered everywhere, around the highly-paid morons who are busy writing memos to cover their asses: "how could we have foreseen this particular event? Clearly, it wasn't our fault, but pay us 20% extra and we might catch or mitigate the next such event."

Go looking for those people. Fire them, and throw them into a den of hungry pigs.

Leaving the doors open

By far the most horrific aspect, however, was the general relaxed attitude about government agencies - and heaven only knows what other NGOs, cabals, and individuals - having under-the-table access to Twitter's data. Just the tolerance of user-installed spyware on privileged devices would be enough for any sane security engineer to be tearing out their hair, but actually letting in individuals known to be employed by foreign - and even domestic - governments for the purposes of obtaining intelligence information, and potentially affecting the flow of information to their and other countries... one is lost for words.

At some stage, Twitter had to either grow up, or close down. Under Dorsey's crew, the latter was inevitable - and likely not far away. It's still too early to tell if Musk can get them to option 1, but there's still hope.

The (Karin Jean-)Pierre Principle

Many of my readers (at least two!) may be familar with the long-established Peter Principle which states that in many organizations, people rise to - and are left at - the level of their incompetence. And I'm sure that we can think of many examples in our own corporate experiences.

Now, dear readers, I would like to propose the (Karin Jean-)Pierre Principle which states:

Promoting someone based on their identity, rather than ability, leads almost inevitably to incompetence which you are unable to fire.

For evidence, I'd like to offer the examples:

• The titular Karin Jean-Pierre, promoted to White House Press Sec based on being black and a lesbian. If anyone would like to argue that ability featured in this promo, I'm all ears.
• Kamala Harris as VP of the USA. Heaven help us.
• Pete Buttgig as Sec of Transportation, advising all Americans to buy electric cars to save money.
• Keisha Lance-Bottoms as mayor of Atlanta, Georgia; subsequently failing-up to be senior advisor and director of the White House Office of Public Engagement
• Jayson Blair, former staff writer at the New York Times; they were eventually able to fire him but my goodness, it was hard going.

Notable counter-examples: Hillary Clinton (barely female, was kept out of any important jobs), Barack Obama (could actually give a good speech), Margaret Thatcher (one of the few competent UK Prime Ministers, and elected despite being female).

Elon Musks' employees might be racist

A Happy New Year to all my occasional readers! May your 2022 be less fucked-up than your 2021, which is probably the best we can hope for.

I was inspired to put pen to... LCD? whatever... reading a breathless Los Angeles Times article on racism and discrimination at Tesla's Fremont, California plant:

4:05 p.m. Feb. 12, 2022: An earlier version of this article said that at least 167 racial and sexual harassment suits were filed against Tesla since 2006. At least 160 worker lawsuits were filed over various grievances, not just harassment.

Oh, sorry, that was an article correction. I'm sure that most of the 160 lawsuits were about racism and sexism. Many, at least. Some, for sure. I wonder why they didn't give a more specific breakdown?

Anyway, let's get on to the meat of the allegations from California's (government) civil rights agency, who are clearly in no influenced by Tesla's move from California to Texas that will take billions of dollars out of their income:

Tesla segregated Black workers into separate areas that its employees referred to as “porch monkey stations,” “the dark side,” “the slave ship” and “the plantation,” the lawsuit alleges.
Only Black workers had to scrub floors on their hands and knees, and they were relegated to the Fremont, Calif., factory’s most difficult physical jobs, the suit states.
Graffiti — including “KKK,” “Go back to Africa,” the hangman’s noose, the Confederate Flag and “F-- [N-word]” — were carved into restroom walls, workplace benches and lunch tables and were slow to be erased, the lawsuit says.

Ooohkayyy... Let me show you where this factory is located:

It's squarely in the Bay Area, which is one of the most liberal areas known to man persons. It's just down the road from Oakload, which is heavily Black. If this behavior was really happening at the scale indicated, I would fully expect mobs from Oakland to come and burn down the factory - while mobs from San Jose and San Francisco with artificially colored hair paraded cleverly-worded signs outside.

This claim is almost certainly massively exaggerated bullshit, based on a few events. I mean, the Confederate Flag? If you ever flew it here, your house and car would be burned down in short order. But, and here's the kicker, it might contain a kernel of truth.

Thing is, these "triggering events" aren't coming from the well-established white supremacists in the Bay Area (either of them). They're coming from the near-minimum wage factory worker class - and, to no-one's surprise, this is dominated by recent immigrants: Very few of whom are in any way white.

The dirty secret which the California civil rights agency skirts around is, and this goes back to Harper Lee's "To Kill a Mockingbird", practical racism mostly happens around the same strata in society. The Ewells were racist - eventually, fatally so - to Tom Robinson because he was only one rung below them in society - a hard-working black man, not far off from doing better than the wastrel crackers who formed the Ewells. Things have not changed much in 60 years.

The relevant strata of society in the case of Tesla - and any other manufacturing company in the Bay Area - is recent immigrants in manual labour, earning not much above the minimum wage. There is strenuous competition for these jobs, and indeed for the positions just above them that have more security and better prospects. As such, you tend to see the various communities form "views" on the other communities. And, unfortunate as it might be, African-Americans are seen as the least hard workers and least dependable.

From reported experience in major cities, in both USA and Canada, newly immigrated Hispanics and East Asian Americans tend to be reasonably coherent groups, with intra-group solidarity (if you were Manuel Labor on the factory floor, you'd support Javi in his attempt to get a full-time position because he's "one of you", you'd grudgingly agree that Binh from Vietnam could do the job but he's clearly trying to climb to the management ladder, but there's no way you want Marland from Jamaica because he's always complaining and making other people fix his mistakes, but the boss won't fire him because he's black and they can't be doing with the inevitable racial discrimination claim...

This is deliberately a caricature, but it's the practical reality in a wide range of jobs around here. There's strenuous competition for jobs and advancement, so the natural defensive tendency of the human is to join a tribal group for self-protection, and absorb their attitude towards other groups.

Interestingly you don't get so many (proportionate to immigration rates) South Asians in these jobs - they tend to come to the Bay Area on graduate-level visas, or head towards their own small business rather than working for The Man. But still, they have a view on Black Americans, and it's often informed by the relative racial proportion of people robbing their - and their extended family's - retail establishments.

I particularly liked this claim from the article:

One was lodged by a female Black employee who said her female white boss struck her with a hot grinding tool and called her “stupid” and the N-word and insulted her intelligence. The suit says the supervisor was fired but later rehired.

A white woman, in the Bay Area, using physical violence and calling a Black woman the N-word in earshot of anyone else? And the company re-hires her? I would love to hear the details of this case. I suspect the actual facts are quite different to the stated ones.

I understand that the California governing class is mad at Elon for taking his money away from them, and that an accusation of racial discrimination is the easiest tool to exact revenge. But let's not be fooled - these claims are likely 90% bullshit, and the core of real racism arises from the struggle for good jobs and money between non-white ethnic groups.