The Twitter Whistleblower report - how bad was Twitter, really?

Prompted by a post by everyone's favourite Portugal-based squirrel-torturing blogger, Tim Worstall, I thought I'd dive into the practical implications of all the (frankly, horrendous) technical, security and privacy problems that Twitter was identified as having before Elon Musk rocked up as owner and CEO.

Usual disclaimer: I'm going by the reports. Reality might be different. I cite where I can.

For background: both USA and European authorities take a dim view of corporate access to, and usage of, individual user data. Remember the European "ePrivacy Directive"? Also known as the "'f+ck these annoying cookie pop-ups' law"... Governments in both Europe and the USA are keenly interested in companies tracking individual users' activities, though my personal opinion is that they're just jealous; they'd like to do it too, but they're just not competent. Anyway, a company doing individual tracking at large scale for profit - Twitter, Google, YouTube, Meta, Amazon - attracts their attention, and their laws.

Security

Let's talk about security - and, more importantly, access to secure data. A fundamental principle of security is "least privilege" - everyone should have the smallest set of access privileges to be able to do their job. You could argue that 5000+ people in Twitter "need" to be able to change things in production at some point to do their jobs, but they certainly don't "need" to have always-on, cross-production access. Not least, because someone running a command they found on an internal playbook as an experiment, could easily break a large chunk of the service. But don't rely on me, ask their job candidates:

Twitter's practice was a huge red flag for job candidates, who universally expressed disbelief. One Vice President of Information Technology [his current role, not the target role] considered withdrawing his application on the (accurate) rationale that Twitter's lack of basic engineering hygiene in their arrangement presaged major headaches.

Hire that guy.

Certainly, every company is far from perfect in this area, but those with regulators are continually seeking to narrow the number of people with access, and the scope of access those people have. Twitter pre-Musk clearly did not give a crap about the count and scope of access. One can only imagine why; were they, for instance, relying on a large base of pre-approved employees to intercept and downgrade/block opinions outside the mainstream? How would we tell if this were not the case? Can Twitter show that they were engaged in a systematic reduction of number and scope of access to production? If not, who will be held to account?

Auditing

Control is one thing - but at least, if a human performs an action in the production environment (change, or query), that action should at least be logged, so future audit can see what happened. This is not a high bar, but was apparently too high for pre-2022 Twitter:

There was no logging of who went into the production environment or what they did.

FFS
To make clear the implications: in general, there was no way of finding out who queried (for their own purposes) or changed (deleted posts, down-rated users, etc) the production environment at any particular time. "Why did [event] happen?" "Beats the hell out of me, someone probably changed something." "Who? When?" "No idea."

This is particularly interesting because Twitter's Chief Information Security Officer - who resigned post-Musk - was also their former head of privacy engineering, and before that, apparently, global lead of privacy technology at Google. One could only imagine what that implies.

Control

There is also a wide range of engineering issues. Data integrity (not losing user-entered data) was obviously a critical issue, but Twitter had been aware for a while that they teetered on the edge of a catastrophic production data loss:

even a temporary but overlapping outage of a small number of datacenters would likely [my italics] result in the service going offline for weeks, months, or permanently.

This is not quite as bad as it first seems. After a year or so in operation, companies have a fairly good idea what happens with a datacenter outage - because they're more frequent than you imagine. Say, Henry the intern accidently leans against the Big Red Button on the datacenter floor, that cuts power to everywhere. Or you do a generator test, only to discover that a family of endangered hawks have made their nest in the generator housing for Floor 2... So you get used to (relatively) small-scale interruptions.

If you want to run a global service, though, you need to be able to tolerate single site outages as routine, and multiple site outages (which turn out to be inevitable) have to be managed within the general bounds of your service's promised availability - and latency, and data availability. Even if all your physical locations are very separate, there will inevitably be common cause failures - not least, when you're pushing binary or config changes to them. So, don't wait for these events to sneak up on you - rather, anticipate them.

This means that you have to plan for, and practice these events. If you're not doing so, than a) it will be obvious to anyone asking questions in this area, and b) when things inevitably do run off the rails, there will be bits of burning infrastructure scattered everywhere, around the highly-paid morons who are busy writing memos to cover their asses: "how could we have foreseen this particular event? Clearly, it wasn't our fault, but pay us 20% extra and we might catch or mitigate the next such event."

Go looking for those people. Fire them, and throw them into a den of hungry pigs.

Leaving the doors open

By far the most horrific aspect, however, was the general relaxed attitude about government agencies - and heaven only knows what other NGOs, cabals, and individuals - having under-the-table access to Twitter's data. Just the tolerance of user-installed spyware on privileged devices would be enough for any sane security engineer to be tearing out their hair, but actually letting in individuals known to be employed by foreign - and even domestic - governments for the purposes of obtaining intelligence information, and potentially affecting the flow of information to their and other countries... one is lost for words.

At some stage, Twitter had to either grow up, or close down. Under Dorsey's crew, the latter was inevitable - and likely not far away. It's still too early to tell if Musk can get them to option 1, but there's still hope.

The (Karin Jean-)Pierre Principle

Many of my readers (at least two!) may be familar with the long-established Peter Principle which states that in many organizations, people rise to - and are left at - the level of their incompetence. And I'm sure that we can think of many examples in our own corporate experiences.

Now, dear readers, I would like to propose the (Karin Jean-)Pierre Principle which states:

Promoting someone based on their identity, rather than ability, leads almost inevitably to incompetence which you are unable to fire.

For evidence, I'd like to offer the examples:

• The titular Karin Jean-Pierre, promoted to White House Press Sec based on being black and a lesbian. If anyone would like to argue that ability featured in this promo, I'm all ears.
• Kamala Harris as VP of the USA. Heaven help us.
• Pete Buttgig as Sec of Transportation, advising all Americans to buy electric cars to save money.
• Keisha Lance-Bottoms as mayor of Atlanta, Georgia; subsequently failing-up to be senior advisor and director of the White House Office of Public Engagement
• Jayson Blair, former staff writer at the New York Times; they were eventually able to fire him but my goodness, it was hard going.

Notable counter-examples: Hillary Clinton (barely female, was kept out of any important jobs), Barack Obama (could actually give a good speech), Margaret Thatcher (one of the few competent UK Prime Ministers, and elected despite being female).

Master/slave terminology - bears repeating

Two years ago I made fun of the effort to remove the terms "master", "slave", "whitelist" and "blacklist" from tech vocabulary, and made the assertion:

I don't have any intrinsic objection to using alternative terminology for master/slave, or for blacklist/whitelist. But I've scrutinised the people calling for this change, and I'm going to keep using the original terminology because civilised people should not yield an inch to these totalitarian fuckers.

With the developments in crimethink in the past two months, I stand by this more strongly than ever.

Surviving Diversity Training

If you work in a bureaucratic or enterprise organisation, the inevitable result of 2020’s bout of intellectual masturbation on the topic of Black Lives Mattering is going to be more "training" based on the apparent need to increase diversity in your organisation at all costs. One is reminded of the remarks of the great American satirist Tom Lehrer in the intro to his song "It Makes a Fellow Proud to be a Soldier”:

“...one of the many fine things one has to admit is the way that the Army has carried the American democratic ideal to its logical conclusion, in the sense that not only do they prohibit discrimination on the grounds of race, creed and color, but also on the grounds of ability.”

How then can we unenlightened peons “ride the wave” of diversity training so as to achieve the maximum benefit? Allow your humble correspondent to guide you.

Assuming that you're taking an in-person or video-driven interactive training session, the first thing to understand are your own objectives:

• Are you hoping to benefit from what’s being taught;
• Are you looking for the minimum interaction and distraction for your life; or
• do you resent the intrusion on your life and intend to resist actively?

All are valid choices, but make sure you pick one and plan for it. Are you aiming to avoid risk of being disciplined/fired? That's also an important factor to be aware of. If you're 63 and planning to retire soon, you might well not care at all, and in fact being fired could be a trigger for a lucrative constructive dismissal suit. If you've just accepted a job at another company which is not full of woke scolds, you can probably go to town. If you're 22 and just starting your career, your risk appetite is likely very small. Know before you go.

You also need to understand your “facilitator” (session leader), what they’re trying to achieve, and how they’re intending to do it. Unless they’ve just come down with the last shower, they will be mentally grouping their audience as above, and labelling each of you "learner", "holidaymaker" or "prisoner" - who says that stereotyping doesn't serve a purpose? Generally they try to focus on the first group and minimize contact with the others, but you might come across someone who is on an evangelical mission to convert everyone - in that case, holidaymakers and prisoners are no longer safe.

Course pre-reading and objectives

A tactic I've recently seen is to distribute “pre-reading” around the group. To make sure it gets done, they may also require everyone to prepare a small amount of pre-work based on the reading. This might be a form you have to submit before the course, or to assemble some written notes on your response to it, or to prepare a scenario for discussion in the group. (e.g. to answer: “what is a time that you felt uncomfortable in a racially charged situation? How did you respond? How did you think the other parties felt?”)

If your corporate training department is doing its job - and it’s quite possible that they aren't - the course description should come with some stated learning objectives. This will tell you a lot about what the course is trying to get you to do, and it will also clue you in to what evidence they might expect the course to produce. After all, the instructor can’t blithely claim "everyone is racism free now, that will be $5000 please!", or at least they shouldn't be able to. Therefore your company will be expecting objectives that are at least somewhat measurable, and expect to measure them.

The objectives will therefore tell you what the instructor wants you to be thinking or to have achieved by the end of the class, giving you a clear signpost to the topics under discussion. This can be a useful prompt to do pre-reading around the topics. For instance, if it requires "understanding the causes and result of implicit bias" then understanding the original study and the various debunkings can prepare you well for raising pointed questions.

Conducting yourself in session

If you intend to be a holiday maker, the worst thing to do is to make this apparent in the first ten minutes of class. Turn up a bit early, be chatty with the facilitator and others, take active part in the icebreaker activity (there's almost always an icebreaker). That should buy you some initial credit with the instructor, even though you disengage for the rest of the session. I'd also recommend doing the same burst of activity when you come back from each break. That way the instructor starts to get the impression that he or she is boring you, despite your best efforts to be attentive. Or possibly that you're snorting cocaine in the bathroom in the breaks, which I cannot endorse.

If you're a "prisoner" looking to tunnel out, I'd actually recommend a similar approach. At the very least, you don't want to look hostile in the first ten minutes. (And you shouldn't be hostile then anyway, because even the most painfully right-on diversity trainer deserves a chance to show that they're trying to make this session useful and interesting.) However, it's fine to develop a more quizzical look and increasingly more defensive body language over time. Evolve from an occasional raised eyebrow to stroking/holding your chin (showing disagreement with what the facilitator is saying), to crossed arms, to switching between crossed arms and crossed legs, depending on how obvious you want disagreement to be. This raises an implicit challenge to the facilitator, and whether they take you up on it can indicate their level of confidence in their content.

These events are normally "laptops down, phones down" so that everyone is "present". However taking a notebook and pen won't raise any eyebrows, and these can be useful tools. Whenever the facilitator says something weird or objectionable, you can take a direct note. This can be great material for feedback (see later). At worst, you can draft a love letter to your garage mechanic - no-one's going to ask to see your notes, and if they do then you can indignantly refuse. Make sure it's your own pen and own notebook.

Video sessions come with their own opportunities and challenges. For holidaymakers, feigning attention on video is a whole topic in itself, but you want to watch out for the facilitator calling on you for a response when you have no idea what they just said. Normally they require everyone to have video on and be on mute by default, which buys you a few seconds to "fumble" for un-mute. I recommend using an external, wired mike (e.g. on headphones), and so if you're stuck then gently move the plug out and in while you're talking to fake a dodgy connection. Practice makes perfect! Remember to temporarily stop using that mike and move to just using the laptop mike - it's then fine to "reset" your machine at the next break and report things "working" again.

I also recommend setting up poor lighting conditions in your room; either a strong light source causing glare (sun through a window), or curtains drawn to "remove the glare" but leaving a dark, grainy picture. Touching an unwashed thumb on the camera lens can help too - be sure to clean it after the session! For dark rooms, remember that glasses can reflect the colours of your screen so check before the meeting to see if they give anything away.

Group and pair exercises

Ideally, have a good friend in the same session who has similar views on these mandatory training events, and pair off with them for exercises if you get the chance. However, experienced facilitators will make people rotate round groups, so you've got to plan on group exercises with one or more "committed" learners. In my experience the best way to handle this is to realise that people love to talk about themselves - and the self-righteous adore it. So lead them into talking about their experiences, ask "concerned" and "interested" open questions such as "how did that change how you thought about...", and let them run out the clock for you.

You're going to need to have one or two experiences to talk about, if you don't have an obvious topic then I recommend something like:

• the time you were the only white person / male / non-tranvestite in a place and had an epiphany "this must be what it's like to be black / female / a cross dresser all the time!"; or
• childhood experience being once-off mean to someone black / female / cross-dressing which you only ten years later realising how that affected their relationships in school and oppressed them; or
• if you're got military experience, something on perception of veterans - veteran status is protected in the USA, and companies often have diversity outreach efforts focused on veteran hiring not least because they work hard and are good at execution and leadership but strangely diversity courses seldom talk much about this.

Stirring the pot

If you've got a best friend / partner / family member in a "protected" category, and frowns on "diversity" training, don't be afraid to cite them. "My mate is black and he says it's daft to focus on deaths caused by police as long as we're all shooting each other all the time" - as long as he'll back you up on that, you can probably get away with it. It will derail the hell out of the conversation, so make sure you're comfortable with raised emotions if you're going to try this.

The Equity, Diversity and Inclusion (EDI) label is itself ripe for manipulation. You could refer to it as Diversity, Inclusion and Equity (DIE); or perhaps Inclusion, Equity and Diversion (IED), making sure to use the appropriate abbreviation. Every firm should have a constructive IED program, after all!

Agreeing with "Black Lives Matter" and raising the game by suggesting "Unborn black lives matter too!" addressing the 35% abortion percentage in the black community is probably only viable if you don't mind a meeting-without-coffee with your boss and HR afterwards. Mind you, if you're a well-established Christian (or Muslim!) you might be able to get away with it.

If they do mention deaths-at-the-hand-of-police, you can refer to my previous analysis of the 2015 black unarmed fatalities in the USA. Short version: if you're not armed, and don't actively try to attack a police officer, you're actually remarkably safe even if you're black.

Raising the well-documented differences between genders is only for the suicidally determined. The facts might be with you, but you can expect prompt personal attacks on you during and after the course.

Constructive feedback

There is almost inevitably a post-course survey. The temptation is to blow this off with the minimum of effort, but If you actually want to effect changes then this is one of the best bang-for-your-buck places to make effort.

In terms of scoring, try not to be too much of an outlier. Score the content and the facilitator honestly and separately if you can. You can have the world's best facilitator, but if they're stuck with bad or dishonest material then there's only so much they can do. Highlight how the course has failed to help you meet the objectives or failed to be convinced about the assertions.

Where possible, make recommendations for "improving" the exercises. Say that there needs to be more time / quicker switching around / better briefing / more guidance on picking the right focus topic / content that you can better relate to. Again, you want to show helpfulness so they don't dismiss your scores and feedback as "grumpy prisoner".

If the facilitator or someone in the class has said something actually offensive, don't hold back from flagging it - I'd suggest without naming the person, at least initially. Make sure you wrote it down verbatim at the time in your notes, with the point in the session marked, so you can point to your notes as evidence.

Next steps

With luck, that will be it for the next year or so. No matter how woke the company, diversity training is expensive in terms of lost productivity, so they're unlike to repeat it unless they feel they have to. Sit back and enjoy 364 days of just treating people like people without fear or favour no matter what their skin colour, gender, age, or religion.