Evanston - an inadvertent experiment in racial math

Hat tip for this to @DanProft on Twitter:

AP Calculus at Evanston Township H.S.

"...the course is restricted to students who identify as Black, all genders."

"...the course is restricted to students who identify as Latinx, all genders."

Neo-segregation is the absurdity of identitarianism taken to its logical end. pic.twitter.com/Jo8I9yzZCa

— Dan Proft (@DanProft) April 30, 2023

Note: I'll be giving references throughout these blog posts, but I have high confidence that they will soon be wiped out, so I'm preserving as much relevant text as possible.

Evanston Township high school in Illinois (1600 Dodge Avenue, Evanston, Illinois, 60201), is in the suburbs of Chicago, about 7 miles north of Chicago city center. It has just achieved notoriety by racially segregating its math classes. For background, Evanston's population demographics are 16.1% Black, 11% Hispanic or Latino - not far off the USA average.

In case you think I'm indulging in hyperbole, here is the curriculum:

• AP Calculus AB - MA0515: Students will study the equivalent of one semester of college calculus... This code for the course is restricted to students who identify as Black, all genders
• AP Calculus AB - MA0555: Students will study the equivalent of one semester of college calculus... [No racial restriction]
• AP Calculus AB - MA0565: Students will study the equivalent of one semester of college calculus... This code for the course is restricted to students who identify as Latinx, all genders.

The same pattern is repeated for the Pre-Calculus course, which is a precursor to AP Calculus, and in the case of Latinx for 2 Algebra, which is the precursor for Pre-Calculus.

Note: I looked at the Science courses, and they did not have any kind of racial restriction.

Some background: the "AP" classes are usually taken at ages 16-18, though can theoretically be taken any time in the four years of high school. They are 1 year long, have a well-defined national curriculum, and culminate in an exam which is marked completely separately from the coursework and other scoring for the class; so, for a 1 year AP course you get a regular grade (0.0 to 4.0, approximately, higher being better) and also the final exam grade (integers 1 to 5, 5 being best). You could in theory get a 1.0 in class (terrible) and 5 in final exam (amazing) - or 4.0 in class (great!) and 1 in final exam (did you even spell your name correctly?). They are notorious, especially in mathematics topics, for needing coaching to do well.

In UK terms, AP Calculus AB is something around A-level Mathematics. The separate course AP Calculus BC - which is offered here but not racially segregated is more like A-level Further Mathematics, and only for hard-core nerds.

The obvious question: why is the school doing this, and what are the implications?

A tempting answer is: "to cheat!" Why else would you split out by race? You give the Latinx and Black students a teacher who marks easy, say about +1.0 on the curve, and the rest of the students a teacher who marks hard, say -0.5 on the curve. That way, a Latinx student who is actually 1 whole grade point behind a white/Asian student, would show up in final results as 0.5 points ahead, and therefore tempting material for a college recruitment.

Problem! The AP final exam is hard to cheat on. All students in the school should be taking it at the same time, under the same conditions, and probably in the same room. It would be very hard - though not impossible - to give the Black/Latinx students an advantage by writing down or correcting answers. If a school took this route, it would show up for any sample size other than tiny, by the white/Asian students having much higher final AP exam scores than their course grades would predict, relevant to Black/Latinx.

What I think might be going on instead: the school assigns the Black/Latinx classes their best teacher in the area, and a time-serving loser for the white/Asian class. This will muddy the waters in the scoring differentials. With a good teacher, Black/Latinx will achieve their maximum potential in both class and final exam, whereas white/Asians will depend heavily on external coaching - or high innate ability - to do well, since the teacher is useless. And, as an explanation for why they're not doing it for AP Calculus BC, they probably only have one teacher who can lead this subject, and in any case lots of external parental help / tuition ends up being important, which Black/Latinx students are less likely to get in any case.

Short version: in my opinion, this school wants to boost Black/Latinx student achievement in mid-level math, at the expense of white/Asian students. If you're one of the latter, consider identifying as Black.

I'm really looking forward to how this experiment works out, and I sincerely hope that legal action doesn't kill this segregation - as it well might - because getting this relative performance data would be fascinating.

Testing for determinism

Apropos of nothing^[1], here's a view on testing a complicated system for deterministic behaviour. The late, great John Conway proposed the rules for "Game of Life", an environment on an arbitrary-sized "chess board" where each square could be either alive or dead, and potentially change at every "tick" of a clock according to the following rules.

1. Any live cell with two or three live neighbours survives.
2. Any dead cell with three live neighbours becomes a live cell.
3. All other live cells die in the next generation. Similarly, all other dead cells stay dead.

You'd think that this would be a very boring game, given such simple rules - but it in fact generates some very interesting behaviour. You find eternally iterating structures ("oscillators"), evolving structures that travel steadily across the board ("spaceships"), and even "glider guns" that fire a repeated sequence of spaceships.

Building a simulation of Conway's Game of Life is something of a rite of passage for programmers - doing it in a coding language new to the programmer generally shows that they have figured out the language enough to do interesting things. But how do they know that they have got it right? This is where "unit testing" comes into play.

Unit testing is a practice where you take one function F in your code, figure out what it should be doing, and write a test function that repeatedly calls F with specific inputs, and checks in each case that the output is what's expected. Simple, no? If F computes multiplication, you check that F(4,5)=20, F(0,10)=0, F(45,1)=45 etc.

Here's a unit test script. It's written in Go, for nerds, ^[2] but should be understandable based on function names to most people with some exposure to programming. First, you need to check the function that you've written to see whether two Life boards are equivalent, so you create empty 4x4, 4x5, 5x4 boards and see if your comparison function thinks they're the same.
(In Go, read "!" as "not", and "//" marks a comment which the computer will ignore but programmers can, and should, read)

  b1 := life.NewBoard(4,4)
  b2 := life.NewBoard(4,4)
  // These should be equivalent
  if ! life.AreEqual(b1,b2) {
     t.Error("blank 4x4 boards aren't the same")
  }
  b3 := life.NewBoard(5,4)
  b4 := life.NewBoard(4,5)
  if life.AreEqual(b1,b3) {
    t.Error("different size boards are the same")
  }

That's easy, but you also need to check that adding a live cell to a board makes it materially different:

  // Add in a block to b1 and compare with b2
  life.AddBlock(0,0,b1)
  if life.AreEqual(b1,b2) {
    t.Error("one board has a block, blank board is equivalent")
  }
  // Add the same block to b2 in same place, they should be equal
  life.AddBlock(0,0,b2)
  if ! life.AreEqual(b1,b2) {
    t.Error("2 boards, same block, unequal")
  }

This is helpful, but we still don't know whether that "block" (live cell) was added in the right place. What if a new block is always added at (2,3) rather than the coordinates specified? Our test above would still pass. How do we check for this failure case?

One of the spaceships in Life, termed a glider, exists in a 3x3 grid and moves (in this case) one row down and one column across every 4 generations. Because we understand this fundamental but fairly complex behaviour, we can build a more complicated test. Set up a 5x5 board, create a glider, and see if

1. the board is different from its start state at time T+1;
2. the board does not return to its start state at time T+2 through T+19; and
3. the board does return to its start start at time T+20.

Code to do this:

  b5 := life.NewBoard(5,5)
  life.AddGlider(0, 0, b5, life.DownRight)
  b6 := life.CopyBoard(b5)
  if ! life.AreEqual(b5,b6) {
    t.Error("Copied boards aren't the same")
  }
  // A glider takes 4 cycles to move 1 block down and 1 block across.
  // On a 5x5 board, it will take 5 x 4 cycles to completely cycle
  for i := 0 ; i< 19 ; i++ {
    life.Cycle(b5)
    if life.AreEqual(b5,b6) {
      t.Error(fmt.Sprintf("Glider cycle %d has looped, should not", i))
  }
  life.Cycle(b5)
  if ! life.AreEqual(b5,b6) {
    t.Error("Glider on 5x5 board did not cycle with period 20")
  }

Now, even if you assume AreEqual(), NewBoard(), CopyBoard() work fine, you could certainly construct functions AddGlider(), Cycle() which pass this test. However you'd have to try pretty hard to get them right enough to pass, but still wrong. This is the essence of unit testing - you make it progressively harder, though not impossible, for a function to do the wrong thing. One plausible failure scenario is to make the adjacent-cells locator in Cycle() incorrect such that the glider goes up-and-across rather than down-and-across. To fix that, you could add some code to turn-on a critical cell at (say) time 8, such that that cell would be live in the expected motion, so no effect, but empty in the other motion.

Clearly, for unit testing to work, you want a unit tester who is at least as ingenious (and motivated) as the coder. In most cases, the coder is the unit tester, so "soft" unit tests are unfortunately common - still, at least they're a basis to argue that the code meets some kind of spec. And if the client isn't happy with the tests, they're free to add their own.

Why am I so mad at Neil Ferguson? He's free to make whatever epidemiological assumptions that he wants, but he usurped the "authority" of computer modelling to assert that his model should be trusted, without actually undertaking the necessary and fundamental computer science practices - not least, unit testing.

[1] Lies: Neil Ferguson, take note
[2] Object-oriented model avoided for clarity to readers

Symbolic of California's struggle against reality - car insurance

Dear readers, it has been a busy couple of months, but I thought I'd check in after reading a barnstormer of a story from CALmatters. First, a little bit of background.

In October 2018, California Governor Jerry Brown signed the California Senate Bill SB 179, which created a new gender "nonbinary" designation for all forms of state ID - including, of course, driving licenses. This then gave drivers the option of listing their gender as nonbinary. Regular readers of the antics of the Californian government will be forgiven for not falling off their chairs at the realization that this has had some unexpected consequences.

It seems that as a follow-on, California's outgoing Insurance Commissioner Dave Jones issued a regulation last month prohibiting the use of gender in automobile insurance rating. One can speculate why he did this, but (judging from the self-centeredness of transgender people I've met around here) if I were a car insurance company, I'd be giving a hefty premium bump to anyone checking that box ; perhaps this was Mr. Jones' attempt to get out in front of that problem. Ten out of ten for forward thinking, minus several thousand for economic illiteracy - of which, more later.

The proximate effect: California's Department of Insurance has decreed that auto insurance companies can no longer grant breaks in insurance rates to teen drivers who are female, or charge young men more. So if you're a woman - in particular a young woman - in California looking to insure a car, you can expect your new rate to take a sharp move skywards:

[California auto insurers' rep] Frazier said the gender of teen drivers can result in an additional cost for boys or discount for girls of about 6 percent on their premiums.

Honestly, that seems low-ball to me:

The association also cited a 2016 Insurance Institute report saying: "Men typically drive more miles than women and more often engage in risky driving practices, including not using safety belts, driving while impaired by alcohol, and speeding."

Yep, I'd agree with all of that, especially for late teens/early twenties. Given that, a 6% male-over-female premium seems really low. I'd expect it to be more like 25%. We'll know for sure when insurance renewal rolls around and California girls start yelling on Twitter.

The real prize for willful ignorance or brazen lies, however, must go to the new Insurance Commissioner Ricardo Lara:

Lara supports that policy, saying in a statement: “Gender, race, ethnicity or sexual orientation are beyond your control, and it is not a fair or even an effective way to predict risk.”

That's right, Ricky. Any insurance company could charge boys the same as girls with no additional risk, and quickly get every single boy in the country insuring their car with them, but they don't do so because they... like leaving huge sums of money on the table? Yes, that must be it.

Commissioner Ricardo Lara made history in 2018 by becoming the first openly gay person elected to statewide office in California’s history. Commissioner Lara previously served in the California Legislature, representing Assembly District 50 from 2010 to 2012 and Senate District 33 from 2012 to 2018. Commissioner Lara earned a BA in Journalism and Spanish with a minor in Chicano Studies from San Diego State University.

I guess he skipped all the finite math classes in high school. Or worse, he know's he's talking bollocks but simply doesn't care because no-one will call him on it.

Consider yourself called on it, Ricardo.

It's possible, however, that it won't actually work out in practice as Ricky intends. We saw how this worked out in Europe after a European Court of Justice ruling. [note: link from that notorious right-of-center rag The Guardian]:

But what has happened since the rules came into force? Instead of the gap between men’s and women’s premiums narrowing, as expected, it has actually widened. In 2012, men on average paid £27 more for a car insurance policy than a woman, but rather remarkably they now they pay £101 more – nearly a four-fold increase.
...
What appears to be at work is that car insurance companies set a price very much according to all the other data they can find on you – without actually asking your gender. So the quote you get back reflects the risks attached to your occupation, how much you drive, the sort of car you drive and whether you have made any modifications to the car.

Perhaps this is fine with Ricky - as long as there is the appearance of fairness, and he's protecting his favored class of people from reality, this is all Working As Intended.

Life of an actuary: more exciting than you might believe

When I was in university and hanging around the mathematicians - those students who dedicated their life to math(s), and I use "life" in the loosest sense possible - the standard joke was that the really successful ones would get involved in professional gambling^[1], most of them would be accountants, and the ones who couldn't handle the excitement of accountancy^[2] would become actuaries.

At least two of the three were true. Approximately the majority of the mathematicians I knew ended up in accountancy or related consultancy, and the actuaries were definitely around the low end of the social scale in the subject. Given the starting point of a mathematics degree, that's a scale needing very fine calibration. Anyone who chose optional courses in computer science ended up working for Big Tech and making out like a bandit.

Today I stumbled across a site Be An Actuary, which is (as far as I can tell) not a spoof site. It contains invaluable advice and guidance on what to do if you feel that actuarialism^[3] is your calling, and a critical piece of information is what a day in the life of an actuary is like.

Before reading these quotes, you may wish to equip yourself with a spoon^[4].

So far today, I've researched the applicable accounting rules and written a report for a client who's acquiring a small life insurance company.

Be still my beating heart.

I am constantly asking myself "Does this make sense?".

I'm assuming that "this" doesn't apply to "my terrible career choice". But it should.

In a midsize company like mine, there is also opportunity to price a new product, which takes creativity, or respond to an insurance department inquiry, which requires communication skills and tact.

Or, in despair at my life, throw myself through a 10th floor window, which requires a good run-up.

I currently manage three reserving analysts and we spend most of our time doing reserve analyses and projects like catastrophe modeling, loss modeling for some of our low frequency/high severity lines, and supporting our Corporate Actuary as he writes Actuarial Opinions and Reports.

You should spend some time on serious introspection on how your life got to be this way.^[5]

If you still have more than one eye remaining to view the remainder of this blog post, you're a more resilient person than me.

[1] Specifically, running the numbers games in the casinos rather than playing them.
[2] Yes, that's irony.
[3] Probably not a word, at least I hope not.
[4] Because it's DULL, you twit. It'll hurt more.
[5] Probably, you have Korean/Indian/Chinese parents and you paid more attention to their ambitions than your desires.

Scentrics worth half a billion quid - and other fiction

Regular readers (both of you) will recall my previous scepticism regarding IT "security" company Scentrics. TL;DR - they're pushing the idea that a key part of "secure" email is sending a copy of every email to a central server, encrypted with a key that only gives access to a trusted party - your local government, for instance. Singapore seemed very interested in their proposals, for reasons one can imagine.

Out of idle curiosity, I thought I'd check the Scentrics accounts for 2016-2017. Well, gosh.

	30 June 2017 £	30 June 2016 £
Fixed assets
Intangible assets	504,014,092	20,455
Property, plant and equipment	6,463	8,618
Investments	10	-
	504,020,565	29,073
Current assets
Debtors	1,051,556	1,047,027
Cash at bank	893,815	2,793,822
	1,945,371	3,840,849
Creditors within 1 year	(893,718)	(893,232)
Net current assets	1,051,653	2,947,617
Total assets less current liabilities	505,072,218	2,976,690
Provision for liabilities	(99,546,235)
Net assets	405,525,983	2,976,690
Capital and reserves
Called up share capital	130	130
Share premium	5,778,596	5,778,596
Retained earnings	399,747,257	(2,802,036)
	405,525,983	2,976,690

How would I read this? They spent £1.9M of their cash on various things during the year; about half of that on medium-to-long term debt servicing, and the rest presumably on overheads (salary, office, patent office fees, other professional service fees). This is clearly not sustainable, and indeed last year they had a net worth (retained earnings) of minus 2.8 million pounds. How could this be fixed?

Well, they've just gained £504 million in intangible assets. The associated notes indicate a "revaluation" of their intangibles happened, which changed from £22K to £560M. There was a 10% amortisation charge ("spreading out") over the year, taking them down to a measly £504M. That's quite a change, what was involved?

Patents and licences were valued on an open market basis on 20 August 2018 by the Directors

There's also the useful information:

Patents and licences are being amortised evenly over their estimated useful life of ten years.

But there's no obvious licence revenue in the company accounts that I can see, and there's still only 4 employees (the directors) so they're not doing anything substantial with the resources, so I'd bet this £560M change is an evaluation of the worth of their patents. Let's look at these, shall we?

The main Scentrics patents pivot around the previously discussed system where a client (mobile, in the most recent patents, but there's nothing specifically "mobile" about them) talks to a centralised mail server to obtain encryption keys to safely send messages to it for routing onwards a destination, and then separately sends a copy of the message (asynchronously! wow, there's some modern thinking) to a "monitoring" server using a different encryption key.

Basically, it's a system for a company or government to enable scanning of email sent by its employees/citizens - as long as they're using its mail application, of course. If the employees use Outlook.com, Gmail, or any number of other public webmail services, they are sunk. So companies will block all the webmail applications by restricting the web browsers in their corporate devices, forcing use of the corporate mail server (Outlook, most likely) which they can snoop on. They don't need Scentrics' patents. Governments would need a willing population to live with the (likely) crappy, unreliable custom email application and not look elsewhere for their email needs. Even China struggles to keep up with restricting their population to approved websites, and they're a gosh-darned communist dictatorship.

It's not impossible that Scentrics reckons they can get a major corporation or government to licence their patents, but I'd have to rate it as unlikely at best. Why would someone pay £500M for it, rather than (say) £5M to get a moderately competent cryptographer to design a better system? The patent is extremely dubious to defend in my personal technical opinion; there are alternative strategies such as encrypting the message with a randomized key, encrypting that key with a) the recipient's key and b) the monitoring service's key, and enclosing both encrypted keys in the message. Then the client only has to send one message, and the monitoring service can store it and decrypt it on demand. But hey, what do I know.

Guru Paran Chandrasekaran and Andrea Bittau - happy to bring you gents up to speed on the state of modern cryptography, if you're interested. No charge!

(They've finally fixed their https problem. Guess it got a bit embarrassing.)

Update: Looks like Andrea Bittau was killed in a motorcycle crash last year. Nothing sinister, just terribly sad - 34 years old.

Good news about Hawaii's ballistic missile warning service

It works!

Watching the 1pm (Hawaii) press conference, the Governor and the Administrator for Emergency Management are going through the expected self-flagellation. The Administrator commented "Our process is to have no more false alarms from now" and that now two people will be required to send out an alert.

The interesting questions, which the journalists don't seem to be asking:

1. How many false alarms are acceptable - rather, what rate of false alarming is acceptable? Once in 30 years? Once in 10 years? Once a year?
2. What are the benefits from a false alarm - e.g. testing the alert channel, prompting people to think about their emergency plans - and what are the costs - e.g. mental health events, car accidents, heart attacks, premature consumption of expensive whisky
3. What actions taken to reduce the risk of false alarms increase the risk of a real alarm being delayed in sending?

Everything comes with a trade-off. The last question is probably the most important. If you only have 10 minutes from alert going out until missile impact (on the current plan), what happens if e.g. your requirement for two people to trigger the alert sending ends up causing a delay because one person isn't around? You just know it's going to happen:

"Hey Akamu, can you watch the console for the next few minutes, I just gotta go to ABC Stores to get some more chocolate macadamias?"
"Sure Alika, I don't want to call in Ula the backup guy if we don't really need to."

I'd like to see a public written postmortem about this incident. Redact names - replace them with roles e.g. "the coming-on-duty emergency alerts worker", "the going-off-duty emergency worker" - and explain:

• what went wrong,
• why it went wrong (following the 5 Whys technique),
• what actions are being taken to remediate the risk, and
• what do they aim to achieve in terms of the false alarm rate and the failure to alert probability?

Write it in a blameless fashion; assume good faith and basic competence by the people involved. If someone made a bad choice, or slipped and hit the wrong button, the problem isn't with the person - it's the process and technology that let them make that bad choice or press the button in a non-deliberate way.

One interesting question that was raised in the conference: why did some but not all of the sirens trigger? You'd want the process to be that both the sirens team and the alert message should monitor each others' output. If you're the siren operator and get the alert on your phone, the best strategy is to trigger the siren immediately to increase coverage of the alert. The impact of a false siren is much lower than impact of not playing the siren when a missile really is inbound because of the PACOM-to-sirens message channel failing. So maybe this was individual siren operator initiative - reward those folks, and make it standard procedure.

This is a great opportunity for the state government to demonstrate transparency and a commitment to making the systems objectively work better, rather than just playing to the press. Unfortunately, you just know that it's not going to happen like that.

Motivations behind the California gas tax increase

This weekend, a 66% increase went into effect for the state gasoline (petrol) tax in California. The stated intention is to raise money for transportation projects:

Under Senate Bill No. 1, the gasoline tax will increase by 12 cents, from 18 cents to 30 cents per gallon, the Los Angeles Times reported after Gov. Jerry Brown signed the legislation in April.
[...]
As part of the legislation, motorists will also have to pay an annual vehicle fee, though that doesn't take effect until Jan. 1, 2018. The fees range from $25 for cars worth less than $5,000 to $175 for those valued at more than $60,000. Additionally, a $100 annual fee on electric vehicles will be imposed on owners in lieu of gas taxes beginning on July 1, 2020.
Most California drivers can expect the taxes and fees to cost them less than $10 each month, according to Brown.

The revenue increase

Well Jerry, let's look at the math. I'm ignoring the costs on diesel users here, just focusing on regular gasoline, so this will be an underestimate of revenue raised.

California population is approximately 39M. There are approximately as many registered vehicles as people. So the stated overall income delta to the state of California for the imminent change is $120/year x 39M vehicles, or $4.6 billion per year, based on Jerry's numbers.

How much gasoline did Californians buy in the past year? About 14.57 billion gallons. Add $0.12/gallon and that's $1.8 billion; the bulk of the increase will come from the per-vehicle fee. Looking at the scales, let's say $80 is average; that yields $3.12 billion.

Interestingly, electric vehicle users really get screwed by this. If I've read the spec correctly, they pay $100/year plus the value-of-the car tax. $100 looks to be at least 2x the average gas tax impact based on mileage.

So California shouldn't be exercised about the gas tax per se, they should be much more indignant about the per-vehicle charge. And electric vehicle owners should be marching on Sacramento to burn Jerry in effigy, if their vehicles can get them that far.

What gas costs, and why

The California tax situation is worse than it appears. In San Jose currently you'll expect to pay about $2.70/gallon; you'll pay $0.18/gallon federal gas tax, $0.12/gallon state gas tax, plus $0.06/gallon state sales tax. But there's also a secondary excise tax and an underground storage fee - yes, California taxes the right to store gasoline underground - which bumps the total California per-gallon tax to about $0.58; over 21% of your gas cost goes to the state of California.

Also note for completeness: in summer gas costs much more because California requires a special anti-pollution blend. This isn't crazy, especially if you live in a major conurbation like LA or the Bay, but you can easily find gas prices over $4/gallon. The extra $1/gallon or so yields another $0.02/gallon in sales tax for about half of the year.

Where it goes

Why is the California state government doing this? Well, because politicians will tax anything they can get away with. California already has more than enough money from gas taxes to address the issue of highway repair. Don't believe me? Check the math.

The stated need for highway repair in the California 2016-2017 budget was $3.6 billion annually, plus a $12 billion backlog:

Highway and Road Maintenance and Repair Needs. In order to assist the Legislature in its deliberations regarding increased funding for state highway and road repairs, we assess the costs to maintain and rehabilitate core aspects of the state highway system—pavement, bridges, and culverts—as well as local roads. We find that the state has ongoing highway repair needs of about $3.6 billion annually as well as an existing backlog of needed repairs totaling roughly $12 billion. This need is significantly higher than can be addressed through the existing funding of about $1.6 billion for these purposes.

Interesting. California is currently extracting $0.583 in tax per gallon of gasoline on 14.57 billion gallons of gas per year - $8.5 billion - and yet it's spending only 20% of that money on highway repair. What is it spending the rest on? You can bet that the excess is finding its way in disguise to be able to fund the usual payoffs: pensions and pet projects like the brain-meltingly dumb high speed SFO-LA rail.

We should ask: where does Governor Brown think this money will be spent?

“Safe and smooth roads make California a better place to live and strengthen our economy,” Brown said. “This legislation will put thousands of people to work.”

Oh, Jezza. Jobs are a cost, not a benefit. The minimum-wage Bay Area or LA commuter spending literally thousands of dollars per year on gas is not going to thank you for taxing him about $150/year extra to fund union workers doing fuck-all to fix potholes. (Believe me, I've had ample opportunity to watch them work while sitting in traffic jams. They could not move slower if they were quadriplegic.)

"But California is so big! It has the most roads!"
Nope. Texas has nearly 2x the road mileage of California. I've driven quite a lot in Texas, and their pothole situation feels better than California. Heck, you occasionally get actual freezes in Texas, which are practically unknown in California, and a classic source of potholes. The top 4 worst pothole cities are Californian. What the hell?

"California has other transport costs! Like congestion reduction!"
California does f-all congestion reduction, believe me. The only congestion reduction it likes which works is higher gas taxes. Building additional roads is waaaaayy down their priorities list, only just above "donate to Donald Trump's reelection campaign".
There are also additional separate revenue streams from vehicles in California. Existing annual vehicle registration is proportionate to the vehicle's current estimated value, and for a $30K car will set you back $100-$200/year. "Use tax" applies to all vehicle purchases - even second hand! - and the state gets 7.5% of the purchase. So for a $30K car you'd pay $2250; assuming you keep the car for 6 years, that's $375/year income for the state.

California doesn't have an income problem for its transport costs. It has a "not spending that money on transport costs" problem.

Measuring success

Where I work, if I were to propose recurrent spending of a significant chunk of my company's annual income on a project, I'd rightly be asked to define a measurement of its impact, and "success" criteria by which we could judge if the spending had an acceptably high impact, or if it should be stopped and I should be fired.

Jerry Brown: what will be your measure of success in spending the $5 billion+/year raised from the additional gas tax and vehicle fee? Is it just in "sustained union support of Edmund Gerald Brown, Jr"? You mendacious over-taxing git.

28 hours of racial lies

One of the latest bits of social justice posturing is the play "Every 28 hours", a project produced by the Oregon Shakespeare Festival:

Every 28 Hours is a national partnership focused on the widely shared and contested statistic that every twenty-eight hours a black person is killed by vigilante, security guard, or the police in the United States.

Regular readers will know that a maths-based arse-kicking is coming. But perhaps, disregarding the numbers, this play is still a compelling work? After all, Harold Pinter was a complete arse, but his plays could still pull in the crowds. Might it be the same here?

The Every 28 Hours Plays consist of 72 one-minute plays inspired by the Black Lives Matter movement, with participation from artists across the nation.

OK, maybe I'll save myself the price of the ticket and just gently gouge out my eyes with a spoon.

One black person killed every 28 hours is 312 black people murdered a year. This is 312 murders too many, no matter who's doing it - and, let's be clear, I'm not quibbling with this . However, let's put this in some numerical context, shall we? I'm assuming that the "Every 28 Hours" authors are mostly liberal arts majors, so I promise to go slow and show my working. (Which, I'd guess, is a sight more than they do.)

The Facts

I'm using the FBI 2015 crime figures, specifically Expanded Homicide Data Table 6 (Race, Ethnicity, and Sex of Victim by Race, Ethnicity, and Sex of Offender, 2015).

Race/Ethnicity of victim	Total	Race of offender
		White	Black / African-American	Other	Unknown
White	3,167	2,574	500	49	44
Black / African-American	2,664	229	2,380	13	42
Other race	222	60	34	126	2
Unknown race	84	34	20	6	24

The other key stat is that, as of 2010, 12.6% of Americans are black or African-American - 1 citizen in 8. I'm making a leap of faith that this fraction has not changed significantly in the past 6 years. Since white people are about 63% of the population, they outnumber black Americans 5 to 1.

The Math(s)

The obvious stat that leaps out - though is hard to state grammatically: white people kill approximately as many white people as black people kill black people. White-on-black and black-on-white killings are actually relatively infrequent. This is also true for the "other" racial category (Asian, mixed-race, Native America etc) which turns out to be a similar fraction of the US population as black / African-American, but only about 7% of the number of racial colleagues killed even if you incorporate the "unknown" category.

So we could produce a companion play "Every 220 Minutes" representing the time interval between one black person killing another black person. We could also write "Every 17 1/2 hours" for a black person killing a white person, and "Every 38 hours" for a white person killing a black person.

But wait! If we have to wait 38 hours for a white person to kill a black person, and a black person is killed by a vigilante / security guard / police officer every 28 hours, doesn't that mean that some of those vigilantes / security guards / police officers must be black (or other ethnic minority)? Why yes, it does. I wonder if "Every 28 Hours" brings out this aspect of the statistics.

The truly terrifying stat is simply that black Americans kill about the same number of people as white Americans despite being outnumbered 5:1. The fear of young black American males held by many white people is visceral rather than statistical - the rate at which black people kill white people is about what you'd expect given the relative proportion of population - but black people in the 20-29 age range should be fucking terrified of black males aged 17 to 24 because they are the ones doing most of the killing of victims in that age range.

Why in the name of all that is holy are the "Every 28 Hours" folks talking about (white) police officers as a deadly influence, when young black men do 10 times more killing?

The Weasels

Let's go back to the Every 28 Hours claim:

...every twenty-eight hours a black person is killed by vigilante, security guard, or the police in the United States [my italics]

Now why, do you think, they added those two extra categories? If they could say:

...every twenty-eight hours a black person is killed by the police in the United States

then wouldn't that be a more powerful message? Perhaps they're not using it because it's not true. The Washington Post reports 258 black people killed by police in 2015. If "Every 28 Hours" used that figure as its basis, it would be called "Every 34 Hours" instead.

The statutory ad hominem

"Every 28 Hours" producer Claudia Alick is big on artistic direction, with a minor in hip-hop coaching, but it seems that her MA from NYU and BA from GWU have not equipped her with the ability to do math. Or perhaps she has the ability, but also gained the power to ignore the figures for the greater good of spreading propaganda. She certainly doesn't seem to be concerned with actually improving the lives of, and reducing the horrific body count in, the black American community in any meaningful way.

Analysing the blue-red hat problem in the face of user error

Everyone knows computers are getting smarter - unless they're being programmed by a major corporation for a government contract - but there has recently been another leap in the level of smart. DeepMind (now part of Google) has built an AI that has successfully deduced the optimal solution to the hat problem:

100 prisoners stand in line, one in front of the other. Each wears either a red hat or a blue hat. Every prisoner can see the hats of the people in front – but not their own hat, or the hats worn by anyone behind. Starting at the back of the line, a prison guard asks each prisoner the colour of their hat. If they answer correctly, they will be pardoned [and if not, executed]. Before lining up, the prisoners confer on a strategy to help them. What should they do?

Tricky, n'est ce pas?

The obvious part first: the first prisoner to answer, whom we'll designate number 1, has no information about his hat colour. Assuming blue and red hats are assigned with equal probability, he can answer either "red" or "blue" with a 50% chance of success and 50% chance of getting executed; he has no better strategy for self-survival. What about the other prisoners?

Applying information theory, our system has 100 binary bits of state - 100 people, each with 1 bit of state relating to whether their hat is blue or not. We generate 99 bits of knowledge about that state as the hat-wearers give answers. So the maximum we can expect to discover reliably is 99/100 hat values. How can we get close to this?

If everyone just guesses their own hat colour randomly, or everyone says "blue", or everyone says "red", then on average 50% of people survive. How to do better? We need to communicate information to people further down their line about their hat colour.

Let's get the first 50 people in line to tell the next 50 people in line about their hat colour. Person 1 announces the hat colour of person 51, person 2 of person 52 and so on. So the last 50 people are guaranteed to survive because they have been told their hat colour. The first 50 people each have a 50-50 chance of survival because the colour they "guess" has no necessary relation to the colour of their hat. On average 25 of them survive, giving an average survival of 75% of people.

The DeepMind algorithm relies on an insight based on the concept of parity: an 0/1 value encapsulating critical state, in this case the number of blue hats seen and guessed, modulo 2. The first user counts the number of blue hats seen and says "blue" if that number is even, and "red" if odd. He still has a 50-50 chance of survival because he has no information about his hat. The second user counts the number of blue hats. If even, and the first person said "blue", then he and the first person both saw the same number of blue hats - so his hat must be red. If even, and the first person said "red", his hat must be blue because it changed the number of blue hats seen between the first person and him. Similar reasoning on the odd case means that he can announce his hat colour with full confidence.

What about person 3? He has to listen to person 1 and person 2, and observe the hat colours in front of him, to deduce whether his hat is blue; his strategy, which works for all others after him too, is to add the parity values (0 for blue, 1 for red) for heard and seen hats modulo 2, and if 0 then announce "blue", if 1 then announce "red". Follow this down the line, and persons 2 through 100 are guaranteed survival while person 1 has a 50-50 chance, for an average 99.5% survival rate.

Of course, this is a fairly complicated algorithm. What if someone mis-counts - what effect does it have? We don't want a fragile algorithm where one person's error can mess up everyone else's calculations, such as with "Chinese whispers." Luckily, a bit of thought (confirmed by experiment) shows us that both the future-casting and parity approaches are resilient to individual error. For future-casting, if one of the first 50 people makes an error then it makes no difference to their chance of survival but their correspondent in the second half of the line is doomed. If one of the second 50 people makes an error then they are doomed unless their correspondent also makes a mistake - generally unlikely, a 10% chance. So if 10% of users make errors then the approximate number of survivors is (75 - 10) + 1, i.e. 66%.

Surprisingly, the parity approach is also robust. It turns out that if user N makes a mistake then they doom themselves, and also doom user N+1 who relies on user N's calculation. But because both user N and N+1 make erroneous guesses, this brings the parity value back in line for user N+2, whose guess will be correct (absent any other errors). So the approximate number of survivors given a 10% error rate is 99.5 - 10*2 = 79.5%

Here's Python code to test the various algorithms: save it as "hats.py" and run it (e.g. "chmod 0755 hats.py ; ./hats.py" on OS X or Linux). It runs 10 trials of 100 people and reports the average number of survivors, based on a 10% error rate in hat wearers following their strategy. Default strategy is the parity algorithm.

#!/usr/bin/python

import random

person_count = 100
half_person_count = person_count / 2
# Hat choices
hat_choices = ['r','b']
hat_opposite = {'b':'r', 'r':'b'}
# 10% error rate in guesses
error_rate = 0.1

def guess_constant(heard_guesses, seen_hats):
    return 'b'

def guess_random(heard_guesses, seen_hats):
    return random.choice(hat_choices)

def guess_future(heard_guesses, seen_hats):
    """ First half of list calls out hat of correspondent in second half of list """
    full_list = heard_guesses + ['x'] + seen_hats
    my_index = len(heard_guesses)
    if my_index < half_person_count:
        # Call out the hat of the person in the second half of the list, hope same as mine
        return full_list[my_index+half_person_count]
    else:
        # Remember what was called out by my corresponding person in first half of list
        return heard_guesses[my_index - half_person_count]

def guess_parity(heard_guesses, seen_hats):
    """ Measure heard and seen parity of blue hats, call out blue for even, red for odd."""
    heard_blue_count = len([g for g in heard_guesses if g == 'b'])
    seen_blue_count = len([s for s in seen_hats if s == 'b'])
    if (heard_blue_count + seen_blue_count) % 2 == 0:
        return 'b'
    else:
        return 'r'

def run_test(guess_fun):
    hat_list = [ random.choice(hat_choices) for i in range(0, person_count) ]
    print "Actual: " + "".join(hat_list)
    answer_list = []
    score_list = []
    error_list = []
    correct = 0
    for i in range(0, person_count):
        guess = guess_fun(answer_list, hat_list[i+1:])
        if random.random() < error_rate:
            guess = hat_opposite[guess]
            error_list.append('X')
        else:
            error_list.append('-')
        answer_list.append(guess)
        if guess == hat_list[i]:
            correct += 1
            score_list.append('-')
        else:
            score_list.append('X')
    print "Called: " + "".join(answer_list)
    print "Score:  " + "".join(score_list)
    print "Errors: " + "".join(error_list)
    print "%d correct" % correct
    return correct

if __name__ == "__main__":
    trial_count = 10
    correct_total = 0
    for i in range(0, trial_count):
        print "\nTrial %d" % (i+1)
        correct_total += run_test(guess_parity)
    print "\nAverage correct: %d" % (correct_total / trial_count)

You can change the "guess_parity" value in the run_test() invocation on the penultimate line to "guess_future" for the "warn the second half" strategy, or "guess_random" for the random choice.

This is a lousy problem for use in software engineering job interviews, by the way. It's a famous problem, so candidates who have heard it are at a major advantage to those who haven't. It relies on a key and non-obvious insight. A candidate who hasn't encountered the problem before and solves it gives a very strong "hire" signal, but a candidate who fails to find the optimal solution could still be a valid hire. The least worst way to assess candidates based on this problem is whether they can write code to evaluate these algorithms, once the algorithms are described to them.

Delays are good for you - the MTA proves it

No, really, they do. New York's Metropolitan Transit Authority (something like Transport for London) has produced an outstanding video that shows why making some subway trains late makes others less late:

Yes, the idea is that sometimes delaying a train can prevent further delays by not compounding the gap between trains. Anyone who has waited impatiently on a hot subway platform might find this concept counterintuitive, but transportation experts generally agree that that the evenness of service is as crucial as avoiding individual delays.

The MTA video makes a compelling case. The key insight is that once a platform gets crowded enough, due to constant feed of new passengers and a delayed train, it becomes slower for the next train to debark and embark passengers. So an already delayed train gets more delayed as it progresses down the line. The solution? Spot a train that's getting near the critical delay time and give it priority to progress through the network even if this involves delaying other (less delayed trains).

It's a great example that, even in what we regard as relatively simple systems, there can be a complex interplay between entities that produce highly unintuitive results. Deliberately delaying trains can actually be good for the system as a whole (if not for the passengers sitting in the delayed train with their faces pressed into a fellow passenger's unwashed armpit).

A voice of reason in CiF

It would have to be a mathmo, wouldn't it? Sam Howison, an applied maths professor, looks at why the first 50 Fields medal winners were uniformly male and, refreshingly, comes up with a range of explanations with the starting point that there just aren't many female mathmos:

Data is scarce in this rarefied region, and hypotheses are hard to test; so, too, is the influence of the culture of their chosen field. Nevertheless, such astronomical odds of a woman winning the medal are disturbing, and they are just an extreme point of a range of evidence that women are underrepresented in mathematics at many levels.

It's indisputably true that you don't find anything like a 50% proportion of women at the top level of maths, or theoretical computer science for that matter. On the other hand, in my experience the women that you do find there aren't obviously any less smart and capable than the men, so if you were making randomized choices based on intellect you'd expect women to be far more frequent in Fields medal holders than they are.

This year, Stanford professor Maryam Mirzakhani won a Fields medal. She's clearly a hard-core pure mathmo; I defy anyone with anything less than a Ph.D. in maths to read about her research interests and not have their brain leak out of their ears. This is not just "I don't understand what this is about", this is "I can't even picture the most basic explanation of this in my head". Compared to that, even Fermat's Last Theorem was a walk in the park - solving polynomial equations is standard A-level fare, and even if you can't understand what Andrew Wiles did to prove it you can at least understand the problem. With Mirzakhani's work, you have no frame of reference, you're like a child who wanders into the middle of a movie.

Howison's point about the astronomical odds of the Fields medal award gender distribution (50 tails in 51 unbiased coin tosses) is a nice point of probability, but of course the first place you'd start is to look at the eligible pool - top-flight mathematicians, generally at (UK) professor level, with a substantial track record of publishing. That will tell you your bias; if 1 in 10 people in the pool are female, you're tossing a biased coin which will show tails 9 times out of 10. Still, it's pretty clear that even with that pool the Fields medal gender split is way out of line with what you'd expect.

Howison makes an interesting point that I hadn't considered up to now:

[...] people with successful careers have usually had a high degree of support from a mentor. As well as providing academic guidance and inspiration (as Mirzakhani freely acknowledges she had when a student), the mentor will introduce their charge to influential colleagues on the conference circuit and elsewhere, and arrange invitations to speak at seminars and workshops. That is one way for a young mathematician to get their work noticed, and to improve their chances of getting a position in a world-leading department where they can thrive. Is this perhaps (if only subconsciously) difficult for women in a community where the majority are men?

The usual reason for explaining the lack of women in senior positions in Fortune 500 firms (banks, Big Pharma etc.) is that they're not as good at men at talking their own book, preferring to be more even-handed in giving credit for the achievements in which they'd participated. However, Howison tantalisingly hints at a squaring function in gender representation here - will junior female mathmos only get good support and PR from a senior female mentor, and do such senior female mathmos pick up juniors with a blind eye to gender? It would be fascinating to get some data here.

I do wonder whether that perennial topic in gender discrimination, motherhood, plays a role here. Because the Fields medal only goes to people younger than 40 - Andrew Wiles, who cracked Fermat's Last Theorem, was a notable omission from its holders due to his age - if you take time out from academe to have children then this disproportionately affects your time where you're eligible for a Fields medal. The Guardian interviewed this year's sole female awardee, Maryam Mirzakhani but she didn't make any comment about her family life so I have no idea if she has kids.

So mad props to Maryam Mirzakhani for being the first female winner of the Fields medal, and here's to hoping for many more. Apart from anything else, if we can start to get some data on what factors determine female Fields medal winners we might have a hazy glimpse of what we need to fix in the academic lifecycle to get more top-flight women choosing to follow it.

Strategy in 2048

I've been playing the 2048 game which (if you haven't played it yet) is the most phenomenonal time sink invented. To save people from sanity, here are some general game hints; using this strategy I manage to obtain the 2048 tile in roughly 50% of games.

1. Start by building up the big numbers in the lower left corner, spreading along the bottom row.
2. As the bottom row is nearly full (say you have [16, 8, 4, _] start filling the third row with numbers starting from half the lower left number (say, [8, 4, 2, _)
3. When the opportunity arises to have the bottom row full and the third row filling 3 of 4, right shift the board so the third row numbers line up above the same numbers on the bottom row and then drop them down and shift left to increase your bottom row numbers by a factor of 2.
4. When the bottom row starts getting big (say, 64+ as the left corner number) start trying to order the third row in the opposite direction. If the bottom row is [128, 64, 32, 16] start trying to create 16 at the end of the third row. Whenever you can, drop numbers into the bottow row.
5. At nearly all cost, avoid filling row 2 so that you have no option but to move the board up - that will trap small numbers under your row of big numbers. If that happens, drop the board again immediately (and hope).
6. If you end up with 2 or 4 tile to the left of a big number tile on your bottom row, focus on increasing that tile number until it matches your big number so you can left-shift the bottom row and have the biggest tile in the bottom left corner.

Good luck!

Claire Perry finds maths hard as well

Not content with landing herself with a libel writ due to appalling misunderstanding of the Internet, everyone's favourite Helen Lovejoy - Devizes MP Claire Perry - now has the cunning idea that the Church of England should divest itself of shares in Google. And why?

Mrs Perry said: "It is quite clear that many companies, in particular British Internet Service Providers are finally now taking a really responsible approach to this [Internet porn filtering]. They are seeing that we want a level of social responsibility. “There are others out there who have not got that attitude. The Prime Minister was saying Google have a responsibility, they are effectively helping people for which there can be no case made."

Not the most coherent message, but you get her general drift: Google is not dancing to my tune, what impudence! we must slap them on the nose with a rolled-up newspaper.

Let's assume that this isn't a piece of pointless political posturing and that Perry MP thinks this will actually make a difference. So how much are we talking here?

Accounts published later this year will show that the Church’s pension fund has a £5.7 million stake in the internet search giant, alongside significant investments in a number of internet service providers.

That's about $8.7M, or slightly under 10,000 shares at Google's current stock price of $885. Since about 1.7M Google shares traded yesterday, and Google's market cap is a little under $300 billion, the effectiveness of this divestment could be measured at the same scale as a dog breaking wind in the middle of a tornado.

If I had a more suspicious mind, I'd think that Claire Perry was waving such pointlessly flimsy proposals at friendly journos to distract (squirrel!) from her little contretemps with Guido Fawkes... I'm glad however that the Archbish has a much more honest approach to the issue:

"If you exclude any contact with anything that directly or indirectly gets in any way bad, you can't do anything at all."

For a good take on the very real problem which Claire Perry is trying, desperately ineptly, to tackle, I recommend Greg Ferenstein's piece on Internet filtering. Go read the whole thing, but here's a taste:

The problem with this approach is that the world isn't PG-13. Politics, business, and personal health regularly intersect with adult issues. The (very) savvy engineers at Apple have already discovered that you have to apply a tourniquet to the First Amendment to effectively block children from seeing naughty pictures.

Want more tax? Buy more engines!

The usual suspects are out in the streets tearing their clothes at the news that Rolls Royce paid no corporation tax in the UK last year:

Rolls-Royce's annual financial statement, released in February, shows it made £1.4bn in pre-tax profit in 2012, an increase of 24% on 2011.

OK, corporation tax is on profit; there's profit, why isn't there tax? Well, it seems that Rolls Royce did pay a fair amount of tax - just not in the UK:

According to its records, last year Rolls-Royce paid £218m in taxes abroad where it said it conducts 85% of its business.

Indeed, Rolls Royce isn't selling many engines in the UK - we're not building many planes. The RAF Typhoon uses a Eurojet EJ200 which is based on a Rolls Royce design but produced by a consortium and (as far as I can tell) isn't in current production. The major airliner manufacturers are all based abroad. So it's not surprising that the sales of Rolls Royce engines are being booked abroad. The UK operation must be substantially loss-making in isolation - presumably they receive indirect revenue from the sales abroad, but not enough to generate any actual profit.

Someone needs to be given short shrift:

Chris Williamson, Labour MP for Derby North, said he had written to the chief executive of Rolls-Royce for more information.
He said: "We do need to get to the bottom of the story. All companies, irrespective of how many people they employ, have an obligation to pay tax if they are making profits here.

Well, because Rolls Royce employ a lot of people (many in Derby) and aren't selling engines in the UK, they're not making any profits here, are they? So, by your argument, the obligation to pay UK tax is negated.

Now the tax paid on profits worldwide isn't huge - £218m tax /£1.4bn profit is 15%, so I was curious about the nitty gritty of the figures as I'd have naively expected something over 20%. So I looked at the breakdown of their 2012 results. It rather looks to me that their taxation was £318m not £218m. Oopsie, BBC journalists. That's a 22% tax fraction which looks far more reasonable.

It seems entirely unsurprising to me that Rolls Royce has paid no corporation tax in the UK. They have over 20,000 people in the UK; if we assume a low average wage of £25,000 and a tax + employee/employer NI take of about £6000 per person, that's £120m contribution to the Exchequer right there, in good years and bad. This is before any local taxes on RR's substantial commercial properties, and the knock-on effect on Derby's economy - I remember vividly having to pay an unconscionable amount for a dingy hotel room with a sagging bed in Derby, and not much less for a passable Indian meal for four, when meeting with some RR folks. Good times.

Rolls Royce are one of the few UK engineering firms who actually appear to know what they're doing; contrast them with the sharks at BAE Systems if you want to see how good they really are. RR build engines at the top of the line which really perform, don't seem to overrun much and, unlike BAE, repeatedly win in a truly competitive world market. If Chris Williamson doesn't like them conducting all their sales overseas, he should be lobbying the Government to buy more RR engines. Of course, even he can see that without an actual need for those engines, the tax take would be more than offset by the engine sticker price. In the meantime, the USA is probably reaping much of the tax take from RR. You could try asking for it to be repatriated, of course. Good luck with that.

How to run a political campaign like you want to win

The Obama campaign wrote a document "Inside the Cave" documenting their perception of the reasons that their online campaign was (unarguably) so much more successful than Romney's "Project Orca" team.

For me, the key takeaways that emerge from the presentation on why they won:

• Four times as many resources, people, targets than the GOP campaign;
• A massive focus on analytics, which the GOP apparently ignored;
• Recruiting individual credentialed tech staff rather than political wonks and entire corporations (Microsoft in the case of the GOP) for the technology campaign;
• Pitching technical jobs as undesirable ("It won't pay very well. The hours are terrible ... Most people who come to work here will take a pay cut.");
• Dynamic daily reallocation of campaigning resources based on per-state simulations;
• Daily calling of a large number of randomised voters in key states with short questionnaires to obtain data to feed the simulations;
• Tracking "persuadeability" for voter groups to determine whether it was worth trying to convince them to switch/stay with voting intent;
• Use of off-the-shelf open-source software such as R for stats analysis;
• Tracking and categorising Twitter accounts to gauge reaction to local and national political events;
• Greater online fundraising (which I don't think was a big deal - the campaign was going to spend money whether or not it had it, since it would backstopped by the major unions and private contributors);
• Using the mail subject line "Hey" (which I'd bin out of hand, but then I wasn't a USA election voter...)
• Invoking Michelle Obama's name in an email would reduce the amount raised by it;
• Never mind gut feeling for selecting strategy - use hard data;
• Allow people to store credit card info on central website but donate from mobile views via a button click, facilitating "drunk donation";
• Multi-step pages for donations were less off-putting than single-page large forms;
• Recruit experienced devs from major Silicon Valley social media companies (Facebook, Twitter, Google);
• Developing tools earlier in the campaign cycle than they actually did would have improved their campaign efficiency;
• Prepare and drill for the worst (major regional outages) so that when it happens you won't have problems, and you'll have a runbook telling you what to do so you just have to react, not try to problem-solve;
• Use large-scale hosting and content distribution (AWS, Akamai) to obtain well-managed distributed hardware and robust connectivity so you only have to worry about your apps;
• Program backends in Python and associated frameworks (Django, Flask) for rapid development
• Strong presence on social media posting frequent and interesting (reshareable) content;
• Target heavy online and TV ads spend in key states, age, gender, racial sectors.

I don't think it's too much to say that Obama's "Project Narwhal" spanked Romney's "Project Orca" in terms of efficiency and efficacy. Orca appears to have been a relatively traditional software and system development, using major technology vendors and consultants rather than dedicated individuals, and produced the result that anyone who has worked with large companies on a business-critical system has experienced - farce. This is ironic, as Orca typifies the Big Government approach to solving a problem, whereas Narwhal was a very libertarian project - find motivated people, give them a cause and let them work out what to do. It appears that any major companies involved in Orca quickly ran for cover after Romney's defeat; no surprise there.

Cameron and Miliband should be taking notes if they actually intend to win the next UK election. Following in the steps of Narwhal would seem to be critical to winning the social information war that modern elections are becoming.

HR by the numbers

A fascinating post on Slate about how Google HR applies engineering principles to its job:

After crunching the data, Carlisle found that the optimal interview rate—the number of interviews after which the candidate’s average score would converge on his final score—was four. "After four interviews," Carlisle says, "you get diminishing returns." Presented with this data, Google's army of engineers was convinced. Interview times shrunk, and Google's hiring sped up.

That rings true with me. The banks in particular are notable for grilling candidates with ten or more interviews, often back-to-back in a single day. The implication from these findings - and remember, Google has engineering offices all over the globe, so it's not just confined to the West Coast hippies - is that the only benefit of this interviewing approach is as a rite of passage or test of endurance, because it sure doesn't seem to be optimised for finding people whom interviewers agree are suitable. You'd think that a bank would value the time of its interviewers more.

Google have something called PiLab ("people and innovation lab") that seems to be dedicated to running HR experiments on Google engineers and interviewees. They seem to be ruthlessly data-driven, and I'm reminded of the maxim "Talk is cheap, show me the code". I wonder how many HR organisations that I've encountered would handle that kind of demand for proposals backed by (relatively) hard data and stats. I suspect most of the would be curled up weeping under their desks by the time the engineers were dissecting the ludicrously unfounded stats in the second paragraph of their report.

They even determined that good managers, defined by those getting good 360-degree rankings, make a measurable difference to their teams:

When analysts compared the highest- and lowest-performing managers, they found a stark difference—the best managers had lower attrition rates (meaning fewer people left their teams), and their teams were much more productive across a range of criteria.

Of course, with that knowledge, the trick is finding and hiring or promoting people who are going to be those good managers; there's always the risk that a whiff of power might make an otherwise engaging and competent recruit go psycho.

The more mischievous part of me wonders how many of the facts put out by Google in this article are true, and how many are false (or, more likely, "true but misleading") to get Facebook, Cisco, Apple, IBM et al to run down blind alleys in their own recruiting/HR practices...

Dear Daily Mail editors, a mathmo is yanking your chain

The photo accompanying an article claiming "Maths 'too hard for students and dons': Universities drop subject from science courses" is of a blackboard with maths that makes no sense.

I see:

1. a nonsense vector equation
2. a pointless series adjoined without explanation to a meaningless delta equivalence (assuming the series is of t, y is +/1)
3. an equation with a missing right bracket
4. a bastardisation of Boyle's Law
5. the function product differentiation identity
6. some trivial differentiation identities

Whoever is the mathmo who wrote that up for you, he holds journalists in complete contempt. I'm with him.