Scentrics finds that security is hard

Two years ago I wrote about Scentrics and their "Key Man" security proposal. I wondered idly what had happened there so did some Googling. Turns out that I'm the top two hits for [scentrics key man] which is heart-warming for me but suggests that their world-beating security patent might have sunk like a stone...

I went to their website www.scentrics.com and noted that it didn't redirect to https. I tried https://www.scentrics.com and lo! Chrome's Red "Not secure" Warning of Death appears. Seems that Scentrics can't even secure their website, which is not a little ironic when their home page trumpets "Secure with Scentrics".

All the pages on the site - even "Overview and Vision" and "Careers" - are hidden behind a sign-on box, declaring the website "invitation only" and inviting you to contact "admin@scentrics.com" if you'd like access. You can view headers, but that's about it. You wonder why they would be so sensitive about exposing information like that.

The 2016 news included a nugget from the Daily Telegraph in June:

Scentrics is poised to seek new funding that would value the company at more than $1 billion as it prepares to rollout its infrastructure for the first time.

"Poised", huh? I like that. I read that as "not yet ready". I also like the uncritical write-up of the company's pitch:

Individual messages and documents sent over the internet can be unlocked without compromising the overall security of the network, according to Scentrics's pitch to operators and governments.

Remember that this essentially involved encrypting one copy of a message with the recipient's public key, and another with a government/agency public key, and storing the latter to give the agency access on demand. The government and security agencies involved might not think that this "compromises" the overall security of the network, but as a consumer of the network's function I can assure them that I'd feel very differently. And of course for this to be effective all network users would have to use a very small ecosystem of only approved apps / browsers which implemented this dual encryption, and maintained the central repository of government-friendly encrypted messages. I'm sure there's no risk of systematic system compromise there by insiders at all.

Companies House shows three officers plus a secretarial company including our old friend Guruparan "Paran" Chandrasekaran. Looks like Sir Francis Mackay, David Rapoport and Dr. Thaksin Shinawatra resigned since 2014, which is interesting because the latter gent used to be the Prime Minister of Thailand, and Scentrics trumpted his role in the Telegraph piece, but as of 1 month ago he's out of his company role.

According to their June 2015 accounts they have about GBP4.2M in net assets, looks like they had an infusion of about GBP4.5M during the year. Going from this to a $1bn valuation seems... optimistic.

Update: Looks like Scentrics are diving into Singapore with advertisements for Project Manager and Devops roles there. This seems to be part of the Singapore government's "Smart Nation" project for a unified network in Singapore:

• A Smart Nation is one where people are empowered by technology to lead meaningful and fulfilled lives.
• A Smart Nation harnesses the power of networks, data and info-comm technologies to improve living, create economic opportunity and build a closer community.
• A Smart Nation is built not by Government, but by all of us - citizens, companies, agencies. This website chronicles some of our endeavours and future directions.

Cutting through the marketing speak, Singaporeans will be using a government-provided network for all services including personal and business communication. With Scentrics playing a role, the benevolent semi-dictatorship of Singapore will be able to snoop on all its citizens' internal communications at will.

Scentrics seems to be very comfortable enabling a government's surveillance on its citizens. I wonder how this is going to work out for them long-term given the distinctly libertarian tilt of most software engineers.

[Disclaimer: no share position in Scentrics. Financially I don't care if they live or die. Personally, I'd incline towards the latter.]

Don't blame the tech industry for its "lack of diversity"

Tekla S. Perry, who's experienced enough in the technology world to know better, wrote a provocative piece in IEEE Spectrum this week titled "Why Isn't the Tech Industry Doing Better on Diversity? It's Google's and Facebook's Fault". This sprang from a discussion at "Inclusion In Silicon Valley" where Leslie Miley, Slack's director of engineering, excoriated Bay Area tech companies for their alleged lack of inclusion:

You come to Silicon Valley and you don't see people that look like me in positions of power [Miley is black]. If that's not hostile, what is?

You don't see Chinese Americans or Indian Americans in positions of power in the Federal government, despite 8 years of a black president. If that's not hostile to Chinese and Indian Americans, what is?

Leslie Miley is a mendacious asshole. There are many legitimate points to make about the disproportionately small number of black software engineers, and the horrendous educational and societal failings behind that - and let's be clear, prejudice against academically successful black engineers is a real thing from both the black and white communities - but Leslie's point is not one of those. He is jumping from "X is not happening" (observation) to "X must be being blocked by Y" (assumption). You'd think that a competent engineer would be better acquainted with logical reasoning. But looking at Miley's LinkedIn profile he's only spent a series of 2-3 year stints at a list of major tech companies (Google, Apple, Twitter) in engineering management roles; since you spend 3-6 months coming up to speed with a job like that, and assume you draw down effort in the 3 months looking for a replacement job before you leave, his actual engineering experience doesn't seem that great, and you wonder why he kept leaving each firm before his stock options started to vest in quantity... (This is of course the "play the man, not the ball" approach to argument, which is intellectually facile but no less well founded that Miley's approach to argument.)

I've said this before but let's say it again. The main reason that people of Afro-Caribbean descent are under-represented in the software engineering industry is because the dominant education requirement for that industry is a bachelor's degree in a numerical subject (STEM), and such people are correspondingly under-represented in that qualification bucket. Such under-representation is a major issue that needs fixing, but it's happening way before the Silicon Valley and other engineering companies get involved. There's a secondary issue that engineering companies in general should get better at finding bright numerate non-STEM-degree holders who will do well in software engineering with a small investment of training, but that's another blog post entirely - and in any case, Silicon Valley big firms do spend time and money looking in that general area.

It's not just Miley who's making dumb remarks at this diversity love-fest, of course:

The lack of diversity stems from hidden and systemic bias, believes Monique Woodard, a partner in 500 startups. "If you turned off the imported talent, would you look to Oakland and Atlanta? I'm not sure people would," she said.

This is bollocks on stilts, but not just for the reasons you think. Oakland is stuffed full of Bay Area tech workers, especially junior engineers. They live there because it is relatively cheap compared to San Francisco, Palo Alto, San Jose, Milpitas etc. Tech companies recruit people from Oakland all the gosh-darn time. What Monique Woodard means is that she doesn't believe that tech companies will go looking for the black talent in Oakland and Atlanta. Why isn't she saying this explicitly? You be the judge.

"Changing the practices that perpetuate the overwhelmingly white and male character of the Silicon Valley workforce are not going to be easy"

Male: yep. White: nope. In Silicon Valley, Caucasians are actually under-represented per the general population; Chinese and Indians are significantly overrepresented. In my experience, people who openly identify as gay or transgender are also markedly over-represented. By many reasonable measures, Silicon Valley is one of the most diverse environments there is - there is a huge population of people whose national original is not the USA, and they aren't just Indians and Chinese: there are substantial Russian, Korean, Polish, Filipino, Vietnamese and other nationalities.

What Ms. Woodard is actually saying is: "there aren't enough engineers with dark skin - excluding Indians - in Silicon Valley." Well, Ms. Woodard, why is that? Is there a peculiar conspiracy in hiring where the recruiters and hiring deciders are wide open to all sorts of people except those who are of Afro-Caribbean extraction? Is that what you are saying, or is it such a ridiculous notion that you have to resort to camouflaging it behind the umbrella of "diversity"?

Behind Miley's comments, at least, there's a nugget of good sense. The competition for engineers in Silicon Valley and its environs, and to some extent other places like Seattle (Microsoft/Amazon) and New York (Big Finance) is intense. If big firms want to find a cheaper source of good engineers then they should look at other major cities, such as Atlanta, Dallas, Austin. This is something of a risk though: you need to start a new engineering office, which means recruiting many tens of new engineers in addition to migrating some of your existing senior engineers down there to help build and train the teams, reinforce company culture and keep strong communication with the root offices. Up until now, this has been more of a risk than just upping the game in recruiting from the Bay: I suspect soon the numbers will cross a threshold that makes new engineering offices sufficiently financially attractive to be worth a try.

Bringing in new engineers from Republican states such as Texas and Georgia is also excellent for increasing diversity in the heavily Democratic (and worse, Californian) engineering cohorts of Silicon Valley. Yet, why is it that I suspect that Miley, Woodard et al don't regard that kind of diversity as desirable?

neveragain.tech virtue signalling

In the past couple of days I've seen all manner of prompts to add my name to the petition at neveragain.tech, solemnly swearing to:

1. refuse to participate in the creation of databases of identifying information for the United States government to target individuals based on race, religion, or national origin.
2. advocate within our organizations:
• to minimize the collection and retention of data that would facilitate ethnic or religious targeting.
• to scale back existing datasets with unnecessary racial, ethnic, and national origin data.
to responsibly destroy high-risk datasets and backups.
• to implement security and privacy best practices, in particular, for end-to-end encryption to be the default wherever possible. to demand appropriate legal process should the government request that we turn over user data collected by our organization, even in small amounts.
3. if I discover misuse of data that I consider illegal or unethical in my organizations:
• I will work with our colleagues and leaders to correct it.
• If we cannot stop these practices, we will exercise our rights and responsibilities to speak out publicly and engage in responsible whistleblowing without endangering users.
• If we have the authority to do so, we will use all available legal defenses to stop these practices.
• If we do not have such authority, and our organizations force us to engage in such misuse, we will resign from our positions rather than comply.
4. raise awareness and ask critical questions about the responsible and fair use of data and algorithms beyond my organization and our industry.

The more perceptive readers will be surprised at how closely this declaration follows the election of Donald Trump as President of the USA, and wonder why - following the past 8 years of progressive weaponization of the Federal government - the tech industry has suddenly decided that unlimited government power is A Bad Thing to be strenuously resisted.

OK, maybe it's not much of a mystery.

Seriously though, one has to wonder why so many tecchies - who are, on average, very intelligent and somewhat resistant to regular bullshit - are signing this petition. The classic excuse comes from the role of IBM's equipment in the Holocaust, used by the Nazis to process the data around selection and slaughter of Jews in Europe. IBM itself acknowledges its role:

It has been known for decades that the Nazis used Hollerith equipment and that IBM's German subsidiary during the 1930s -- Deutsche Hollerith Maschinen GmbH (Dehomag) -- supplied Hollerith equipment. As with hundreds of foreign-owned companies that did business in Germany at that time, Dehomag came under the control of Nazi authorities prior to and during World War II. It is also widely known that Thomas J. Watson, Sr., received and subsequently repudiated and returned a medal presented to him by the German government for his role in global economic relations.

It's a bit unfair to single out IBM here. The premise is that equipment from an IBM-owned subsidiary was instrumental to the Nazis being able to kill Jews more efficiently. Nowadays, how would we feel if Syria's Bashar Assad used an Excel spreadsheet or two to organise slaughter of non-Alawite citizens? I'm fairly sure that Microsoft's Excel developers couldn't realistically be held accountable for this. Even if a Microsoft sales rep sold a 1000-seat Excel license to the Syrian regime, it would be a bit of a stretch to blame them for any resulting massacre. After all, the regime could always use OpenOffice for a free-as-in-beer-and-freedom solution to programmatic pogrom.

As you might expect from a Silicon Valley initiative, this is primarily intended as strenuous virtue-signalling. "Look at me, how right-thinking I am and how willing to prevent persecution of minorities!" Really though, it will have zero effect. The US Government does not contract out to random Silicon Valley firms for immigration and related database work. They have their own information systems for this, developed at horrific expense and timescales by the Beltway Bandit consulting firms and government IT workers. The US Citizenship and Immigration Services department isn't going to ask Twitter or a San Francisco start-up to develop a new immigrant tracking system - even though I suspect they'd get one with 10% of the downtime and 20% of the cost of the one that the Bandits will develop for them.

The most plausible concern of the signatories is the existing social graph and personally identifiable information in systems like Facebook and Twitter. Religion and national origin isn't stored systematically, and visa status isn't stored at all, but from analysis of posts and relationship activities I can imagine that you could fairly reliably infer areas of the relationship graph that are likely to be e.g. Guatemalan in origin and using Latin American Spanish as their primary language, working in low-wage industries, and physically located in Southern California (checking in from IPs known to be in LA and its environment). If you wanted to identify a pool of likely illegal immigrants, that would be a good place to start. Since Facebook already has this data, and sells access to parts of their information to advertisers, I wonder what these signatories are going to do about it?

$20 says "not a damn thing." They like their jobs and status too much. They won't find other companies as accepting of their social activism and public posturing. They won't take on new jobs targeting minorities, but then no-one sane is going to ask them to take on that kind of work because the D.C. consulting firms want the money instead and have lobbyists ensuring that they'll get it.

Expensive integer overflows, part N+1

Now the European Space Agency has published its preliminary report into what happened with the Schiaparelli lander, it confirms what many had suspected:

As Schiaparelli descended under its parachute, its radar Doppler altimeter functioned correctly and the measurements were included in the guidance, navigation and control system. However, saturation – maximum measurement – of the Inertial Measurement Unit (IMU) had occurred shortly after the parachute deployment. The IMU measures the rotation rates of the vehicle. Its output was generally as predicted except for this event, which persisted for about one second – longer than would be expected. [My italics]

This is a classic software mistake - of which more later - where a stored value becomes too large for its storage slot. The lander was spinning faster than its programmers had estimated, and the measured rotation speed exceeded the maximum value which the control software was designed to store and process.

When merged into the navigation system, the erroneous information generated an estimated altitude that was negative – that is, below ground level.

The stream of estimated altitude reading would have looked something like "4.0km... 3.9km... 3.8km... -200km". Since the most recent value was below the "cut off parachute, you're about to land" altitude, the lander obligingly cut off its parachute, gave a brief fire of the braking thrusters, and completed the rest of its descent under Mars' gravitational acceleration of 3.8m/s^2. That's a lot weaker than Earth's, but 3.7km of freefall gave the lander plenty of time to accelerate; a back-of-the-envelope calculation (v^2 = 2as) suggests a terminal velocity of 167 m/s, minus effects of drag.

Well, there goes $250M down the drain. How did the excessive rotation speed cause all this to happen?

When dealing with signed integers, if - for instance - you are using 16 bits to store a value then the classic two's-complement representation can store values between -32768 and +32767 in those bits. If you add 1 to the stored value 32767 then the effect is that the stored value "wraps around" to -32768; sometimes this is what you actually want to happen, but most of the time it isn't. As a result, everyone writing software knows about integer overflow, and is supposed to take account of it while writing code. Some programming languages (e.g. C, Java, Go) require you to manually check that this won't happen; code for this might look like:

/* Will not work if b is negative */
if (INT16_MAX - b >= a) {
   /* a + b will fit */
   result = a + b
} else {
   /* a + b will overflow, return the biggest
    * positive value we can
    */
   result = INT16_MAX
}

Other languages (e.g. Ada) allow you to trap this in a run-time exception, such as Constraint_Error. When this exception arises, you know you've hit an overflow and can have some additional logic to handle it appropriately. The key point is that you need to consider that this situation may arise, and plan to detect it and handle it appropriately. Simply hoping that the situation won't arise is not enough.

This is why the "longer than would be expected" line in the ESA report particularly annoys me - the software authors shouldn't have been "expecting" anything, they should have had an actual plan to handle out-of-expected-value sensors. They could have capped the value at its expected max, they could have rejected the use of that particular sensor and used a less accurate calculation omitting that sensor's value, they could have bounded the calculation's result based on the last known good altitude and velocity - there are many options. But they should have done something.

Reading the technical specs of the Schiaparelli Mars Lander, the interesting bit is the Guidance, Navigation and Control system (GNC). There are several instruments used to collect navigational data: inertial navigation systems, accelerometers and a radar altimeter. The signals from these instruments are collected, processed through analogue-to-digital conversion and then sent to the spacecraft. The spec proudly announces:

Overall, EDM's GNC system achieves an altitude error of under 0.7 meters

Apparently, the altitude error margin is a teeny bit larger than that if you don't process the data robustly.

What's particularly tragic is that arithmetic overflow has been well established as a failure mode for ESA space flight for more than 20 years. The canonical example is the Ariane 5 failure of 4th June 1996 where ESA's new Ariane 5 rocket went out of control shortly after launch and had to be destroyed, sending $500M of rocket and payload up in smoke. The root cause was an overflow while converting a 64 bit floating point number to a 16 bit integer. In that case, the software authors had actually explicitly identified the risk of overflow in 7 places of the code, but for some reason only added error handling code for 4 of them. One of the remaining cases was triggered, and "foom!"

It's always easy in hindsight to criticise a software design after an accident, but in the case of Schiaparelli it seems reasonable to have expected a certain amount of foresight from the developers.

ESA's David Parker notes "...we will have learned much from Schiaparelli that will directly contribute to the second ExoMars mission being developed with our international partners for launch in 2020." I hope that's true, because they don't seem to have learned very much from Ariane 5.

Journalist ecomonic understanding makes me cry

The megalopolis of San Jose, CA has approved a rise in the minimum wage to $15 by January 1 2019. The usual suspects are weighing in approvingly, but my eye was drawn in fascinated horror to the way that the journalist (or press release author) expressed the financial changes expected:

Mayor Liccardo launched the effort last fall to follow the lead of five other cities in Santa Clara County and to come up with a regional approach to raise minimum wage throughout Silicon Valley.
City statistics show it would mean a $300,000 raise for 115,000 workers.

To which I can only say huh? Assuming they're on $12/hour now, they're working 100,000 hours per year?

What the author means, one assumes, is that each worker is going to benefit by just under $3 per hour, but that's a horrible way of expressing that statistic. And of course, the statistic itself is misleading. The workers are going to pay a varying amount of tax on that additional money, other benefits they are currently paid may change, and of course that assumes that otherwise their salary would not have risen at all by January 2019 despite the extra 2 years of experience and possible promotion they would have achieved by then.

But let's look at what the author believes is the downside of this measure - because they're trying to be even-handed, yes?

Some small business owners and non-profits worry raising the minimum wage would reduce their share of the economic pie. The result could either mean service reduction for non profits or price increases for mainstay businesses.

Or, you know, firings left and right for any worker whose skills aren't valued at $15/hour (plus additional costs) by the business they work at. Or businesses closing down because they're no longer economically viable. Or employers cutting existing worker benefits to offset the new costs. Heck, ask workers and business owners in Seattle how their new $15/hour minimum is working out.

You can just taste the disdain for business owners in the expression "reduce their share of the economic pie". Why exactly does the author think the owners have put in all the work and risk to create the businesses that create the jobs for these good people in the first place?

Always consider what happens when the shoe switches feet

The recent panic from the LGBT+ / Black / Hispanic communities about increased violence in the wake of Trump's victory has caused a sharp uptick in blogs and forum posts from various West Coast people, notably those of the transgender persuasion, claiming a new fear for the personal safety of them and their families. This seems to be based around the assumption that a Trump presidency will embolden the less savoury side of society prone to gay-bashing to perpetrate physical violence on them. Let's say, for arguments' sake, this is true: what should they do about it?

Larry Correia, author of the "Monster Hunter Nation" and related high-output high-sales fantasy book series, penned "A Handy Guide For Liberals Who Are Suddenly Interested In Gun Ownership" which is as sympathetic to the political gripes of Hillary/Bernie supporters as the title suggests, but does provide a lot of good practical advice about how you can go about getting armed and trained in effective self-defence. Correia owned a gun store and did a lot of concealed-carry training before his literary career properly started, so seems to know what he's talking about.

What he really nails is the ever-increasing squeeze on firearms possession, gun ranges and ammo purchase that has been happening in Democrat-controlled states over the past few years, and why it's relevant now:

When the already super powerful government wants to make you even more powerless, that scares the crap out of regular Americans, but you guys have been all in favor of it. Take those nasty guns! Guns are scary and bad. Don't you stupid rednecks know what's good for you? The people should live at the whim of the state!
But now that the shoe is on the other foot, and somebody you distrust and fear is in charge for a change, the government having all sorts of unchecked power seems like a really bad idea, huh?

It's hard enough owning a gun in California anyway, but cities like San Francisco have taken it to extremes. They have used local law changes to force all the gun shops to close down. In last week's voting, there was a strong San Francisco representation pushing state Proposition 63 to make ammunition purchases harder and more expensive. The net effect is that you can guarantee that no-one in San Francisco is carrying a gun unless they're a law enforcement officer or a criminal.

Gay bashing is far from a new crime in San Francisco. Despite the city's image as gay-friendly, there are enough unreconstructed citizens who are not keen on public displays of homosexuality or trans people for there to be a significant risk of violence. Since these folk know that their victims won't be armed, they have no disincentive to engage in these attacks. But if there were a few well-publicised self-defence shootings in reaction to gay bashing attempts, you can bet that the rate of gay bashing attempts would decline rapidly.

For now, California citizens have to deal with the laws as they stand - and as Correia notes, those laws make it hard for law-abiding citizens to be armed effectively:

See, traditionally Democrats don't like the 2nd Amendment and historically have done everything in their power to screw with it. Your gun laws are going to vary dramatically based upon where you live. It might be really difficult and expensive for you to exercise your 2nd Amendment rights, or it might be relatively easy.
But you’re scared right now! Well, that's too bad. Because for the most part Democrats have tried to make it so that citizens have to abdicate their responsibilities and instead entrust that only [the] state can defend everyone... That doesn't seem like such a bright idea now that you don't trust who is running the state, huh?

Perhaps San Francisco Mayor Ed Lee could take time out from his crusade against the gun industry to ensure that his vulnerable constituents can defend themselves against the increasing violence in his city. I'm not holding my breath for this to happen, but if the LGBT+ community wants to be able to protect themselves then Ed might be a good target for their lobbying. "Mayor Lee, why don't you want the gay community to be safe in your city?". They could recommend that Lee work with past SF Democrat mayoral candidate Leland Yee to draw on the latter's expertise in firearms supply.

Silicon Valley in the Time of Trump

The past few days have given me a great view into how the famously liberal population of the Bay Area has taken the election of Donald Trump. "Not well" is fair, but a yuuuuge understatement.

Do you know what California's principal export is? Whine.

The Bay Area is probably the most pro-Clinton anti-Trump group outside the island of Manhattan, and the residents were never going to be entirely happy with a Trump victory. I predicted butthurt-ness, and was I ever right. However even I, with my jaundiced view of human nature, never expected the level of rage and opprobrium directed at Trump and his voting enablers. So far I've seen - not heard but actually seen written on group emails and forums - the following:

• claims of suicidal feelings, particularly from trans and gender-fluid folks;
• assertions that anyone voting for Trump needs to publicly denounce Trump's perceived opinions about Black Lives Matter, Hispanics, gays (wut?) and immigrants;
• statements that anyone voting for Trump needs to go work for another company;
• room-sized group hugs to support each other post-election; and
• claims that Trump and Pence wanted to electrocute people who were gay or trans.

Thank goodness Trump has elephant-thick skin, because there's probably enough libel in every Bay Area tech company's emails to pay for the building of another Trump Tower.

The straw that broke the camel's back for me was a bundle of complaints around the theme:

"I was hoping to teach my girls that, if you work hard and dream big, you can be anything you want to be. I would like to thank 2016 for putting me right."

It seems that a large number of people were going to use "Hillary as first woman president" as the totem for their children to show that the glass ceiling had been shattered. While I'm all in favour of showing children role models, is Hillary really the model you want to use?

I actually found it inspiring, in a way. The lesson I took from the election was that if you are a woman, even if you are a revolting and corrupt human being, you can make it to within a gnat's chuff of being the President of the United States, and your party organisation will happily screw over men to help you get its nomination. It wouldn't have taken much of a vote change in one or two swing states for Hillary to be elected, at which point I guarantee that no-one on the Dems side would be talking about upsetting the electoral college applecart.

Hillary is (of course) not happy and blames FBI Director Comey for her narrow defeat:

But our analysis is that [FBI Director James B.] Comey's letter raising doubts that were groundless, baseless, proven to be, stopped our momentum,” she said. “We dropped, and we had to keep really pushing ahead to regain our advantage — which going into the last weekend, we had."

She's right, of course. Comey's letter was quite possibly enough to cause Hillary voters in key states to stay home on polling day.

On the other hand, there were many other what-ifs, any one of which was probably enough to get her elected:

• what if she had actually achieved something of note as Secretary of State?
• what if she and Bill hadn't gone around the world soliciting hundreds of millions of dollars from various dubious countries and individuals?
• what if she were actually personally likeable?
• what if she'd not blown her chance to land a kill-shot on The Donald in the debates?
• what if she'd insisted that the DNC not put its thumb on the scales, and instead beat Bernie fairly in the nomination?

All these were in her control, so to blame solely Comey for her loss seems rather obtuse.

And on the flip side, what if Comey had taken the - apparently quite reasonable - step to indict her for her recklessness in running her own email server and exposing any amount of State classified material to any intelligence service worth its name? Isn't she grateful to him for not doing that, at least?

Trump triumphant

Blimey, he actually did it. Just how poor a candidate must Hillary have been, with all the media, technical, organisational and financial advantages she had, to go down so badly to Trump? I'm guessing that Hillary 2020 is not going to be a thing.

I continue to feel very comfortable in my prediction of an unprecedent wave of butthurt about to appear from the Guardian opinion pages (and indeed all other articles) and the BBC US correspondents.

2016 US election prediction

It's less than 24 hours before we'll have a good idea whether Hillary Clinton has made it to the 270 electoral college votes needed to secure the presidency to which she clearly believes she's entitled. At this stage, although I wouldn't write off Trump, I'd have to say that Hillary is likely to make it. Her Get-Out-The-Vote ground game is much better organised than Trump's, Wikileaks and the FBI haven't landed a killer blow on her, and the media have carried water faithfully enough to keep most of her followers following. I'm sure a lot of Bernie supporters are extremely unhappy with the revelations of past weeks, but I suspect most of them will hold their noses and vote Hillary nevertheless.

Should The Donald continue his trend of confounding predictions and actually pull off an upset - winning Florida, Pennsylvania and such other states as needed to break 270 - I confidently predict the most ear- splitting snit of all times from 95% of the US media. Hillary herself might actually evaporate in a toxic plume of rage. It would be quite something to watch.

DDoS and the Tragedy of the Commons of the Internet of Things

On Friday there was a massive Distributed Denial of Service attack on DynDNS, who provide Domain Name services to a number of major companies including Twitter, Spotify and SoundCloud, effectively knocking those sites offline for a significant fraction of the global population. Brian Krebs provides a useful summary of the attack; he is unusually well versed in these matters because his website "Krebs on Security" was taken offline on 20th September after a massive Internet-of-Things-sourced DDoS against it. It seems that Krebs' ongoing coverage and analysis of DDoS with a focus on the Internet of Things (IoT) - "smart" Internet connected home devices such as babycams and security monitors - raised the ire of those using the IoT for their nefarious purposes. It proved necessary to stick Krebs' blog behind Google's Project Shield which protects major targets of information suppression behind something resembling +5 enchanted DDoS armour.

Where did this threat to the Internet come from? Should we be worried? What can we do? And why is this whole situation a Tragedy of the Commons?

Primer on DNS

Let's look at Friday's outage first. Dyn DNS is a DNS hosting company. They provide an easy way for companies who want a worldwide web presence to distribute information about the addresses of their servers - in pre-Internet terms, they're like a business phone directory. Your company Cat Grooming Inc., which has bought the domain name catgrooming.com, has set up its web servers on Internet addresses 1.2.3.4 and 1.2.3.5, and its mail server on 1.2.4.1. Somehow, when someone types "catgrooming.com" in their internet brower, they need that translating to the right numerical Internet address. For that translation, their browser consults the local Domain Name Service (DNS) server, which might be from their local ISP, or a public one like Google's Public DNS (8.8.4.4 and 8.8.8.8).

So if Cat Grooming wants to change the Internet address of their webservers, they either have to tell every single DNS server of the new address (impractical), or run a special service that every DNS server consults to discover up to date information for the hostnames. Running a dedicated service is expensive, so many companies use a third party to run this dedicated service. Dyn DNS is one such company: you tell them whenever you make an address change, and they update their records, and your domain's information says that Dyn DNS does its address resolution.

To check whether a hostname on the web uses DynDNS, you can use the "dig" command which should work from the Linux, MacOS or FreeBSD command line:

$ dig +short -t NS twitter.com
ns3.p34.dynect.net.
ns2.p34.dynect.net.
ns1.p34.dynect.net.
ns4.p34.dynect.net.

This shows that twitter.com is using Dyn DNS because it has dynect.net hostnames as its name servers.

Your browser doesn't query Dyn DNS for every twitter.com URL you type. Each result you get back from DNS comes with a "time to live" (TTL) which specifies for how many seconds the answer is valid. If your twitter.com query came back as 199.59.150.7 with a TTL of 3600 then your browser would use that address for the next hour without bothering to check Dyn DNS. Only after 1 hour (3600 seconds) would it re-check Dyn DNS for an update.

Attack mechanism

The Internet of Things includes devices such as "babycams" which enable neurotic parents to keep an eye on their child's activities from elsewhere in the house, or even from the restaurant to which they have sneaked out for a couple of hours of eating that does not involve thrown or barfed food. The easiest way to make these devices accessible from the public Internet is to give them their own Internet address, so you can enter that address on a mobile phone or whatever and connect to the device. Of course, the device will challenge any new connection attempt for a username and password; however, many devices have extremely stupid default passwords and most users won't bother to change them.

Over the past decade, Internet criminals have become very good at scanning large swathes of the Internet to find devices with certain characteristics - unpatched Windows 2000 machines, webcams, SQL servers etc. That lets them find candidate IoT devices on which they can focus automated break-in attempts. If you can get past the password protection for these devices, you can generally make them do anything you want. The typical approach is to add code that makes them periodically query a central command-and-control server for instructions; those instructions might be "hit this service with queries randomly selected from this list, at a rate of one query every 1-2 seconds, for the next 4 hours."

The real problem with this kind of attack is that it's very hard to fix. You have to change each individual device to block out the attackers - there's generally no way to force a reset of passwords to all devices from a given manufacturer. The manufacturer has no real incentive to do this since it has the customer's money already and isn't obviously legally liable for the behavior. The owner has no real incentive to do this because this device compromise doesn't normally materially affect the device operation. You can try to sell the benefits of a password fix - "random strangers on the internet can see your baby!" but even then the technical steps to fix a password may be too tedious or poorly explained for the owner to action. ISPs might be able to detect compromised devices by their network traffic patterns and notify their owners, but if they chase them to fix the devices too aggressively then they might piss off the owners enough to move to a different ISP.

Why don't ISPs pre-emptively fix devices if they find compromised devices on their network? Generally, because they have no safe harbour for this remedial work - they could be prosecuted for illegal access to devices. They might survive in court after spending lots of money on lawyers, but why take the risk?

Effects of the attack

Dyn DNS was effectively knocked off the Internet for many hours. Any website using Dyn DNS for their name servers saw incoming traffic drop off as users' cached addresses from DNS expired and their browsers insisted on getting an up-to-date address - which was not available, because the Dyn DNS servers were melting.

Basic remediation for sites in this situation is to increase the Time-to-Live setting on their DNS records. If Cat Grooming Inc's previous setting was 3600 seconds, then after 1 hour of the Dyn DNS servers being down their traffic would be nearly zero. If their TTL was 86400 seconds (1 day) then a 12 hour attack would only block about half their traffic - not great, but bearable. A TTL of 1 week would mean that a 12 hour attack would be no more than an annoyance. Unfortunately, if the attack downs Dyn DNS before site owners can update their TTL this doesn't really help.

Also, the bigger a site is, the more frequently it needs to update DNS information. Twitter will serve different Internet addresses for twitter.com to users in different countries, trying to point users to the closest Twitter server to them. You don't want a user in Paris pointed to a Twitter server in San Francisco if there is one available in Amsterdam, 500 millseconds closer to them. And when you have many different servers, every day some of them are going offline for maintenance or coming online as new servers, so you need to update DNS to stop users going to the former and start sending them to the latter.

Therefore the bigger your site, the shorter your DNS TTL is likely to be, and the more vulnerable you are to this attack. If you're a small site with infrequent DNS updates, and your DNS TTL is short, then make it longer right the hell now.

Alternative designs

The alternative to this exposed address approach is to have a central service which all the baby monitors from a given manufacturer connect to, e.g. the hostname cams.babycamsRus.com; users then connect to that service as well and the service does the switching to connect Mr. and Mrs. Smith to the babycam chez Smith. This prevents the devices from being found by Internet scans - they don't have their own Internet address, and don't accept outside connections. If you can crack the BabyCams-R-Us servers then you could completely control a huge chunk of IoT devices, but their sysadmins will be specifically looking out for these attacks and it's a much more tricky proposition - it's also easy to remediate once discovered.

Why doesn't every manufacturer do this, if it's more secure? Simply, it's more expensive. You have to set up this central service, capable of servicing all your sold devices at once, and keep it running and secure for many years. In a keenly price-competitive environment, many manufacturers will say "screw this" and go for the cheaper alternative. They have no economic reason not to, no-one is (yet) prosecuting them for selling insecure devices, and customers still prefer cheap over secure.

IPv6 will make things worse

One brake on this run-away cheap-webcams-as-DoS-tool is the shortage of Internet addresses. When the Internet addressing scheme (Internet Protocol version 4, or IPv4 for short) was devised, it was defined as four numbers between 0 and 255, conventionally separated by dots e.g. 1.2.3.4. This gives you just under 4.3 billion possible addresses. Back in 2006 large chunks of this address space were free. This is no longer the case - we are, in essence, out of IPv4 addresses, and there's an active trade in them from companies which are no longer using much of their allocated space. Still, getting large blocks of contiguous addresses is challenging. Even a /24 (shorthand for 256 contiguous IPv4) is expensive to obtain. Father of the Internet Vint Cerf recently apologised for the (relatively) small number of IPv4 addresses - they thought 4.3 billion addresses would be enough for the "experiment" that IPv4 was. The experiment turned into the Internet. Oops.

This shortage means that the current model where webcams and other IoT devices have their own public Internet address is unsustainable: the cost of that address will become prohibitive, and customers will need something that sits behind their single home Internet address given to them by their ISP. You can have many devices behind one address via a mechanism called Network Address Translation NAT) where the router connecting your home to the Internet lets each of your devices start connections to the Internet and allocates them a "port" which is passed to the website they connect to: when the website server responds, it sends the web page back to your router along with the port number, so the router knows which of your home devices the web page should be sent to.

The centralized service described above is (currently) the only practical solution in this case of one IP for many devices. More and more devices on the Internet will be hidden from black-hat hacker access in this way.

Unfortunately (for this problem) we are currently transitioning to use the next generation of Internet addressing - IPv6. This uses 128 bits, which is a staggering number: 340 with an additional 36 zeroes after it. Typically your ISP would give you a "/64" for your home devices to use for their public Internet addresses - a mere 18,000,000,000,000,000,000 (18 quintillion) addresses. Since there are 18 quintillion /64s in the IPv6 address space, we're unlikely to run out of them for a while even if ever person on earth is given a fresh one every day and there's no re-use.

IPv6 use is not yet mainstream, but more and more first world ISPs are giving customers IPv6 access if they want it. Give it a couple of years and I suspect high-end IoT devices will be explicitly targeted at home IPv6 setups.

Summary: we're screwed

IPv4 pressures may temporarily push IoT manufacturers to move away from publicly addressable IoT devices, but as IPv6 becomes more widely used the commercial pressures may once more become too strong to resist and the IoT devices will be publicly discoverable and crackable once more. Absent a serious improvement in secure, reliable and easy dynamic updates to these devices, the IoT botnet is here to stay for a while.

Hillary doesn't deserve to be President

I've just finished watching the #2 US Presidential Debate, chaired by Anderson Cooper - for whom I have a reasonable amount of respect as a more-fair-than-average interviewer - and Martha Raddatz, who was hopelessly out of her depth and showing awful bias. Coming out of the debate, I have one question for Hillary: how, with all the advantages you had two hours ago, did you manage to lose?

Going into this debate, Hillary had Donald cornered by the media after his not terribly edifying 2005 remarks about pussy-grabbing opportunities in showbiz were reported. Near-universal media agreement was that The Donald was fatally holed beneath the waterline. Even Trump's own Vice President pick, Pence, was publicly disapproving of Trump's comments. Republican senators and Congress critters were denouncing Trump and saying they wouldn't vote for it. In golf, this would be like being 2 inches away from the hole when your opponent is 200 yards away in a bunker, and it has just started to rain.

And yet... Hillary missed the putt, kept missing it, and Donald chipped his ball onto the green and snuck it into the hole before Hillary found her game.

Trump is not a great public speaker. His train of thought wanders as he speaks, and he assumes technical and factual knowledge in the audience rather than explaining as he goes along. These traits were in full display this evening. A great example was in the "birther" issue where Hillary accused Trump of asking the "racist" question about whether President Obama had actually been born in the USA. Trump (accurately) pointed out that this issue had first been raised by Hillary's consiglieri Sidney Blumenthal, but he did it in such an indirect way that anyone not substantially familiar with the people concerned would have had no idea what he was talking about and how it was tied to Hillary.

Still, somehow he did a better job of debating than Hillary herself. Tonight's debate format seemed to work better for him, because he's comfortable doing spontaneous exposition on topics. Hillary is awful at this, visibly working her way through pre-prepared points on each topic rather than going with the flow of the question and debate. Trump was prone to wander off the thread to include the attacks he wanted to make on Hillary (Bill's disbarring, Russia, black poverty, Syria, tax policy and of course her email server) but seemed to make most of it stick and force Hillary to respond.

Raddatz did her best to cover for Hillary's poor quality responses - Cooper, to his credit, did not - but it seemed clear to me that Trump had managed to bring up nearly all the Hillary dirty laundry that he had avoided in the first debate. Hillary did a variable job in responding to these points, but looked really weak on Russia/Syria, and her responses on the email server were strong but - frankly - flat-out lies. If Donald could learn to speak with more clarity and focus, he'd crucify her. As it was, this was a win on points only, but compared to expectations Donald killed it tonight.

Why was the pussy-grab tape such a non-event in this debate? I think it was because of the apology. Trump apologised for what he said on the tape a few hours after it was publicised, and did so again in the debate as soon as it was brought up. Once he'd done that, it was much harder for Hillary to use it as leverage. "He said these horrible things!" "I've apologised for that, you heard me." Where do you go from there? You can try "this shows what he thinks about women!" but Trump was willing to go on the offense about Bill Clinton and his bimbo eruptions - perhaps the lack of challenge in this area is a sign of how vulnerable Hillary thinks she is here.

By contrast, Hillary's mea culpa for the email server still had a whiff of "I'm sorry I got caught" - her assertions around "no evidence that anyone hacked the server" were incredibly weaselly. A responsible candidate would have agreed that it was quite likely that unfriendly nations had got at least some access to that server, and taken personal responsibility for any consequences arising from their decision to use it.

Conclusion? It's still game on for November 8th. Somehow Donald has mitigated the worst of the impact of the pussy-grab, and is challenging Hillary on the issues again. What other gotchas for him has she got left to leak? Are they good enough to be game-ending, or are they just "the same again"?

Why jail women at all?

I've noticed increasing concern among UK media column writers over the past year about the situation of women in prison, with a clamour to reduce - if not eliminate - the practice of sending women to jail. A good example is this column from Eric Allison in (where else?) the Guardian, late last year: "Women are dying in jails they should not have been sent to":

Many female prisoners are mothers and primary carers. Every year, around 18,000 children are affected by their mother being sent to jail. As women are usually the main caregiver, many end up in care. We can only guess how much that adds to the anguish of mothers behind bars.

A compelling argument to be sure.

Let us turn to the case of Eunice Spry from Gloucestershire, who was sent down for 14 years at Bristol Crown Court in 2007:

Judge Simon Darwall-Smith told the devout Jehovah's Witness that this was the worst case he had come across in 40 years.
Passing sentence, he said: "It's difficult for anyone to understand how any human being could have even contemplated what you did, let alone with the regularity and premeditation you employed."
As punishment for misbehaving, she would beat the children on the soles of their feet and force them to drink washing-up liquid and bleach.

I'm sure Eunice Spry's children were affected by her being sent to jail, but I'd imagine it's more along the lines of thanking God that she was finally kept away from them.

Her defence brief did his best to mitigate, but had something of an uphill struggle:

Mr Mitchell also revealed that Spry had needed protection in prison following her convictions and it was a "particularly unpleasant" place for her.

To which I'd be minded to respond "Et alors?" I hadn't realized before reading the detailed verdict that she was also convicted of "Intimidating a juror or witness or person assisting, or who has assisted, the investigation of an offence" - this is not just a woman who made a few bad choices.

Spry was of course eligible for parole in April 2014 and (of course) was released on schedule - the 14 year imprisonment sentence was reduced to 12 years on appeal.

There's certainly an argument that people are being sent to jail for crimes which are not obviously harmful to society - for example, possession of substantial quantities of narcotics but no obvious intent to supply outside their circle of dysfunctional friends - but let's not special-case women in this argument. If we are serious about gender equality, we should apply the same standards to the decision about jailing a father that we do about deciding to jail a mother. Otherwise we're perpetuating serious inequality in the application of the law to men and women - and isn't that something an enlightened society should want to fix?

Does Putin want Trump as President?

I'm a huge fan of thoughtful blogger Richard Fernandez from Belmont Club, but respectfully have to disagree on his take on the current Wikileaks leaking of Democratic National Catfight emails and voicemails:

By striking at Hillary's aura, the Russians may be attempting the same thing. Democratic voters looked up to her to protect and defend the nation because that's what presidents do. By hacking Hillary and humiliating her, Putin has sent the message that she cannot even defend herself -- and what's the use of a president who can't defend herself?

This is an excellent point, except that - despite the tone of publicity - Hillary is not actually President of the United States. She's locked in a deadly struggle with Donald Trump for the title, and the decision won't happen until November.

I have no trouble at all believing that the Russians have the goods on Hillary. FBI Director Comey's statement on the Clinton private email server left little doubt that any competent foreign security service would have gained complete access to her communications, and have any amount of blackmail material on her and on her confidants. But if you're playing poker and have four kings, why would you all-but-announce this at the start of bidding?

Wikileaks has doubtlessly been compromised by Russian security services, but such compromise is covert - the SVR doesn't have an editorial veto - and it still provides a low-friction platform for publicising controversial data. This is a classic example of a disgruntled insider publicising information to hurt someone they loathe; Wikileaks is just the medium.

If you doubt this assertion, ask yourself: if you were Putin, with whom would you want to negotiate? Trump who is well-established as a wildcard who could say or do anything, and is (in practice) very hard to blackmail because of all the unsavory facts which are already public? or Hillary who still tries to project an aura of robustness and foreign intelligence savvy from her time at State, and whose private email correspondence you have available on request?

I'm starting to believe that May is trolling the Guardianistas

I thought that the chorus of butthurt from the why-didn't-the-plebs-listen-to-ME part of the Remain camp was finally starting to die down, but then May appointed Johnson as Foreign Secretary, and oh my goodness. My Twitter feed and Farcebook timeline have erupted in caterwauling once again.

Note that this has the effect of focusing the limited Guardianista attention on Johnson and his various alleged[1] faux pas, and there's been very little comment on the appointment of the sharp and strongly pro-Brexit David Davis as "Minister for Brexit". I rather suspect Davis is going to be the source of most of the actual heartache for the Remainers in the next couple of years.

[1] Most of which I suspect they're overselling. Johnson has his flaws, Heaven knows, but he's a smart cookie, extremely well travelled, with a highly multinational family. And I'd endorse him as Foreign Secretary solely on the basis of his trolling of the Chinese about ping pong at the Beijing Olympics.

Denatonium Benzoate loses its crown

Also known as Bitrex, Denatonium Benzoate held the record for the most bitter substance on earth until 24th June 2016. A teaspoon of the substance added to an Olympic size swimming pool (volume 2.5M litres) makes the water noticeably bitter. Bitrex has been a very successful additive to poisonous substances to prevent accidental ingestion, such as car antifreeze.

Sadly for manufacturer Macfarlan Smith, since 24th June Bitrex's record has been overtaken by the UK Guardian opinion page. One opening paragraph has the same bitterness impact as approximately 300ml of Denatonium Benzoate. Rumours suggest that Macfarlan Smith has opened negotiations with Jonathan Freedland, Nick Cohen and Polly Toynbee for purchase of their spleens as a manufacturing source of the Bitrex successor.

It is serendipitous that the name "Bitrex" is an anagram of the new product: "Brexit".

Toys firmly out of prams

I predicted a certain amount of tantrums, but really didn't think it would get this bad this quickly. Scotland and London wanting to split off and rejoin Europe, Labour Party stalwarts gunning for Corbyn (who, up until a couple of hours ago, must have thought he'd played a blinder) and Twitter and Facebook in meltdown with Remainers calling Leavers "racist idiots" and worse.

Heavens sake, you're all adults, bloody act like it. This was a full national referendum with a turnout of 74% which is way above recent elections. If your side lost, sit down and put up with it. Don't whine like a three year old deprived of an ice cream. Leave seem to have been a heck of a lot more restrained in their unexpected win than you'd have been in their place.

Not entirely surprised by Cameron chucking the towel in. He seems to be one of the few people today (and maybe the only Remainer) acting with dignity.

Referendum predictions

I have no idea on the actual result. I don't think I could place a bet if I was offered 50:50 odds on each choice. That said, the breakdown by region is going to be very interesting, and I wonder if the rain/floods will hit turnout in the SE, and whether that will make a material difference.

If "Remain" wins: The Guardian (and, less obviously, BBC) will be insufferable. Juncker et al will keep true to their promise not to give any concessions to the UK, even if the result is knife-edge. UKIP effectively dissolves in a frenzied pit of backbiting. Who knows what the UKIP voters will do at the next election?

If "Leave" wins: Immediate witch-hunt from Guardian, BBC. Cameron resigns. Panic in Europe. Stock markets burning. Sweden and maybe Denmark start feeling popular pressure to exit or form referendum. Juncker et al refuse any trade deals with the UK. Boris's hair a fixture on the international news.

I've observed my Facebook stream becoming increasingly stridently pro-Remain over the past 2 weeks. The Leavers are keeping very quiet, presumably because they're swamped by insufferable Remainers if they post anything. Remain posts seem to be relatively free of Leaver comments. So is this due to Remain having an insurmountable majority, due to me having a supermajority of Remain friends, or because the Leavers don't care what the Remainers think or do?

Going by their selection of stories and interviewees, the BBC have steadily abandoned impartiality over the past couple of weeks. The only really studiously neutral Beebite I've seen has been the indefatigueable Kuenssberg.

Weasel will find a way

After the furore last year when it turned out that UK airport shops were demanding boarding passes to save themselves VAT but not save you any money I assumed that this was the effective end of the weasel. From my recent experience at Birmingham International (motto: "We put the 'slack jaw' in 'security'") it seems not.

First stop: the bookshop, to buy some doorstop-sized illiterate literature. No shortage of supply. I present the volume to the lady at the till who demands: "Boarding pass?" with no hint of shame. I enquire whether it's actually mandatory, at which point she rings up the transaction with no further questions. 1-0.

Next stop: W H Smith, for a magazine. Avoiding the single human-manned till I opt for the self-service till. I scan the magazine for a grand total of £2.50 - and it asks for a boarding pass, and won't proceed until I scan one. I hit the "my boarding pass won't scan" button, wait a minute for the roaming attendant to punch the override and proceed on my way. But hell, I remember the huge fuss in August 2015 about this. It seems that the airport shops were content to let the hubbub die down, then go back to their old ways.

Don't let them do this! Make them pay a cost in salaried worker time for each time they demand a boarding pass. Once the average worker salary rate times delay is more than the expected VAT, they will shut up about the boarding passes and let us buy our dubious literature un-monitored and without delay. (Until 1-2 years later when some bright MBA spark spots an opportunity to re-introduce the practice, at which point we hang them from the Heathrow radar pillar as a warning to others.)

The implications of the "Out" threats

With the UK In/Out referendum less than three weeks away, and the BetFair odds on "Leave" starting to come down - albeit still very far from 50-50 - it has been instructive to listen to the veiled, and not so veiled, threats about what will happen if the populace vote for "Leave".

A good example was the comment in late May from Jean-Claude "Piss Artist" Juncker, European Commission President:

"The United Kingdom will have to accept being regarded as a third country, which won't be handled with kid gloves.
"If the British leave Europe, people will have to face the consequences -- we will have to, just as they will. It's not a threat but our relations will no longer be what they are today."

Apparently EU officials don't want to have lengthy negotiations[1] with a Brexited UK, which makes sense. But of course, the easiest course for both sides would be to retain status quo ante: continue trade under the same conditions and tariff schedule as before. Why wouldn't this be the starting point? In general, trade tariffs hurt the populace of the country / countries that impose them: they make imported goods more expensive for their populace. The main function of trade tariffs is to protect local industry from "abuse" from "dumping" by foreign manufacturers: selling goods below the cost of local produce. This may not be good for local industry, but it's certainly good for anyone who wants to buy those goods, at least in the short term.

It seems fairly clear that, whatever the merits of the "Remain" and "Leave" positions, the EU establishment is happy to cut off its population's noses to spite the UK's faces. One has to ask: if national membership of the EU is supposed to be of benefit to the population, why would the EU take action to screw over all their population in order to punish a member nation that wanted to leave?

[1] Note that the EU can apparently spare the manpower to negotiate a mostly free trade agreeement with Canada, which has half the population and a bit more than half the GDP of the UK.

I'm starting to think that Trump might just pull this off...

Trump's political opponents seem hell-bent on getting him elected. Dixit Linus Torvalds, father of Linux and otherwise political moderate:

It used to be that the only thing that made Donald Trump look good was comparing him with the other Republican candidates. Because even a whiny five-year old megalomaniac looks positively stellar when compared to a religious nut who loves the death penalty.
Now, those other Republican candidates are gone. That should make for a saner baseline, no?
No.
These days, it's the anti-Trump protesters that make "the Donald" look good in comparison.
Christ, people. You're doing it wrong.

One can only assume that this is in reference to the sustained violence at the Trump rally in San Jose, CA last night which seemed to be perpetrated by a motley crew of students, Mexican nationalists and union-backed thugs and involved Trump supporters being pelted with eggs, sucker-punched, and clubbed on the side of the head. I watched the videos and it was indisputably appalling. The American Constitution has the First Amendment which guarantees the right to free speech; as P. J. O'Rourke remarked, it also implies the responsibility to live with the consequences. If you vocally support Trump because you hate people with brown skin, you're an asshat and the concomitant public opprobrium is your problem. But if you are physically attacked for supporting the Republican party candidate for President, then there are other laws which should come into play and they should be squarely aimed at - and enforced on - your attacker.

The Bay Area news organisations - with the commendable exception of KRON 4 were carefully keeping the lid on reports of the violence last night. Even CNN sat on it until reporting on the violence was unavoidable; even then, there were strenuous efforts to deflect the blame towards Trump. San Jose mayor Sam Liccardo's comments were particularly awful:

"Our police officers have done an extremely courageous and professional job so far," Liccardo told The Associated Press Thursday night. "At some point Donald Trump needs to take responsibility for the irresponsible behavior of his campaign."

Yes, heaven forfend that a Presidential candidate actually speak clearly about his intentions to enforce the law of the land and secure a nation's borders. There are very reasonable arguments to be had about whether this is a good idea or not, but the implicit blaming of Trump for the actions of the protestors was disgraceful. Liccardo has the luxury of an electorate who would vote him in based on party affiliation even if it came out that he framed OJ, spied for China, and buggered raccoons on his free weekends, so the concept of trying to win an election based on popular policy is doubtless alien to him. His blatant repudiation of the First Amendment might well be related to metropolitan California's sustained attack on the Second Amendment, but neither does him any credit.

Faced with a Twitter firestorm, he tried to walk this back later on:

I condemn all acts of violence committed against people who exercise their rights to free speech and assembly pic.twitter.com/PumCWdfWnT

— Sam Liccardo (@sliccardo) June 3, 2016

but it's clear where his sympathies lie.

If I were Donald Trump, I'd be campaigning from now until November in Democrat stronghold cities around the USA. It won't win me those states, but the widely-reported predictable riots and abuse from the opposition will steadily win me marginal voters in every marginal state around the country. Even if those marginal voters can't stand me (or my hair), they'd rather be with me than the scumballs throwing eggs and beating up women.

Hillary Clinton to give counter-terrorism speech at Stanford University

"Do as I say, not as I do!" Hillary Clinton will urge listeners to stand in solidarity with Europe in order to defeat the Islamic State group:

"Today's attacks will only strengthen our resolve to stand together as allies and defeat terrorism and radical jihadism around the world."

Heaven forfend that we strenuously deny any connection between organized terrorism and this event and instead speciously blame random YouTube videos for incitement.

Hillary certainly has balls. She probably took Bill's in exchange for keeping quiet about his dalliances...

Trump's Republican problem

Long-time readers (both of them) will know of my affection for "my favorite wonk", Megan McArdle. She has been canvassing on Twitter for information about where Donald Trump support is coming from and where it won't ever come from, and has just published a great list of anonymous quotes from lifelong Republicans who won't vote Trump even if he's the Republican candidate:

• I've always voted Republican [...] I have generally avoided voting third-party for fear of helping the Democratic candidate win. However, if Trump wins the nomination, I will vote for the winner of the Libertarian party nomination. I will not support Trump under any circumstances.
• [I will] stay home or not vote for President if Trump is the Republican nominee. After voting basically a straight ticket Republican since I have been eligible to vote, this is truly amazing.
• I have never voted for a Democrat before, but I care too much about the future of this country to let a blithering imbecile become the President.
• I hate Hillary Clinton, but at the very least I know she will do what [she thinks] is best for this country. I cannot say the same about Trump.

I'd say "read the whole thing", but be aware that you'll be there a while.

I really can't imagine many Democrats voting for (say) Republican Ted Cruz if the Democrat candidate was either Sanders or Clinton, no matter what they though of the Democrat. I wonder if this will turn out to be the most compelling reason for the Republicans to band together and stop Trump - not so much to stop him being President, but to stop him being such a horrible Republican candidate that he would keep Republican supporters at home and let in Clinton or Sanders.

Analysing the blue-red hat problem in the face of user error

Everyone knows computers are getting smarter - unless they're being programmed by a major corporation for a government contract - but there has recently been another leap in the level of smart. DeepMind (now part of Google) has built an AI that has successfully deduced the optimal solution to the hat problem:

100 prisoners stand in line, one in front of the other. Each wears either a red hat or a blue hat. Every prisoner can see the hats of the people in front – but not their own hat, or the hats worn by anyone behind. Starting at the back of the line, a prison guard asks each prisoner the colour of their hat. If they answer correctly, they will be pardoned [and if not, executed]. Before lining up, the prisoners confer on a strategy to help them. What should they do?

Tricky, n'est ce pas?

The obvious part first: the first prisoner to answer, whom we'll designate number 1, has no information about his hat colour. Assuming blue and red hats are assigned with equal probability, he can answer either "red" or "blue" with a 50% chance of success and 50% chance of getting executed; he has no better strategy for self-survival. What about the other prisoners?

Applying information theory, our system has 100 binary bits of state - 100 people, each with 1 bit of state relating to whether their hat is blue or not. We generate 99 bits of knowledge about that state as the hat-wearers give answers. So the maximum we can expect to discover reliably is 99/100 hat values. How can we get close to this?

If everyone just guesses their own hat colour randomly, or everyone says "blue", or everyone says "red", then on average 50% of people survive. How to do better? We need to communicate information to people further down their line about their hat colour.

Let's get the first 50 people in line to tell the next 50 people in line about their hat colour. Person 1 announces the hat colour of person 51, person 2 of person 52 and so on. So the last 50 people are guaranteed to survive because they have been told their hat colour. The first 50 people each have a 50-50 chance of survival because the colour they "guess" has no necessary relation to the colour of their hat. On average 25 of them survive, giving an average survival of 75% of people.

The DeepMind algorithm relies on an insight based on the concept of parity: an 0/1 value encapsulating critical state, in this case the number of blue hats seen and guessed, modulo 2. The first user counts the number of blue hats seen and says "blue" if that number is even, and "red" if odd. He still has a 50-50 chance of survival because he has no information about his hat. The second user counts the number of blue hats. If even, and the first person said "blue", then he and the first person both saw the same number of blue hats - so his hat must be red. If even, and the first person said "red", his hat must be blue because it changed the number of blue hats seen between the first person and him. Similar reasoning on the odd case means that he can announce his hat colour with full confidence.

What about person 3? He has to listen to person 1 and person 2, and observe the hat colours in front of him, to deduce whether his hat is blue; his strategy, which works for all others after him too, is to add the parity values (0 for blue, 1 for red) for heard and seen hats modulo 2, and if 0 then announce "blue", if 1 then announce "red". Follow this down the line, and persons 2 through 100 are guaranteed survival while person 1 has a 50-50 chance, for an average 99.5% survival rate.

Of course, this is a fairly complicated algorithm. What if someone mis-counts - what effect does it have? We don't want a fragile algorithm where one person's error can mess up everyone else's calculations, such as with "Chinese whispers." Luckily, a bit of thought (confirmed by experiment) shows us that both the future-casting and parity approaches are resilient to individual error. For future-casting, if one of the first 50 people makes an error then it makes no difference to their chance of survival but their correspondent in the second half of the line is doomed. If one of the second 50 people makes an error then they are doomed unless their correspondent also makes a mistake - generally unlikely, a 10% chance. So if 10% of users make errors then the approximate number of survivors is (75 - 10) + 1, i.e. 66%.

Surprisingly, the parity approach is also robust. It turns out that if user N makes a mistake then they doom themselves, and also doom user N+1 who relies on user N's calculation. But because both user N and N+1 make erroneous guesses, this brings the parity value back in line for user N+2, whose guess will be correct (absent any other errors). So the approximate number of survivors given a 10% error rate is 99.5 - 10*2 = 79.5%

Here's Python code to test the various algorithms: save it as "hats.py" and run it (e.g. "chmod 0755 hats.py ; ./hats.py" on OS X or Linux). It runs 10 trials of 100 people and reports the average number of survivors, based on a 10% error rate in hat wearers following their strategy. Default strategy is the parity algorithm.

#!/usr/bin/python

import random

person_count = 100
half_person_count = person_count / 2
# Hat choices
hat_choices = ['r','b']
hat_opposite = {'b':'r', 'r':'b'}
# 10% error rate in guesses
error_rate = 0.1

def guess_constant(heard_guesses, seen_hats):
    return 'b'

def guess_random(heard_guesses, seen_hats):
    return random.choice(hat_choices)

def guess_future(heard_guesses, seen_hats):
    """ First half of list calls out hat of correspondent in second half of list """
    full_list = heard_guesses + ['x'] + seen_hats
    my_index = len(heard_guesses)
    if my_index < half_person_count:
        # Call out the hat of the person in the second half of the list, hope same as mine
        return full_list[my_index+half_person_count]
    else:
        # Remember what was called out by my corresponding person in first half of list
        return heard_guesses[my_index - half_person_count]

def guess_parity(heard_guesses, seen_hats):
    """ Measure heard and seen parity of blue hats, call out blue for even, red for odd."""
    heard_blue_count = len([g for g in heard_guesses if g == 'b'])
    seen_blue_count = len([s for s in seen_hats if s == 'b'])
    if (heard_blue_count + seen_blue_count) % 2 == 0:
        return 'b'
    else:
        return 'r'

def run_test(guess_fun):
    hat_list = [ random.choice(hat_choices) for i in range(0, person_count) ]
    print "Actual: " + "".join(hat_list)
    answer_list = []
    score_list = []
    error_list = []
    correct = 0
    for i in range(0, person_count):
        guess = guess_fun(answer_list, hat_list[i+1:])
        if random.random() < error_rate:
            guess = hat_opposite[guess]
            error_list.append('X')
        else:
            error_list.append('-')
        answer_list.append(guess)
        if guess == hat_list[i]:
            correct += 1
            score_list.append('-')
        else:
            score_list.append('X')
    print "Called: " + "".join(answer_list)
    print "Score:  " + "".join(score_list)
    print "Errors: " + "".join(error_list)
    print "%d correct" % correct
    return correct

if __name__ == "__main__":
    trial_count = 10
    correct_total = 0
    for i in range(0, trial_count):
        print "\nTrial %d" % (i+1)
        correct_total += run_test(guess_parity)
    print "\nAverage correct: %d" % (correct_total / trial_count)

You can change the "guess_parity" value in the run_test() invocation on the penultimate line to "guess_future" for the "warn the second half" strategy, or "guess_random" for the random choice.

This is a lousy problem for use in software engineering job interviews, by the way. It's a famous problem, so candidates who have heard it are at a major advantage to those who haven't. It relies on a key and non-obvious insight. A candidate who hasn't encountered the problem before and solves it gives a very strong "hire" signal, but a candidate who fails to find the optimal solution could still be a valid hire. The least worst way to assess candidates based on this problem is whether they can write code to evaluate these algorithms, once the algorithms are described to them.

Putting Twitter's loss in perspective

Today, Twitter (NYSE symbol TWTR) lost 7% of its value to close at $16.69/share at a market cap of $11.4bn. That's a loss of approximately $800m of of share capital.

To put that in perspective, that's 8M $100 bills. The NYSE (New York Stock Exchange) is open from 9:30am to 4pm; 6.5 hours, or 23,400 seconds. A well-tuned toilet flush cycle is 35 seconds, so you could get in 668 back-to-back flushes during NYSE opening hours. Therefore you'd have to flush 12,000 $100 bills each time in order to match TWTR's loss. At 150 bills/stack that's 80 stacks, and I can't see you getting more than 1 stack per flush in a single toilet, so I would characterise today's loss as a rate of 80 NYSE-toilets.

I hesitate to ascribe all this loss to Twitter's de-verification of arch-gay-conservative @Nero on 9th January when Twitter was $20, but its share price has descended in more or less a straight line since then. Today the NYSE actually went very slightly up but Twitter still plummeted.

It certainly wasn't helped by 6 hours of partial unavailability of Twitter today, but I suspect that was the straw breaking the camel's back.