The pace of PACER

Permit me a brief, hilarious diversion into the world of US Government corporate IT. PACER is a USA federal online system - "Public Access to Court Electronic Records" which lets people and companies access transcribed records from the US courts. One of their judges has been testifying to the House Judiciary Committee’s Subcommittee on Courts, IP, and the internet and in the process revealed interesting - and horrifying - numbers.

TL;DR -

1. it costs at least 4x what it reasonably should; but
2. any cost savings will be eaten up by increased lawyer usage; nevertheless,
3. rampant capitalism might be at least a partial improvement; so
4. the government could upload the PACER docs to the cloud, employ a team of 5-10 to manage the service in the cloud, and save beaucoup $$.

Of course, I could be wrong on point 2, but I bet I'm not.

Background

PACER operates with all the ruthless efficiency we have come to expect from the federal government.^[1] It's not free; anyone can register for it, usage requires a payment instrument (credit card) but it is free if you use less than $15 per quarter. The basis of charging is:

All registered agencies or individuals are charged a user fee of $0.10 per page. This charge applies to the number of pages that results from any search, including a search that yields no matches (one page for no matches). You will be billed quarterly.

You would think that, at worst, it would be cost-neutral. One page of black+white text at reasonably high resolution is a bit less than 1MB, and (for an ISP) that costs less than 1c to serve on the network. Therefore you spend less than 9c on the machines and people required to store and serve the data, and profit!

Apparently not...

The PACER claims

It was at this point in the article that I fell off my chair:

Fleissig said preliminary figures show that court filing fees would go up by about $750 per case to “produce revenue equal to the judiciary’s average annual collections under the current public access framework.” That could, for example, drive up the current district court civil filing fee from $350 to $1,100, she said.

What the actual expletive? This implies that:

1. the average filing requests 7500 pages of PACER documents - and that the lawyers aren't caching pages to reduce client costs (hollow laughter); or
2. the average filing requests 25 PACER searches; or
3. the average client is somewhere on the continuum between these points.

It seems ridiculously expensive. One can only conclude, reluctantly, that lawyers are not trying to drive down costs for their clients; I know, it's very hard to credit. ^[2]

And this assumes that 10c/page and $30/search is the actual cost to PACER - let us dig into this.

The operational costs

Apparently PACER costs the government $100M/year to operate:

“Our case management and public access systems can never be free because they require over $100 million per year just to operate,” [Judge Audrey] Fleissig said [in testimony for the House Judiciary Committee’s Subcommittee on Courts, IP, and the internet]. “That money must come from somewhere.”

Judge Fleissig is correct in the broad sense - but hang on, $100M in costs to run this thing? How much traffic does it get?

The serving costs

Let's look at the serving requirements:

PACER, which processed more than 500 million requests for case information last fiscal year

Gosh, that's a lot. What's that per second? 3600 seconds/hour x 24 hours/day x 365 days/year is 32 million seconds/year, so Judge Fleissig is talking about... 16 queries per second. Assume that's one query per page. That's laughably small.

Assume that peak traffic is 10x that, and you can serve comfortably 4 x 1MB pages per second on a 100Mbit network connection from a single machine; that's 40 machines with associated hardware, say amortized cost of $2,000/year per machine - implies order of $100K/year on hardware, to ensure a great user experience 24 hours per day 365 days per year. Compared to $100M/year budget, that's noise. And you can save 50% just by halving the number of machines and rejecting excess traffic at peak times.

The ingestion and storage costs

Perhaps the case ingestion is intrinsically expensive, with PACER having to handle non-standard formats? Nope:

The Judiciary is planning to change the technical standard for filing documents in the Case Management and Electronic Case Filing (CM/ECF) system from PDF to PDF/A. This change will improve the archiving and preservation of case-related documents.

So PACER ingests PDFs from courts - plus, I assume, some metadata - and serves PDFs to users.

How much data does PACER ingest and hold? This is a great Fermi question; here's a good worked example of answer, with some data.

There's a useful Ars Technica article on Aaron Swartz that gives us data on the document corpus as of 2013:

PACER has more than 500 million documents

Assume it's doubled as of 2019, that's 1 billion documents. Assume 1MB/page, 10 pages/doc, that's 10^9 docs x 10 MB per doc = 10^10 MB = 1x10^4 TB. That's 1000 x 10TB hard drives. Assume $300/drive, and drives last 3 years, and you need twice the number of drives to give redundancy, that's $200 per 10TB per year in storage costs, or $200K for 10,000 TB. Still, noise compared to $100M/year budget. But the operational costs of managing that storage can be high - which is why Cloud services like Amazon Web Services, Azure and Google Cloud have done a lot of work to offer managed services in this area.

Amazon, for instance, charges $0.023 per GB per month for storage (on one price model) - for 10^9 x 1MB docs, that's 1,000,000 GB x $0.023 or $23K/month, $276K/year. Still way less than 1% of the $100M/year budget.

Incidentally Aaron Swartz agrees with the general thrust of my article:

Yet PACER fee collections appear to have dramatically outstripped the cost of running the PACER system. PACER users paid about $120 million in 2012, thanks in part to a 25 percent fee hike announced in 2011. But Schultze says the judiciary's own figures show running PACER only costs around $20 million.

A rise in costs of 5x in 6 years? That's approximately doubling every 2 years. As noted above, it seems unlikely to be due to serving costs - even though volumes have risen, serving and storage costs have got cheaper. Bet it's down to personnel costs. I'd love to see the accounts break-down. How many people are they employing, and what are those people doing?

The indexing costs - or lack thereof

Indexing words and then searching a large corpus of text is notoriously expensive - that's what my 10c per electronic page is paying for, right? Apparently not:

There is a fee for retrieving and distributing case information for you: $30 for the search, plus $0.10 per page per document delivered electronically, up to 5 documents (30 page cap applies).

It appears that PACER is primarily constructed to deliver responses to "show me the records of case XXXYYY" or "show me all cases from court ZZZ", not "show me all cases that mention 'Britney Spears'." That's a perfectly valid decision but makes it rather hard to justify the operating costs.

Security considerations

Oh, please. These docs are open to anyone who has an account. The only thing PACER should be worried about is someone in Bangalore or Shanghai scraping the corpus, or the top N% of cases, and serving that content for much less cost. Indeed, that's why they got upset at Aaron Swartz. Honestly, though, the bulk of their users - law firms - are very price-insensitive. Indeed, they quite possibly charge their clients 125% or more of their PACER costs, so if PACER doubled costs overnight they'd celebrate.

I hope I'm wrong. I'm afraid I'm not.

Public serving alternatives

I don't know how much Bing costs to operate, but I'd bet a) that its document corpus is bigger than PACER, b) that its operating costs are comparable, c) that its indexing is better than PACER, d) that its search is better than PACER, e) that its page serving latency is better than PACER... you get the picture.

Really though, if I were looking for a system to replace this, I'd build off an off-the-shelf solution to translate inbound PDFs to indexed text - something like OpenText - and run a small serving stack on top. That reduces the regular serving cost, since pages are a few KB of text rather than 1MB of PDF, and lets me get rid of all the current people costs associated with the customized search and indexing work on the current corpus.

PACER is a terrible use of government money

Undoubtedly it's not the worst^[3], but I'd love for the House Judiciary Committee’s Subcommittee on Courts, IP, and the internet to drag Jeff Bezos in to testify and ask him to quote a ballpark number for serving PACER off Amazon Web Services, with guaranteed 100% profit margin.

Bet it's less than 1/4 of the current $100M/year.

[1] Yes, irony
[2] Why does New Jersey have the most toxic waste dumps and California the most lawyers? California New Jersey got first choice. [Thanks Mr Worstall!]
[3] Which is terribly depressing.

Downsides of an IT monolith (NHS edition)

I have been watching, with no little schadefreude (trans. "damage joy") today's outage of many NHS services as a result of a ransomware attack.

This could happen to anyone, n'est ce pas? The various NHS trusts affected were just unlucky. They have many, many users (admin staff in each GP's surgery; nurses, auxiliaries and doctors rushing to enter data before dashing off to the next patient). Why is it unsurprising that this is happening now?

The NHS is an organisational monolith. It makes monolithic policy announcements. As a result of those policies, Windows XP became the canonical choice for NHS PCs. It is still the canonical choice for NHS PCs. Windows XP launched to the public in late 2001. Microsoft ended support for Windows XP in April 2014. Honestly, I have to give Microsoft kudos for this (oh, that hurts) because they kept XP supported way beyond any reasonable timeframe. But all good things come to an end, and security updates are no longer built for XP. The NHS paid Microsoft for an extra year of security patches but decided not to extend that option beyond 2015, presumably because no-one could come up with a convincing value proposition for it. Oops.

The consequences of this were inevitable, and today we saw them. A huge userbase of Internet-connected PCs no longer receiving security updates is going to get hit by something - they were a bit unlucky that it was ransomware, which is harder to recover from than a straight service-DoS, but this was entirely foreseeable.

Luckily the NHS mandates that all critical operational data be backed up to central storage services, and that its sites conduct regular data-restore exercises. Doesn't it? Bueller?

I don't want to blame the central NHS IT security folks here - I'm sure they do as good a job as possible in an impossible-to-manage environment, and that the central patient data is fairly secure. However, if you predicate effective operations for most of the NHS on data stored on regular PCs then you really want to be sure that they are secure. Windows XP has been end-of-support for three gold-durned years at this point, and progress in getting NHS services off it has been negligible. You just know that budget for this migration got repurposed for something else more time-sensitive "temporarily".

This is a great example of organisational inertia, in fact maybe a canonical one. It was going to be really hard to argue for a massively expensive and disruptive change, moving all NHS desktops to a less-archaic OS - Windows 10 seems like a reasonable candidate, but would still probably require a large proportion of desktops and laptops to be replaced. As long as nothing was on fire, there would be a huge pushback on any such change with very few people actively pushing for it to happen. So nothing would happen - until now...

Please check back in 2027 when the NHS will have been on Windows 10 for 8 years, 2 years end-of-life, and the same thing will be happening again.

Scentrics finds that security is hard

Two years ago I wrote about Scentrics and their "Key Man" security proposal. I wondered idly what had happened there so did some Googling. Turns out that I'm the top two hits for [scentrics key man] which is heart-warming for me but suggests that their world-beating security patent might have sunk like a stone...

I went to their website www.scentrics.com and noted that it didn't redirect to https. I tried https://www.scentrics.com and lo! Chrome's Red "Not secure" Warning of Death appears. Seems that Scentrics can't even secure their website, which is not a little ironic when their home page trumpets "Secure with Scentrics".

All the pages on the site - even "Overview and Vision" and "Careers" - are hidden behind a sign-on box, declaring the website "invitation only" and inviting you to contact "admin@scentrics.com" if you'd like access. You can view headers, but that's about it. You wonder why they would be so sensitive about exposing information like that.

The 2016 news included a nugget from the Daily Telegraph in June:

Scentrics is poised to seek new funding that would value the company at more than $1 billion as it prepares to rollout its infrastructure for the first time.

"Poised", huh? I like that. I read that as "not yet ready". I also like the uncritical write-up of the company's pitch:

Individual messages and documents sent over the internet can be unlocked without compromising the overall security of the network, according to Scentrics's pitch to operators and governments.

Remember that this essentially involved encrypting one copy of a message with the recipient's public key, and another with a government/agency public key, and storing the latter to give the agency access on demand. The government and security agencies involved might not think that this "compromises" the overall security of the network, but as a consumer of the network's function I can assure them that I'd feel very differently. And of course for this to be effective all network users would have to use a very small ecosystem of only approved apps / browsers which implemented this dual encryption, and maintained the central repository of government-friendly encrypted messages. I'm sure there's no risk of systematic system compromise there by insiders at all.

Companies House shows three officers plus a secretarial company including our old friend Guruparan "Paran" Chandrasekaran. Looks like Sir Francis Mackay, David Rapoport and Dr. Thaksin Shinawatra resigned since 2014, which is interesting because the latter gent used to be the Prime Minister of Thailand, and Scentrics trumpted his role in the Telegraph piece, but as of 1 month ago he's out of his company role.

According to their June 2015 accounts they have about GBP4.2M in net assets, looks like they had an infusion of about GBP4.5M during the year. Going from this to a $1bn valuation seems... optimistic.

Update: Looks like Scentrics are diving into Singapore with advertisements for Project Manager and Devops roles there. This seems to be part of the Singapore government's "Smart Nation" project for a unified network in Singapore:

• A Smart Nation is one where people are empowered by technology to lead meaningful and fulfilled lives.
• A Smart Nation harnesses the power of networks, data and info-comm technologies to improve living, create economic opportunity and build a closer community.
• A Smart Nation is built not by Government, but by all of us - citizens, companies, agencies. This website chronicles some of our endeavours and future directions.

Cutting through the marketing speak, Singaporeans will be using a government-provided network for all services including personal and business communication. With Scentrics playing a role, the benevolent semi-dictatorship of Singapore will be able to snoop on all its citizens' internal communications at will.

Scentrics seems to be very comfortable enabling a government's surveillance on its citizens. I wonder how this is going to work out for them long-term given the distinctly libertarian tilt of most software engineers.

[Disclaimer: no share position in Scentrics. Financially I don't care if they live or die. Personally, I'd incline towards the latter.]

DDoS and the Tragedy of the Commons of the Internet of Things

On Friday there was a massive Distributed Denial of Service attack on DynDNS, who provide Domain Name services to a number of major companies including Twitter, Spotify and SoundCloud, effectively knocking those sites offline for a significant fraction of the global population. Brian Krebs provides a useful summary of the attack; he is unusually well versed in these matters because his website "Krebs on Security" was taken offline on 20th September after a massive Internet-of-Things-sourced DDoS against it. It seems that Krebs' ongoing coverage and analysis of DDoS with a focus on the Internet of Things (IoT) - "smart" Internet connected home devices such as babycams and security monitors - raised the ire of those using the IoT for their nefarious purposes. It proved necessary to stick Krebs' blog behind Google's Project Shield which protects major targets of information suppression behind something resembling +5 enchanted DDoS armour.

Where did this threat to the Internet come from? Should we be worried? What can we do? And why is this whole situation a Tragedy of the Commons?

Primer on DNS

Let's look at Friday's outage first. Dyn DNS is a DNS hosting company. They provide an easy way for companies who want a worldwide web presence to distribute information about the addresses of their servers - in pre-Internet terms, they're like a business phone directory. Your company Cat Grooming Inc., which has bought the domain name catgrooming.com, has set up its web servers on Internet addresses 1.2.3.4 and 1.2.3.5, and its mail server on 1.2.4.1. Somehow, when someone types "catgrooming.com" in their internet brower, they need that translating to the right numerical Internet address. For that translation, their browser consults the local Domain Name Service (DNS) server, which might be from their local ISP, or a public one like Google's Public DNS (8.8.4.4 and 8.8.8.8).

So if Cat Grooming wants to change the Internet address of their webservers, they either have to tell every single DNS server of the new address (impractical), or run a special service that every DNS server consults to discover up to date information for the hostnames. Running a dedicated service is expensive, so many companies use a third party to run this dedicated service. Dyn DNS is one such company: you tell them whenever you make an address change, and they update their records, and your domain's information says that Dyn DNS does its address resolution.

To check whether a hostname on the web uses DynDNS, you can use the "dig" command which should work from the Linux, MacOS or FreeBSD command line:

$ dig +short -t NS twitter.com
ns3.p34.dynect.net.
ns2.p34.dynect.net.
ns1.p34.dynect.net.
ns4.p34.dynect.net.

This shows that twitter.com is using Dyn DNS because it has dynect.net hostnames as its name servers.

Your browser doesn't query Dyn DNS for every twitter.com URL you type. Each result you get back from DNS comes with a "time to live" (TTL) which specifies for how many seconds the answer is valid. If your twitter.com query came back as 199.59.150.7 with a TTL of 3600 then your browser would use that address for the next hour without bothering to check Dyn DNS. Only after 1 hour (3600 seconds) would it re-check Dyn DNS for an update.

Attack mechanism

The Internet of Things includes devices such as "babycams" which enable neurotic parents to keep an eye on their child's activities from elsewhere in the house, or even from the restaurant to which they have sneaked out for a couple of hours of eating that does not involve thrown or barfed food. The easiest way to make these devices accessible from the public Internet is to give them their own Internet address, so you can enter that address on a mobile phone or whatever and connect to the device. Of course, the device will challenge any new connection attempt for a username and password; however, many devices have extremely stupid default passwords and most users won't bother to change them.

Over the past decade, Internet criminals have become very good at scanning large swathes of the Internet to find devices with certain characteristics - unpatched Windows 2000 machines, webcams, SQL servers etc. That lets them find candidate IoT devices on which they can focus automated break-in attempts. If you can get past the password protection for these devices, you can generally make them do anything you want. The typical approach is to add code that makes them periodically query a central command-and-control server for instructions; those instructions might be "hit this service with queries randomly selected from this list, at a rate of one query every 1-2 seconds, for the next 4 hours."

The real problem with this kind of attack is that it's very hard to fix. You have to change each individual device to block out the attackers - there's generally no way to force a reset of passwords to all devices from a given manufacturer. The manufacturer has no real incentive to do this since it has the customer's money already and isn't obviously legally liable for the behavior. The owner has no real incentive to do this because this device compromise doesn't normally materially affect the device operation. You can try to sell the benefits of a password fix - "random strangers on the internet can see your baby!" but even then the technical steps to fix a password may be too tedious or poorly explained for the owner to action. ISPs might be able to detect compromised devices by their network traffic patterns and notify their owners, but if they chase them to fix the devices too aggressively then they might piss off the owners enough to move to a different ISP.

Why don't ISPs pre-emptively fix devices if they find compromised devices on their network? Generally, because they have no safe harbour for this remedial work - they could be prosecuted for illegal access to devices. They might survive in court after spending lots of money on lawyers, but why take the risk?

Effects of the attack

Dyn DNS was effectively knocked off the Internet for many hours. Any website using Dyn DNS for their name servers saw incoming traffic drop off as users' cached addresses from DNS expired and their browsers insisted on getting an up-to-date address - which was not available, because the Dyn DNS servers were melting.

Basic remediation for sites in this situation is to increase the Time-to-Live setting on their DNS records. If Cat Grooming Inc's previous setting was 3600 seconds, then after 1 hour of the Dyn DNS servers being down their traffic would be nearly zero. If their TTL was 86400 seconds (1 day) then a 12 hour attack would only block about half their traffic - not great, but bearable. A TTL of 1 week would mean that a 12 hour attack would be no more than an annoyance. Unfortunately, if the attack downs Dyn DNS before site owners can update their TTL this doesn't really help.

Also, the bigger a site is, the more frequently it needs to update DNS information. Twitter will serve different Internet addresses for twitter.com to users in different countries, trying to point users to the closest Twitter server to them. You don't want a user in Paris pointed to a Twitter server in San Francisco if there is one available in Amsterdam, 500 millseconds closer to them. And when you have many different servers, every day some of them are going offline for maintenance or coming online as new servers, so you need to update DNS to stop users going to the former and start sending them to the latter.

Therefore the bigger your site, the shorter your DNS TTL is likely to be, and the more vulnerable you are to this attack. If you're a small site with infrequent DNS updates, and your DNS TTL is short, then make it longer right the hell now.

Alternative designs

The alternative to this exposed address approach is to have a central service which all the baby monitors from a given manufacturer connect to, e.g. the hostname cams.babycamsRus.com; users then connect to that service as well and the service does the switching to connect Mr. and Mrs. Smith to the babycam chez Smith. This prevents the devices from being found by Internet scans - they don't have their own Internet address, and don't accept outside connections. If you can crack the BabyCams-R-Us servers then you could completely control a huge chunk of IoT devices, but their sysadmins will be specifically looking out for these attacks and it's a much more tricky proposition - it's also easy to remediate once discovered.

Why doesn't every manufacturer do this, if it's more secure? Simply, it's more expensive. You have to set up this central service, capable of servicing all your sold devices at once, and keep it running and secure for many years. In a keenly price-competitive environment, many manufacturers will say "screw this" and go for the cheaper alternative. They have no economic reason not to, no-one is (yet) prosecuting them for selling insecure devices, and customers still prefer cheap over secure.

IPv6 will make things worse

One brake on this run-away cheap-webcams-as-DoS-tool is the shortage of Internet addresses. When the Internet addressing scheme (Internet Protocol version 4, or IPv4 for short) was devised, it was defined as four numbers between 0 and 255, conventionally separated by dots e.g. 1.2.3.4. This gives you just under 4.3 billion possible addresses. Back in 2006 large chunks of this address space were free. This is no longer the case - we are, in essence, out of IPv4 addresses, and there's an active trade in them from companies which are no longer using much of their allocated space. Still, getting large blocks of contiguous addresses is challenging. Even a /24 (shorthand for 256 contiguous IPv4) is expensive to obtain. Father of the Internet Vint Cerf recently apologised for the (relatively) small number of IPv4 addresses - they thought 4.3 billion addresses would be enough for the "experiment" that IPv4 was. The experiment turned into the Internet. Oops.

This shortage means that the current model where webcams and other IoT devices have their own public Internet address is unsustainable: the cost of that address will become prohibitive, and customers will need something that sits behind their single home Internet address given to them by their ISP. You can have many devices behind one address via a mechanism called Network Address Translation NAT) where the router connecting your home to the Internet lets each of your devices start connections to the Internet and allocates them a "port" which is passed to the website they connect to: when the website server responds, it sends the web page back to your router along with the port number, so the router knows which of your home devices the web page should be sent to.

The centralized service described above is (currently) the only practical solution in this case of one IP for many devices. More and more devices on the Internet will be hidden from black-hat hacker access in this way.

Unfortunately (for this problem) we are currently transitioning to use the next generation of Internet addressing - IPv6. This uses 128 bits, which is a staggering number: 340 with an additional 36 zeroes after it. Typically your ISP would give you a "/64" for your home devices to use for their public Internet addresses - a mere 18,000,000,000,000,000,000 (18 quintillion) addresses. Since there are 18 quintillion /64s in the IPv6 address space, we're unlikely to run out of them for a while even if ever person on earth is given a fresh one every day and there's no re-use.

IPv6 use is not yet mainstream, but more and more first world ISPs are giving customers IPv6 access if they want it. Give it a couple of years and I suspect high-end IoT devices will be explicitly targeted at home IPv6 setups.

Summary: we're screwed

IPv4 pressures may temporarily push IoT manufacturers to move away from publicly addressable IoT devices, but as IPv6 becomes more widely used the commercial pressures may once more become too strong to resist and the IoT devices will be publicly discoverable and crackable once more. Absent a serious improvement in secure, reliable and easy dynamic updates to these devices, the IoT botnet is here to stay for a while.

Putting Twitter's loss in perspective

Today, Twitter (NYSE symbol TWTR) lost 7% of its value to close at $16.69/share at a market cap of $11.4bn. That's a loss of approximately $800m of of share capital.

To put that in perspective, that's 8M $100 bills. The NYSE (New York Stock Exchange) is open from 9:30am to 4pm; 6.5 hours, or 23,400 seconds. A well-tuned toilet flush cycle is 35 seconds, so you could get in 668 back-to-back flushes during NYSE opening hours. Therefore you'd have to flush 12,000 $100 bills each time in order to match TWTR's loss. At 150 bills/stack that's 80 stacks, and I can't see you getting more than 1 stack per flush in a single toilet, so I would characterise today's loss as a rate of 80 NYSE-toilets.

I hesitate to ascribe all this loss to Twitter's de-verification of arch-gay-conservative @Nero on 9th January when Twitter was $20, but its share price has descended in more or less a straight line since then. Today the NYSE actually went very slightly up but Twitter still plummeted.

It certainly wasn't helped by 6 hours of partial unavailability of Twitter today, but I suspect that was the straw breaking the camel's back.

The spectacular kind of hardware failure

Gentle reader, I have attempted several times to pen my thoughts on the epic hack of the US Office of Personnel Management that compromised the security information of pretty much everyone who works for the US government, but I keep losing my vision and hearing a ringing in my ears when I try to do so. So I turn to a lesser-known and differently-awesome fail: the US visa system.

Since a computer failure on the 26th of May - over three weeks ago - the US embassies and consulates worldwide have been basically unable to issue new visas except in very limited circumstances. You haven't heard much about this because it hasn't really affected most US citizens, but believe me it's still a big issue. It seems that they're not expecting the system to be working again until next week at the earliest. Estimates of impacted users are on the order of 200,000-500,000; many people are stuck overseas, unable to return to the USA until their visa renewal is processed.

What happened? The US Department of State has a FAQ but it is fairly bland, just referring to "technical problems with our visa systems" and noting "this is a hardware failure, and we are working to restore system functions".

So a hardware failure took out nearly the entire system for a month. The most common cause of this kind of failure is a large storage system - either a mechanical failure that prevents access to all the data you wrote on the disks, or a software error that deleted or overwrote most of the data on there. This, of course, is why we have backups - once you discover the problem, you replace the drive (if broken) and then restore your backed up data from the last known good state. You might then have to apply patches on top to cover data that was written after the backup, but the first step should get you 90%+ of the way there. Of course, this assumes that you have backups and that you are regularly doing test restores to confirm that what you're backing up is still usable.

The alternative failure is of a relatively large machine. If you're running something comparable to the largest databases in the world you're going to be using relatively custom hardware. If it goes "foom", e.g. because its motherboard melts, you're completely stuck until an engineer can come over with the replacement part and fix it. If the part is not replaceable, you're going to have to buy an entirely new machine - and move the old one out, and install the new one, and test it, and hook it up to the existing storage, and run qualification checks... But this should still be on the order of 1 week.

A clue comes from a report of the State Department:

"More than 100 engineers from the government and the private sector [my emphasis] are working around the clock on the problem, said John Kirby, State Department spokesman, at a briefing on Wednesday.

You can't use 100 engineers to replace a piece of hardware. They simply won't fit in your server room. This smells for all the world like a mechanical or software failure affecting a storage system where the data has actually been lost. My money is on backups that weren't actually backing up data, or backing it up in a form that needed substantial manual intervention to restore, e.g. a corrupted database index file which would need every single piece of data to be reindexed. Since they've roped in private sector engineers, they're likely from whoever supplied the hardware in question: Oracle or IBM, at a guess.

The US Visa Office issues around 10 million non-immigrant visas per year, which are fairly simple, and about 500,000 immigrant visas per year which are a lot more involved with photos, other biometrics, large forms and legal papers. Say one of the latter takes up 100MB (a hi-res photo is about 5MB) and one of the former takes up 5MB; then that's a total of about 100TB per year. That's a lot of data to process, particularly if you have to build a verification system from scratch.

I'd love to see a report on this from the Government Accountability Office when the dust settles, but fear that the private sector company concerned will put pressure on to keep the report locked up tight "for reasons of commercial confidentiality and government security". My arse.

You should care about moving to HTTPS

Eric Mill's "We're Deprecating HTTP and it's going to be okay" is a must-read call-to-arms for everyone with a site on the Internet, explaining why the transition from unencrypted web traffic (HTTP) to encrypted (HTTPS) is actually fundamental to the future existence of the democratic web-as-we-know it.

For the 90% of my reading audience who are already saying "Bored now!" here's why it matters to you. Sir Tim Berners-Lee invented HTTP (the language of communication between web browser and web server) in CERN, a European haven of free thought, trust and international co-operation. The 1930s idea that "Gentlemen do not read each other's mail" was - surprisingly, given the history of cryptographic war in WW2 - fundamental to HTTP; messages might have transited systems owned by several different groups, but none of them would have thought to copy the messages passing through their system, let alone amend them.

This worked fine as long as no-one was interested in the communication of harmless nerds about their hobbies, much as the government-owned Royal Mail doesn't bother to copy the contents of postcards passing through their sorting offices because they only contain inane drivel about sun, sea and sand. However, once people realized that they could communicate freely about their occasionally subversive ideas across borders and continents, and financial institutions woke to the possibility of providing services without paying for expensive un-scalable fallible human cashiers, many governments and other less-legal entities wanted to read (and sometimes alter) Internet traffic.

Mills gives two great examples of where HTTPS prevented - and could have prevented further - nation-state abuse of Internet content:

- The nation of India tried and failed to ban all of GitHub. HTTPS meant they couldn't censor individual pages, and GitHub is too important to India's tech sector for them to ban the whole thing.
- The nation of China weaponized the browsers of users all over the world to attack GitHub for hosting anti-censorship materials (since like India, they can't block only individual pages) by rewriting Baidu's unencrypted JavaScript files in flight.

And closer to home, Cameron's plan to make all online communication subject to monitoring is so stupidly illiberal and expensively pointless that it deserves to be made impractical by general adoption of HTTPS. GCHQ and friends can tap all the Internet traffic they like: if it's protected by HTTPS, the traffic is just taking up disk space to no practical purpose. Brute-forcing, even with nation-state resources, is so expensive that it's reserved for really high-value targets. GCHQ would have to go after something fundamental like a Certificate Authority, which would leave big and obvious fingerprints, or compromise a particular user's machine directly, which doesn't scale.

As long as users are still relaxed about the absence of a padlock in their browser bar, HTTP will continue to provide a route for governments to snoop on their citizens' traffic. So let's give up on HTTP - it has had its day - and move to a world where strongly encrypted traffic is the default.

Journos writing about trading and high-speed computing

I have to admit, this amused me - the Daily Mail trying to write about high-frequency trading:

Suspected rogue trader Navinder Sarao lived in his parents' modest home because it gave him a split-second advantage worth millions of pounds, it was claimed yesterday.
His family's semi-detached house in suburban West London is closer to an internet server used by one of the major financial exchanges, giving him a nanosecond advantage over rivals in the City.
[...]
Sarao, 36, was dubbed the 'Hound of Hounslow' after it emerged he lived at home with his parents, despite allegedly making £26.7million in just four years of dealing from their home.

And yet you'd think that renting a small flat in Slough and paying for Internet access there would have improved his speed advantage; at a cost of about £50K for four years, that would have been a bargain. Why, it's almost as if the Daily Mail journalists had no idea what they were talking about....

Active attack on an American website by China Unicom

I wondered what the next step in the ongoing war between Western content and Chinese censorship might be. Now we have our answer.

"Git" is a source code repository system which allows programmers around the world to collaborate on writing code: you can get a copy of a software project's source code onto your machine, play around with it to make changes, then send those changes back to Git for others to pick up. Github is a public website (for want of a more pedantic term) which provides a repository for all sorts of software and similar projects. The projects don't actually have to be source code: anything which looks like plain text would be fine. You could use Github to collaborate on writing a book, for instance, as long as you used mostly text for the chapters and not e.g. Microsoft Word's binary format that makes it hard for changes to be applied in sequence.

Two projects on Git are "greatfire" and "cn-nytimes" which are, respectively, a mirror for the Greatfire.org website focused on the Great Firewall of China, and a Chinese translation of the New York Times stories. These are, obviously, not something to which the Chinese government wants its citizenry to have unfettered access. However, Github has many other non-controversial software projects on it, and is actually very useful to many software developers in China. What to do?

Last week a massive Distributed Denial of Service (DDoS) attack hit Github:

The attack began around 2AM UTC on Thursday, March 26, and involves a wide combination of attack vectors. These include every vector we've seen in previous attacks as well as some sophisticated new techniques that use the web browsers of unsuspecting, uninvolved people to flood github.com with high levels of traffic. Based on reports we've received, we believe the intent of this attack is to convince us to remove a specific class of content. [my italics]

Blocking Github at the Great Firewall - which is very easy to do - was presumably regarded as undesirable because of its impact on Chinese software businesses. So an attractive alternative was to present the Github team with a clear message that until they discontinued hosting these projects they would continue to be overwhelmed with traffic.

If this attack were just a regular DDoS by compromised PCs around the world it would be relatively trivial to stop: just block the Internet addresses (IPs) of the compromised PCs until traffic returns to normal levels. But this attack is much more clever. It intercepts legitimate requests from worldwide web browsers for a particular file hosted on China's Baidu search engine, and modifies the request to include code that commands repeated requests for pages from the two controversial projects on Github. There's a good analysis from NetreseC:

In short, this is how this Man-on-the-Side attack is carried out:
1. An innocent user is browsing the internet from outside China.
2. One website the user visits loads a JavaScript from a server in China, for example the Badiu Analytics script that often is used by web admins to track visitor statistics (much like Google Analytics).
3. The web browser's request for the Baidu JavaScript is detected by the Chinese passive infrastructure as it enters China.
4. A fake response is sent out from within China instead of the actual Baidu Analytics script. This fake response is a malicious JavaScript that tells the user's browser to continuously reload two specific pages on GitHub.com.

The interesting question is: where is this fake response happening? We're fairly sure that it's not at Baidu themselves, for reasons you can read in the above links. Now Errata Security has done a nice bit of analysis that points the finger at the Great Firewall implementation in ISP China Unicom:

By looking at the IP addresses in the traceroute, we can conclusive prove that the man-in-the-middle device is located on the backbone of China Unicom, a major service provider in China.

That existing Great Firewall implementors have added this new attack functionality fits with Occam's Razor. It's technically possible for China Unicom infrastructure to have been compromised by patriotically-minded independent hackers in China, but given the alternative that China Unicom have been leant on by the Chinese government to make this change, I know what I'd bet my money on.

This is also a major shift in Great Firewall operations: this is the first major case I'm aware of that has them focused on inbound traffic from non-Chinese citizens.

Github look like they've effectively blocked the attack, after a mad few days of scrambling, and kudos to them. Now we have to decide what the appropriate response is. It seems that any non-encrypted query to a China-hosted website would be potential fair game for this kind of attack. Even encrypted (https) requests could be compromised, but that would be a huge red arrow showing that the company owning the original destination (Baidu in this case) had been compromised by the attacker: this would make it 90%+ probable that the attacker had State-level influence.

If this kind of attack persists, any USA- or Europe-focused marketing effort by Chinese-hosted companies is going to be thoroughly torpedoed by the reasonable expectation that web traffic is going to be hijacked for government purposes. I wonder whether the Chinese government has just cut off its economic nose to spite its political face.

What does "running your own email server" mean?

There's lots of breathless hyperbolae today about Hillary Clinton's use of a non-government email address during her tenure as Secretary of State. The Associated Press article is reasonably representative of the focus of the current debate:

The email practices of Hillary Rodham Clinton, who used a private account exclusively for official business when she was secretary of state, grew more intriguing with the disclosure Wednesday that the computer server she used traced back to her family's New York home, according to Internet records reviewed by The Associated Press.
[...]
It was not immediately clear exactly where Clinton's computer server was run, but a business record for the Internet connection it used was registered under the home address for her residence in Chappaqua, New York, as early as August 2010. The customer was listed as Eric Hoteham.

Let's apply a little Internet forensics to the domain in question: clintonemail.com. First, who owns the domain?

$ whois clintonemail.com
[snip]
Domain Name: CLINTONEMAIL.COM
Registry Domain ID: 1537310173_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.networksolutions.com
Registrar URL: http://networksolutions.com
Updated Date: 2015-01-29T00:44:01Z
Creation Date: 2009-01-13T20:37:32Z
Registrar Registration Expiration Date: 2017-01-13T05:00:00Z
Registrar: NETWORK SOLUTIONS, LLC.
Registrar IANA ID: 2
Registrar Abuse Contact Email: abuse@web.com
Registrar Abuse Contact Phone: +1.8003337680
Reseller:
Domain Status:
Registry Registrant ID:
Registrant Name: PERFECT PRIVACY, LLC
Registrant Organization:
Registrant Street: 12808 Gran Bay Parkway West
Registrant City: Jacksonville
Registrant State/Province: FL
Registrant Postal Code: 32258
Registrant Country: US
Registrant Phone: +1.5707088780
Registrant Phone Ext:
Registrant Fax:
Registrant Fax Ext:
Registrant Email: kr5a95v468n@networksolutionsprivateregistration.com

So back in January this year the record was updated, and we don't necessarily know what it contained before that, but currently Perfect Privacy, LLC are the owners of the domain. They register domains on behalf of people who don't want to be explicitly tied to that domain. That's actually reasonably standard practice: any big company launching a major marketing initiative wants to register domains for their marketing content, but doesn't want the launch to leak. If Intel are launching a new microbe-powered chip, they might want to register microbeinside.com without their competitors noticing that Intel are tied to that domain. That's where the third party registration companies come in.

The domain record itself was created on the 13th of January 2009, which is a pretty strong indicator of when it started to be used. What's interesting, though, is who operates the mail server which receives email to this address. To determine this, you look up the "MX" (mail exchange) records for the domain in question, which is what any email server wanting to send email to hillary@clintonemail.com would do:

$ dig +short clintonemail.com MX
10 clintonemail.com.inbound10.mxlogic.net.
10 clintonemail.com.inbound10.mxlogicmx.net.

mxlogic.net were an Internet hosting company, bought by McAfee in 2009. So they are the ones running the actual email servers that receive email for clintonemail.com and which Hillary's email client (e.g. MS Outlook) connected to in order to retrieve her new mail.

We do need to take into account though that all we can see now is what the Internet records point to today. Is there any way to know where clintonemail.com's MX records pointed to last year, before the current controversy? Basically, no. Unless someone has a hdr22@clintonemail.com mail from her home account which will have headers showing the route that emails took to reach her, or has detailed logs from their own email server which dispatched an email to hdr22@clintonemail.com, it's probably not going to be feasible to determine definitively where she was receiving her email. However, CBS News claims that the switch to mxlogic happened in July 2013 - that sounds fairly specific, so I'll take their word for it for now. I'm very curious to know how they determined that.

All of this obscures the main point, of course, which is that a US federal government representative using a non-.gov email address at all for anything related to government business is really, really bad. Possibly going-to-jail bad, though I understand that the specific regulation requiring a government employee to use a .gov address occurred after Hillary left the role of SecState (Feb 2013). Still, if I were the Russian or Chinese foreign intelligence service, I'd definitely fancy my chances in a complete compromise of either a home-run server, or of a relatively small-scale commercial email service (mxlogic, for instance).

Desperately attempting to spin this whole situation is Heidi Przybyla from Bloomberg:

Jeb Bush owns his own email server http://t.co/6EEPZbeL0q via @msnbc

— Heidi Przybyla (@HeidiPrzybyla) March 4, 2015

OK, let's apply our forensics to jeb.org:

$ dig +short jeb.org MX
5 mx1.emailsrvr.com.
10 mx2.emailsrvr.com.

emailsrvr.com is, like mxlogic.net, a 3rd party email hosting service, apparently specialising in blocking spam. I'm not surprised that someone like Jeb Bush uses it. And, like Hillary, he isn't "running his own email server", he's using an existing commercial email server. It's not Gmail/Outlook.com/Yahoo, but there's not reason to think it's not perfectly serviceable, and it's not controlled by Bush so if they log or archive incoming or outgoing email his correspondence is legally discoverable.

The difference between Jeb Bush and Hillary Clinton of course, as many others note, is that Jeb is not part of the US federal government and hence not subject to federal rules on government email...

Net neutrality - be careful what you wish for

I'm driving my forehead into an ever-deepening dent on my desk in despair at the news that the US Federal Communications Commission has approved new rules governing net neutrality in the USA. This may seem like the sort of news that a progressive geek like your humble bloghost would welcome, but it turns out to involve some inconvenient wrinkles.

The EFF, guardians of liberty, were originally cheering on behalf of net neutrality. Then, 2 days ago, they started to get a little concerned with some of the details being proposed by the FCC:

Unfortunately, if a recent report from Reuters is correct, the general conduct rule will be anything but clear. The FCC will evaluate "harm" based on consideration of seven factors: impact on competition; impact on innovation; impact on free expression; impact on broadband deployment and investments; whether the actions in question are specific to some applications and not others; whether they comply with industry best standards and practices; and whether they take place without the awareness of the end-user, the Internet subscriber.

In essence, the proposed rules for Net Neutrality gave the FCC - a US government agency, headed by a former lobbyist for the cable and wireless industry - an awfully wide scope for deciding whether innovations in Internet delivery were "harmful" or not. There's no way that this could go horribly wrong, surely?

Broadband in the USA

Now, let's start with the assertion that there is an awful lot wrong with broadband provision in the USA currently. It's a lot more expensive than in the UK, it's almost always supplied by the local cable TV provider, and in general there is very little if any choice in most regions. See the broadband provider guide and choose min, max of 1 - there's an awful lot of the USA with monopoly provision of wired high-speed internet.

The dominant ISPs with high-speed provision are Comcast, AT+T, Time Warner, CenturyLink and Verizon. It would be fair to say that they are not particularly beloved. Comcast in particular is the target of a massive amount of oppprobium: type "Comcast are " in your favourite search engine, and you get autocompletion suggestions including "liars", "crooks", "criminals". American broadband is approximately twice the price of British, and you generally get lower speeds and higher contention ratios (you share a pipe of fixed size with a lot of people, so if your neighbours are watching streaming video then you're out of luck). As effective monopolies, ISPs were in a very powerful position to charge Internet services for streaming data to their customers, as last year's Comcast-Netflix struggle showed - and it ended with Netflix effectively forced to pay Comcast to ship the bytes that Netflix customers in Comcast regions were demanding.

Google's upstart "Google Fiber" offering of 1 Gbps (125 MB per second) fiberoptic service tells a story in itself. It's targeting a relatively short list of cities but has been very popular whenever it opened signups. It has spurred other broadband providers to respond, but in a very focused way: AT+T is planning to offer 1Gbps service, but only in Google Fiber's inaugural area of Kansas City which is impressive in its brazenness. Other community-based efforts are starting to bear fruit, e.g. NAP is proposing their Avalon gigabit offering in part of Atlanta, Georgia. However, most of the USA is still stuck with practical speeds that have not changed noticeably in half a decade. Entrenched cable ISPs have spent plenty of money on lobbyists to ensure that states and cities make it expensive and difficult for newcomers to compete with them, requiring extensive studies and limiting rights to dig or string fiber-optic cable to residential addresses.

So there's clearly a problem; why won't Net Neutrality solve it?

The ISP problem

Net neutrality essentially says that you (an ISP) can't discriminate between bytes from one service and bytes from a different service. Suppose you have two providers of streaming Internet movies: Netflix and Apple iTunes. Suppose Comcast subscribers in rural Arkansas pay Comcast for a 20Mbps service, easily sufficient for HD streaming video. Comcast controls the network which ends at their customers' home routers, and when it receives a TCP or UDP packet (small chunk of data) from their customers they will look at its destination address and forward it either to its destination - e.g. a server in the Comcast network - or to one of the other Internet services they "peer" to. Peering is a boundary across which Internet entities exchange Internet data. When data comes back across that boundary with the address of one of their customers, Comcast routes the data to the customer in question. So far, so good.

Now the customer is paying Comcast for their connection, so it's not really reasonable for Comcast to force them to pay more for more data above and beyond the plan they've agreed. If you've got a 20 Mbps connection, you expect to be able to send / receive 20Mbps more or less forever. Comcast might have a monthly bandwidth cap beyond which you pay more or get a lower speed, but that should be expressed in your plan. Comcast might weight certain kinds of traffic lower than others, so that when 20 people are contending for use of a 100 Mbps pipe traffic which is less sensitive to being dropped (e.g. streaming video) is dropped more often than more sensitive traffic (web page fetches), but that's all reasonable as long as you know how many people you're contending with and what the rules are.

Streaming video is one kind of traffic that's problematic for ISPs: it requires very little bandwidth from the paying customer. They send an initial message "I want to see this video" and then a low volume of following messages to control the video stream and assure the video streaming service that someone really is still watching it. From Comcast's point of view, though, they have a large amount of latency-sensitive traffic coming into their network from a peering point, so they need to route it through to the destination user and use up a large chunk of their network capacity in the process. If lots of people want to watch videos at once, they'll have to widen the incoming pipe from their peer; that will involve buying extra hardware and paying for its associated management overhead so that they can handle the traffic, as long as they are the limiting factor. (Their peer might also be the limiting factor, but that's less likely).

So the more data users stream concurrently, the more it costs Comcast. This can be mitigated to some extent by caching - storing frequently used data within the Comcast network so that it doesn't have to be fetched from a peer each time - and indeed this is a common strategy used by content delivery networks like Akamai and video streaming firms like YouTube. They provide a bunch of their own PCs and hard disks which Comcast stores inside its datacenters, and when a user requests a resource (video, image, music file, new operating system image) which might be available in that cache they will be directed to the cache computers. The cache will send the data directly if it's available; if not, it will download it and send it on, but store it locally so if someone else requests it then it's ready to send to them directly. This has the effect of massively reducing the bandwidth for popular data (large ad campaigns, "Gangnam Style" videos, streaming video releases), and also increases reliability and reduces latency of the service from the user's perspective, but costs the provider a substantial overhead (and operational expertise) to buy, emplace and maintain the hardware and enable the software to use it.

The non-neutral solution

If Netflix aren't willing or able to pay for this, Comcast is stuck with widening their pipe to their peers. One might argue that that's what they're supposed to do, and that their customers are paying them to be able to access the Greater Internet at 20Mbps, not just Comcast's local services. But Comcast might not see it this way. They know what destination and source addresses belong to Netflix, so they might decide "we have 100 Gbps of inbound connectivity on this link, and 50 Gbps of that is Netflix video streaming source addresses at peak. Let's reduce Netflix to a maximum of 20 Gbps - at peak, any packet from Netflix video streaming sources has a 60% chance of being dropped - and see what happens."

You see where the "neutrality" aspect comes in? Comcast is dropping inbound traffic based solely on its source address - what company it comes from. Only internal Comcast configuration needs to be changed. From the customer's point of view, Netflix traffic is suddenly very choppy or even nonfunctional at peak times - but YouTube, Facebook, Twitter etc. all work fine. So Netflix must be the problem. Why am I paying them money for this crap service? (Cue angry mail to Netflix customer support).

Net Neutrality says that Comcast can't do this - it can't discriminate based on source or destination address. Of course, it's not really neutral because ISPs might still blacklist traffic from illegal providers e.g. the Pirate Bay, but since that's normally done at the request of law enforcement it's regarded as OK by most.

The problem

The USA has handed the Federal Communications Commission, via the "general conduct" rules, a massive amount of control of and discretion in the way in which ISPs handle Internet traffic. It presumes that the FCC has the actual best interests of American consumers at heart, and is intelligent and foresighted enough to apply the rules to that effect. Given the past history of government agencies in customer service and in being effectively captured by the industries they are supposed to regulate, this seems... unwise.

Scentrics, "Key Man" and mobile security, oh my

From a story in the Daily Mail today I found this October article in the Evening Standard about security firm Scentrics which has been working with UCL

In technical parlance, Scentrics has patented the IP for “a standards-based, fully automatic, cryptographic key management and distribution protocol for UMTS and TCP/IP”. What that translates as in layman’s language is “one-click privacy”, the pressing of a button to guarantee absolute security.
Where issues of national security are concerned, the ciphers used are all government-approved, which means messages can be accessed if they need to be by the security services. What it also signals in reality is a fortune for Scentrics and its wealthy individual shareholders, who each put in £500,000 to £10 million.

Hmm. That's a fairly vague description - the "government-approved" language makes it look like key escrow, but it's not clear. I was curious about the details, but there didn't seem to be any linked from the stories. Chandrasekaran was also touting this in the Independent in October, and it's not clear why the Mail ran with the story now.

I tried googling around for any previous news from Scentrics. Nada. So I tried "Paran Chandrasekaran" and found him back in 2000 talking about maybe netting £450M from the prospective sale of his company Indicii Salus. I couldn't find any announcements about the sale happening, but it looks like email security firm Comodo acquired the IP from Indicii Salus in March 2006. According to Comodo's press release

The core technology acquired under this acquisition includes Indicii Salus Limited's flagship security solution which, unlike other PKI offerings, is based on server-centric architecture with all information held securely in a central location thus providing a central platform necessary to host and administer central key management solutions.

That's a single-point-of-failure design of course - when your central server is down, you are screwed, and all clients need to be able to authenticate your central server so they all need its current public key or similar signature validation. It's not really world-setting-on-fire, but hey it's 8 years ago.

Then LexisWeb turns up an interesting court case: Indicii Salus Ltd v Chandrasekaran and others with summary "Claimant [Indicii Salus] alleging defendants [Chandrasekaran and others] intending to improperly use its software - Search order being executed against defendants - Defendants applying to discharge order - Action being disposed of by undertakings not to improperly use software"

Where the claimant had brought proceedings against the defendants, alleging that they intended to improperly use its software in a new business, the defendants' application to discharge a search order, permitting a search of the matrimonial home of the first and second defendants, would be dismissed.

The case appears to be fairly widely quoted in discussions of search+seizure litigation. I wonder whether Paran Chandrasekaran was one of the defendants here, or whether they were other family members? There's no indications of what happened subsequently.

How odd. Anyway, here's a sample of the Scentrics patent (USPTO Patent Application 20140082348):

The invention extends to a mobile device configured to:
send to a messaging server, or receive from a messaging server, an electronic message which is encrypted with a messaging key;
encrypt a copy of the message with a monitoring key different from the messaging key; and
send the encrypted copy to a monitoring server remote from the messaging server.
[...]
Thus it will be seen by those skilled in the art that, in accordance with the invention, an encrypted copy of a message sent securely from the mobile device, or received securely by it, is generated by the device itself, and is sent to a monitoring server, where it can be decrypted by an authorized third party who has access to a decryption key associated with the monitoring key. In this way, an authorized third party can, when needed, monitor a message without the operator of the messaging server being required to participate in the monitoring process.
Because both the message and its copy are encrypted when in transit to or from the mobile device, unauthorized eavesdropping by malicious parties is still prevented.

This reads to me like "given a message and a target, you encrypt it with a public key whose private key is held by your target and send it to the target as normal, but you also encrypt it with a separate key known to a potential authorized snooper and send it to their server so that they can access if they want to."

WTF? That's really not a world-beating million-dollar idea. Really, really it's not. Am I reading the wrong patent here? Speaking personally, I wouldn't invest in this idea with five quid I found on the street.

A caricature of Civil Service placement and rhetoric

The new director of GCHQ was announced earlier this year as Robert Hannigan, CMG (Cross of St Michael and St George, aka "Call Me God") replacing the incumbent Sir Iain Lobban, KCMG (Knight's Cross of St Michael and St George, aka "Kindly Call Me God"). Whereas Sir Iain was a 30 year veteran of GCHQ, working his way up from a language specialist post, Hannigan was an Oxford classicist - ironically at Wadham, one of the few socialist bastions of the university - and worked his way around various government communications and political director posts before landing a security/intelligence billet at the Cabinet office. Hannigan is almost a cliché of the professional civil servant.

Hannigan decided to write in the FT about why Facebook, Twitter and Google increasing user security was a Bad Thing:

The extremists of Isis use messaging and social media services such as Twitter, Facebook and WhatsApp, and a language their peers understand. The videos they post of themselves attacking towns, firing weapons or detonating explosives have a self-conscious online gaming quality. [...] There is no need for today’s would-be jihadis to seek out restricted websites with secret passwords: they can follow other young people posting their adventures in Syria as they would anywhere else.

Right - but the UK or US governments can already submit requests to gain access to specific information stored by Facebook, Google, Twitter et al. What Hannigan leaves out is: why is this not sufficient? The answer, of course, is that it's hard to know where to look. Far easier to cast a dragnet through Internet traffic, identify likely sources of extremism, and use intelligence based on their details to ask for specific data from Facebook, Google, Twitter et al. But for the UK in the first half of 2014, the UK issued over 2000 individual requests for data, covering an average of 1.3 people per request. How many terrorism-related arrests (never mind convictions) correspond to this - single digits? That's a pretty broad net for a very small number of actual offenders.

Hannigan subsequently received a bitchslap in Comment is Free from Libdem Julian Huppert:

Take the invention of the radio or the telephone. These transformed the nature of communication, allowing people to speak with one another across long distances far more quickly than could have ever been imagined. However, they also meant that those wishing to do us harm, whether petty criminals or terrorists, could communicate with each other much more quickly too. But you wouldn’t blame radio or phone manufacturers for allowing criminals to speak to each other any more than you would old Royal Mail responsible for a letter being posted from one criminal to another.

Good Lord, I'm agreeing with a Libdem MP writing in CiF. I need to have a lie down.

Hannigan is so dangerous in his new role because he's never really had to be accountable to voters (since he's not a politician), nor influenced by the experience and caution of the senior technical staff in GCHQ (since he never worked there). He can view GCHQ as a factory for producing intelligence to be consumed by the civil service, not as a dangerous-but-necessary-in-limited-circumstances intrusion into the private lives of UK citizens. After all, he knows that no-one is going to tap his phone or read his email.

Personally, I'd like to see a set of 10 MPs, selected by public lottery (much like the National Lottery draw, to enforce fairness) read in on GCHQ and similar agency information requests. They'd get to see a monthly summary of the requests made and information produced, and would be obliged to give an annual public report (restricted to generalities, and maybe conducted 6 months in arrears of the requests to give time for data to firm up) on their perception of the width of the requests vs information retrieved. That's about 40 Facebook personal data trawls per MP, which is a reasonably broad view of data without excessive work. Incidentally, I'd also be interested in a breakdown of the immigration status of the people under surveillance.

State-endorsed web browsers turn out to be bad news

Making the headlines in the tech world this week has been evidence of someone trying to man-in-the-middle Chinese iCloud users:

Unlike the recent attack on Google, this attack is nationwide and coincides with the launch today in China of the newest iPhone. While the attacks on Google and Yahoo enabled the authorities to snoop on what information Chinese were accessing on those two platforms, the Apple attack is different. If users ignored the security warning and clicked through to the Apple site and entered their username and password, this information has now been compromised by the Chinese authorities. Many Apple customers use iCloud to store their personal information, including iMessages, photos and contacts. This may also somehow be related again to images and videos of the Hong Kong protests being shared on the mainland.

MITM attacks are not a new phenomenon in China but this one is widespread, and clearly needs substantial resources and access to be effective. As such, it would require at least government complicity to organise and implement.

Of course, modern browsers are designed to avoid exactly this problem. This is why the Western world devotes so much effort to implementing and preserving the integrity of the "certificate chain" in SSL - you know you're connecting to your bank because the certificate is signed by your bank, and the bank's signature is signed by a certificate authority, and your browser already knows what the certificate authority's signature looks like. But it seems that in China a lot of people use Qihoo 360 web browser. It claims to provide anti-virus and malware protection, but for the past 18 months questions have been asked about its SSL implementation:

If your browser is either 360 Safe Browser or Internet Explorer 6, which together make up for about half of all browsers used in China, all you need to do is to click continue once. You will see no subsequent warnings. 360's so-called "Safe Browser" even shows a green check suggesting that the website is safe, once you’ve approved the initial warning message.

I should note, for the sake of clarity, that both the 2013 and the current MITM reports come from greatfire.org, whose owners leave little doubt that they have concerns about the current regime in China. A proper assessment of Qihoo's 360 browser would require it to be downloaded on a sacrificial PC and used to check out websites with known problems in their SSL certificates (e.g. self-signed, out of date, being MITM'd). For extra points you'd download it from a Chinese IP. I don't have the time or spare machine to test this thoroughly, but if anyone does then I'd be interested in the results.

Anyway, if the browser compromise checks out then I'm really not surprised at this development. In fact I'm surprised it hasn't happened earlier, and wonder if there have been parallel efforts at compromising IE/Firefox/Opera/Chrome downloads in China: it would take substantial resources to modify a browser installer to download and apply a binary patch to the downloaded binary which allowed an additional fake certificate authority (e.g. the Chinese government could pretend to be Apple), and more resources to keep up to date with browser releases so that you could auto-build the patch shortly after each new browser version release, but it's at least conceivable. But if you have lots of users of a browser developed by a firm within China, compromising that browser and its users is almost as good and much, much easier.

Corporate welfare from Steelie Neelie and the EU

I used to be the starry-eyed person who thought that governments pouring into a new concept for "research" was a good thing. That didn't last long. Now I read The Reg on the EU's plan to chuck 2.5 billion euros at "Big Data" "research" and wonder why, in an age of austerity, the EU thinks that pissing away the entire annual defence budget of Austria is a good idea.

First, a primer for anyone unfamiliar with "Big Data". It's a horrendously vague term, as you'd expect. The EU defines the term thus:

Big data is often defined as any data set that cannot be handled using today’s widely available mainstream solutions, techniques, and technologies.

Ah, "mainstream". What does this actually mean? It's a reasonable lower bound to start with what's feasible on a local area network. If you have a data set with low hundreds of terabytes of storage, you can store and process this on some tens of regular PCs; if you go up to about 1PB (petabyte == 1024 terabytes, 1 terabyte is the storage of a regular PC hard drive) then you're starting to go beyond what you can store and process locally, and need to think about someone else hosting your storage and compute facility.

Here's an example. Suppose you have a collection of overhead imagery of the United Kingdom, in the infra-red spectrum, sampled at 1m resolution. Given that the UK land area is just under 250 thousand square kilometers, if you represent this in an image with 256 levels of intensity (1 byte per pixel) you'll need 250,0000 x (1000 x 1000) = 250 000 000 000 pixels or 250 gigabytes of storage. This will comfortably fit on a single hard drive. If you reduce this to 10cm resolution - so that at maximum resolution your laptop screen of 1200 pixel width will show 120m of land - then you're looking at 25 TB of data, so you'll need a network of tens of PCs to store and process it. If, instead of a single infra-red channel, you have 40 channels of different electromagnetic frequencies, from low infra-red up to ultra violet, you're at 1PB and need Big Data to solve the problem of processing the data.

Another example, more privacy-concerning: if you have 1KB of data about each of the 7bn people in the world (say, their daily physical location over 1 year inferred from their mobile phone logs), you'll have 7 terabytes of information. If you have 120 KB of data (say, their physical location every 10 minutes) then this is around 1PB and approaches the Big Data limits.

Here's the press release:

Mastering big data could mean:

• up to 30% of the global data market for European suppliers;
• 100,000 new data-related jobs in Europe by 2020;
• 10% lower energy consumption, better health-care outcomes and more productive industrial machinery.

My arse, but let's look at each claim in turn.

• How is this project going to make it more likely for European suppliers to take over more of the market? Won't all the results of the research be public? How, then, will a European company be better placed to take advantage of them than a US company? Unless one or more US-based international company has promised to attribute a good chunk of its future Big Data work to its European operations as an informal quid-pro-quo for funding from this pot.
• As Tim Worstall is fond of saying, jobs are a cost not a benefit. These need to be new jobs that are a prerequisite for larger Big Data economic gains to be realized, not busywork to meet artificial Big Data goals
• [citation required] to quote Wikipedia. I'll believe it when I see it measured by someone without financial interest in the Big Data project.

The EU even has a website devoted to the topic: Big Data Value. Some idea of the boondoggle level of this project can be gleaned from the stated commitment:

... to build a data-driven economy across Europe, mastering the generation of value from Big Data and creating a significant competitive advantage for European industry, boosting economic growth and jobs. The BDV PPP will commence in 2015[,] start with first projects in 2016 and will run until 2020. Covering the multidimensional character of Big Data, the PPP activities will address technology and applications development, business model discovery, ecosystem validation, skills profiling, regulatory and IPR environment and social aspects.

So how will we know if these 2.5bn Euros have been well spent? Um. Well. Ah. There are no deliverables specified, no ways that we can check back in 2020 to see if the project was successful. We can't even check in 2017 whether we're making the required progress, other than verifying that the budget is being spent at the appropriate velocity - and believe me, it will be.

The fundamental problem with widespread adoption of Big Data is that you need to accumulate the data before you can start to process it. It's surprisingly hard to do this - there really isn't that much new data generated in most fields and you can do an awful lot if you have reasonably-specced PCs on a high-speed LAN. Give each PC a few TB in storage, stripe your data over PCs for redundancy (not vulnerable to failure of a single drive or PC) and speed, and you're good to go. Even if you have a huge pile of storage, if you don't have the corresponding processing power then you're screwed and you'll have to figure out a way of copying all the data into Amazon/Google/Azure to allow them to process it.

Images and video are probably the most ripe field for Big Data, but still you can't avoid the storage/processing problem. If you already have the data in a cloud storage provider like Amazon/Google/Azure, they likely already have the processing models for your data needs; if you don't, where are all the CPUs you need for your processing? It's likely that the major limitations processing Big Data in most companies is appropriate reduction of the data to a relatively small secondary data set (e.g. processing raw images into vectors via edge detection) before sending it somewhere for processing.

The EU is about to hand a couple billion euros to favoured European companies and university research departments, and it's going to get nine tenths of squat all out of it. Mark my words, and check back in 2020 to see what this project has produced to benefit anyone other than its participants.

New clamping down on information in China

Spotted this on a net security research blog yesterday: someone is trying to snoop on the web traffic of Chinese students and researchers:

All evidence indicates that a MITM [man-in-the-middle] attack is being conducted against traffic between China’s nationwide education and research network CERNET and www.google.com. It looks as if the MITM is carried out on a network belonging to AS23911, which is the outer part of CERNET that peers with all external networks. This network is located in China, so we can conclude that the MITM was being done within the country.

To decipher this, readers should note that CERNET is the Chinese network for education and research - universities and the like. The regular Great Firewall of China blocking is fairly crude and makes it practically difficult for researchers to get access to the information they need, so CERNET users have mostly free access to the Internet at large - I'm sure their universities block access to dodgy sites, but to be fair so do Western universities. What's happening is that someone is intercepting - not just snooping on - their requests to go to www.google.com and is trying to pretend to be Google.

The reason the intercept is failing is because Google - like Facebook, Yahoo, Twitter and other sites - redirects plain HTTP requests to its homepage to a HTTPS address, so most people bookmark those sites with an HTTPS address. Therefore the users were requesting https://www.google.com/ and the attackers had to fake Google's SSL certificate. Because of of the way SSL is designed, this is quite hard; they couldn't get a reputable Certificate Authority to sign their certificate saying "sure, this is Google" so they signed it themselves, much like a schoolchild signing a note purportedly from their parent but with their own name. Modern browsers (Chrome, Firefox, modern versions of IE) warn you when this is happening, which is how the users noticed. The Netresec team's analysis showed that the timings of the steps of the connection indicated strongly that the interceptor was somewhere within China.

The attack doesn't seem to be very sophisticated, but it does require reasonable resources and access to networking systems - you've got to reprogram routers in the path of the traffic to redirect the traffic going to Google to come to your own server instead, so you either need to own the routers to start with or compromise the routers of an organisation like a university. Generally, the further you get from the user you're intercepting, the greater your resources need to be. It would be interesting to know what fraction of traffic is being intercepted - the more users you're intercepting, the more computing resource you need to perform the attack because you've got to intercept the connection, log it, and then connect to Google/Twitter/Yahoo yourself to get the results the user is asking for.

The attempted intercepts were originally reported on the Greatfire.org blog which observes that there were several reports from around CERNET of this happening. Was this a trial run? If so it has rather blown up in the faces of the attackers; now the word will circulate about the eavesdropping and CERNET users will be more cautious when faced with odd connection errors.

If the attackers want to press on, I'd expect the next step to be more sophisticated. One approach would be SSL stripping where the interceptor tries to downgrade the connection - the user requests https://www.twitter.com/ but the attacker rewrites that request to be http://www.twitter.com/. The user's browser sees a response for http instead of https and continues with an unencrypted connection. Luckily, with Twitter this will not work well. If you run "curl -I https://www.twitter.com/" from a command line, you'll see this:

HTTP/1.1 301 Moved Permanently
content-length: 0
date: Sat, 06 Sep 2014 17:23:21 UTC
location: https://twitter.com/
server: tsa_a
set-cookie: guest_id=XXXXXXXXXXXXXXXXX; Domain=.twitter.com; Path=/; Expires=Mon, 05-Sep-2016 17:23:21 UTC
strict-transport-security: max-age=631138519
x-connection-hash: aaaaaaaaaaaaaaaa

That "strict-transport-security" line tells the browser that future connections to this site for the next N seconds must use HTTPS, and the browser should not continue the connection if the site tries to use HTTP. This is HTTP Strict Transport Security (HSTS) and Twitter is one of the first big sites I've seen using it - Google and Facebook haven't adopted it yet, at least for their main sites.

Alternatively the interceptor may try to compromise a reputable certificate authority so it can forge SSL certificates that browsers will actually accept. This would be a really big investment, almost certainly requiring nation-state-level resources, and would probably not be done just to snoop on researchers - if you can do this, it's very valuable for all sorts of access. It also won't work for the major sites as browsers like Chrome and Firefox use certificate pinning - they know what the current version of those sites' SSL certs look like, and will complain loudly if they see something different.

The most effective approach, for what it's worth, is to put logging software on all the computers connected to CERNET, but that's probably logistically infeasible - it only works for targeting a small number of users.

So someone with significant resources in China is trying to find out what their researchers are searching for. Is the government getting nervous about what information is flowing into China via this route?

Formalising success in a bureaucracy

It's only natural, when you've managed to get out of a hole against all odds, that you want to re-use the people and/or planning that made the difference. You'd be wasteful if you didn't, to be honest. Following this line of thinking, and after a small team of digital fixers managed to save the flagship Healthcare.Gov federal healthcare exchange from near-certain doom, the White House is trying to do just that.

Today they announced the launch of the new U.S. Digital Service which aims to replicates the lessons of the (relative) success in saving Healthcare.Gov with other troubled US federal government IT projects. Heaven knows that there's no shortage of potential targets for USDS to help with. The question of the moment is: can this new government team actually succeed? If so, what does success look like?

US CIO Steve van Roekel outlined the USDS role:

"This isn't going to be a group that we parachute in to write code," as Van Roekel put it in a call earlier this summer, and with perhaps the Department of Health and Human's experience with HealthCare.gov on the brain, "This isn't decending a group of developers onto the scene." Rather, the focus is going to be on helping agencies figure out where their weak points are and how to fix them.

Note that therefore the role of USDS staff isn't actually the same as the Healthcare.Gov fixers, but that might be OK as the fixing itself wouldn't scale; if you want to solve the key IT problems of more than one government agency at at time then you can't have most your staff embedded in one project, and there's no reason to think that the government can recruit multiples of the motivated team that fixed Healthcare.gov. They're going to have to strike a balance, though. They won't be able to determine the principal IT problems of an agency without spending time working with and talking to the agency's tech team. The more time they spend there, the more trust they'll gain and the better the quality of information they'll gather - but then they won't be able to help as many agencies.

The danger with any new government agency is that after a time it accumulates bureaucrats whose primary purpose is propagating their own employment and importance. Van Roekel seems to be aware of this and planning to bring in people for 2-4 year rotations. With placements of 3-6 months this may be about right; long enough for the new people to spend a placement or two with the veterans and absorb the institutional knowledge, do a couple more placements as peers while encouraging their friends to join up, then lead new recruits in placements as the veterans leave.

What's going to be interesting is to see how the USDS embeds are treated in the troubled agencies. Are they going to have the influence and effective power to remove obstructions - such as long-term barnacle workers who hoard knowledge and obstruct progress? If not, they're unlikely to be able to change much. If so, the agency's workers are going to hunker down and be terrified of being fired or reassigned. It's going to be quite a challenge for tech sector workers to get their heads around the government worker mindset sufficiently to influence those workers into getting things fixed.

Incidentally, www.usds.gov was not resolving as of posting time; I actually consider that a potential sign of success as the new team is focusing on getting operational before getting any marketing/PR in place; still, they're going to need a portfolio of some form after a few months in order to attract their new short-term hires.

The joys of hard drive death

The IRS (US tax service) ex-head Lois Lerner has been under the spotlight in the past year about the IRS allegedly targeting organisations for audit based on their political allegiances. Apparently Tea Party related organisations were much more likely to be targeted than left-leaning organisation. Lerner retired from the agency in September last year, but the Republican party has unsurprisingly been chasing her. Lerner took the 5th at a hearing in March, refusing to testify to avoid the risk of incriminating herself, so the investigators have been looking for other sources of information.

Of course, most communication these days is done by email, and the IRS is no exception. The obvious place to start in finding the details of Lerner's involvement - if any - would be to trawl her email. Except that this appears to be difficult:

Today, Ways and Means Committee Chairman Dave Camp (R-MI) issued the following statement regarding the Internal Revenue Service informing the Committee that they have lost Lois Lerner emails from a period of January 2009 – April 2011. Due to a supposed computer crash, the agency only has Lerner emails to and from other IRS employees during this time frame.

Oopsie. Still, these things happen occasionally. It's just bad luck, right?

The IRS has 89,500 employees. It's not unreasonable to estimate that every one of them has an email account, and most of them have a computer. Say they have 70,000 personal computers on their network. Every computer has at least one hard drive. A hard drive's average life is 2-3 years; let's say 1000 days. On average, if you have 1000 hard drives, one will be failing each day. In the case of the IRS we'd expect to see 70 hard drives a day, nearly 500 per week, failing. Hard drives failing are a completely normal part of IRS IT operations.

Given that, you put together an IT system that lets your executives lose all their emails whenever their personal computer hard drive crashes? This seems... not the approach one would normally take.

What I find interesting is an IRS note from 1998 announcing that they were standardising on Exchange:

The new e-mail package will use Microsoft Exchange Server Version 5.5 along with the Microsoft Outlook 98 desktop product. The IRS will switch over to the new system during the next 12 months

I'm assuming that by now they've done several migrations to more modern versions of Exchange. By 2009 they should have been on Exchange 2003 at least, maybe 2007. A user's emails would be in folders on replicated central storage, not just on a personal machine; the Outlook client would copy mails from the central storage to the local computer for speed and ease of access, but they would remain in the central storage precisely because personal computers fail all the time. Suppose the power supply exploded, or the motherboard shorted, or coffee spilled into the CD-ROM drive slot, or the user has to get email access out of office hours (e.g. via Outlook Web Access) - there has to be a way to get to their data when the PC is not available. The replicated storage copies the data to several physically separate machines, using a scheme such as RAID which lets you trade off the number of copies of data, read performance and write performance.

What I would believe, and I should make it clear that this is pure speculation, is that someone was deleting old emails off the replicated storage for some purpose; perhaps for perfectly legitimate purposes. They ended up deleting much more than they expected. Once this was discovered, they tried to recover the data from the daily / weekly tape backups that were almost certainly being made from the central storage. When they did this, they discovered that for the past 1-2 years the backup data being written wasn't being written correctly - taken from the wrong source, missing indexes, taken from a source that was being updated as it was being read, whatever. This was so embarrassing given the amount of money that they were spending on their storage and backups that they cooked up a story about a hard drive failing and hoped no-one would ask any inconvenient questions. Bad luck, boys!

If the details of IRS's excuse haven't been mis-reported - a possibility we should not reject out of hand - then either they have a painfully badly assembled and operated IT system, or someone is telling pork pies.