Public goods can also be public bads

Imagine a village in the middle of the woods. The villagers use wood for many of their needs - building houses, firewood etc. - but it's rather tedious carrying axes into the wood to find a suitable tree, and then axing down a tree is slow and error-prone. The village elders, mindful of this, buy a job lot of high-spec chainsaws and distribute them around the wood marked by big red flags. A villager can wander into the wood, spot a handy-looking tree, find the nearest chainsaw, drop the tree in no time and then leave the chainsaw by the flag before pulling his tree back home.

Unfortunately, some of the village youth have a nihilistic bent. One evening they go into the woods, get a chainsaw each, bring them back to the village and chop down the structural pillars of a number of houses before they can be caught and stopped. The village elders are embarrassed that their good intentions in improving village life have been turned on them to wreak industrial havoc. Perhaps leaving high-powered machinery around the woods for anyone to use has its downsides?

This is roughly the situation that the Internet finds itself in currently, as described in CloudFlare's account of the ongoing DDoS that nearly broke the Internet. Anyone with a technical bent should go read the original, for it is a very good (if frightening) piece. For those less technical or with less time, here's the short version.

Spamhaus is a long-existing Internet establishment that does its best to identify email spammers and the machines they use to spam, and feeds data to major Internet Service Providers and other entities enabling them to identify that spam and cut it off early before it overwhelms users' inboxes. They have been very successful at this; the email spam problem today is still substantial but much, much better than it used to be even a couple of years ago. Recently they identified a fairly "liberal" Dutch hosting company "Cyberbunker" as spammers and started including them in their blacklist. It would be safe to say that Cyberbunker did not appreciate this.

Last week Spamhaus was on the receiving end of a big Distributed Denial of Service (DDoS) attack, thousands of compromised computers being used to drown Spamhaus's website in a flood of requests. This was initially very successful. Spamhaus asked for help, and distributed hosting provider CloudFlare stepped in to host Spamhaus. Their defences and capacity could cope with the attack. But this did not stop the attackers, who have raised their game in recent days:

An engineer at one of the largest Internet communications firms said the attacks in recent days have been as many as five times larger than what was seen recently in attacks against major American banks. [my emphasis] He said the attacks were not large enough to saturate the company's largest routers, but they had overwhelmed important equipment.

The attacks have been so big (up to 300 gigabits per second - enough data every second to represent the text of 100,000 novels) that they have started to saturate some of the networking hardware of the Internet exchanges, the entities which "glue" the major parts of the Internet together. If you've been seeing slower-than-usual Internet speeds over the past few days, this may have been part of the problem.

What does this have to do with chainsaws in the woods? Well, the attackers have a lot of computers under their control, but those computers are mostly on regular home Internet connections and can't get near the upload rate they'd need to each 300Gbps. Instead they are sending forged requests to open DNS recursor hosts. These computers, which are the chainsaws in our example, are part of the Internet's naming system - "DNS", the Domain Name System - which translates human-readable names into the numeric addresses used by the Internet. As an example, www.dailymail.co.uk translates into the Internet (version 4) address 23.59.191.33.

Normally these computers are provided by ISPs and serve only that ISP's customers. However a number of them, either by misconfiguration or by design, accept requests from anyone. Worse, a) there are certain queries where a very small request can result in the DNS host returning a large amount of data (the "amplification" problem) and b) it is possible for the requesting computer to forge its sending address to pretend that it's a different computer. The result of this is that a single computer with a very slow link to the rest of the Net can command an open DNS host to send a much larger stream of data to any Internet address it chooses. This is a big problem, allowing distributed denial of service attacks of much more traffic than the compromised computers can send.

I expect that as a result of the Spamhaus attack more work will be done to lock down open DNS hosts, or at least get them to react much more slowly to unknown users. Still, this situation is a reminder that providing public goods can come with unexpected public costs.

Update: a sysadmin with an open DNS server confesses. Well, that's 0.0005 Gbit/s down, only 299.9995 Gbit/s to go.