Verisign saw MASSIVE query spike during Facebook outage

Verisign’s .com and .net name servers saw a huge spike in queries when Facebook went offline for hours last October, Verisign said this week.

Queries for facebook.com, instagram.com, and whatsapp.net peaked at over 900,000 per second during the outage, up from a normal rate of 7,000 per second, a more than 100x increase, the company said in a blog post.

The widely publicized Facebook outage was caused by its IP addresses, including the IP addresses of its DNS servers, being accidentally withdrawn from routing tables. At first it looked to outside observers like a DNS failure.

When computers worldwide failed to find Facebook on their recursive name servers, they went up the hierarchy to Verisign’s .com and .net servers to find out where they’d gone, which led to the spike in traffic to those zones.

Traffic from DNS resolver networks run by Google and Cloudflare grew by 7,000x and 2,000x respectively during the outage, Verisign said.

The company also revealed that the failure of .club and .hsbc TLDs a few days later had a similar effect on the DNS root servers that Verisign operates.

Queries for the two TLDs at the root went up 45x, from 80 to 3,700 queries per second, Verisign said.

While the company said its systems were not overloaded, it subtly criticized DNS resolver networks such as Google and Cloudflare for “unnecessarily aggressive” query-spamming, writing:

We believe it is important for the security, stability and resiliency of the internet’s DNS infrastructure that the implementers of recursive resolvers and public DNS services carefully consider how their systems behave in circumstances where none of a domain name’s authoritative name servers are providing responses, yet the parent zones are providing proper referrals. We feel it is difficult to rationalize the patterns that we are currently observing, such as hundreds of queries per second from individual recursive resolver sources. The global DNS would be better served by more appropriate rate limiting, and algorithms such as exponential backoff, to address these types of cases

Verisign said it is proposing updates to internet standards to address this problem.

Paranoid ICANN opens another root server in China

ICANN has announced the creation of another root server instance in China, which definitely, DEFINITELY won’t let the Chinese government mess with the interwebs.
ICANN said this week that it’s opened an instance of the L-root that it manages in Shanghai.
It’s the third L-root in China but only the first outside of Beijing.
In a press release announcing the installation, which was carried out with technical support from CNNIC and Shanghai Telecom, ICANN decided to preemptively head off any concerns that putting an important piece of internet infrastructure in China comes with added security risk:

Contrary to common misconception, root servers do not control the Internet. The operation of an instance also does not provide any mechanism to alter content of the DNS. Any modification of root zone content will be mitigated by a part of the DNS protocol known as the DNS Security Extensions (DNSSEC) and if an instance fail to respond to a query, resolvers will ask the same question to another instance or root server.

It’s merely the latest of 168 L-root installations and 1,015 copies of the 13 logical root servers, which all use IP Anycast to more quickly serve DNS answers to their local users.
Given how big and populous China is, there are surprisingly few root server instances in the country, according to root-servers.org.
In addition to ICANN’s three boxes, Verisign’s J-root and Internet Systems Consortium’s F-root have three in Beijing and two in Hangzhou between them. The K, I and F roots each have one instance in Beijing.
That’s eight nodes in China proper, which has 800 million internet users. Cross the border into semi-autonomous Hong Kong, which has a population of under eight million people, and there are nine root instances.
The city of Bucharest, Romania (pop. 1.8 million) has the same number of root instances as China.

New ICANN director named

A member of the root server community has been named to the ICANN board of directors.
The Nominating Committee yesterday revealed its three selections for the board, two of whom are already seated.
The new director is Tripti Sinha of the University of Maryland, where she heads the Advanced Cyber Infrastructure and Internet Global Services division, which manages the D-root server.
Sinha is currently co-chair of ICANN’s Root Server System Advisory Committee.
She will replace fellow North American George Sadowsky who, after joining the board in 2009 and being reselected twice, is term-limited and will be given his marching orders this October.
NomCom also reaffirmed current directors Lousewies van der Laan, a former Dutch politician, and Rafael “Lito” Ibarra, founder of the El Salvadorean ccTLD .sv.
New directors will take their seats at the conclusion of the ICANN 63 meeting in Barcelona in October.
NomCom’s other selections to various leadership positions at ICANN can be found here.

NTIA alarmed as Verisign hints that it will not delegate new gTLDs

Verisign has escalated its war against competition by telling its government masters that it is not ready to add new gTLDs to the DNS root, raising eyebrows at NTIA.
The company told the US National Telecommunications and Information Administration in late May that the lack of uniform monitoring across the 13 root servers means it would put internet security and stability at risk to start delegating new gTLDs now.
In response, the NTIA told Verisign that its recent position on DNS security is “troubling”. It demanded confirmation that Verisign is not planning to block new gTLDs from being delegated.
The letters (pdf and pdf) were published by ICANN over the weekend, over two months after the first was sent.
Verisign senior VP Pat Kane wrote in the May letter:

we strongly believe certain issues have not been addressed and must be addressed before any root zone managers, including Verisign, are ready to implement the new gTLD Program.
We want to be clearly on record as reporting out this critical information to NTIA unequivocally as we believe a complete assessment of the critical issues remain unaddressed which left unremediated could jeopardize the security and stability of the DNS.
…
we strongly recommend that the previous advice related to this topic be implemented and the capability for root server system monitoring, instrumentation, and management capabilities be developed and operationalized prior to beginning delegations.

Kane’s concerns were first outlined by Verisign in its March 2013 open letter to ICANN, which also expressed serious worries about issues such as internal name collisions.
Verisign is so far the only root server operator to publicly express concerns about the lacking of coordinated monitoring, and many people believe that the company is simply desperately trying to delay competition for its $800 million .com business for as long as possible.
These people note that in early November 2012, Verisign signed a joint letter with ICANN and NTIA that said:

the Root Zone Partners are able to process at least 100 new TLDs per week and will commit the necessary resources to meet all root zone management volume increases associated with the new gTLD program

That letter was signed before NTIA stripped Verisign of its right to increase .com prices every year, depriving it of tens or hundreds of millions of dollars of additional revenue.
Some say that Verisign is raising spurious security concerns now purely because it’s worried about its bottom line.
NTIA is beginning to sound like one of these critics. In its response to the May 30 letter, sent by NTIA and published by ICANN on Saturday, deputy associate administrator Vernita Harris wrote:

NTIA and VeriSign have historically had a strong working relationship, but inconsistencies in VeriSign’s position in recent months are troubling… NTIA fully expects VeriSign to process change requests when it receives an authorization to delegate a new gTLD. So that there will be no doubt on this point, please provide me a written confirmation no later than August 16, 2013 that VeriSign will process change requests for the new gTLD program when authorized to delegate a new gTLD.

Harris said that a system is already in place that would allow the emergency rollback of the root zone, basically ‘un-delegating’ any gTLD that proves to cause a security or stability problem.
This would be “sufficient for the delegation of new gTLDs”, she wrote.
Could Verisign block new gTLDs?
It’s worth a reminder at this point that ICANN’s power over the DNS root is something of a facade.
Verisign, as operator of the master A root server, holds the technical keys to the kingdom. Under its NTIA contract, it only processes changes to the root — such as adding a TLD — when NTIA tells it to.
NTIA in practice merely passes on the recommendations of IANA, the department within ICANN that has the power to ask for changes to the root zone, also under contract with NTIA.
Verisign or NTIA in theory could refuse to delegate new gTLDs — recall that when .xxx was heading to the root the European Union asked NTIA to delay the delegation.
In practice, it seems unlikely that either party would stand in the way of new gTLDs at the root, but the Verisign rhetoric in recent months suggests that it is in no mood to play nicely.
To refuse to delegate gTLDs out of commercial best interests would be seen as irresponsible, however, and would likely put its role as custodian of the root at risk.
That said, if Verisign turns out to be the lone voice of sanity when it comes to DNS security, it is ICANN and NTIA that will ultimately look like they’re the irresponsible parties.
What’s next?
Verisign now has until August 16 to confirm that it will not make trouble. I expect it to do so under protest.
According to the NTIA, ICANN’s Root Server Stability Advisory Committee is currently working on two documents — RSSAC001 and RSSAC002 — that will outline “the parameters of the basis of an early warning system” that will address Verisign’s concerns about root server management.
These documents are likely to be published within weeks, according to the NTIA letter.
Meanwhile, we’re also waiting for the publication of Interisle Consulting’s independent report into the internal name collision issue, which is expected to recommend that gTLDs such as .corp and .home are put on hold. I’m expecting this to be published any day now.

Oh, yeah, and the DNS root got signed

It happened yesterday night.
To crushing silence from ICANN.

ICANN to stream DNSSEC ceremony live

ICANN is to webcast the second of its root server DNSSEC key generation ceremonies, this coming Monday.
You’ll be able to find the stream here, from 2000 UTC, according to a message ICANN’s DNS director Joe Abley just sent to the DNS-Ops mailing list.
The ceremony, which will likely take several hours, takes place in El Segundo, California.
In it, staff will create the Key Signing Key used in cryptographically signing the very root of the DNS according to the DNSSEC standard.
The first such ceremony took place last month at a facility in Virginia. While it was recorded, as well as witnessed by several well-known security experts, it was not streamed live.
The full transition to a validatable DNSSEC-signed root is still scheduled for next Thursday, July 15.
Abley’s update is likely to be available here shortly.

ICANN creates DNSSEC root keys

ICANN took the penultimate step towards adding DNSSEC to the root of the domain name system, during in a lengthy ceremony in Virginia yesterday.
The move means we’re still on track to have the DNSSEC “trust anchor” go live in the root on July 15, which will make end-to-end validation of DNS answers feasible for the first time.
DNSSEC is an extension to the DNS protocol that enables resolvers to validate that the DNS answers they receive come from the true owner of the domain.
Yesterday, ICANN generated the Key Signing Key for the root zone. That’s one of two keys required when adding DNSSEC to a zone.
The KSK is used to sign the DNSKey record, the public half of a key pair used to validate DNS responses. It has a longer expiration date than the Zone Signing Key used to sign other records in the zone, so its security is more important.
The videotaped ceremony, held at a facility in Culpeper, Virginia, was expected to take six hours, due to a lengthy check-list of precautions designed to instil confidence in the security of the KSK.
ICANN said:

During the ceremony, participants were present within a secure facility and witnessed the preparations required to ensure that the so-called key-signing-key (KSK) was not only generated correctly, but that almost every aspect of the equipment, software and procedures associated with its generation were also verified to be correct and trustworthy.

Ten hand-picked independent observers were present to bear witness.
ICANN expects to perform the ceremony four times a year. The second will be held at a backup facility in California next month.

ICANN staff need to get their pee tested

I imagine it’s a pretty hard job, largely thankless, working at ICANN. No matter what you do, there’s always somebody on the internet bitching at you for one reason or another.
The job may be about to get even more irksome for some staffers, if ICANN decides to implement new security recommendations made by risk management firm JAS Communications.
In a report published yesterday, JAS suggests that senior IANA staff – basically anyone with critical responsibilities over the DNS root zone – should be made to agree to personal credit checks, drug screening and even psych evaluations.
To anyone now trying to shake mental images of Rod Beckstrom peeing into a cup for the sake of the internet, I can only apologise.
This is what the report says:

JAS recommends a formal program to vet potential new hires, and to periodically re‐vet employees over time. Such a vetting program would include screening for illegal drugs, evaluation of consumer credit, and psychiatric evaluation, which are all established risk factors for unreliable and/or malicious insider activity and are routinely a part of employee screening in government and critical infrastructure providers.

I’ve gone for the cheap headline here, obviously, but there’s plenty in this report to take seriously, if you can penetrate the management consultant yadda yadda.
There are eight other recommendations not related to stoners running the root, covering contingencies such as IANA accidentally unplugging the internet and Los Angeles sinking into the Pacific.
Probably most interesting of all is the bit explaining how ICANN’s custom Root Zone Management System software, intended to reduce the possibility of errors creeping into the root after hundreds of new TLDs are added, apparently isn’t being built with security in mind.
“No formal requirements exist regarding the security and resiliency of these systems, making it impossible to know whether the system has been built to specification,” the report says.
It also notes that ICANN lacks a proper risk management strategy, and suggests that it improve communications both internally and with VeriSign.
It discloses that “nearly all critical resources are physically located in the greater Los Angeles area”, which puts the IANA function at risk of earthquake damage, if nothing else.
JAS recommends spreading the risk geographically, which should give those opposed to ICANN bloat something new to moan about.
There’s a public comment forum over here.
UPDATE (2010-06-13): As Michael Palage points out over at CircleID, ICANN has pulled the PDF from its web site for reasons unknown.
On the off-chance that there’s a good security reason for this, I shall resist the temptation to cause mischief by uploading it here. This post, however, remains unedited.

Root DNSSEC push delayed two weeks

The final rollout of DNSSEC to the internet’s root servers, a major security upgrade for the domain name system, has been pushed back two weeks to July 15.
ICANN’s DNS director Joe Abley said in an update on root-dnssec.org and in email to the dns-ops mailing list:

The schedule change is intended to allow ICANN and VeriSign an additional two weeks for further analysis of the DURZ rollout, to finalise testing and best ensure the secure, stable and resilient implementation of the root DNSSEC production processes and systems.

The Deliberately-Unvalidatable Root Zone is a way for the root operators to test how normal DNS resolution copes with fatter DNSSEC responses coming from the root, before worrying about issues concerning DNSSEC validation itself.
The DURZ has been cautiously rolled out over the last few months and has been operational across all 13 root servers since May 5.
The original plan called for the roots to become validatable following a key signing ceremony on July 1
The schedule change from ICANN also comes with a notice that the US government will be asking for public comment before the decision is made to properly sign the root.

Prior to 2010-07-15 the U.S. Department of Commerce (DoC) will issue a public notice announcing the publication of the joint ICANN-VeriSign testing and evaluation report as well as the intent to proceed with the final stage of DNSSEC deployment. As part of this notice the DoC will include a public review and comment period prior to taking any action.

I may be just a little forgetful, but I can’t remember hearing about this Commerce involvement before.
Still, DNSSEC is a big change, so there’s nothing wrong with more of the softly-softly approach.

I-Root yanks Beijing node

Autonomica, which runs i-root-servers.net, has stopped advertising its Anycast node in Beijing, after reports last week that its responses were being tampered with.
In the light of recent tensions between China and the US, people got a bit nervous after the Chilean ccTLD manager reported some “odd behaviour” to the dns-ops mailing list last week.
It seemed that DNS lookups for Facebook, Twitter and YouTube were being censored as they returned from I-Root’s node in China, which is hosted by CNNIC.
There was no suggestion that Autonomica was complicit in any censorship, and chief executive Karl Erik Lindqvist has now confirmed as much.
“Netnod/Autonomica is 100% committed to serving the root zone DNS data as published by the IANA. We have made a clear and public declaration of this, and we guarantee that the responses sent out by any i.root-servers.net instance consist of the appropriate data in the IANA root zone,” he wrote.
While Lindqvist is not explicit, the suggestion seems to be that somebody on the Chinese internet not associated with I-Root has been messing with DNS queries as they pass across the network.
This is believed to be common practice in China, whose citizens are subject to strict censorship, but any such activity outside its borders obviously represents a threat to the internet’s reliability.
The CNNIC node is offline until further notice.