One of ICANN’s Seven Secret Key-Holders To The Internet got taken out as part of an elaborate heist or something on American TV this week.
In tense scenes, a couple of secret agents or something with guns were forced to break into one of ICANN’s quarterly root zone key signing ceremonies to prevent a hacker or terrorist or something from something something, something something.
The stand-off came after the secret agents or whatever discovered that a hacker called Mayhew had poisoned a guy named Adler, causing a heart attack, in order to secure his position as a replacement ICANN key-holder and hijack the ceremony.
This all happened on a TV show called Blacklist: Redemption that aired in the US March 16.
I’d be lying if I said I fully understood what was supposed to be going on in the episode, not being a regular viewer of the series, but here’s the exposition from the beginning of the second act.
Botox Boss Lady: Seven keys control the internet? That can’t be possible.
Neck Beard Exposition Guy: They don’t control what’s on it, just how to secure it. All domain names have an assigned number. But who assigns the numbers?
Soap Opera Secret Agent: Key holders?
Neck Beard Exposition Guy: Seven security experts randomly selected by ICANN, the Internet Corporation for Assigned Names and Numbers.
Bored Secret Agent: Max Adler’s wife mentioned a key ceremony.
Neck Beard Exposition Guy: Yeah, four times a year the key holders meet to generate a master key and to assign new numbers, to make life difficult for hackers who want to direct folks to malicious sites or steal their credit card information.
Botox Boss Lady: But by being at the ceremony, Mayhew gets around those precautions?
Neck Beard Exposition Guy: Oh, he does more than that. He can route any domain name to him.
That’s the genuine dialogue. ICANN, jarringly, isn’t fictionalized in the way one might usually expect from US TV drama.
The scene carries on to explain the elaborate security precautions ICANN has put in place around its key-signing ceremonies, including biometrics, smart cards and the like.
The fast-moving show then cuts to the aforementioned heist situation, in which our villain of the week takes an ICANN staffer hostage before using the root’s DNSSEC keys to somehow compromise a government data drop and download a McGuffin.
Earlier this week I begged Matt Larson, ICANN’s VP of research and a regular participant in the ceremonies (which are real) to watch the show and explain to me what bits reflect reality and what was plainly bogus.
“There are some points about it that are quite close to how the how the root KSK administration works,” he said, describing the depiction as “kind of surreal”.
“But then they take it not one but two steps further. The way the ceremony happens is not accurate, the consequences of what happens at the ceremony are not accurate,” he added.
“They talk about how at the ceremony we generate a key, well that’s not true. It’s used for signing a new key. And then they talk about how as a result of the ceremony anyone can intercept any domain name anywhere and of course that’s not true.”
The ceremonies are used to sign the keys that make end-to-end DNSSEC possible. By signing the root, DNSSEC resolvers have a “chain of trust” that goes all the way to the top of the DNS hierarchy.
The root keys just secure the bit between the root at the TLDs. Compromising them would not enable a hacker to immediately start downloading data from the site of his choosing, as depicted in the show. He’d then have to go on to compromise the rest of the chain.
“You’d have to create an entire path of spoofed zones to who you wanted to impersonate,” Larson said. “Your fake root zone would have to delegate to a fake TLD zone to a fake SLD zone and so on so you could finally convince someone they were going to the address that you wanted.”
“If you could somehow compromise the processes at the root, that alone doesn’t give you anything,” he said.
But the show did present a somewhat realistic description of how the ceremony rooms (located in Virginia and California, not Manhattan as seen on TV) are secured.
Among other precautions, the facilities are secured with smart cards and PINs, retina scans for ICANN staff, and have reinforced walls to prevent somebody coming in with a sledgehammer, Larson said.
Blacklist: Redemption airs on Thursday nights on NBC in the US, but I wouldn’t bother if I were you.
ICANN is about to embark on a year-long effort to warn the internet that it plans to replace the top-level cryptographic keys used in DNSSEC for the first time.
CTO David Conrad told DI today that ICANN will rotate the so-called Key Signing Key that is used as the “trust anchor” for all DNSSEC queries that happen on the internet.
Due to the complexity of the process, and the risk that something might go wrong, the move is to be announced in the coming days even though the new public key will not replace the existing one until October 2017.
The KSK is a cryptographic key pair used to sign the Zone Signing Keys that in turn sign the DNS root zone. It’s basically at the top of the DNSSEC hierarchy — all trust in DNSSEC flows from it.
It’s considered good practice in DNSSEC to rotate keys every so often, largely to reduce the window would-be attackers have to compromise them.
The Zone Signing Key used by ICANN and Verisign to sign the DNS root is rotated quarterly, and individual domain owners can rotate their own keys as and when they choose, but the same KSK has been in place since the root was first signed in 2010.
Conrad said that ICANN is doing the first rollover partly to ensure that the procedures in has in place for changing keys are effective and could be deployed in case of emergency.
That said, this first rotation is going to happen at a snail’s pace.
Key generation is a complex matter, requiring the physical presence of at least three of seven trusted key holders.
These seven individuals possess physical keys to bank-style strong boxes which contain secure smart cards. Three of the seven cards are needed to generate a new key.
Each of the quarterly ZSK signing ceremonies — which are recorded and broadcast live over the internet — takes about five hours.
The first step in the rollover, Conrad said, is to generate the keys at ICANN’s US east coast facility in October this year. A copy will be moved to a facility on the west coast in February.
The first time the public key will appear in DNS will be July 11, 2017, when it will appear alongside the current key.
It will finally replace the current key completely on October 11, 2017, by which time the DNS should be well aware of the new key, Conrad said.
There is some risk of things going wrong, which could affect domains that are DNSSEC-signed, which is another reason for the slowness of the rollover.
If ISPs that support DNSSEC do not start supporting the new KSK before the final switch-over, they’ll fail to correctly resolve DNSSEC-signed domains, which could lead to some sites going dark for some users.
There’s also a risk that the increased DNS packet sizes during the period when both KSKs are in use could cause queries to be dropped by firewalls, Conrad said.
“Folks who have things configured the right way won’t actually need to do anything but because DNSSEC is relatively new and this software hasn’t really been tested, we need to get the word out to everyone that this change is going to be occurring,” said Conrad.
ICANN will conduct outreach over the coming 15 months via the media, social media and technology conferences, he said.
It is estimated that about 20% of the internet’s DNS resolvers support DNSSEC, but most of those belong to just two companies — Google and Comcast — he said.
The number of signed domains is tiny as a percentage of the 326 million domains in existence today, but still amounts to millions of names.
Verisign this morning confirmed yesterday’s reports that the .gov top-level domain went down for some internet users due to a DNSSEC problem, which it said was related to an algorithm change.
In a posting to various mailing lists, Verisign principal engineer Duane Wessels said:
On the morning of August 14, a relatively small number of networks may have experienced an operational disruption related to the signing of the .gov zone. In preparation for a previously announced algorithm rollover, a software defect resulted in publishing the .gov zone signed only with DNSSEC algorithm 8 keys rather than with both algorithm 7 and 8. As a result .gov name resolution may have failed for validating recursive name servers. Upon discovery of the issue, Verisign took prompt action to restore the valid zone.
Verisign plans to proceed with the previously announced .gov algorithm rollover at the end of the month with the zone being signed with both algorithms for a period of approximately 10 days.
This clarifies that the problem was slightly different to what had been assumed yesterday.
It was related to change of the cryptographic algorithm used to create .gov’s DNSSEC keys, a relatively rare event, rather than a scheduled key rollover, which is a rather more frequent occurrence.
The problem would only have made .gov domains (and consequently web sites, email, etc) inaccessible for users of networks where DNSSEC validation is strictly enforced, which is quite small.
The US ISP with the strongest support for DNSSEC is Comcast. Since turning on its validators it has reported dozens of instances of DNSSEC failing — mostly in second-level .gov domains, where DNSSEC is mandated by US policy.
On two other occasions Comcast has blogged about the whole .gov TLD failing DNSSEC validation due to problems keeping keys up to date.
The general problem is widespread enough, and the impact severe enough, that Comcast has had to create an entirely new technology to prevent borked key rollovers making web sites go dark for its customers.
Called Negative Trust Anchors, it’s basically a Band-Aid that allows the ISP to deliberately ignore DNSSEC on a given domain while it waits for that domain’s owner to sort out its key problem.
The technology was created following the widely reported nasa.gov outage last year.
It’s really little wonder that so few organizations are interested in deploying DNSSEC today.
Yesterday’s .gov problem may have been minor, lasting only an hour or two, but had the affected TLD been .com, and had DNSSEC deployment been more widespread, everyone on the planet would have noticed.
Under ICANN contract, DNSSEC is mandatory for new gTLDs at the top level, but not the second level.
The .gov top-level domain suffered a DNSSEC problem today and was unavailable to some internet users, according to reports.
According to mailing lists and the SANS Internet Storm Center, it appeared that .gov rolled one of its DNSSEC keys without telling the root zone about the update.
This meant that anyone whose DNS servers do strict DNSSEC validation — a relatively small number of networks — would have been unable to access .gov web sites, email and other resources.
As a matter of policy, all second-level .gov domains have to be DNSSEC-signed.
The problem was corrected quite quickly — looks like within an hour or two — but as SANS noted, caching issues may prolong the impact.
Both .gov and the root zone are managed by Verisign, which isn’t on the best of terms with the US government at the moment.
Google has started fully supporting DNSSEC, the domain name security standard, on its Public DNS service.
According to a blog post from the company, while the free-to-use DNS resolution service has always passed on DNSSEC requests, now its resolvers will also validate DNSSEC signatures.
What does this mean?
Well, users of Public DNS will get protected from DNS cache poisoning attacks, but only for the small number of domains (such as domainincite.com) that are DNSSEC-signed.
It also means that if a company borks its DNSSEC implementation or key rollover, it’s likely to cause problems for Public DNS users. Comcast, an even earlier adopter, sees such problems pretty regularly.
But the big-picture story is that a whole bunch of new validating resolvers have been added to the internet, providing a boost to DNSSEC’s protracted global roll-out.
Currently Google Public DNS is serving more than 130 billion DNS queries on average (peaking at 150 billion) from more than 70 million unique IP addresses each day. However, only 7% of queries from the client side are DNSSEC-enabled (about 3% requesting validation and 4% requesting DNSSEC data but no validation) and about 1% of DNS responses from the name server side are signed. Overall, DNSSEC is still at an early stage and we hope that our support will help expedite its deployment.
One has to wonder whether Google’s participation in the ICANN new gTLD program — with its mandatory DNSSEC at the registry level — encouraged the company to adopt the technology.