Latest news of the domain name industry

Recent Posts

Second delay for domain security key rollover

Kevin Murphy, December 18, 2017, Domain Tech

ICANN has decided to delay changing the security keys to the DNS for the second time.

The “KSK Rollover” had been rescheduled from October 11 to some time in the first quarter 2018, but that will no longer happen. We’re now looking at Q3 at the earliest.

“We have decided that we do not yet have enough information to set a specific date for the rollover,” VP of research Matt Larson said in a blog post. “We want to make clear, however, that the ICANN org is committed to rolling the root zone KSK”.

The root KSK, or Key Signing Key, is the cryptographic key pair at the very top of the security hierarchy specified by DNSSEC, the security extension for DNS.

The current, first-ever, root KSK has been in operation since 2010, but ICANN’s policy is to roll it every five years or so.

The October date was delayed after newly available data showed that hundreds of DNS resolvers were still only configured to use the 2010 keys and not the 2017 keys that have already been deployed in tandem.

This would mean a rollover would cut off access to DNSSEC-signed zones to potentially millions of internet users.

ICANN found that 4% of the 12,000 DNSSEC-validating resolvers — roughly 500 IP addresses — it surveyed in September were not ready for KSK-2017.

Larson told us last month that at least 176 organizations in 41 countries were affected.

Since the first delay, ICANN has been trying to contact the owners of the 500 incompatible IP addresses but has run into some serious problems, Larson blogged.

First, a significant number of these addresses are dynamically allocated (such as to home broadband hubs) meaning tracking down the owners of the misconfigured devices would be next to impossible. Others were forwarding DNS queries on behalf of other devices, creating a similar problem.

Additionally, it seems ICANN has still not received responses from owners of 80% of the affected IP addresses.

Due to the lack of reliable data, it’s difficult for ICANN to figure out how many users’ internet access will be affected by a rollover.

The threshold called for by current policy is about 20 million people.

So ICANN has delayed the event to some point after Q1. Larson wrote that the organization will publish a plan on January 18 which will be open for public comment and discussed at the ICANN 61 meeting in Puerto Rico next March.

A final plan is not expected until ICANN 62, which happens in late June, so Q3 would be the earliest the rollover could actually occur.

Larson encouraged anyone interested in discussing the plan to join this mailing list.

Up to 20 million people could get broken internet in domain security rollover

Kevin Murphy, November 9, 2017, Domain Tech

Twenty million people losing access to parts of the internet is considered an acceptable level of collateral damage for ICANN’s forthcoming DNS root security update.

That’s one of a number of facts and figures to emerge from recent updates from the organization, explaining its decision to delay the so-called “KSK rollover” from October 11 to some time in the first quarter next year.

The rollover will see a new Key Signing Key, used as the trust anchor for all DNSSEC-signed domains, replace the seven-year-old original.

DNSSEC protects internet users and registrants from domain-based man-in-the-middle attacks. It’s considered good practice to roll keys at each level of the DNS hierarchy periodically, to reduce the risk of successful brute-force attacks.

The root KSK update will affect hundreds of millions of people who currently use DNSSEC-compatible resolvers, such as Google DNS.

ICANN delayed the rollover after it, rather fortuitously, spotted that not all of these resolvers are configured to correctly handle the change.

The number of known incompatible servers is quite small — only about 500 of the 11,982 DNSSEC-using recursive servers initially surveyed (pdf). That represents only a very small minority of the world’s internet users, as most are not currently using DNSSEC.

Subsequent ICANN research, presented by principal researcher Roy Arends at ICANN 60 last week, showed that:

  • There are currently about 4.2 million DNS resolvers in the world.
  • Of those, 27,084 are configured to tell the root servers which KSKs they support (currently either the KSK-2011 or KSK-2017).
  • Of those, 1,631 or 6.02% do not support KSK-2017

It was only possible to survey servers that have turned on a recent update to DNS software such as BIND and Unbound, so the true number of misconfigured servers could be much higher.

Matt Larson, ICANN’s VP of research, told DI that ICANN has identified 176 organizations in 41 countries that are currently not prepared to handle the new KSK. These organizations are fairly evenly spread geographically, he said.

Since making the decision to delay the rollover, ICANN has hired a contractor to reach out to these network operators to alert them to potential problems.

ICANN’s CEO Goran Marby has also been writing to telecommunications regulators in all countries to ask for assistance.

After the rollover, people using an incompatible resolver would be unable to access DNSSEC-signed domains. Again, that’s still quite a small minority of domains — there are only about 750,000 in .com by some accounts and apparently none of the top 25 site support it.

ICANN could roll back the change if it detects that a sufficiently large number of people are negatively affected, but that number turns out to be around 20 million.

According to its published rollover plan:

Rollback of any step in the key roll process should be initiated if the measurement program indicated that a minimum of 0.5% of the estimated Internet end-user population has been negatively impacted by the change 72 hours after each change has been deployed into the root zone.

According to InternetWorldStats, there were around 3,885,567,619 internet users in the world this June. It’s very likely more people now.

So a 0.5% threshold works out to about 19 million to 20 million people worldwide.

Larson agreed that in absolute terms, it’s a big number.

“The overall message to take away from that number, I suggest, is that a problem would have to be pretty serious for us to consider rolling back,” Larson, who was not on the team that came up with the threshold, said.

“I think that’s a reasonable position considering that, in the immediate aftermath of the rollover, there are two near-immediate fixes available to any operator experiencing problems: update their systems’ trust anchors with the new key or (less desirable from my perspective but still effective) simply disable DNSSEC validation,” he said.

He added that the 0.5% level is not a hard and fast rule, and that ICANN could be flexible in the moment.

“For example, if when we roll the key, we find out there’s some critical system with a literal life or death impact that is negatively affected by the KSK roll, I think I can pretty confidently state that we wouldn’t require the 0.5% of Internet user threshold to be met before rolling back if it looked like there would be a significant health and safety risk not easily mitigated,” he said.

The chances of such an impact are very slim, but not impossible, he suggested.

It’s not ICANN’s intention to put anyone’s internet access at risk, of course, which is why there’s a delay.

ICANN’s plan calls for any rollover to happen on the eleventh day of a given calendar quarter, so the soonest it could happen would be January 11.

Given the complexity of the outreach task in hand, the relative lack of data, and the holiday periods approaching in many countries, and ICANN’s generally cautious nature, I’d hazard a guess we might be looking at April 11 at the earliest instead.

ICANN just came thiiis close to breaking the internet

Kevin Murphy, September 28, 2017, Domain Tech

ICANN has decided to postpone an unprecedented change at the DNS root after discovering it could break internet for potentially millions of users.

The so-called KSK Rollover was due to go ahead on October 11, but it’s now been pushed back to — tentatively — some time in the first quarter 2018.

The delay was decided after ICANN realized that there were still plenty of ISPs and network operators that weren’t ready for the change.

Had ICANN gone ahead anyway with the change anyway, it could have seen subscribers of affected ISPs lose access to millions of DNSSEC-supporting domain names.

So the postponement is a good thing.

A KSK or Key Signing Key is a public-private cryptographic key pair used to sign other keys called Zone Signing Keys. The root KSK signs the root ZSK and is in effect the apex of the DNSSEC hierarchy.

The same KSK has been in operation at the root since 2010, when the root was first signed, but it’s considered good practice to change it every so often to mitigate the risk of brute-force attacks against the public key.

While it’s important enough to get dramatized in US spy shows, in practice it only affects ISPs and domain names that voluntarily support DNSSEC.

ICANN estimates that 750 million people use DNSSEC, which is designed to prevent problems such as man-in-the-middle attacks against domain names.

That’s a hell of a lot of people, but it’s still a minority of the world’s internet-using population. It’s not been revealed how many of those would have been affected by a premature rollover.

When DNSSEC fails, people whose DNS resolvers have DNSSEC turned on (Comcast and Google are two of the largest such providers) can’t access domain names that have DNSSEC turned on (such as domainincite.com).

Preventing the internet breaking is pretty much ICANN’s only job, so it first flagged up its intention to roll the root KSK back in July last year.

In July this year, the new public KSK was uploaded as part of a transition phase that is seeing the 2010 keys and 2017 keys online simultaneously.

Last year, CTO David Conrad told us the long lead time and cautious approach was necessary to get the word out that ISPs needed to test their resolvers to make sure they would work with the new keys.

In June, ICANN CEO Goran Marby spammed the telecommunications regulators in every country in the world with a letter (pdf) asking them to coordinate their home ISPs to be ready for the change.

The organization’s comms teams has also been doing a pretty good job getting word of the rollover into the tech press over the last few months.

But, with a flashback to the new gTLD program, that outreach doesn’t seem to have reached out as far as it needed to.

ICANN said last night that a “significant number” of ISPs are still not ready for the rollover.

It seems ICANN only became aware of this problem due to a new feature of DNS that reports back to the root which keys it is configured to use.

Without being able to collate that data, it’s possible it could have been assumed that the situation was hunky-dory and the rollover might have gone ahead.

ICANN still isn’t sure why so many resolvers are not yet ready for the 2017 KSK. It said in a statement:

There may be multiple reasons why operators do not have the new key installed in their systems: some may not have their resolver software properly configured and a recently discovered issue in one widely used resolver program appears to not be automatically updating the key as it should, for reasons that are still being explored.

It’s not clear why the broken resolver software has not been named — one would assume that getting the word out would be a priority unless issues of responsible disclosure were in play.

ICANN said it is “reaching out to its community, including its Security and Stability Advisory Committee, the Regional Internet Registries, Network Operator Groups and others to help explore and resolve the issues.”

The organization is hopeful that it will be able to go ahead with the rollover in Q1 2018, but noted that would be dependent on “more fully understanding the new information and mitigating as many potential failures as possible.”

While it’s excellent news that ICANN is on top of the situation, the delay is unlikely to do anything to help the perception that DNSSEC is mainly just an administrative ball-ache and far more trouble than it’s worth.

Want to be one of the internet’s SEVEN SECRET KEY-HOLDERS? Apply now!

Kevin Murphy, May 22, 2017, Domain Tech

ICANN has put out a call for volunteers, looking for people to become what are sometimes referred to as “the internet’s seven secret key holders”.

Specifically, it needs Trusted Community Representatives, people of standing in the internet community who don’t mind carrying around a small key and getting a free trip to Los Angeles or Virginia once or twice a year.

The TCRs are used in the paranoia-inducing cryptographic key-signing ceremonies that provide DNSSEC at the root of the domain name system.

The ceremonies take place at ICANN data centers four times a year. The ceremonies themselves take hours, involve multiple layers of physical and data security, and the volunteers are expected to hang around for a day or two before and after each.

There’s no compensation involved, but the TCRs are allowed to apply to ICANN for travel reimbursements.

ICANN expects TCRs to stick around for about five years, but the large majority of the 28 people who act as TCRs (yeah, it’s not seven, it’s 28) have been in the role since 2010 and ICANN is probably planning a cull.

Other than knowing what the DNS is and how it works, the primary requirements are “integrity, objectivity, and intelligence, with reputations for sound judgment and open minds”.

If you think you tick those boxes, head here to apply.

IANA boss quits ICANN

Kevin Murphy, April 19, 2017, Domain Policy

The head of IANA is to leave the organization, ICANN announced this week.

Elise Gerich, currently vice president of IANA Services at ICANN and president of Public Technical Identifiers (PTI), will leave in October, according to a blog post.

She’ll stick around long enough to oversee the DNS root’s first DNSSEC Key-Signing Key rollover, which is due to go ahead October 11.

Gerich has been VP of IANA since May 2010, and took on the job of PTI president last October when the IANA function was restructured to remove the US government from the mix.

ICANN said it will start the hunt for her replacement shortly.

Hacker hostage crisis at ICANN secret key ceremony! (on TV)

Kevin Murphy, March 24, 2017, Gossip

One of ICANN’s Seven Secret Key-Holders To The Internet got taken out as part of an elaborate heist or something on American TV this week.

In tense scenes, a couple of secret agents or something with guns were forced to break into one of ICANN’s quarterly root zone key signing ceremonies to prevent a hacker or terrorist or something from something something, something something.

The stand-off came after the secret agents or whatever discovered that a hacker called Mayhew had poisoned a guy named Adler, causing a heart attack, in order to secure his position as a replacement ICANN key-holder and hijack the ceremony.

This all happened on a TV show called Blacklist: Redemption that aired in the US March 16.

I’d be lying if I said I fully understood what was supposed to be going on in the episode, not being a regular viewer of the series, but here’s the exposition from the beginning of the second act.

Black List

Botox Boss Lady: Seven keys control the internet? That can’t be possible.

Neck Beard Exposition Guy: They don’t control what’s on it, just how to secure it. All domain names have an assigned number. But who assigns the numbers?

Soap Opera Secret Agent: Key holders?

Neck Beard Exposition Guy: Seven security experts randomly selected by ICANN, the Internet Corporation for Assigned Names and Numbers.

Bored Secret Agent: Max Adler’s wife mentioned a key ceremony.

Neck Beard Exposition Guy: Yeah, four times a year the key holders meet to generate a master key and to assign new numbers, to make life difficult for hackers who want to direct folks to malicious sites or steal their credit card information.

Botox Boss Lady: But by being at the ceremony, Mayhew gets around those precautions?

Neck Beard Exposition Guy: Oh, he does more than that. He can route any domain name to him.

That’s the genuine dialogue. ICANN, jarringly, isn’t fictionalized in the way one might usually expect from US TV drama.

The scene carries on to explain the elaborate security precautions ICANN has put in place around its key-signing ceremonies, including biometrics, smart cards and the like.

The fast-moving show then cuts to the aforementioned heist situation, in which our villain of the week takes an ICANN staffer hostage before using the root’s DNSSEC keys to somehow compromise a government data drop and download a McGuffin.

Earlier this week I begged Matt Larson, ICANN’s VP of research and a regular participant in the ceremonies (which are real) to watch the show and explain to me what bits reflect reality and what was plainly bogus.

“There are some points about it that are quite close to how the how the root KSK administration works,” he said, describing the depiction as “kind of surreal”.

“But then they take it not one but two steps further. The way the ceremony happens is not accurate, the consequences of what happens at the ceremony are not accurate,” he added.

“They talk about how at the ceremony we generate a key, well that’s not true. It’s used for signing a new key. And then they talk about how as a result of the ceremony anyone can intercept any domain name anywhere and of course that’s not true.”

The ceremonies are used to sign the keys that make end-to-end DNSSEC possible. By signing the root, DNSSEC resolvers have a “chain of trust” that goes all the way to the top of the DNS hierarchy.

Black ListThe root keys just secure the bit between the root at the TLDs. Compromising them would not enable a hacker to immediately start downloading data from the site of his choosing, as depicted in the show. He’d then have to go on to compromise the rest of the chain.

“You’d have to create an entire path of spoofed zones to who you wanted to impersonate,” Larson said. “Your fake root zone would have to delegate to a fake TLD zone to a fake SLD zone and so on so you could finally convince someone they were going to the address that you wanted.”

“If you could somehow compromise the processes at the root, that alone doesn’t give you anything,” he said.

But the show did present a somewhat realistic description of how the ceremony rooms (located in Virginia and California, not Manhattan as seen on TV) are secured.

Among other precautions, the facilities are secured with smart cards and PINs, retina scans for ICANN staff, and have reinforced walls to prevent somebody coming in with a sledgehammer, Larson said.

Blacklist: Redemption airs on Thursday nights on NBC in the US, but I wouldn’t bother if I were you.

ICANN to flip the secret key to the internet

Kevin Murphy, July 20, 2016, Domain Tech

ICANN is about to embark on a year-long effort to warn the internet that it plans to replace the top-level cryptographic keys used in DNSSEC for the first time.

CTO David Conrad told DI today that ICANN will rotate the so-called Key Signing Key that is used as the “trust anchor” for all DNSSEC queries that happen on the internet.

Due to the complexity of the process, and the risk that something might go wrong, the move is to be announced in the coming days even though the new public key will not replace the existing one until October 2017.

The KSK is a cryptographic key pair used to sign the Zone Signing Keys that in turn sign the DNS root zone. It’s basically at the top of the DNSSEC hierarchy — all trust in DNSSEC flows from it.

It’s considered good practice in DNSSEC to rotate keys every so often, largely to reduce the window would-be attackers have to compromise them.

The Zone Signing Key used by ICANN and Verisign to sign the DNS root is rotated quarterly, and individual domain owners can rotate their own keys as and when they choose, but the same KSK has been in place since the root was first signed in 2010.

Conrad said that ICANN is doing the first rollover partly to ensure that the procedures in has in place for changing keys are effective and could be deployed in case of emergency.

That said, this first rotation is going to happen at a snail’s pace.

Key generation is a complex matter, requiring the physical presence of at least three of seven trusted key holders.

These seven individuals possess physical keys to bank-style strong boxes which contain secure smart cards. Three of the seven cards are needed to generate a new key.

Each of the quarterly ZSK signing ceremonies — which are recorded and broadcast live over the internet — takes about five hours.

The first step in the rollover, Conrad said, is to generate the keys at ICANN’s US east coast facility in October this year. A copy will be moved to a facility on the west coast in February.

The first time the public key will appear in DNS will be July 11, 2017, when it will appear alongside the current key.

It will finally replace the current key completely on October 11, 2017, by which time the DNS should be well aware of the new key, Conrad said.

There is some risk of things going wrong, which could affect domains that are DNSSEC-signed, which is another reason for the slowness of the rollover.

If ISPs that support DNSSEC do not start supporting the new KSK before the final switch-over, they’ll fail to correctly resolve DNSSEC-signed domains, which could lead to some sites going dark for some users.

There’s also a risk that the increased DNS packet sizes during the period when both KSKs are in use could cause queries to be dropped by firewalls, Conrad said.

“Folks who have things configured the right way won’t actually need to do anything but because DNSSEC is relatively new and this software hasn’t really been tested, we need to get the word out to everyone that this change is going to be occurring,” said Conrad.

ICANN will conduct outreach over the coming 15 months via the media, social media and technology conferences, he said.

It is estimated that about 20% of the internet’s DNS resolvers support DNSSEC, but most of those belong to just two companies — Google and Comcast — he said.

The number of signed domains is tiny as a percentage of the 326 million domains in existence today, but still amounts to millions of names.

Verisign confirms .gov downtime, blames algorithm

Kevin Murphy, August 15, 2013, Domain Tech

Verisign this morning confirmed yesterday’s reports that the .gov top-level domain went down for some internet users due to a DNSSEC problem, which it said was related to an algorithm change.

In a posting to various mailing lists, Verisign principal engineer Duane Wessels said:

On the morning of August 14, a relatively small number of networks may have experienced an operational disruption related to the signing of the .gov zone. In preparation for a previously announced algorithm rollover, a software defect resulted in publishing the .gov zone signed only with DNSSEC algorithm 8 keys rather than with both algorithm 7 and 8. As a result .gov name resolution may have failed for validating recursive name servers. Upon discovery of the issue, Verisign took prompt action to restore the valid zone.

Verisign plans to proceed with the previously announced .gov algorithm rollover at the end of the month with the zone being signed with both algorithms for a period of approximately 10 days.

This clarifies that the problem was slightly different to what had been assumed yesterday.

It was related to change of the cryptographic algorithm used to create .gov’s DNSSEC keys, a relatively rare event, rather than a scheduled key rollover, which is a rather more frequent occurrence.

The problem would only have made .gov domains (and consequently web sites, email, etc) inaccessible for users of networks where DNSSEC validation is strictly enforced, which is quite small.

The US ISP with the strongest support for DNSSEC is Comcast. Since turning on its validators it has reported dozens of instances of DNSSEC failing — mostly in second-level .gov domains, where DNSSEC is mandated by US policy.

On two other occasions Comcast has blogged about the whole .gov TLD failing DNSSEC validation due to problems keeping keys up to date.

The general problem is widespread enough, and the impact severe enough, that Comcast has had to create an entirely new technology to prevent borked key rollovers making web sites go dark for its customers.

Called Negative Trust Anchors, it’s basically a Band-Aid that allows the ISP to deliberately ignore DNSSEC on a given domain while it waits for that domain’s owner to sort out its key problem.

The technology was created following the widely reported nasa.gov outage last year.

It’s really little wonder that so few organizations are interested in deploying DNSSEC today.

Yesterday’s .gov problem may have been minor, lasting only an hour or two, but had the affected TLD been .com, and had DNSSEC deployment been more widespread, everyone on the planet would have noticed.

Under ICANN contract, DNSSEC is mandatory for new gTLDs at the top level, but not the second level.

Reports: .gov fails due to DNSSEC error

Kevin Murphy, August 14, 2013, Domain Tech

The .gov top-level domain suffered a DNSSEC problem today and was unavailable to some internet users, according to reports.

According to mailing lists and the SANS Internet Storm Center, it appeared that .gov rolled one of its DNSSEC keys without telling the root zone about the update.

This meant that anyone whose DNS servers do strict DNSSEC validation — a relatively small number of networks — would have been unable to access .gov web sites, email and other resources.

As a matter of policy, all second-level .gov domains have to be DNSSEC-signed.

The problem was corrected quite quickly — looks like within an hour or two — but as SANS noted, caching issues may prolong the impact.

Both .gov and the root zone are managed by Verisign, which isn’t on the best of terms with the US government at the moment.

Google starts supporting DNSSEC

Kevin Murphy, March 21, 2013, Domain Tech

Google has started fully supporting DNSSEC, the domain name security standard, on its Public DNS service.

According to a blog post from the company, while the free-to-use DNS resolution service has always passed on DNSSEC requests, now its resolvers will also validate DNSSEC signatures.

What does this mean?

Well, users of Public DNS will get protected from DNS cache poisoning attacks, but only for the small number of domains (such as domainincite.com) that are DNSSEC-signed.

It also means that if a company borks its DNSSEC implementation or key rollover, it’s likely to cause problems for Public DNS users. Comcast, an even earlier adopter, sees such problems pretty regularly.

But the big-picture story is that a whole bunch of new validating resolvers have been added to the internet, providing a boost to DNSSEC’s protracted global roll-out.

Google said:

Currently Google Public DNS is serving more than 130 billion DNS queries on average (peaking at 150 billion) from more than 70 million unique IP addresses each day. However, only 7% of queries from the client side are DNSSEC-enabled (about 3% requesting validation and 4% requesting DNSSEC data but no validation) and about 1% of DNS responses from the name server side are signed. Overall, DNSSEC is still at an early stage and we hope that our support will help expedite its deployment.

One has to wonder whether Google’s participation in the ICANN new gTLD program — with its mandatory DNSSEC at the registry level — encouraged the company to adopt the technology.