Latest news of the domain name industry

Recent Posts

Up to 20 million people could get broken internet in domain security rollover

Kevin Murphy, November 9, 2017, Domain Tech

Twenty million people losing access to parts of the internet is considered an acceptable level of collateral damage for ICANN’s forthcoming DNS root security update.

That’s one of a number of facts and figures to emerge from recent updates from the organization, explaining its decision to delay the so-called “KSK rollover” from October 11 to some time in the first quarter next year.

The rollover will see a new Key Signing Key, used as the trust anchor for all DNSSEC-signed domains, replace the seven-year-old original.

DNSSEC protects internet users and registrants from domain-based man-in-the-middle attacks. It’s considered good practice to roll keys at each level of the DNS hierarchy periodically, to reduce the risk of successful brute-force attacks.

The root KSK update will affect hundreds of millions of people who currently use DNSSEC-compatible resolvers, such as Google DNS.

ICANN delayed the rollover after it, rather fortuitously, spotted that not all of these resolvers are configured to correctly handle the change.

The number of known incompatible servers is quite small — only about 500 of the 11,982 DNSSEC-using recursive servers initially surveyed (pdf). That represents only a very small minority of the world’s internet users, as most are not currently using DNSSEC.

Subsequent ICANN research, presented by principal researcher Roy Arends at ICANN 60 last week, showed that:

  • There are currently about 4.2 million DNS resolvers in the world.
  • Of those, 27,084 are configured to tell the root servers which KSKs they support (currently either the KSK-2011 or KSK-2017).
  • Of those, 1,631 or 6.02% do not support KSK-2017

It was only possible to survey servers that have turned on a recent update to DNS software such as BIND and Unbound, so the true number of misconfigured servers could be much higher.

Matt Larson, ICANN’s VP of research, told DI that ICANN has identified 176 organizations in 41 countries that are currently not prepared to handle the new KSK. These organizations are fairly evenly spread geographically, he said.

Since making the decision to delay the rollover, ICANN has hired a contractor to reach out to these network operators to alert them to potential problems.

ICANN’s CEO Goran Marby has also been writing to telecommunications regulators in all countries to ask for assistance.

After the rollover, people using an incompatible resolver would be unable to access DNSSEC-signed domains. Again, that’s still quite a small minority of domains — there are only about 750,000 in .com by some accounts and apparently none of the top 25 site support it.

ICANN could roll back the change if it detects that a sufficiently large number of people are negatively affected, but that number turns out to be around 20 million.

According to its published rollover plan:

Rollback of any step in the key roll process should be initiated if the measurement program indicated that a minimum of 0.5% of the estimated Internet end-user population has been negatively impacted by the change 72 hours after each change has been deployed into the root zone.

According to InternetWorldStats, there were around 3,885,567,619 internet users in the world this June. It’s very likely more people now.

So a 0.5% threshold works out to about 19 million to 20 million people worldwide.

Larson agreed that in absolute terms, it’s a big number.

“The overall message to take away from that number, I suggest, is that a problem would have to be pretty serious for us to consider rolling back,” Larson, who was not on the team that came up with the threshold, said.

“I think that’s a reasonable position considering that, in the immediate aftermath of the rollover, there are two near-immediate fixes available to any operator experiencing problems: update their systems’ trust anchors with the new key or (less desirable from my perspective but still effective) simply disable DNSSEC validation,” he said.

He added that the 0.5% level is not a hard and fast rule, and that ICANN could be flexible in the moment.

“For example, if when we roll the key, we find out there’s some critical system with a literal life or death impact that is negatively affected by the KSK roll, I think I can pretty confidently state that we wouldn’t require the 0.5% of Internet user threshold to be met before rolling back if it looked like there would be a significant health and safety risk not easily mitigated,” he said.

The chances of such an impact are very slim, but not impossible, he suggested.

It’s not ICANN’s intention to put anyone’s internet access at risk, of course, which is why there’s a delay.

ICANN’s plan calls for any rollover to happen on the eleventh day of a given calendar quarter, so the soonest it could happen would be January 11.

Given the complexity of the outreach task in hand, the relative lack of data, and the holiday periods approaching in many countries, and ICANN’s generally cautious nature, I’d hazard a guess we might be looking at April 11 at the earliest instead.

ICANN just came thiiis close to breaking the internet

Kevin Murphy, September 28, 2017, Domain Tech

ICANN has decided to postpone an unprecedented change at the DNS root after discovering it could break internet for potentially millions of users.

The so-called KSK Rollover was due to go ahead on October 11, but it’s now been pushed back to — tentatively — some time in the first quarter 2018.

The delay was decided after ICANN realized that there were still plenty of ISPs and network operators that weren’t ready for the change.

Had ICANN gone ahead anyway with the change anyway, it could have seen subscribers of affected ISPs lose access to millions of DNSSEC-supporting domain names.

So the postponement is a good thing.

A KSK or Key Signing Key is a public-private cryptographic key pair used to sign other keys called Zone Signing Keys. The root KSK signs the root ZSK and is in effect the apex of the DNSSEC hierarchy.

The same KSK has been in operation at the root since 2010, when the root was first signed, but it’s considered good practice to change it every so often to mitigate the risk of brute-force attacks against the public key.

While it’s important enough to get dramatized in US spy shows, in practice it only affects ISPs and domain names that voluntarily support DNSSEC.

ICANN estimates that 750 million people use DNSSEC, which is designed to prevent problems such as man-in-the-middle attacks against domain names.

That’s a hell of a lot of people, but it’s still a minority of the world’s internet-using population. It’s not been revealed how many of those would have been affected by a premature rollover.

When DNSSEC fails, people whose DNS resolvers have DNSSEC turned on (Comcast and Google are two of the largest such providers) can’t access domain names that have DNSSEC turned on (such as domainincite.com).

Preventing the internet breaking is pretty much ICANN’s only job, so it first flagged up its intention to roll the root KSK back in July last year.

In July this year, the new public KSK was uploaded as part of a transition phase that is seeing the 2010 keys and 2017 keys online simultaneously.

Last year, CTO David Conrad told us the long lead time and cautious approach was necessary to get the word out that ISPs needed to test their resolvers to make sure they would work with the new keys.

In June, ICANN CEO Goran Marby spammed the telecommunications regulators in every country in the world with a letter (pdf) asking them to coordinate their home ISPs to be ready for the change.

The organization’s comms teams has also been doing a pretty good job getting word of the rollover into the tech press over the last few months.

But, with a flashback to the new gTLD program, that outreach doesn’t seem to have reached out as far as it needed to.

ICANN said last night that a “significant number” of ISPs are still not ready for the rollover.

It seems ICANN only became aware of this problem due to a new feature of DNS that reports back to the root which keys it is configured to use.

Without being able to collate that data, it’s possible it could have been assumed that the situation was hunky-dory and the rollover might have gone ahead.

ICANN still isn’t sure why so many resolvers are not yet ready for the 2017 KSK. It said in a statement:

There may be multiple reasons why operators do not have the new key installed in their systems: some may not have their resolver software properly configured and a recently discovered issue in one widely used resolver program appears to not be automatically updating the key as it should, for reasons that are still being explored.

It’s not clear why the broken resolver software has not been named — one would assume that getting the word out would be a priority unless issues of responsible disclosure were in play.

ICANN said it is “reaching out to its community, including its Security and Stability Advisory Committee, the Regional Internet Registries, Network Operator Groups and others to help explore and resolve the issues.”

The organization is hopeful that it will be able to go ahead with the rollover in Q1 2018, but noted that would be dependent on “more fully understanding the new information and mitigating as many potential failures as possible.”

While it’s excellent news that ICANN is on top of the situation, the delay is unlikely to do anything to help the perception that DNSSEC is mainly just an administrative ball-ache and far more trouble than it’s worth.

Want to be one of the internet’s SEVEN SECRET KEY-HOLDERS? Apply now!

Kevin Murphy, May 22, 2017, Domain Tech

ICANN has put out a call for volunteers, looking for people to become what are sometimes referred to as “the internet’s seven secret key holders”.

Specifically, it needs Trusted Community Representatives, people of standing in the internet community who don’t mind carrying around a small key and getting a free trip to Los Angeles or Virginia once or twice a year.

The TCRs are used in the paranoia-inducing cryptographic key-signing ceremonies that provide DNSSEC at the root of the domain name system.

The ceremonies take place at ICANN data centers four times a year. The ceremonies themselves take hours, involve multiple layers of physical and data security, and the volunteers are expected to hang around for a day or two before and after each.

There’s no compensation involved, but the TCRs are allowed to apply to ICANN for travel reimbursements.

ICANN expects TCRs to stick around for about five years, but the large majority of the 28 people who act as TCRs (yeah, it’s not seven, it’s 28) have been in the role since 2010 and ICANN is probably planning a cull.

Other than knowing what the DNS is and how it works, the primary requirements are “integrity, objectivity, and intelligence, with reputations for sound judgment and open minds”.

If you think you tick those boxes, head here to apply.

Hacker hostage crisis at ICANN secret key ceremony! (on TV)

Kevin Murphy, March 24, 2017, Gossip

One of ICANN’s Seven Secret Key-Holders To The Internet got taken out as part of an elaborate heist or something on American TV this week.

In tense scenes, a couple of secret agents or something with guns were forced to break into one of ICANN’s quarterly root zone key signing ceremonies to prevent a hacker or terrorist or something from something something, something something.

The stand-off came after the secret agents or whatever discovered that a hacker called Mayhew had poisoned a guy named Adler, causing a heart attack, in order to secure his position as a replacement ICANN key-holder and hijack the ceremony.

This all happened on a TV show called Blacklist: Redemption that aired in the US March 16.

I’d be lying if I said I fully understood what was supposed to be going on in the episode, not being a regular viewer of the series, but here’s the exposition from the beginning of the second act.

Black List

Botox Boss Lady: Seven keys control the internet? That can’t be possible.

Neck Beard Exposition Guy: They don’t control what’s on it, just how to secure it. All domain names have an assigned number. But who assigns the numbers?

Soap Opera Secret Agent: Key holders?

Neck Beard Exposition Guy: Seven security experts randomly selected by ICANN, the Internet Corporation for Assigned Names and Numbers.

Bored Secret Agent: Max Adler’s wife mentioned a key ceremony.

Neck Beard Exposition Guy: Yeah, four times a year the key holders meet to generate a master key and to assign new numbers, to make life difficult for hackers who want to direct folks to malicious sites or steal their credit card information.

Botox Boss Lady: But by being at the ceremony, Mayhew gets around those precautions?

Neck Beard Exposition Guy: Oh, he does more than that. He can route any domain name to him.

That’s the genuine dialogue. ICANN, jarringly, isn’t fictionalized in the way one might usually expect from US TV drama.

The scene carries on to explain the elaborate security precautions ICANN has put in place around its key-signing ceremonies, including biometrics, smart cards and the like.

The fast-moving show then cuts to the aforementioned heist situation, in which our villain of the week takes an ICANN staffer hostage before using the root’s DNSSEC keys to somehow compromise a government data drop and download a McGuffin.

Earlier this week I begged Matt Larson, ICANN’s VP of research and a regular participant in the ceremonies (which are real) to watch the show and explain to me what bits reflect reality and what was plainly bogus.

“There are some points about it that are quite close to how the how the root KSK administration works,” he said, describing the depiction as “kind of surreal”.

“But then they take it not one but two steps further. The way the ceremony happens is not accurate, the consequences of what happens at the ceremony are not accurate,” he added.

“They talk about how at the ceremony we generate a key, well that’s not true. It’s used for signing a new key. And then they talk about how as a result of the ceremony anyone can intercept any domain name anywhere and of course that’s not true.”

The ceremonies are used to sign the keys that make end-to-end DNSSEC possible. By signing the root, DNSSEC resolvers have a “chain of trust” that goes all the way to the top of the DNS hierarchy.

Black ListThe root keys just secure the bit between the root at the TLDs. Compromising them would not enable a hacker to immediately start downloading data from the site of his choosing, as depicted in the show. He’d then have to go on to compromise the rest of the chain.

“You’d have to create an entire path of spoofed zones to who you wanted to impersonate,” Larson said. “Your fake root zone would have to delegate to a fake TLD zone to a fake SLD zone and so on so you could finally convince someone they were going to the address that you wanted.”

“If you could somehow compromise the processes at the root, that alone doesn’t give you anything,” he said.

But the show did present a somewhat realistic description of how the ceremony rooms (located in Virginia and California, not Manhattan as seen on TV) are secured.

Among other precautions, the facilities are secured with smart cards and PINs, retina scans for ICANN staff, and have reinforced walls to prevent somebody coming in with a sledgehammer, Larson said.

Blacklist: Redemption airs on Thursday nights on NBC in the US, but I wouldn’t bother if I were you.

ICANN to flip the secret key to the internet

Kevin Murphy, July 20, 2016, Domain Tech

ICANN is about to embark on a year-long effort to warn the internet that it plans to replace the top-level cryptographic keys used in DNSSEC for the first time.

CTO David Conrad told DI today that ICANN will rotate the so-called Key Signing Key that is used as the “trust anchor” for all DNSSEC queries that happen on the internet.

Due to the complexity of the process, and the risk that something might go wrong, the move is to be announced in the coming days even though the new public key will not replace the existing one until October 2017.

The KSK is a cryptographic key pair used to sign the Zone Signing Keys that in turn sign the DNS root zone. It’s basically at the top of the DNSSEC hierarchy — all trust in DNSSEC flows from it.

It’s considered good practice in DNSSEC to rotate keys every so often, largely to reduce the window would-be attackers have to compromise them.

The Zone Signing Key used by ICANN and Verisign to sign the DNS root is rotated quarterly, and individual domain owners can rotate their own keys as and when they choose, but the same KSK has been in place since the root was first signed in 2010.

Conrad said that ICANN is doing the first rollover partly to ensure that the procedures in has in place for changing keys are effective and could be deployed in case of emergency.

That said, this first rotation is going to happen at a snail’s pace.

Key generation is a complex matter, requiring the physical presence of at least three of seven trusted key holders.

These seven individuals possess physical keys to bank-style strong boxes which contain secure smart cards. Three of the seven cards are needed to generate a new key.

Each of the quarterly ZSK signing ceremonies — which are recorded and broadcast live over the internet — takes about five hours.

The first step in the rollover, Conrad said, is to generate the keys at ICANN’s US east coast facility in October this year. A copy will be moved to a facility on the west coast in February.

The first time the public key will appear in DNS will be July 11, 2017, when it will appear alongside the current key.

It will finally replace the current key completely on October 11, 2017, by which time the DNS should be well aware of the new key, Conrad said.

There is some risk of things going wrong, which could affect domains that are DNSSEC-signed, which is another reason for the slowness of the rollover.

If ISPs that support DNSSEC do not start supporting the new KSK before the final switch-over, they’ll fail to correctly resolve DNSSEC-signed domains, which could lead to some sites going dark for some users.

There’s also a risk that the increased DNS packet sizes during the period when both KSKs are in use could cause queries to be dropped by firewalls, Conrad said.

“Folks who have things configured the right way won’t actually need to do anything but because DNSSEC is relatively new and this software hasn’t really been tested, we need to get the word out to everyone that this change is going to be occurring,” said Conrad.

ICANN will conduct outreach over the coming 15 months via the media, social media and technology conferences, he said.

It is estimated that about 20% of the internet’s DNS resolvers support DNSSEC, but most of those belong to just two companies — Google and Comcast — he said.

The number of signed domains is tiny as a percentage of the 326 million domains in existence today, but still amounts to millions of names.

  • Page 1 of 2
  • 1
  • 2
  • >