Set buttocks to clench! ICANN approves risky KSK rollover

ICANN has approved the first rollover of the domain name system’s master security key, setting the clock ticking on a change that could cause internet access issues for millions.
The so-called KSK rollover, when ICANN deletes the key-signing key that has been used as the trust anchor for the DNSSEC ecosystem since 2011 and replaces it with the new one — will now go ahead as planned on October 11.
The decision was made yesterday at the ICANN board of directors’ retreat in Brussels.
ICANN chief technology officer David Conrad posted this to an ICANN mailing list this morning:

The Board voted to approve the resolution for ICANN org to move forward with the revised KSK rollover plan. So barring unforeseen circumstances, the KSK-2017-signed ZSK will be used to sign the root zone on 11 October 2018.

The rollover was due to happen October 11 last year, but ICANN delayed it when it emerged that many DNS resolvers weren’t yet configured to use the new key.
That’s still a problem, and nobody knows for sure how many endpoints will stop functioning properly when the new KSK goes solo.
While most experts weighing in on the rollover, including Conrad, agreed that the risk of more delay outweighed the risk of rolling now, that feeling was not unanimous.
Five members of the 22-member Security and Stability Advisory Committee — including top guys from Google and Verisign — last month dissented from the majority view and said ICANN should delay again.
The question now is not whether internet users will see a disruption in the days following October 11, but how many users will be affected and how serious their disruptions will be.
Based on current information, as many as two million internet users could be affected.
ICANN is likely to take flak for even relatively minor disruptions, but the alternative was to continue with the delays and risk an even bigger impact, and even more flak, in future.
The text of ICANN’s resolution and the rationale behind it will be published in the next day or so.

Root crypto rollover now slated for October

ICANN has penciled in October 11 as the new date for rolling the DNS root’s cryptographic keys, a delay of a year from its original plan.
The so-called KSK rollover will see ICANN remove the deprecated 2010 Key Signing Key, leaving only the 2017 KSK active.
The KSK acts as the “trust anchor” for DNSSEC across the whole internet.
After the rollover, any network not configured to use the latest KSK would see a service interruption.
This could mean many millions of internet users being affected, but ICANN doesn’t know the extent of the possible impact for sure.
ICANN told us in November that it knows of 176 organizations in 41 countries, fairly evenly spread across the globe, that are currently not prepared to handle the new KSK.
But its data is patchy because only a tiny number of DNS resolvers are actually configured to automatically report which KSKs they’re set up to use.
Key rollovers are recommended by DNSSEC experts to reduce the risk of brute force attacks against old keys. At the root, the original plan was to roll the keys every five years.
ICANN had named October 11 2017 as the date for the first such rollover, but this was pushed back to some time in the first quarter after ICANN became aware of the lack of support for the 2017 KSK.
This was pushed back again in December to Q3 at the earliest, after ICANN admitted it still didn’t have good enough data to measure the impact of a premature roll.
Since then, ICANN has been engaged in (not always successful) outreach to networks it knows are affected and has kicked off discussions among network operators (there’s a fairly lively mailing list on the topic) to try to gauge how cautious it needs to be.
It’s now published an updated plan that’s the same as the original plan but with a date exactly one year late — October 11, 2018.
Between now and then, it will continue to try to get hold of network operators not ready to use the new keys, but it’s not expecting to completely eliminate damage. The plan reads:

Implicit in the outreach plan is the same assumption that the community had for the earlier (postponed) plan: there will likely be some systems that will fail to resolve names starting on the day of the rollover. The outreach will attempt to minimize the number of affected users while acknowledging that the operators of some resolvers will be unreachable.

The plan is open for public comment and will require the assent of the ICANN board of directors before being implemented. You have until April 2 to respond.

Second delay for domain security key rollover

ICANN has decided to delay changing the security keys to the DNS for the second time.
The “KSK Rollover” had been rescheduled from October 11 to some time in the first quarter 2018, but that will no longer happen. We’re now looking at Q3 at the earliest.
“We have decided that we do not yet have enough information to set a specific date for the rollover,” VP of research Matt Larson said in a blog post. “We want to make clear, however, that the ICANN org is committed to rolling the root zone KSK”.
The root KSK, or Key Signing Key, is the cryptographic key pair at the very top of the security hierarchy specified by DNSSEC, the security extension for DNS.
The current, first-ever, root KSK has been in operation since 2010, but ICANN’s policy is to roll it every five years or so.
The October date was delayed after newly available data showed that hundreds of DNS resolvers were still only configured to use the 2010 keys and not the 2017 keys that have already been deployed in tandem.
This would mean a rollover would cut off access to DNSSEC-signed zones to potentially millions of internet users.
ICANN found that 4% of the 12,000 DNSSEC-validating resolvers — roughly 500 IP addresses — it surveyed in September were not ready for KSK-2017.
Larson told us last month that at least 176 organizations in 41 countries were affected.
Since the first delay, ICANN has been trying to contact the owners of the 500 incompatible IP addresses but has run into some serious problems, Larson blogged.
First, a significant number of these addresses are dynamically allocated (such as to home broadband hubs) meaning tracking down the owners of the misconfigured devices would be next to impossible. Others were forwarding DNS queries on behalf of other devices, creating a similar problem.
Additionally, it seems ICANN has still not received responses from owners of 80% of the affected IP addresses.
Due to the lack of reliable data, it’s difficult for ICANN to figure out how many users’ internet access will be affected by a rollover.
The threshold called for by current policy is about 20 million people.
So ICANN has delayed the event to some point after Q1. Larson wrote that the organization will publish a plan on January 18 which will be open for public comment and discussed at the ICANN 61 meeting in Puerto Rico next March.
A final plan is not expected until ICANN 62, which happens in late June, so Q3 would be the earliest the rollover could actually occur.
Larson encouraged anyone interested in discussing the plan to join this mailing list.

Up to 20 million people could get broken internet in domain security rollover

Twenty million people losing access to parts of the internet is considered an acceptable level of collateral damage for ICANN’s forthcoming DNS root security update.
That’s one of a number of facts and figures to emerge from recent updates from the organization, explaining its decision to delay the so-called “KSK rollover” from October 11 to some time in the first quarter next year.
The rollover will see a new Key Signing Key, used as the trust anchor for all DNSSEC-signed domains, replace the seven-year-old original.
DNSSEC protects internet users and registrants from domain-based man-in-the-middle attacks. It’s considered good practice to roll keys at each level of the DNS hierarchy periodically, to reduce the risk of successful brute-force attacks.
The root KSK update will affect hundreds of millions of people who currently use DNSSEC-compatible resolvers, such as Google DNS.
ICANN delayed the rollover after it, rather fortuitously, spotted that not all of these resolvers are configured to correctly handle the change.
The number of known incompatible servers is quite small — only about 500 of the 11,982 DNSSEC-using recursive servers initially surveyed (pdf). That represents only a very small minority of the world’s internet users, as most are not currently using DNSSEC.
Subsequent ICANN research, presented by principal researcher Roy Arends at ICANN 60 last week, showed that:

• There are currently about 4.2 million DNS resolvers in the world.
• Of those, 27,084 are configured to tell the root servers which KSKs they support (currently either the KSK-2011 or KSK-2017).
• Of those, 1,631 or 6.02% do not support KSK-2017

It was only possible to survey servers that have turned on a recent update to DNS software such as BIND and Unbound, so the true number of misconfigured servers could be much higher.
Matt Larson, ICANN’s VP of research, told DI that ICANN has identified 176 organizations in 41 countries that are currently not prepared to handle the new KSK. These organizations are fairly evenly spread geographically, he said.
Since making the decision to delay the rollover, ICANN has hired a contractor to reach out to these network operators to alert them to potential problems.
ICANN’s CEO Goran Marby has also been writing to telecommunications regulators in all countries to ask for assistance.
After the rollover, people using an incompatible resolver would be unable to access DNSSEC-signed domains. Again, that’s still quite a small minority of domains — there are only about 750,000 in .com by some accounts and apparently none of the top 25 site support it.
ICANN could roll back the change if it detects that a sufficiently large number of people are negatively affected, but that number turns out to be around 20 million.
According to its published rollover plan:

Rollback of any step in the key roll process should be initiated if the measurement program indicated that a minimum of 0.5% of the estimated Internet end-user population has been negatively impacted by the change 72 hours after each change has been deployed into the root zone.

According to InternetWorldStats, there were around 3,885,567,619 internet users in the world this June. It’s very likely more people now.
So a 0.5% threshold works out to about 19 million to 20 million people worldwide.
Larson agreed that in absolute terms, it’s a big number.
“The overall message to take away from that number, I suggest, is that a problem would have to be pretty serious for us to consider rolling back,” Larson, who was not on the team that came up with the threshold, said.
“I think that’s a reasonable position considering that, in the immediate aftermath of the rollover, there are two near-immediate fixes available to any operator experiencing problems: update their systems’ trust anchors with the new key or (less desirable from my perspective but still effective) simply disable DNSSEC validation,” he said.
He added that the 0.5% level is not a hard and fast rule, and that ICANN could be flexible in the moment.
“For example, if when we roll the key, we find out there’s some critical system with a literal life or death impact that is negatively affected by the KSK roll, I think I can pretty confidently state that we wouldn’t require the 0.5% of Internet user threshold to be met before rolling back if it looked like there would be a significant health and safety risk not easily mitigated,” he said.
The chances of such an impact are very slim, but not impossible, he suggested.
It’s not ICANN’s intention to put anyone’s internet access at risk, of course, which is why there’s a delay.
ICANN’s plan calls for any rollover to happen on the eleventh day of a given calendar quarter, so the soonest it could happen would be January 11.
Given the complexity of the outreach task in hand, the relative lack of data, and the holiday periods approaching in many countries, and ICANN’s generally cautious nature, I’d hazard a guess we might be looking at April 11 at the earliest instead.

ICANN to flip the secret key to the internet

ICANN is about to embark on a year-long effort to warn the internet that it plans to replace the top-level cryptographic keys used in DNSSEC for the first time.
CTO David Conrad told DI today that ICANN will rotate the so-called Key Signing Key that is used as the “trust anchor” for all DNSSEC queries that happen on the internet.
Due to the complexity of the process, and the risk that something might go wrong, the move is to be announced in the coming days even though the new public key will not replace the existing one until October 2017.
The KSK is a cryptographic key pair used to sign the Zone Signing Keys that in turn sign the DNS root zone. It’s basically at the top of the DNSSEC hierarchy — all trust in DNSSEC flows from it.
It’s considered good practice in DNSSEC to rotate keys every so often, largely to reduce the window would-be attackers have to compromise them.
The Zone Signing Key used by ICANN and Verisign to sign the DNS root is rotated quarterly, and individual domain owners can rotate their own keys as and when they choose, but the same KSK has been in place since the root was first signed in 2010.
Conrad said that ICANN is doing the first rollover partly to ensure that the procedures in has in place for changing keys are effective and could be deployed in case of emergency.
That said, this first rotation is going to happen at a snail’s pace.
Key generation is a complex matter, requiring the physical presence of at least three of seven trusted key holders.
These seven individuals possess physical keys to bank-style strong boxes which contain secure smart cards. Three of the seven cards are needed to generate a new key.
Each of the quarterly ZSK signing ceremonies — which are recorded and broadcast live over the internet — takes about five hours.
The first step in the rollover, Conrad said, is to generate the keys at ICANN’s US east coast facility in October this year. A copy will be moved to a facility on the west coast in February.
The first time the public key will appear in DNS will be July 11, 2017, when it will appear alongside the current key.
It will finally replace the current key completely on October 11, 2017, by which time the DNS should be well aware of the new key, Conrad said.
There is some risk of things going wrong, which could affect domains that are DNSSEC-signed, which is another reason for the slowness of the rollover.
If ISPs that support DNSSEC do not start supporting the new KSK before the final switch-over, they’ll fail to correctly resolve DNSSEC-signed domains, which could lead to some sites going dark for some users.
There’s also a risk that the increased DNS packet sizes during the period when both KSKs are in use could cause queries to be dropped by firewalls, Conrad said.
“Folks who have things configured the right way won’t actually need to do anything but because DNSSEC is relatively new and this software hasn’t really been tested, we need to get the word out to everyone that this change is going to be occurring,” said Conrad.
ICANN will conduct outreach over the coming 15 months via the media, social media and technology conferences, he said.
It is estimated that about 20% of the internet’s DNS resolvers support DNSSEC, but most of those belong to just two companies — Google and Comcast — he said.
The number of signed domains is tiny as a percentage of the 326 million domains in existence today, but still amounts to millions of names.