ICANN faces critical choice as security experts warn against key rollover

Members of ICANN’s top security body have advised the organization to further delay plans to change the domain name system’s top cryptographic key.
Five dissenting members of the influential, 22-member Security and Stability Advisory Committee said they believe “the risks of rolling in accordance with the current schedule are larger than the risks of postponing”.
Their comments relate to the so-called KSK rollover, which would see ICANN for the first time ever change the key-signing key that acts as the trust anchor for all DNSSEC queries on the internet.
ICANN is fairly certain rolling the key will cause DNS resolution problems for some — possibly as much as 0.05% of the internet or a couple million people — but it currently lacks the data to be absolutely certain of the scale of the impact.
What it does know — explained fairly succinctly in this newly published guide (pdf) — is that within 48 hours of the roll, a certain small percentage of internet users will start to see DNS resolution fail.
But there’s a prevailing school of thought that believes the longer the rollover is postponed, the bigger that number of affected users will become.
The rollover is currently penciled in for October 11, but the ultimate decision on whether to go ahead rests with the ICANN board of directors.
David Conrad, the organization’s CTO, told us last week that his office has already decided to recommend that the roll should proceed as planned. At the time, he noted that SSAC was a few days late in delivering its own verdict.
Now, after some apparently divisive discussions, that verdict is in (pdf).
SSAC’s majority consensus is that it “has not identified any reason within the SSAC’s scope why the rollover should not proceed as currently planned.”
That’s in line with what Conrad, and the Root Server System Advisory Committee have said. But SSAC noted:

The assessment of risk in this particular area has some uncertainty and therefore includes a component of subjective judgement. Individuals (including some members of the SSAC) have different assessments of the overall balance of risk of the resumption of this plan.

It added that it’s up to the ICANN board (comprised largely of non-security people) to make the final call on what the acceptable level of risk is.
The minority, dissenting opinion gets into slightly more detail:

The decision to proceed with the keyroll is a complex tradeoff of technical and non-technical risks. While there is risk in proceeding with the currently planned roll, we understand that there is also risk in further delay, including loss of confidence in DNSSEC operational planning, potential for more at-risk users as more DNSSEC validation is deployed, etc.
While evaluating these risks, the consensus within the SSAC is that proceeding is preferable to delay. We personally evaluate the tradeoffs differently, and we believe that the risks of rolling in accordance with the current schedule are larger than the risks of postponing and focusing heavily on additional research and outreach, and in particular leveraging newly developed techniques that provide better signal and fidelity into potentially impacted parties.
We would like to reiterate that we understand our colleagues’ position, but evaluate the risks and associated mitigation prospects differently. We believe that the ultimate decision lies with the ICANN Board, and do not envy them with this decision.

SSAC members are no slouches when it comes to security expertise, and the dissenting members are no exception. They are:

• Lyman Chapin, co-owner of Interisle Consulting, a regular ICANN contractor perhaps best-known to DI readers for carrying out a study into new gTLD name collisions five years ago.
• Kimberly “kc claffy” Claffy, head of the Center for Applied Internet Data Analysis at the University of California in San Diego. CAIDA does nothing but map and measure the internet.
• Jay Daley, a registry executive with a technical background whose career includes senior stints at .uk and .nz. He’s currently keeping the CEO’s chair warm at .org manager Public Interest Registry.
• Warren Kumari, a senior network security engineer at Google, which is probably the largest early adopter of DNSSEC on the resolution side.
• Danny McPherson, Verisign’s chief security officer. As well as .com, Verisign runs the two of the 13 root servers, including the master A-root. It’s running the boxes that sit at the top of the DNSSEC hierarchy.

It may be the first time SSAC has failed to reach a full-consensus opinion on a security matter. If it has ever published a dissenting opinion before, I certainly cannot recall it.
The big decision about whether to proceed or delay is expected to be made by the ICANN board during its retreat in Brussels, a three-day meeting that starts September 14.
Given that ICANN’s primary mission is “to ensure the stable and secure operation of the Internet’s unique identifier systems”, it could turn out to be one of ICANN’s biggest decisions to date.

ICANN CTO: no reason to delay KSK rollover

ICANN’s board of directors will be advised to go ahead with a key security change at the DNS root — “the so-called KSK rollover” — this October, according to the organization’s CTO.
“We don’t see any reason to postpone again,” David Conrad told DI on Monday.
If it does go ahead as planned, the rollover will see ICANN change the key-signing key that acts as the trust anchor for the whole DNSSEC-using internet, for the first time since DNSSEC came online in 2010.
It’s been delayed since last October after it emerged that misconfigurations elsewhere in the DNS cloud could see potentially millions of internet users see glitches when the key is rolled.
Ever since then, ICANN and others have been trying to figure out how many people could be adversely affected by the change, and to reduce that number to the greatest extent possible.
The impact has been tricky to estimate due to patchy data.
While it’s been possible to determine a number of resolvers — about 8,000 — that definitely are poorly configured, that only represents a subset of the total number. It’s also been hard to map that to endpoints due to “resolvers behind resolvers behind resolvers”, Conrad said.
“The problem here is that it’s sort of a subjective evaluation,” he said. “We can’t rely on the data were seeing. We’re seeing the resolvers but we’re not seeing the users behind the resolvers.”
Some say that the roll is still too risky to carry out without better visibility into the potential impact, but others say that more delays would lead to more networks and devices becoming DNSSEC-compatible, potentially leading to even greater problems after the eventual rollover.
ICANN knows of about 8,000 resolver IP addresses that are likely to stop working properly after the rollover, because they only support the current KSK, but that’s only counting resolvers that automatically report their status to the root using a relatively new internet standard. There’s a blind spot concerning resolvers that do not have that feature turned on.
ICANN has also had difficulty reaching out to the network operators behind these resolvers, with good contact information apparently only available for about a quarter of the affected IP addresses, Conrad said.
Right now, the best data available suggests that 0.05% of the internet’s population could see access issues after the October 11 rollover, according to Conrad.
That’s about two million people, but it’s 10 times fewer people than the 0.5% acceptable collateral damage threshold outlined in ICANN’s rollover plan.
The 0.05% number comes from research by APNIC, which used Google’s advertising system to place “zero-pixel ads” to check whether individual user endpoints were using compatible resolvers or not.
If problems do emerge October 11 the temporary solution is apparently quite quick to implement — network operators can simply turn off DNSSEC, assuming they know that’s what they’re supposed to do.
But still, if a million or two internet users could have their day ruined by the rollover, why do it at all?
It’s not as if the KSK is in any danger of being cracked any time soon. Conrad explained that a successful brute-force attack on the 2048-bit RSA key would take longer than the lifetime of the universe using current technology.
Rather, the practice of rolling the key every five years is to get network operators and developers accustomed to the idea that the KSK is not a permanent fixture that can be hard-coded into their systems, Conrad said.
It’s a problem comparable to new gTLD name collisions or the Y2K problem, instances where developers respectively hard-coded assumptions about valid TLDs or the century into their software.
ICANN has already been reaching out to the managers of open-source projects on repositories such as Github that have been seen to hard-code the current KSK into their software, Conrad said.
Separately, Wes Hardaker at the University of Southern California Information Sciences Institute discovered that a popular VPN client was misconfigured. Outreach to the developer saw the problem fixed, reducing the number of users who will be affected by the roll.
“What we’re trying to avoid is having these keys hardwired into firmware, so that that it would never be changeable,” he said. “The idea is if you exercise the infrastructure frequently enough, people will know the that the key is not permanent configuration, it’s not something embedded in concrete.”
One change that ICANN may want to make in future is to change the algorithm used to generate the KSK.
Right now it’s using RSA, but Conrad said it has downsides such as rather large signature size, which leads to heavier DNSSEC traffic. By switching to elliptical curve cryptography, signatures could be reduced by “orders of magnitude”, leading to a more efficient and slimline DNS infrastructure, Conrad said.
Last week, ICANN’s Root Server Stability Advisory Committee issued an advisory (pdf) that essentially gave ICANN the all-clear to go ahead with the roll.
The influential Security and Stability Advisory Committee has yet to issue its own advisory, however, despite being asked to do so by August 10.
Could SSAC be more cautious in its advice? We’ll have to wait and see, but perhaps not too long; the current plan is for the ICANN board to consider whether to go ahead with the roll during its three-day Brussels retreat, which starts September 14.

Digital archery ruled out for next new gTLD round

The oft-mocked “digital archery” system will not be making a return when ICANN finally starts taking more new gTLD applications.
That’s the current thinking of the ICANN community working group looking at subsequent application procedures.
Readers with long memories may recall digital archery as a hack for Californian gambling laws that ICANN org pressed for in 2012 as a way to form its 1,930 applications into an orderly queue for processing.
The idea was that applicants would fire off a bit of data to an ICANN site at a predetermined time and the applicants whose packets arrived closet to the target time, measured by the millisecond, would receive priority in the queue.
It was a bit like drop-catching, and the concept advanced to the stage where companies skilled in such things were offering digital archery services.
But after ICANN changed CEOs later that year, it turned out gambling wasn’t as illegal in California as former management thought it was. The org got itself a license to run a one-off lottery and sold tickets for $100 per application.
That’s now the preferred method for ordering the queue for the next rounds of applications, whenever those may be, according to last week’s Initial Report on the New gTLD Subsequent Procedures Policy Development Process.
Unlike 2012, the WG is proposing that portfolio applicants should be able to swap around their priority numbers according to their commercial interests.
So, if Donuts gets priority #1 for .crappy and #4,000 for .awesome, it would be able to switch priorities to get the better string evaluated earlier.
The WG is also not convinced that internationalized domain names, which received automatic priority in 2012, should get the same preferential treatment this time around.
That’s one of several questions it poses for the community in its public comment period.
While a better place in the evaluation queue had time-to-market advantages in 2012 — Donuts’ .guru sold a tonne of domains largely due to its first-mover status — that’s probably not going to be as big a deal next time around due to domainer skepticism about new gTLDs.

Could crypto solve the Whois crisis?

Could there be a cryptographic solution to some of the problems caused by GDPR’s impact on public Whois databases? Security experts think so.
The Anti-Phishing Working Group has proposed that hashing personal information and publishing it could help security researchers carry on using Whois to finger abusive domain names.
In a letter to ICANN, APWG recently said that such a system would allow registries and registrars to keep their customers’ data private, but would still enable researchers to identify names registered in bulk by spammers and the like.
“Redacting all registration records which were formerly publicly available has unintended and undesirable consequences to the very citizens and residents that electronic privacy legislation intends to protect,” the letter (pdf) says.
Under the proposed system, each registry or registrar would generate a private key for itself. For each Whois field containing private data, the data would be added to the key and hashed using a standard algorithm such as SHA-512.
For items such as physical addresses, all the address-related fields would be concatenated, with the key, before hashing the combined value.
The resulting hash — a long string of gibberish characters — would then be published in the public Whois instead of the [REDACTED] notice mandated by current ICANN policy.
Security researchers would then be able to identify domains belonging to the same purported registrant by searching for domains containing the same hash values.
It’s not a perfect solution. Because each registry or registrar would have their own key, the same registrant would have different hash values in different TLDs, so it would not be possible to search across TLDs.
But that may not be a huge problem, given that bad guys tend to bulk-register names in TLDs that have special offers on.
The hashing system may also be beneficial to interest groups such as trademark owners and law enforcement, which also look for registration patterns when tracking down abuse registrants.
The proposal would create implementation headaches for registries and registrars — which would actually have to build the crypto into their systems — and compliance challenges for ICANN.
The paper notes that ICANN would have to monitor its contracted parties — not all of which may necessarily be unfriendly to spammers — to make sure they’re hashing the data correctly.

ICANN found a zero-day hole in Adobe Connect

It’s looking like ICANN may have found a zero-day vulnerability in Adobe Connect, until recently its default collaboration tool.
The organization on Friday announced the results of a “forensic investigation” into the bug, and said it has reported its findings to Adobe, which is now “working on a software fix to address the root cause of the issue”.
If Adobe didn’t know about it, it looks rather like ICANN — or at least the unnamed member of the security advisory committee who found it — has bagged itself a zero-day.
ICANN had previously said that the glitch “could possibly lead to the disclosure of the information shared in an ICANN Adobe Connect room”.
The review found that the only person who exploited the bug was the person who discovered and disclosed it.
AC is used not only in ICANN’s public meetings but also, I understand, in closed sessions of ICANN staff, board and committees, where secret information is most likely to be shared.
After the bug was discovered, ICANN shut off the system and started using alternatives such as WebEx, to a mixed reception.
In the absence of an immediate patch from Adobe, ICANN has been testing workarounds and said it hopes to have two working ones deployed by May 3.
This would allow the tool to come back online in time for its board workshop, GDD Summit and ICANN 62, the organization said.

Root crypto rollover now slated for October

ICANN has penciled in October 11 as the new date for rolling the DNS root’s cryptographic keys, a delay of a year from its original plan.
The so-called KSK rollover will see ICANN remove the deprecated 2010 Key Signing Key, leaving only the 2017 KSK active.
The KSK acts as the “trust anchor” for DNSSEC across the whole internet.
After the rollover, any network not configured to use the latest KSK would see a service interruption.
This could mean many millions of internet users being affected, but ICANN doesn’t know the extent of the possible impact for sure.
ICANN told us in November that it knows of 176 organizations in 41 countries, fairly evenly spread across the globe, that are currently not prepared to handle the new KSK.
But its data is patchy because only a tiny number of DNS resolvers are actually configured to automatically report which KSKs they’re set up to use.
Key rollovers are recommended by DNSSEC experts to reduce the risk of brute force attacks against old keys. At the root, the original plan was to roll the keys every five years.
ICANN had named October 11 2017 as the date for the first such rollover, but this was pushed back to some time in the first quarter after ICANN became aware of the lack of support for the 2017 KSK.
This was pushed back again in December to Q3 at the earliest, after ICANN admitted it still didn’t have good enough data to measure the impact of a premature roll.
Since then, ICANN has been engaged in (not always successful) outreach to networks it knows are affected and has kicked off discussions among network operators (there’s a fairly lively mailing list on the topic) to try to gauge how cautious it needs to be.
It’s now published an updated plan that’s the same as the original plan but with a date exactly one year late — October 11, 2018.
Between now and then, it will continue to try to get hold of network operators not ready to use the new keys, but it’s not expecting to completely eliminate damage. The plan reads:

Implicit in the outreach plan is the same assumption that the community had for the earlier (postponed) plan: there will likely be some systems that will fail to resolve names starting on the day of the rollover. The outreach will attempt to minimize the number of affected users while acknowledging that the operators of some resolvers will be unreachable.

The plan is open for public comment and will require the assent of the ICANN board of directors before being implemented. You have until April 2 to respond.

Research finds homograph attacks on big brands rife

Apparent domain name homograph attacks against major brands are a “significant” problem, according to research from Farsight Security.
The company said last week that it scanned for such attacks against 125 well-known brands over the three months to January 10 and found 116,113 domains — almost 1,000 per brand.
Homographs are domains that look like other domains, often indistinguishable from the original. They’re usually used to phish for passwords to bank accounts, retailers, cryptocurrency exchanges, and so on.
They most often use internationalized domain names, mixing together ASCII and non-ASCII characters when displayed in browsers.
To the naked eye, they can look very similar to the original ASCII-only domains, but under the hood they’re actually encoded with Punycode with the xn-- prefix.
Examples highlighted by Farsight include baŋkofamerica.com, amazoṇ.com and fàcebook.com
Displayed as ASCII, those domains are actually xn--bakofamerica-qfc.com, xn--amazo-7l1b.com and xn--fcebook-8va.com.
Farsight gave examples including and excluding the www. subdomain in a blog post last week, but I’m not sure if it double-counted to get to its 116,113-domain total.
As you might imagine, almost all of this abuse is concentrated in .com and other TLDs that were around before 2012, judging by Farsight’s examples. That’s because the big brands are not using new gTLDs for their primary sites yet.
Farsight gave a caveat that it had not generally investigated the ownership of the homograph domains it found. It’s possible some of them are defensive registrations by brands that are already fully aware of the security risk they could present.

Second delay for domain security key rollover

ICANN has decided to delay changing the security keys to the DNS for the second time.
The “KSK Rollover” had been rescheduled from October 11 to some time in the first quarter 2018, but that will no longer happen. We’re now looking at Q3 at the earliest.
“We have decided that we do not yet have enough information to set a specific date for the rollover,” VP of research Matt Larson said in a blog post. “We want to make clear, however, that the ICANN org is committed to rolling the root zone KSK”.
The root KSK, or Key Signing Key, is the cryptographic key pair at the very top of the security hierarchy specified by DNSSEC, the security extension for DNS.
The current, first-ever, root KSK has been in operation since 2010, but ICANN’s policy is to roll it every five years or so.
The October date was delayed after newly available data showed that hundreds of DNS resolvers were still only configured to use the 2010 keys and not the 2017 keys that have already been deployed in tandem.
This would mean a rollover would cut off access to DNSSEC-signed zones to potentially millions of internet users.
ICANN found that 4% of the 12,000 DNSSEC-validating resolvers — roughly 500 IP addresses — it surveyed in September were not ready for KSK-2017.
Larson told us last month that at least 176 organizations in 41 countries were affected.
Since the first delay, ICANN has been trying to contact the owners of the 500 incompatible IP addresses but has run into some serious problems, Larson blogged.
First, a significant number of these addresses are dynamically allocated (such as to home broadband hubs) meaning tracking down the owners of the misconfigured devices would be next to impossible. Others were forwarding DNS queries on behalf of other devices, creating a similar problem.
Additionally, it seems ICANN has still not received responses from owners of 80% of the affected IP addresses.
Due to the lack of reliable data, it’s difficult for ICANN to figure out how many users’ internet access will be affected by a rollover.
The threshold called for by current policy is about 20 million people.
So ICANN has delayed the event to some point after Q1. Larson wrote that the organization will publish a plan on January 18 which will be open for public comment and discussed at the ICANN 61 meeting in Puerto Rico next March.
A final plan is not expected until ICANN 62, which happens in late June, so Q3 would be the earliest the rollover could actually occur.
Larson encouraged anyone interested in discussing the plan to join this mailing list.

Davies named new IANA boss

Kim Davies has been named the new head of IANA.
ICANN said today that he’s been promoted from his role as director of technical services to VP of IANA services and president of Public Technical Identifiers, the company that manages the IANA functions.
With ICANN since 2005, he replaces Elise Gerich, who announced her departure, originally scheduled for October, back in April.
Gerich has been IANA’s top staffer since 2010 and was PTI’s first president.
IANA is responsible for overseeing the top-level domain database, as well as the allocation of IP address blocks and protocol numbers.
Starting January 1, Davies will be in the top spot when ICANN executes the first-ever rollover of the root system’s most important DNSSEC keys, due to delays.

Up to 20 million people could get broken internet in domain security rollover

Twenty million people losing access to parts of the internet is considered an acceptable level of collateral damage for ICANN’s forthcoming DNS root security update.
That’s one of a number of facts and figures to emerge from recent updates from the organization, explaining its decision to delay the so-called “KSK rollover” from October 11 to some time in the first quarter next year.
The rollover will see a new Key Signing Key, used as the trust anchor for all DNSSEC-signed domains, replace the seven-year-old original.
DNSSEC protects internet users and registrants from domain-based man-in-the-middle attacks. It’s considered good practice to roll keys at each level of the DNS hierarchy periodically, to reduce the risk of successful brute-force attacks.
The root KSK update will affect hundreds of millions of people who currently use DNSSEC-compatible resolvers, such as Google DNS.
ICANN delayed the rollover after it, rather fortuitously, spotted that not all of these resolvers are configured to correctly handle the change.
The number of known incompatible servers is quite small — only about 500 of the 11,982 DNSSEC-using recursive servers initially surveyed (pdf). That represents only a very small minority of the world’s internet users, as most are not currently using DNSSEC.
Subsequent ICANN research, presented by principal researcher Roy Arends at ICANN 60 last week, showed that:

• There are currently about 4.2 million DNS resolvers in the world.
• Of those, 27,084 are configured to tell the root servers which KSKs they support (currently either the KSK-2011 or KSK-2017).
• Of those, 1,631 or 6.02% do not support KSK-2017

It was only possible to survey servers that have turned on a recent update to DNS software such as BIND and Unbound, so the true number of misconfigured servers could be much higher.
Matt Larson, ICANN’s VP of research, told DI that ICANN has identified 176 organizations in 41 countries that are currently not prepared to handle the new KSK. These organizations are fairly evenly spread geographically, he said.
Since making the decision to delay the rollover, ICANN has hired a contractor to reach out to these network operators to alert them to potential problems.
ICANN’s CEO Goran Marby has also been writing to telecommunications regulators in all countries to ask for assistance.
After the rollover, people using an incompatible resolver would be unable to access DNSSEC-signed domains. Again, that’s still quite a small minority of domains — there are only about 750,000 in .com by some accounts and apparently none of the top 25 site support it.
ICANN could roll back the change if it detects that a sufficiently large number of people are negatively affected, but that number turns out to be around 20 million.
According to its published rollover plan:

Rollback of any step in the key roll process should be initiated if the measurement program indicated that a minimum of 0.5% of the estimated Internet end-user population has been negatively impacted by the change 72 hours after each change has been deployed into the root zone.

According to InternetWorldStats, there were around 3,885,567,619 internet users in the world this June. It’s very likely more people now.
So a 0.5% threshold works out to about 19 million to 20 million people worldwide.
Larson agreed that in absolute terms, it’s a big number.
“The overall message to take away from that number, I suggest, is that a problem would have to be pretty serious for us to consider rolling back,” Larson, who was not on the team that came up with the threshold, said.
“I think that’s a reasonable position considering that, in the immediate aftermath of the rollover, there are two near-immediate fixes available to any operator experiencing problems: update their systems’ trust anchors with the new key or (less desirable from my perspective but still effective) simply disable DNSSEC validation,” he said.
He added that the 0.5% level is not a hard and fast rule, and that ICANN could be flexible in the moment.
“For example, if when we roll the key, we find out there’s some critical system with a literal life or death impact that is negatively affected by the KSK roll, I think I can pretty confidently state that we wouldn’t require the 0.5% of Internet user threshold to be met before rolling back if it looked like there would be a significant health and safety risk not easily mitigated,” he said.
The chances of such an impact are very slim, but not impossible, he suggested.
It’s not ICANN’s intention to put anyone’s internet access at risk, of course, which is why there’s a delay.
ICANN’s plan calls for any rollover to happen on the eleventh day of a given calendar quarter, so the soonest it could happen would be January 11.
Given the complexity of the outreach task in hand, the relative lack of data, and the holiday periods approaching in many countries, and ICANN’s generally cautious nature, I’d hazard a guess we might be looking at April 11 at the earliest instead.