What time is it? For ICANN, even that can be a controversial question

ICANN has found itself involved in a debate about whether Russia’s 2014 annexation of Crimea should be recognized.
It’s not unusual for ICANN to find itself in geopolitical controversies — see .amazon for the most recent example — but this time, it’s not about domain names.
It’s about time zones.
One of the little-known functions ICANN provides via its IANA division is the hosting of the so-called TZ Database, which keeps track of all international time zones, daylight savings time practices, and so on.
The database is referenced by scores of operating systems, web sites, libraries and software development kits. It’s used by MacOS, many major Unix/Linux distributions, Java and PHP.
IANA took over the database in 2011, after the original administrator, David Olson, was hit with a bogus lawsuit from an astrology company.
It’s currently managed by University of California computer scientist Paul Eggert. He’s not an ICANN employee. He’s responsible for making changes to the database, which IANA hosts.
There are no complex layers of policy-making and bureaucracy, just an ICANN-hosted mailing list. it very much harks back to the pre-ICANN/Jon Postel/Just A Guy model of international database administration.
But because time zones are set by the governments of territories, and the ownership of territories is sometimes in dispute, the TZ Database often finds itself involved in political debates.
The latest of these relates to Crimea.
As you will recall, back in 2014 the Russian Federation annexed Crimea — part of Ukraine and formerly part of the Soviet Union.
The United Nations condemned the move as illegal and still refuses to recognize the region as part of Russia. The de facto capital city of Crimea is now Simferopol.
As part of the takeover, Russia switched its new territories over to Moscow Time (MSK), a time zone three hours ahead of UTC that does not observe daylight savings.
The rest of Ukraine continues to use Eastern European Time, which is UTC+2, and Eastern European Summer Time (UTC+3).
This means that in the winter months, Crimea is an hour out of whack with the rest of Ukraine.
Currently, the TZ Database’s entry for Simferpol contains the country code “RU”, instead of “UA”.
This means that if you go to Crimea and try to configure your Unix-based system to the local time, you’ll see an indication in the interface that you’re in Russia, which understandably pisses off Ukrainians and is not in line with what most governments think.
You can check this out on some time zone web sites. The services at time.is and timeanddate.com both refer to Europe/Simferopol as being in Ukraine, while WorldTimeServer says it’s in Russia.
The TZ Database mailing list has recently received a couple of complaints from Ukrainians, including the head of the local cyber police, about this issue.
Serhii Demediuk, head of the Cyberpolice Department of the National Police of Ukraine, wrote in December:

by referring Crimea with the country code “RU”, your organization actually accepts and supports the aggressive actions of the Russian Federation who’s armed forces annexed this part of Ukraine. Such recognition may be considered as a criminal offense by the Ukrainian criminal law and we will be obliged to start formal criminal proceedings

It’s the longstanding principle of the TZ Database administrators that they’re not taking political positions when they assign country-codes to time zones, they’re just trying to be practical.
If somebody shows up for a business meeting in Crimea in December, they don’t want their clock to be an hour behind their local host’s for the sake of political correctness.
But Eggert nevertheless has proposed a patch that he believes may address Ukrainian concerns. It appears to have Simferopol listed as both RU and UA.

ICANN got hacked by crypto bots

ICANN had to take down its community wiki for several hours last week after it got hacked by crypto-currency miners.
The bad guys got in via one of two “critical” vulnerabilities in Confluence, the wiki software that ICANN licences from Atlassian Systems, which ICANN had not yet patched.
ICANN’s techies noticed the wiki, which is used by many of its policy-making bodies to coordinate their work, was running slowly April 11.
They quickly discovered that Atlassian had issued a vulnerability warning on March 20, but ICANN was not on its mailing list (doh!) so hadn’t been directly notified.
They also determined that a malicious “Crypto-Miner” — software that uses spare CPU cycles to attempt to create new cryptocurrency coins — had been installed and was responsible for the poor performance.
ICANN said it took the wiki down, restored it to a recent backup, patched Confluence, and brought the system back online. It seems to have taken a matter of hours from discovery to resolution.
The organization said it has now subscribed to Atlassian’s mailing list, so it will be notified of future vulnerabilities directly.

Root servers whacked after crypto change

The DNS root servers came under accidental attack from name servers across the internet following ICANN’s recent changes to their cryptographic master keys, according to Verisign.
The company, which runs the A and J root servers, said it saw requests for DNSSEC data at the root increase from 15 million a day in October to 1.15 billion a day a week ago.
The cause was the October 11 root Key Signing Key rollover, the first change ICANN had made to the “trust anchor” of DNSSEC since it came online at the root in 2010.
The KSK rollover saw ICANN change the cryptographic keys that rest at the very top of the DNSSEC hierarchy.
The move was controversial. ICANN delayed it for a year after learning about possible disruption at internet endpoints. Its Security and Stability Advisory Committee and even its own board were not unanimous that the roll should go ahead.
But the warnings were largely about the impact on internet users, rather than on the root servers themselves, and the impact was minimal.
Verisign is now saying that requests to its roots for DNSSEC key data increased from 15 million per day to 75 million per day, a five-fold increase, almost overnight.
It was not until January, when the old KSK was marked as “revoked”, did the seriously mahooosive traffic growth begin, however. Verisign’s distinguished engineer Duane Wessels wrote:

Everyone involved expected this to be a non-event. However, we instead saw an even bigger increase in DNSKEY queries coming from a population of root server clients. As of March 21, 2019, Verisign’s root name servers receive about 1.15 billion DNSKEY queries per day, which is 75 times higher than pre-rollover levels and nearly 7 percent of our total steady state query traffic.

Worryingly, the traffic only seemed to be increasing, until March 22, when the revoked key was removed from the root entirely.
Wessels wrote that while the root operators are still investigating, “it would seem that the presence of the revoked key in the zone triggered some unexpected behavior in a population of validating resolvers.”
The root operators hope to have answers in the coming weeks, he wrote.
The next KSK rollover is not expected for years, and the root traffic is now returning to normal levels, so there’s no urgency.

The internet is still working after KSK roll

The first-ever change to the security keys at the top of the DNS tree appears to have been a non-event.
While ICANN received reports of some disruptions after last Thursday’s KSK rollover, the impact appears to have fallen short of the millions of users that had been speculated.
ICANN said yesterday:

After evaluation of the available data, there does not appear to be a significant number of Internet end-users who have been persistently and negatively impacted by the changing of the key.
The few issues that have arisen appear to have been quickly mitigated and none suggested a systemic failure that would approach the threshold (as defined by the ICANN community) to initiate a reversal of the roll. In that context, it appears the rollover to the new Key Signing Key, known as KSK 2017, has been a success.

The KSK, also sometimes called the “trust anchor”, is the ultimate cryptographic key in the chain that secures all DNSSEC queries on the internet.
October 11 was the first time it had been changed since the first version came online in 2010.
While changing the key was broadly considered sound security practice, the roll was delayed by a year after it was discovered that potentially millions of endpoints were using DNS resolvers not properly configured to use the 2017 key.
After much research, outreach and gnashing of teeth, it was decided that the risk posed by rolling the KSK now fell within acceptable parameters of collateral damage.
Experts from the likes of Google and Verisign, and one ICANN director, had urged caution and said perhaps the roll should be delayed further while more data was gathered.
But they were in the minority, ICANN went ahead anyway, and it seems their fears have not come to pass.
The KSK is now likely to be rolled regularly — it could be as little as once every five years, or more frequently.
It also gives ICANN the opportunity to eventually update the system to swap out its current RSA keys for keys based on elliptical curve cryptography, which could reduce the traffic load on the DNS as a whole.

Google adds censorship workaround to Android devices

Google is using experimental DNS to help people in censorious regimes access blocked web sites.
Alphabet sister company Jigsaw this week released an Android app called Intra, which enables users to tunnel their DNS queries over HTTPS to compatible servers, avoiding common types of on-the-wire manipulation.
The company reportedly says it has been testing the app with Venezuelan dissidents recently.
The feature will also be built in to the next version of Android — known as Android 9 or Android Pie — where it will be called Private DNS.
The app is designed for people who for one reason or another are unable to update their device’s OS.
Intra and Private DNS use “DNS over HTTPS”, an emerging protocol Google and others have been working on for a while.
As it’s non-standard, end users will have to configure their devices or Intra apps to use a DoH-compatible DNS server. The public DNS services operated by Google (8.8.8.8) and Cloudflare (1.1.1.1) are both currently compatible.
The release comes even as Google faces controversy for allegedly kowtowing to the Chinese government’s demands for censored search and news results.
You may notice that the new app is being marketed via a .org web site, rather than Google’s own .app gTLD, but intra.app takes visitors directly to the Intra page on the Google Play store.

Emoji domains now easier to use

Emoji domains have become marginally easier to navigate to in the last month, following an update to Google’s Chrome browser.
Google has added “Emoji” to the context menu that appears when users right-click in any editable text field — including the address/search bar.
Clicking the option brings up a searchable list of common emojis that can be inserted into the address bar for either search or, with the addition of a typed-in TLD, navigation.
TLDs currently supporting emojis include .ws, .fm and .to. ICANN has ruled out support for emojis in the gTLDs for security reasons.
When the domain is resolved, the emojis render in the address bar as Punycode-converted Latin characters beginning with the usual “xn--” prefix, at least under my default configuration.
The whole process is still a bit fiddly, so I wouldn’t all rush out to build your businesses on emoji domains just yet.
The context menu feature appears to have been on the experimental track in Chrome for at least a month, but was more recently turned on by default, at least on all the Chrome 69 installs I’ve tested.
If you don’t get the emoji option in your context menu, you should be able to turn it on by navigating to chrome://flags/#enable-emoji-context-menu and selecting the Enabled option.

Whois privacy did NOT increase spam volumes

The advent of more-or-less blanket Whois privacy has not immediately led to the feared uptick in spam, according to researchers.
Data from Cisco’s Talos email data service, first highlighted by security company Recorded Future this week, shows spam levels have been basically flat to slightly down since ICANN’s GDPR-inspired new Whois policy came into effect May 25.
Public Talos data shows that on May 1 this year there were 433.9 billion average daily emails and 370.04 billion spams — 85.28% spam.
This was down to 361.83 billion emails and 308.05 billion spams by August 1, an 85.14% spam ratio, according to Recorded Future.
So, basically no change, and certainly not the kind of rocketing skyward of spam levels that some had feared.
Cisco compiles its data from customers of its various security products and services.
Looking at Talos’ 18-month view, it appears that spam volume has been on the decline since February, when the ratio of spam to ham was pretty much identical to post-GDPR levels.
It also shows a similar seasonal decline during the northern hemisphere’s summer 2017.

There had been a fear in some quarters that blanket Whois privacy would embolden spammers to register more domains and launch more ambitious spam campaigns, and that the lack of public data would thwart efforts to root out the spammers themselves.
While that may well transpire in future, the data seems to show that GDPR has not yet had a measurable impact on spam volume at all.

ICANN faces critical choice as security experts warn against key rollover

Members of ICANN’s top security body have advised the organization to further delay plans to change the domain name system’s top cryptographic key.
Five dissenting members of the influential, 22-member Security and Stability Advisory Committee said they believe “the risks of rolling in accordance with the current schedule are larger than the risks of postponing”.
Their comments relate to the so-called KSK rollover, which would see ICANN for the first time ever change the key-signing key that acts as the trust anchor for all DNSSEC queries on the internet.
ICANN is fairly certain rolling the key will cause DNS resolution problems for some — possibly as much as 0.05% of the internet or a couple million people — but it currently lacks the data to be absolutely certain of the scale of the impact.
What it does know — explained fairly succinctly in this newly published guide (pdf) — is that within 48 hours of the roll, a certain small percentage of internet users will start to see DNS resolution fail.
But there’s a prevailing school of thought that believes the longer the rollover is postponed, the bigger that number of affected users will become.
The rollover is currently penciled in for October 11, but the ultimate decision on whether to go ahead rests with the ICANN board of directors.
David Conrad, the organization’s CTO, told us last week that his office has already decided to recommend that the roll should proceed as planned. At the time, he noted that SSAC was a few days late in delivering its own verdict.
Now, after some apparently divisive discussions, that verdict is in (pdf).
SSAC’s majority consensus is that it “has not identified any reason within the SSAC’s scope why the rollover should not proceed as currently planned.”
That’s in line with what Conrad, and the Root Server System Advisory Committee have said. But SSAC noted:

The assessment of risk in this particular area has some uncertainty and therefore includes a component of subjective judgement. Individuals (including some members of the SSAC) have different assessments of the overall balance of risk of the resumption of this plan.

It added that it’s up to the ICANN board (comprised largely of non-security people) to make the final call on what the acceptable level of risk is.
The minority, dissenting opinion gets into slightly more detail:

The decision to proceed with the keyroll is a complex tradeoff of technical and non-technical risks. While there is risk in proceeding with the currently planned roll, we understand that there is also risk in further delay, including loss of confidence in DNSSEC operational planning, potential for more at-risk users as more DNSSEC validation is deployed, etc.
While evaluating these risks, the consensus within the SSAC is that proceeding is preferable to delay. We personally evaluate the tradeoffs differently, and we believe that the risks of rolling in accordance with the current schedule are larger than the risks of postponing and focusing heavily on additional research and outreach, and in particular leveraging newly developed techniques that provide better signal and fidelity into potentially impacted parties.
We would like to reiterate that we understand our colleagues’ position, but evaluate the risks and associated mitigation prospects differently. We believe that the ultimate decision lies with the ICANN Board, and do not envy them with this decision.

SSAC members are no slouches when it comes to security expertise, and the dissenting members are no exception. They are:

• Lyman Chapin, co-owner of Interisle Consulting, a regular ICANN contractor perhaps best-known to DI readers for carrying out a study into new gTLD name collisions five years ago.
• Kimberly “kc claffy” Claffy, head of the Center for Applied Internet Data Analysis at the University of California in San Diego. CAIDA does nothing but map and measure the internet.
• Jay Daley, a registry executive with a technical background whose career includes senior stints at .uk and .nz. He’s currently keeping the CEO’s chair warm at .org manager Public Interest Registry.
• Warren Kumari, a senior network security engineer at Google, which is probably the largest early adopter of DNSSEC on the resolution side.
• Danny McPherson, Verisign’s chief security officer. As well as .com, Verisign runs the two of the 13 root servers, including the master A-root. It’s running the boxes that sit at the top of the DNSSEC hierarchy.

It may be the first time SSAC has failed to reach a full-consensus opinion on a security matter. If it has ever published a dissenting opinion before, I certainly cannot recall it.
The big decision about whether to proceed or delay is expected to be made by the ICANN board during its retreat in Brussels, a three-day meeting that starts September 14.
Given that ICANN’s primary mission is “to ensure the stable and secure operation of the Internet’s unique identifier systems”, it could turn out to be one of ICANN’s biggest decisions to date.

ICANN CTO: no reason to delay KSK rollover

ICANN’s board of directors will be advised to go ahead with a key security change at the DNS root — “the so-called KSK rollover” — this October, according to the organization’s CTO.
“We don’t see any reason to postpone again,” David Conrad told DI on Monday.
If it does go ahead as planned, the rollover will see ICANN change the key-signing key that acts as the trust anchor for the whole DNSSEC-using internet, for the first time since DNSSEC came online in 2010.
It’s been delayed since last October after it emerged that misconfigurations elsewhere in the DNS cloud could see potentially millions of internet users see glitches when the key is rolled.
Ever since then, ICANN and others have been trying to figure out how many people could be adversely affected by the change, and to reduce that number to the greatest extent possible.
The impact has been tricky to estimate due to patchy data.
While it’s been possible to determine a number of resolvers — about 8,000 — that definitely are poorly configured, that only represents a subset of the total number. It’s also been hard to map that to endpoints due to “resolvers behind resolvers behind resolvers”, Conrad said.
“The problem here is that it’s sort of a subjective evaluation,” he said. “We can’t rely on the data were seeing. We’re seeing the resolvers but we’re not seeing the users behind the resolvers.”
Some say that the roll is still too risky to carry out without better visibility into the potential impact, but others say that more delays would lead to more networks and devices becoming DNSSEC-compatible, potentially leading to even greater problems after the eventual rollover.
ICANN knows of about 8,000 resolver IP addresses that are likely to stop working properly after the rollover, because they only support the current KSK, but that’s only counting resolvers that automatically report their status to the root using a relatively new internet standard. There’s a blind spot concerning resolvers that do not have that feature turned on.
ICANN has also had difficulty reaching out to the network operators behind these resolvers, with good contact information apparently only available for about a quarter of the affected IP addresses, Conrad said.
Right now, the best data available suggests that 0.05% of the internet’s population could see access issues after the October 11 rollover, according to Conrad.
That’s about two million people, but it’s 10 times fewer people than the 0.5% acceptable collateral damage threshold outlined in ICANN’s rollover plan.
The 0.05% number comes from research by APNIC, which used Google’s advertising system to place “zero-pixel ads” to check whether individual user endpoints were using compatible resolvers or not.
If problems do emerge October 11 the temporary solution is apparently quite quick to implement — network operators can simply turn off DNSSEC, assuming they know that’s what they’re supposed to do.
But still, if a million or two internet users could have their day ruined by the rollover, why do it at all?
It’s not as if the KSK is in any danger of being cracked any time soon. Conrad explained that a successful brute-force attack on the 2048-bit RSA key would take longer than the lifetime of the universe using current technology.
Rather, the practice of rolling the key every five years is to get network operators and developers accustomed to the idea that the KSK is not a permanent fixture that can be hard-coded into their systems, Conrad said.
It’s a problem comparable to new gTLD name collisions or the Y2K problem, instances where developers respectively hard-coded assumptions about valid TLDs or the century into their software.
ICANN has already been reaching out to the managers of open-source projects on repositories such as Github that have been seen to hard-code the current KSK into their software, Conrad said.
Separately, Wes Hardaker at the University of Southern California Information Sciences Institute discovered that a popular VPN client was misconfigured. Outreach to the developer saw the problem fixed, reducing the number of users who will be affected by the roll.
“What we’re trying to avoid is having these keys hardwired into firmware, so that that it would never be changeable,” he said. “The idea is if you exercise the infrastructure frequently enough, people will know the that the key is not permanent configuration, it’s not something embedded in concrete.”
One change that ICANN may want to make in future is to change the algorithm used to generate the KSK.
Right now it’s using RSA, but Conrad said it has downsides such as rather large signature size, which leads to heavier DNSSEC traffic. By switching to elliptical curve cryptography, signatures could be reduced by “orders of magnitude”, leading to a more efficient and slimline DNS infrastructure, Conrad said.
Last week, ICANN’s Root Server Stability Advisory Committee issued an advisory (pdf) that essentially gave ICANN the all-clear to go ahead with the roll.
The influential Security and Stability Advisory Committee has yet to issue its own advisory, however, despite being asked to do so by August 10.
Could SSAC be more cautious in its advice? We’ll have to wait and see, but perhaps not too long; the current plan is for the ICANN board to consider whether to go ahead with the roll during its three-day Brussels retreat, which starts September 14.

Digital archery ruled out for next new gTLD round

The oft-mocked “digital archery” system will not be making a return when ICANN finally starts taking more new gTLD applications.
That’s the current thinking of the ICANN community working group looking at subsequent application procedures.
Readers with long memories may recall digital archery as a hack for Californian gambling laws that ICANN org pressed for in 2012 as a way to form its 1,930 applications into an orderly queue for processing.
The idea was that applicants would fire off a bit of data to an ICANN site at a predetermined time and the applicants whose packets arrived closet to the target time, measured by the millisecond, would receive priority in the queue.
It was a bit like drop-catching, and the concept advanced to the stage where companies skilled in such things were offering digital archery services.
But after ICANN changed CEOs later that year, it turned out gambling wasn’t as illegal in California as former management thought it was. The org got itself a license to run a one-off lottery and sold tickets for $100 per application.
That’s now the preferred method for ordering the queue for the next rounds of applications, whenever those may be, according to last week’s Initial Report on the New gTLD Subsequent Procedures Policy Development Process.
Unlike 2012, the WG is proposing that portfolio applicants should be able to swap around their priority numbers according to their commercial interests.
So, if Donuts gets priority #1 for .crappy and #4,000 for .awesome, it would be able to switch priorities to get the better string evaluated earlier.
The WG is also not convinced that internationalized domain names, which received automatic priority in 2012, should get the same preferential treatment this time around.
That’s one of several questions it poses for the community in its public comment period.
While a better place in the evaluation queue had time-to-market advantages in 2012 — Donuts’ .guru sold a tonne of domains largely due to its first-mover status — that’s probably not going to be as big a deal next time around due to domainer skepticism about new gTLDs.