Latest news of the domain name industry

Recent Posts

Another DNSSEC screw-up takes down thousands of .au domains

Kevin Murphy, March 22, 2022, Domain Registries

Australia’s ccTLD has become the latest to see a widespread outage that appears to be the result of a DNSSEC misconfiguration.

A reported 15,000 .au domains were affected, though some suspect it could have been more.

Registry overseer auDA said on Twitter that .au “experienced an error” that affected a “small number of domains” and that an investigation was underway.

Donuts subsidiary Afilias, which runs the back-end for .au’s more that 3.4 million domains, has yet to publicly comment.

Network operators and DNS experts took to social media and mailing lists to observe that .au’s DNSSEC was broken, though it appears the problem was fixed rather quickly.

DNSSEC creates a chain of cryptographic keys all the way to the DNS root, and when that chain is broken by a misconfiguration such as a missing key, most DNSSEC-enabled resolvers treat the affected domains as if they simply don’t exist.

That means services such as web sites and email addresses stop working until the chain is reestablished. People not using DNSSEC resolvers wouldn’t have seen a problem.

It’s the third TLD to experience a significant outage due to DNSSEC in the last six weeks.

In February, thousands of domains in Sweden’s .se went dark for hours, and Fiji’s entire .fj zone disappeared for DNSSEC users less than two weeks ago.

The outage comes at a particularly unfortunate time in terms of public relations for auDA, which on Thursday will start making direct second-level .au registrations available for the first time.

It’s not immediately clear whether the DNSSEC fluff is related to the SLD launch.

DNSSEC claims another victim as entire TLD disappears

Kevin Murphy, March 9, 2022, Domain Tech

A country’s top-level domain disappeared from the internet for many people yesterday, apparently due to a DNSSEC key rollover gone wrong.

All domains in Fiji’s ccTLD, .fj, stopped resolving for anyone behind a strict DNSSEC resolver in the early hours of the morning UTC, afternoon local time, and stayed down for over 12 hours.

Some domains may still be affected due to caching, according to the registry and others.

The University of the South Pacific, which runs the domain, said that it had to contact ICANN’s IANA people to get the problem fixed, which took a while because it had to wait for IANA’s US-based support desk to wake up.

IANA head Kim Davies said that in fact its support runs 24/7 and in this case IANA took Fiji’s call at 2.47am local time.

Analyses on mailing lists and by Cloudflare immediately pointed to a misconfiguration in the country’s DNSSEC.

It seems Fiji rolled one of its keys for the first time and messed it up, meaning its zone was signed with a non-existent key.

Resolvers that implement DNSSEC strictly view such misconfigurations as a potential attack and nix the entire affected zone.

It happens surprisingly often, though not usually at the TLD level. That said, a similar problem hit thousands of Sweden’s .se domains, despite the registry having a decade’s more DNSSEC experience than Fiji, last month.

Domain Incite had a similar problem recently when its registrar carried on publishing DNSSEC information for the domain long after I’d stopped paying for it.

UPDATE: This post was updated with comment from IANA.

Thousands of domains hit by downtime after DNSSEC error

Kevin Murphy, February 7, 2022, Domain Tech

Sweden saw thousands of domains go down for hours on Friday, after DNSSEC errors were introduced to the .se zone file.

Local ccTLD registry IIS said in a statement that around 8,000 domains had a “technical difficulty” that started around 1530 local time and lasted around seven hours:

On the afternoon of 4/2, a problem was discovered that concerned approximately 8,000 .se domains. The problem meant that services, such as email and web, that are linked to the affected domains in some cases could not be used or reached. In total, there are approximately 1.49 million .se domains, of which approximately 8,000 were affected.

During the afternoon and evening, a thorough work was done with the troubleshooting and the error could be fixed for the affected .se domains at approximately 22.25.

The problem is believed to have been caused by incorrect DNSSEC signatures being published in the .se zone file. Any machine using a DNSSEC-validating resolver would have seen the errors and flat-out refused to resolve the domain.

This is probably the key drawback of DNSSEC — typically resolvers will treat badly signed domains as if they do not exist, rather than fail over to an unsigned, but resolving, response.

Sweden is not a DNSSEC newbie — .se was the first TLD to deploy the technology, all the way back in 2005, with services for domain holders coming a couple of years later.

DNS genius and ICANN key-holder Dan Kaminsky dies at 42

Kevin Murphy, April 27, 2021, Domain Tech

Security researcher Dan Kaminsky, best known for uncovering the so-called “Kaminsky Bug” DNS vulnerability, has reportedly died at the age of 42.

It has been widely reported that Kaminsky’s niece confirmed his death from serious complications from his longstanding diabetes.

On Twitter, she rebutted emerging conspiracy theories that his death was linked to the coronavirus vaccine, which he had received April 12, saying her uncle would “laugh” at such views.

During his career as a white-hat hacker, Kaminsky worked for companies including Cisco, Avaya, and IOActive.

He occasionally spoke at ICANN meetings on security issues, and was since 2010 one of IANA’s seven Recovery Key Share Holders, individuals trusted to hold part of a cryptographic key that would be used to reboot root zone DNSSEC in the case of a massive disaster.

But he was best known for his 2008 discovery of a fundamental flaw in the DNS protocol that allowed cache poisoning, and therefore serious man-in-the middle attacks, across millions of name servers worldwide. He worked with DNS software vendors in private to help them with their patches before the problem was publicly disclosed.

His discoveries led in part to the ongoing push for DNSSEC deployment across the internet.

The vulnerability received widespread attention, even in the mainstream media, and quickly came to bear his name.

For me, my standout memory of Kaminsky is one of his series of annual “Black Ops” talks, at the Defcon 12 conference in Las Vegas in 2004, during which he demonstrated to a rapt audience of hackers how it was possible to stream live radio by caching small chunks of audio data in the TXT fields of DNS records and using DNS queries to quickly retrieve and play them in sequence.

As well as being a bit of a DNS genius, he knew how to work a stage: the crowd went mental and I grabbed him for an interview soon after his talk was over.

His death at such a young age is a big loss for the security community.

Fraud checks coming to .ch as SWITCH renews contract

Kevin Murphy, December 15, 2020, Domain Registries

Swiss ccTLD registry SWITCH has agreed to implement new security measures as part of its contract renewal with the government.

The company said Friday that it has extended its contract to run .ch names with the telecoms regulator OFCOM for five more years, bring it up to December 2026.

But as part of the renewal, SWITCH has agreed to “speed up the adoption and implementation of technical security standards”.

This will involved financial incentives for registrars to adopt DNSSEC, the registry said.

It will also introduce measures to combat fraud at the point of registration, with SWITCH saying “in the event of suspected fraudulent intent, newly registered domain names can be used only after an identity check.”

The policy appears similar to those at other ccTLDs, including .uk, where new regs are flagged under certain circumstances (such as containing coronavirus-related terms) and cannot resolve until further checks are carried out.

Coronavirus could cause “high risk of widespread outages”, ICANN says

Kevin Murphy, April 21, 2020, Domain Tech

There’s a “high risk of widespread outages” in the DNS if ICANN can’t get enough people in the same room for its next root DNSSEC ceremony because of the coronavirus pandemic.

That’s according to ICANN’s own board of directors, which yesterday published a contingency plan that — in the worst case scenario — could see parts of the internet come to a screeching halt in July.

The problem is with the elaborate “ceremonies” that ICANN and its IANA/PTI unit uses to make sure the internet can support DNSSEC — the secure version of the DNS protocol — all the way from the root servers down.

Every quarter, ICANN, Verisign and a select few “Trusted Community Representatives” from all over the world meet in person at one of two secure US-based facilities to generate the public Zone Signing Keys for the root.

In addition to the complex cryptographic stuff happening in the computers, there’s a shedload of physical security, such as retinal scans, PIN-based locks, and reinforced walls.

And the “secret key-holders”, memorably fictionalized in a US spy drama a few years ago, actually have physical keys that they must bring to these ceremonies.

The events are broadcast live and archived on YouTube, where they typically get anything from a few hundred to a few thousand views.

Obviously, with the key-holders dotted all over the globe and most under some form of coronavirus-related lockdown, getting a quorum into the same facility at the same time — originally, Culpeper, Virginia on April 23 — isn’t going to be possible.

So IANA has made the decision to instead move the ceremony to the facility in El Segundo, California, within easy driving distance of ICANN’s headquarters, and have it carried out almost entirely by ICANN staff, wrapped in personal protective equipment and keeping their distance from each other.

The TCRs for El Segundo live in Mauritius, Spain, Russia, Tanzania, Uruguay and on the east-coast of the US, according to ICANN.

Four of these key-holders have mailed their keys to different IANA staff “wrapped in opaque material” and sealed in “tamper-evident bags”. These IANA employees will stand in for the TCRs, who will be watching remotely to verify that nothing fishy is going on.

Verisign and the independent auditors will also be watching remotely.

That’s the current plan, anyway, and I’ve no reason to believe it won’t go ahead, but ICANN’s new contingency plans do provide four alternatives.

It’s already discarded the first two options, so if the current, third, plan for the ceremony can’t go ahead before June 19 for some reason, all that would be left is the nuclear option.

Option D: Suspend signing of the DNS root zone

This is the final option if there is no conceivable way to activate the KSK and perform signing operations. There would need to be a massive education campaign at short notice to have resolver operators disable DNSSEC validation. There is a high risk of widespread outages as it is not possible to ensure global implementation, and high risk this will fatally compromise trust in DNSSEC in general as a technology.

This is considered highly unlikely, but nonetheless the final option. Without exercising the option, in the absence of a successful key signing ceremony, DNSSEC validation would be unsuccessful starting in July 2020.

The reason for this scenario is that DNSSEC keys have a finite time-to-live and after that period expires they stop functioning, which means anyone validating DNSSEC on their network may well stop resolving the signed zones.

ICANN typically generates the keys one quarter in advance, so the current key expires at the start of July.

However, the planned April 23 ceremony will generate three quarters worth of keys in advance, so the root should be good until the end of March 2021, assuming everything goes according to plan.

Clearly, the idea that half the planet might be on the verge of lockdown wasn’t taken into consideration on February 12, the last ceremony, when ICANN’s biggest problem was that it couldn’t get into one of its safes.

If you’re interested in more about the ceremony and the coronavirus-related changes, info can be found here.

Root servers whacked after crypto change

Kevin Murphy, March 27, 2019, Domain Tech

The DNS root servers came under accidental attack from name servers across the internet following ICANN’s recent changes to their cryptographic master keys, according to Verisign.
The company, which runs the A and J root servers, said it saw requests for DNSSEC data at the root increase from 15 million a day in October to 1.15 billion a day a week ago.
The cause was the October 11 root Key Signing Key rollover, the first change ICANN had made to the “trust anchor” of DNSSEC since it came online at the root in 2010.
The KSK rollover saw ICANN change the cryptographic keys that rest at the very top of the DNSSEC hierarchy.
The move was controversial. ICANN delayed it for a year after learning about possible disruption at internet endpoints. Its Security and Stability Advisory Committee and even its own board were not unanimous that the roll should go ahead.
But the warnings were largely about the impact on internet users, rather than on the root servers themselves, and the impact was minimal.
Verisign is now saying that requests to its roots for DNSSEC key data increased from 15 million per day to 75 million per day, a five-fold increase, almost overnight.
It was not until January, when the old KSK was marked as “revoked”, did the seriously mahooosive traffic growth begin, however. Verisign’s distinguished engineer Duane Wessels wrote:

Everyone involved expected this to be a non-event. However, we instead saw an even bigger increase in DNSKEY queries coming from a population of root server clients. As of March 21, 2019, Verisign’s root name servers receive about 1.15 billion DNSKEY queries per day, which is 75 times higher than pre-rollover levels and nearly 7 percent of our total steady state query traffic.

Worryingly, the traffic only seemed to be increasing, until March 22, when the revoked key was removed from the root entirely.
Wessels wrote that while the root operators are still investigating, “it would seem that the presence of the revoked key in the zone triggered some unexpected behavior in a population of validating resolvers.”
The root operators hope to have answers in the coming weeks, he wrote.
The next KSK rollover is not expected for years, and the root traffic is now returning to normal levels, so there’s no urgency.

The internet is still working after KSK roll

Kevin Murphy, October 16, 2018, Domain Tech

The first-ever change to the security keys at the top of the DNS tree appears to have been a non-event.
While ICANN received reports of some disruptions after last Thursday’s KSK rollover, the impact appears to have fallen short of the millions of users that had been speculated.
ICANN said yesterday:

After evaluation of the available data, there does not appear to be a significant number of Internet end-users who have been persistently and negatively impacted by the changing of the key.
The few issues that have arisen appear to have been quickly mitigated and none suggested a systemic failure that would approach the threshold (as defined by the ICANN community) to initiate a reversal of the roll. In that context, it appears the rollover to the new Key Signing Key, known as KSK 2017, has been a success.

The KSK, also sometimes called the “trust anchor”, is the ultimate cryptographic key in the chain that secures all DNSSEC queries on the internet.
October 11 was the first time it had been changed since the first version came online in 2010.
While changing the key was broadly considered sound security practice, the roll was delayed by a year after it was discovered that potentially millions of endpoints were using DNS resolvers not properly configured to use the 2017 key.
After much research, outreach and gnashing of teeth, it was decided that the risk posed by rolling the KSK now fell within acceptable parameters of collateral damage.
Experts from the likes of Google and Verisign, and one ICANN director, had urged caution and said perhaps the roll should be delayed further while more data was gathered.
But they were in the minority, ICANN went ahead anyway, and it seems their fears have not come to pass.
The KSK is now likely to be rolled regularly — it could be as little as once every five years, or more frequently.
It also gives ICANN the opportunity to eventually update the system to swap out its current RSA keys for keys based on elliptical curve cryptography, which could reduce the traffic load on the DNS as a whole.

KSK vote was NOT unanimous

Kevin Murphy, September 18, 2018, Domain Policy

ICANN’s board of directors on Sunday voted to approve the forthcoming security key change at the DNS root, but there was some dissent.
Director Avri Doria, a Nominating Committee appointee, said today that she provided the lone vote against the DNSSEC KSK rollover, which is expected to cause temporary internet access problems for potentially a couple million people next month.
I understand there was also a single abstention to Sunday’s vote.
Doria has released a dissenting statement, in which she said the absence of an external, peer-reviewed study of the risks could prove a problem.

The greatest risk is that out of the millions that will fail after the roll over, some that are serious and may even be critical, may occur; if this happens the lack of peer reviewed studies may be a liability for ICANN, perhaps not legal, but in terms of our reputation as protectors of the stability & security of internet system of names.

She added that she was concerned about the extent that the public has been notified of the rollover plan, and questioned whether the current risk mitigation plan is sufficient.
Doria said she found comments filed by Verisign (pdf) particularly informative to her eventual vote, as well as comments from the At-Large Advisory Committee (pdf), Business Constituency (pdf) and Registries Stakeholder Group (pdf).
These groups had called for more study and data, better outreach, more clearly defined success/failure benchmarks, and more delay.
Doria noted in her dissenting statement that the ICANN board did not have a chance to quiz any of the minority of the members of the Security and Stability Advisory Committee who had called for further delay.
The board’s resolution, apparently arrived at after two hours of formal in-person discussions in Brussels at the weekend, is expected to be published shortly.
The rollover, which has already been delayed a year, is now scheduled to go ahead October 11.
Any impact is expected to be felt within a couple of days, as the change ripples out across the DNS.
ICANN says that any network operator impacted by the change has a simple fix: turn off DNSSEC. Then, if they want, they can update their keys and turn it back on again.

Set buttocks to clench! ICANN approves risky KSK rollover

Kevin Murphy, September 17, 2018, Domain Policy

ICANN has approved the first rollover of the domain name system’s master security key, setting the clock ticking on a change that could cause internet access issues for millions.
The so-called KSK rollover, when ICANN deletes the key-signing key that has been used as the trust anchor for the DNSSEC ecosystem since 2011 and replaces it with the new one — will now go ahead as planned on October 11.
The decision was made yesterday at the ICANN board of directors’ retreat in Brussels.
ICANN chief technology officer David Conrad posted this to an ICANN mailing list this morning:

The Board voted to approve the resolution for ICANN org to move forward with the revised KSK rollover plan. So barring unforeseen circumstances, the KSK-2017-signed ZSK will be used to sign the root zone on 11 October 2018.

The rollover was due to happen October 11 last year, but ICANN delayed it when it emerged that many DNS resolvers weren’t yet configured to use the new key.
That’s still a problem, and nobody knows for sure how many endpoints will stop functioning properly when the new KSK goes solo.
While most experts weighing in on the rollover, including Conrad, agreed that the risk of more delay outweighed the risk of rolling now, that feeling was not unanimous.
Five members of the 22-member Security and Stability Advisory Committee — including top guys from Google and Verisign — last month dissented from the majority view and said ICANN should delay again.
The question now is not whether internet users will see a disruption in the days following October 11, but how many users will be affected and how serious their disruptions will be.
Based on current information, as many as two million internet users could be affected.
ICANN is likely to take flak for even relatively minor disruptions, but the alternative was to continue with the delays and risk an even bigger impact, and even more flak, in future.
The text of ICANN’s resolution and the rationale behind it will be published in the next day or so.