Second DNSSEC screw-up takes down Aussie web sites

.au domains failed to resolve for many internet users for almost an hour on Monday, after the registry operator messed up a DNSSEC update.

ccTLD overseer auDA said the issue was caused by a “key re-signing process that generated an incorrect record”. Users on ISPs that strictly enforce DNSSEC would have returned not-found errors for .au domains during the outage.

.au’s technical back-end is managed by Identity Digital, which reportedly said that the outage lasted from 0005 UTC until 0052 UTC.

With over four million domains, .au is I believe the largest TLD zone to fall victim to DNSSEC-related downtime, but it’s not the first time it has happened to the domain.

In March 2022, thousands of .au domains were affected by a DNSSEC snafu that lasted a few hours.

DNSSEC is meant to make the DNS more secure by reducing the risk of man-in-the-middle attacks, but it’s appears to be easy to screw up, judging by a list of TLD outages. Just this year, Mexico, New Zealand and Venezuela have also suffered downtime.

DNSSEC claims another ccTLD victim

A botched DNSSEC upgrade has been fingered as the source of an outage that made .na domain names inaccessible last Tuesday.

Reports and archived DNS records show that names in the Namibian ccTLD suffered as many as 12 hours of downtime following the glitch, which has been blamed on human error.

When DNSSEC-signed domains, including TLDs, are unable to establish a cryptographic chain of trust, anyone using a DNSSEC-compatible resolver will be unable to access web sites or emails of affected domains.

Namibian Network Information Center boss Eberhard Lisse, talking to The Namibian newspaper, blamed the downtime on an unspecified upstream provider pushing an algorithm upgrade “without all prerequisite steps having been completed”.

It’s the second DNSSEC incident to hit .na following a July 2019 glitch, and one of dozens to affect TLDs since the technology started to become more broadly adopted.

Another DNSSEC screw-up takes down thousands of .au domains

Australia’s ccTLD has become the latest to see a widespread outage that appears to be the result of a DNSSEC misconfiguration.

A reported 15,000 .au domains were affected, though some suspect it could have been more.

Registry overseer auDA said on Twitter that .au “experienced an error” that affected a “small number of domains” and that an investigation was underway.

Earlier today the .au domain experienced an error. We regret that a small number of domains were affected but are pleased to report the .au is back up and operating. An investigation as to the cause of this issue is underway.

— auDA (@auda) March 22, 2022

Donuts subsidiary Afilias, which runs the back-end for .au’s more that 3.4 million domains, has yet to publicly comment.

Network operators and DNS experts took to social media and mailing lists to observe that .au’s DNSSEC was broken, though it appears the problem was fixed rather quickly.

(For those following along from home, .AU DNSSEC key management was broken from 05:48 – 06:18 UTC. Quick repair by AuDA.)https://t.co/jmHAbZdWKo https://t.co/SzOi9U7Kwi

— Bill Woodcock (@woodyatpch) March 22, 2022

DNSSEC creates a chain of cryptographic keys all the way to the DNS root, and when that chain is broken by a misconfiguration such as a missing key, most DNSSEC-enabled resolvers treat the affected domains as if they simply don’t exist.

That means services such as web sites and email addresses stop working until the chain is reestablished. People not using DNSSEC resolvers wouldn’t have seen a problem.

It’s the third TLD to experience a significant outage due to DNSSEC in the last six weeks.

In February, thousands of domains in Sweden’s .se went dark for hours, and Fiji’s entire .fj zone disappeared for DNSSEC users less than two weeks ago.

The outage comes at a particularly unfortunate time in terms of public relations for auDA, which on Thursday will start making direct second-level .au registrations available for the first time.

It’s not immediately clear whether the DNSSEC fluff is related to the SLD launch.

DNSSEC claims another victim as entire TLD disappears

A country’s top-level domain disappeared from the internet for many people yesterday, apparently due to a DNSSEC key rollover gone wrong.

All domains in Fiji’s ccTLD, .fj, stopped resolving for anyone behind a strict DNSSEC resolver in the early hours of the morning UTC, afternoon local time, and stayed down for over 12 hours.

Some domains may still be affected due to caching, according to the registry and others.

The University of the South Pacific, which runs the domain, said that it had to contact ICANN’s IANA people to get the problem fixed, which took a while because it had to wait for IANA’s US-based support desk to wake up.

IANA head Kim Davies said that in fact its support runs 24/7 and in this case IANA took Fiji’s call at 2.47am local time.

Analyses on mailing lists and by Cloudflare immediately pointed to a misconfiguration in the country’s DNSSEC.

It seems Fiji rolled one of its keys for the first time and messed it up, meaning its zone was signed with a non-existent key.

Resolvers that implement DNSSEC strictly view such misconfigurations as a potential attack and nix the entire affected zone.

It happens surprisingly often, though not usually at the TLD level. That said, a similar problem hit thousands of Sweden’s .se domains, despite the registry having a decade’s more DNSSEC experience than Fiji, last month.

Domain Incite had a similar problem recently when its registrar carried on publishing DNSSEC information for the domain long after I’d stopped paying for it.

UPDATE: This post was updated with comment from IANA.

Thousands of domains hit by downtime after DNSSEC error

Sweden saw thousands of domains go down for hours on Friday, after DNSSEC errors were introduced to the .se zone file.

Local ccTLD registry IIS said in a statement that around 8,000 domains had a “technical difficulty” that started around 1530 local time and lasted around seven hours:

On the afternoon of 4/2, a problem was discovered that concerned approximately 8,000 .se domains. The problem meant that services, such as email and web, that are linked to the affected domains in some cases could not be used or reached. In total, there are approximately 1.49 million .se domains, of which approximately 8,000 were affected.

During the afternoon and evening, a thorough work was done with the troubleshooting and the error could be fixed for the affected .se domains at approximately 22.25.

The problem is believed to have been caused by incorrect DNSSEC signatures being published in the .se zone file. Any machine using a DNSSEC-validating resolver would have seen the errors and flat-out refused to resolve the domain.

This is probably the key drawback of DNSSEC — typically resolvers will treat badly signed domains as if they do not exist, rather than fail over to an unsigned, but resolving, response.

Sweden is not a DNSSEC newbie — .se was the first TLD to deploy the technology, all the way back in 2005, with services for domain holders coming a couple of years later.

DNS genius and ICANN key-holder Dan Kaminsky dies at 42

Security researcher Dan Kaminsky, best known for uncovering the so-called “Kaminsky Bug” DNS vulnerability, has reportedly died at the age of 42.

It has been widely reported that Kaminsky’s niece confirmed his death from serious complications from his longstanding diabetes.

On Twitter, she rebutted emerging conspiracy theories that his death was linked to the coronavirus vaccine, which he had received April 12, saying her uncle would “laugh” at such views.

During his career as a white-hat hacker, Kaminsky worked for companies including Cisco, Avaya, and IOActive.

He occasionally spoke at ICANN meetings on security issues, and was since 2010 one of IANA’s seven Recovery Key Share Holders, individuals trusted to hold part of a cryptographic key that would be used to reboot root zone DNSSEC in the case of a massive disaster.

But he was best known for his 2008 discovery of a fundamental flaw in the DNS protocol that allowed cache poisoning, and therefore serious man-in-the middle attacks, across millions of name servers worldwide. He worked with DNS software vendors in private to help them with their patches before the problem was publicly disclosed.

His discoveries led in part to the ongoing push for DNSSEC deployment across the internet.

The vulnerability received widespread attention, even in the mainstream media, and quickly came to bear his name.

For me, my standout memory of Kaminsky is one of his series of annual “Black Ops” talks, at the Defcon 12 conference in Las Vegas in 2004, during which he demonstrated to a rapt audience of hackers how it was possible to stream live radio by caching small chunks of audio data in the TXT fields of DNS records and using DNS queries to quickly retrieve and play them in sequence.

As well as being a bit of a DNS genius, he knew how to work a stage: the crowd went mental and I grabbed him for an interview soon after his talk was over.

His death at such a young age is a big loss for the security community.

Fraud checks coming to .ch as SWITCH renews contract

Swiss ccTLD registry SWITCH has agreed to implement new security measures as part of its contract renewal with the government.

The company said Friday that it has extended its contract to run .ch names with the telecoms regulator OFCOM for five more years, bring it up to December 2026.

But as part of the renewal, SWITCH has agreed to “speed up the adoption and implementation of technical security standards”.

This will involved financial incentives for registrars to adopt DNSSEC, the registry said.

It will also introduce measures to combat fraud at the point of registration, with SWITCH saying “in the event of suspected fraudulent intent, newly registered domain names can be used only after an identity check.”

The policy appears similar to those at other ccTLDs, including .uk, where new regs are flagged under certain circumstances (such as containing coronavirus-related terms) and cannot resolve until further checks are carried out.

Coronavirus could cause “high risk of widespread outages”, ICANN says

There’s a “high risk of widespread outages” in the DNS if ICANN can’t get enough people in the same room for its next root DNSSEC ceremony because of the coronavirus pandemic.

That’s according to ICANN’s own board of directors, which yesterday published a contingency plan that — in the worst case scenario — could see parts of the internet come to a screeching halt in July.

The problem is with the elaborate “ceremonies” that ICANN and its IANA/PTI unit uses to make sure the internet can support DNSSEC — the secure version of the DNS protocol — all the way from the root servers down.

Every quarter, ICANN, Verisign and a select few “Trusted Community Representatives” from all over the world meet in person at one of two secure US-based facilities to generate the public Zone Signing Keys for the root.

In addition to the complex cryptographic stuff happening in the computers, there’s a shedload of physical security, such as retinal scans, PIN-based locks, and reinforced walls.

And the “secret key-holders”, memorably fictionalized in a US spy drama a few years ago, actually have physical keys that they must bring to these ceremonies.

The events are broadcast live and archived on YouTube, where they typically get anything from a few hundred to a few thousand views.

Obviously, with the key-holders dotted all over the globe and most under some form of coronavirus-related lockdown, getting a quorum into the same facility at the same time — originally, Culpeper, Virginia on April 23 — isn’t going to be possible.

So IANA has made the decision to instead move the ceremony to the facility in El Segundo, California, within easy driving distance of ICANN’s headquarters, and have it carried out almost entirely by ICANN staff, wrapped in personal protective equipment and keeping their distance from each other.

The TCRs for El Segundo live in Mauritius, Spain, Russia, Tanzania, Uruguay and on the east-coast of the US, according to ICANN.

Four of these key-holders have mailed their keys to different IANA staff “wrapped in opaque material” and sealed in “tamper-evident bags”. These IANA employees will stand in for the TCRs, who will be watching remotely to verify that nothing fishy is going on.

Verisign and the independent auditors will also be watching remotely.

That’s the current plan, anyway, and I’ve no reason to believe it won’t go ahead, but ICANN’s new contingency plans do provide four alternatives.

It’s already discarded the first two options, so if the current, third, plan for the ceremony can’t go ahead before June 19 for some reason, all that would be left is the nuclear option.

Option D: Suspend signing of the DNS root zone

This is the final option if there is no conceivable way to activate the KSK and perform signing operations. There would need to be a massive education campaign at short notice to have resolver operators disable DNSSEC validation. There is a high risk of widespread outages as it is not possible to ensure global implementation, and high risk this will fatally compromise trust in DNSSEC in general as a technology.

This is considered highly unlikely, but nonetheless the final option. Without exercising the option, in the absence of a successful key signing ceremony, DNSSEC validation would be unsuccessful starting in July 2020.

The reason for this scenario is that DNSSEC keys have a finite time-to-live and after that period expires they stop functioning, which means anyone validating DNSSEC on their network may well stop resolving the signed zones.

ICANN typically generates the keys one quarter in advance, so the current key expires at the start of July.

However, the planned April 23 ceremony will generate three quarters worth of keys in advance, so the root should be good until the end of March 2021, assuming everything goes according to plan.

Clearly, the idea that half the planet might be on the verge of lockdown wasn’t taken into consideration on February 12, the last ceremony, when ICANN’s biggest problem was that it couldn’t get into one of its safes.

If you’re interested in more about the ceremony and the coronavirus-related changes, info can be found here.

Root servers whacked after crypto change

The DNS root servers came under accidental attack from name servers across the internet following ICANN’s recent changes to their cryptographic master keys, according to Verisign.
The company, which runs the A and J root servers, said it saw requests for DNSSEC data at the root increase from 15 million a day in October to 1.15 billion a day a week ago.
The cause was the October 11 root Key Signing Key rollover, the first change ICANN had made to the “trust anchor” of DNSSEC since it came online at the root in 2010.
The KSK rollover saw ICANN change the cryptographic keys that rest at the very top of the DNSSEC hierarchy.
The move was controversial. ICANN delayed it for a year after learning about possible disruption at internet endpoints. Its Security and Stability Advisory Committee and even its own board were not unanimous that the roll should go ahead.
But the warnings were largely about the impact on internet users, rather than on the root servers themselves, and the impact was minimal.
Verisign is now saying that requests to its roots for DNSSEC key data increased from 15 million per day to 75 million per day, a five-fold increase, almost overnight.
It was not until January, when the old KSK was marked as “revoked”, did the seriously mahooosive traffic growth begin, however. Verisign’s distinguished engineer Duane Wessels wrote:

Everyone involved expected this to be a non-event. However, we instead saw an even bigger increase in DNSKEY queries coming from a population of root server clients. As of March 21, 2019, Verisign’s root name servers receive about 1.15 billion DNSKEY queries per day, which is 75 times higher than pre-rollover levels and nearly 7 percent of our total steady state query traffic.

Worryingly, the traffic only seemed to be increasing, until March 22, when the revoked key was removed from the root entirely.
Wessels wrote that while the root operators are still investigating, “it would seem that the presence of the revoked key in the zone triggered some unexpected behavior in a population of validating resolvers.”
The root operators hope to have answers in the coming weeks, he wrote.
The next KSK rollover is not expected for years, and the root traffic is now returning to normal levels, so there’s no urgency.

The internet is still working after KSK roll

The first-ever change to the security keys at the top of the DNS tree appears to have been a non-event.
While ICANN received reports of some disruptions after last Thursday’s KSK rollover, the impact appears to have fallen short of the millions of users that had been speculated.
ICANN said yesterday:

After evaluation of the available data, there does not appear to be a significant number of Internet end-users who have been persistently and negatively impacted by the changing of the key.
The few issues that have arisen appear to have been quickly mitigated and none suggested a systemic failure that would approach the threshold (as defined by the ICANN community) to initiate a reversal of the roll. In that context, it appears the rollover to the new Key Signing Key, known as KSK 2017, has been a success.

The KSK, also sometimes called the “trust anchor”, is the ultimate cryptographic key in the chain that secures all DNSSEC queries on the internet.
October 11 was the first time it had been changed since the first version came online in 2010.
While changing the key was broadly considered sound security practice, the roll was delayed by a year after it was discovered that potentially millions of endpoints were using DNS resolvers not properly configured to use the 2017 key.
After much research, outreach and gnashing of teeth, it was decided that the risk posed by rolling the KSK now fell within acceptable parameters of collateral damage.
Experts from the likes of Google and Verisign, and one ICANN director, had urged caution and said perhaps the roll should be delayed further while more data was gathered.
But they were in the minority, ICANN went ahead anyway, and it seems their fears have not come to pass.
The KSK is now likely to be rolled regularly — it could be as little as once every five years, or more frequently.
It also gives ICANN the opportunity to eventually update the system to swap out its current RSA keys for keys based on elliptical curve cryptography, which could reduce the traffic load on the DNS as a whole.