DNSSEC claims another registry victim

DNSSEC is widely touted as a crucial security upgrade for the domain name system, but as so often is the case with security measures, it can also cause serious problems.

DENIC, the registry operator for Germany’s .de, became the latest victim of a DNSSEC screw-up earlier this week, when a botched key rollover led to the entire ccTLD being flagged as bogus by many ISPs.

That’s no small issue — .de is the internet’s third-largest TLD after .com and .cn, with some 17.9 million domains under management.

The company said that the outage began at 2157 UTC on Tuesday night, caused by “incorrect DNSSEC signatures” being deployed during a “routine, scheduled key rollover”, and was fixed by 0115 the following day.

Bad sigs means that any attempt to resolve a .de domain would fail, but only on networks where the DNS resolvers strictly enforce DNSSEC validation. That includes major free resolver networks such as those run by Google and Cloudflare.

The workaround is to temporarily stop enforcing DNSSEC validation, which can be done using a mechanism baked into the IETF standard. Cloudflare has a fairly comprehensive technical description of how it responded here.

DENIC said it has temporarily suspended is key rollover schedule while it figures out what went wrong.

The registry is far from alone when it comes to DNSSEC snafus. Literally dozens of ccTLDs and gTLDs have experiences outages related to the protocol since it was added to the DNS root in 2010.

KeyTrap ‘the most devastating vulnerability ever found in DNSSEC’

A security vulnerability in the DNSSEC standard that could crash DNS resolution in software such as BIND and services such as Cloudflare and Google Public DNS has been called “the most devastating vulnerability ever found in DNSSEC”.

Named KeyTrap, it enables attackers to overwhelm a DNS resolver’s CPU for as long as 16 hours, forcing it to process up to two million times its usual load, using a single malicious DNS packet, making for a potentially crippling denial-of-service attack.

The flaw was discovered last year by Elias Heftrig, Haya Schulmann, Niklas Vogel, and Michael Waidner from ATHENE, the German National Research Center for Applied Cybersecurity, and publicly disclosed last week after vendors were given time to develop and deploy patches.

While KeyTrap has been present in DNS software for almost a quarter-century, the researchers are not aware of it being exploited in the wild.

The vulnerability is actually baked into the DNSSEC technical standards, developed in the 1990s, rather than being specific to any one implementation of the specs, according to the researchers. In fact, in order to patch the problem vendors had to break with the standard RFCs.

KeyTrap works because DNSSEC is (believe it or not) designed to avoid causing downtime when it fails, so it tries too eagerly to validate cryptographic signatures by checking all the keys available to it. Exploiting this helpfulness, an attacker could trick a resolver into eating up all its CPU resources checking huge numbers of keys.

Schulmann wrote in an article explaining the vulnerability:

Our methods show a low-resource adversary can fully attack a DNS resolver with a Denial-of-Service (DoS) for up to 16 hours with a single DNS request. Members from the 31-participant task force of major operators, vendors, and developers of DNS/DNSSEC, to which we disclosed our research, dubbed our attack ‘the most devastating vulnerability ever found in DNSSEC’.

DNSSEC is designed to mitigate the risk of DNS cache-poisoning and man-in-the-middle attacks, but because its default behavior when the crypto fails is to refuse to resolve the affected domains, it can also lead to availability problems.

It’s not uncommon for entire TLDs to fail for hours when the registry screws up a DNSSEC key rollover. The web site you’re reading right now suffered downtime a few years ago due to a DNSSEC fail at the registrar level.

The KeyTrap researchers believe about 31% of web client devices currently use DNSSEC resolvers.

Russia blames DNSSEC, not Ukraine, for internet downtime

Another ccTLD has blamed DNSSEC after seeing hours of downtime affecting its country’s biggest web services yesterday.

This time it’s Russia’s ccTLD.ru, which confirmed today that it was responsible for the widely reported outages on Tuesday, which had sparked speculation that a cyber-attack related to the war in Ukraine might be the culprit.

It was rather a DNSSEC failure that affected both .ru and the Cyrillic .рф domains, the registry said. It was related to a cryptograpghic key rollover, the registry indicated.

“After the failure was detected, the updated keys were revoked, and the functionality of the .RU zone was fully restored, which took about two hours, including the distribution of data through the DNS system,” the registry said on its web site.

“The investigation into the incident is currently ongoing, but it is already clear that the main cause of the failure was the imperfection of the software used to create the encryption keys,” it added.

The explanation was echoed by Russian government officials on social media, and it’s sadly rather plausible. DNSSEC failures at ccTLDs, and to a lesser extent gTLDs, usually related to fluffed key rollovers, are rather common.

There have been similar outages reported in the last few years in Australia (twice), Namibia, Fiji, and Sweden. And those are just the ones reported on this blog. People who track this kind of thing more closely have recorded hundreds of incidents.

Second DNSSEC screw-up takes down Aussie web sites

.au domains failed to resolve for many internet users for almost an hour on Monday, after the registry operator messed up a DNSSEC update.

ccTLD overseer auDA said the issue was caused by a “key re-signing process that generated an incorrect record”. Users on ISPs that strictly enforce DNSSEC would have returned not-found errors for .au domains during the outage.

.au’s technical back-end is managed by Identity Digital, which reportedly said that the outage lasted from 0005 UTC until 0052 UTC.

With over four million domains, .au is I believe the largest TLD zone to fall victim to DNSSEC-related downtime, but it’s not the first time it has happened to the domain.

In March 2022, thousands of .au domains were affected by a DNSSEC snafu that lasted a few hours.

DNSSEC is meant to make the DNS more secure by reducing the risk of man-in-the-middle attacks, but it’s appears to be easy to screw up, judging by a list of TLD outages. Just this year, Mexico, New Zealand and Venezuela have also suffered downtime.

DNSSEC claims another ccTLD victim

A botched DNSSEC upgrade has been fingered as the source of an outage that made .na domain names inaccessible last Tuesday.

Reports and archived DNS records show that names in the Namibian ccTLD suffered as many as 12 hours of downtime following the glitch, which has been blamed on human error.

When DNSSEC-signed domains, including TLDs, are unable to establish a cryptographic chain of trust, anyone using a DNSSEC-compatible resolver will be unable to access web sites or emails of affected domains.

Namibian Network Information Center boss Eberhard Lisse, talking to The Namibian newspaper, blamed the downtime on an unspecified upstream provider pushing an algorithm upgrade “without all prerequisite steps having been completed”.

It’s the second DNSSEC incident to hit .na following a July 2019 glitch, and one of dozens to affect TLDs since the technology started to become more broadly adopted.

Another DNSSEC screw-up takes down thousands of .au domains

Australia’s ccTLD has become the latest to see a widespread outage that appears to be the result of a DNSSEC misconfiguration.

A reported 15,000 .au domains were affected, though some suspect it could have been more.

Registry overseer auDA said on Twitter that .au “experienced an error” that affected a “small number of domains” and that an investigation was underway.

Earlier today the .au domain experienced an error. We regret that a small number of domains were affected but are pleased to report the .au is back up and operating. An investigation as to the cause of this issue is underway.

— auDA (@auda) March 22, 2022

Donuts subsidiary Afilias, which runs the back-end for .au’s more that 3.4 million domains, has yet to publicly comment.

Network operators and DNS experts took to social media and mailing lists to observe that .au’s DNSSEC was broken, though it appears the problem was fixed rather quickly.

(For those following along from home, .AU DNSSEC key management was broken from 05:48 – 06:18 UTC. Quick repair by AuDA.)https://t.co/jmHAbZdWKo https://t.co/SzOi9U7Kwi

— Bill Woodcock (@woodyatpch) March 22, 2022

DNSSEC creates a chain of cryptographic keys all the way to the DNS root, and when that chain is broken by a misconfiguration such as a missing key, most DNSSEC-enabled resolvers treat the affected domains as if they simply don’t exist.

That means services such as web sites and email addresses stop working until the chain is reestablished. People not using DNSSEC resolvers wouldn’t have seen a problem.

It’s the third TLD to experience a significant outage due to DNSSEC in the last six weeks.

In February, thousands of domains in Sweden’s .se went dark for hours, and Fiji’s entire .fj zone disappeared for DNSSEC users less than two weeks ago.

The outage comes at a particularly unfortunate time in terms of public relations for auDA, which on Thursday will start making direct second-level .au registrations available for the first time.

It’s not immediately clear whether the DNSSEC fluff is related to the SLD launch.

DNSSEC claims another victim as entire TLD disappears

A country’s top-level domain disappeared from the internet for many people yesterday, apparently due to a DNSSEC key rollover gone wrong.

All domains in Fiji’s ccTLD, .fj, stopped resolving for anyone behind a strict DNSSEC resolver in the early hours of the morning UTC, afternoon local time, and stayed down for over 12 hours.

Some domains may still be affected due to caching, according to the registry and others.

The University of the South Pacific, which runs the domain, said that it had to contact ICANN’s IANA people to get the problem fixed, which took a while because it had to wait for IANA’s US-based support desk to wake up.

IANA head Kim Davies said that in fact its support runs 24/7 and in this case IANA took Fiji’s call at 2.47am local time.

Analyses on mailing lists and by Cloudflare immediately pointed to a misconfiguration in the country’s DNSSEC.

It seems Fiji rolled one of its keys for the first time and messed it up, meaning its zone was signed with a non-existent key.

Resolvers that implement DNSSEC strictly view such misconfigurations as a potential attack and nix the entire affected zone.

It happens surprisingly often, though not usually at the TLD level. That said, a similar problem hit thousands of Sweden’s .se domains, despite the registry having a decade’s more DNSSEC experience than Fiji, last month.

Domain Incite had a similar problem recently when its registrar carried on publishing DNSSEC information for the domain long after I’d stopped paying for it.

UPDATE: This post was updated with comment from IANA.

Thousands of domains hit by downtime after DNSSEC error

Sweden saw thousands of domains go down for hours on Friday, after DNSSEC errors were introduced to the .se zone file.

Local ccTLD registry IIS said in a statement that around 8,000 domains had a “technical difficulty” that started around 1530 local time and lasted around seven hours:

On the afternoon of 4/2, a problem was discovered that concerned approximately 8,000 .se domains. The problem meant that services, such as email and web, that are linked to the affected domains in some cases could not be used or reached. In total, there are approximately 1.49 million .se domains, of which approximately 8,000 were affected.

During the afternoon and evening, a thorough work was done with the troubleshooting and the error could be fixed for the affected .se domains at approximately 22.25.

The problem is believed to have been caused by incorrect DNSSEC signatures being published in the .se zone file. Any machine using a DNSSEC-validating resolver would have seen the errors and flat-out refused to resolve the domain.

This is probably the key drawback of DNSSEC — typically resolvers will treat badly signed domains as if they do not exist, rather than fail over to an unsigned, but resolving, response.

Sweden is not a DNSSEC newbie — .se was the first TLD to deploy the technology, all the way back in 2005, with services for domain holders coming a couple of years later.

DNS genius and ICANN key-holder Dan Kaminsky dies at 42

Security researcher Dan Kaminsky, best known for uncovering the so-called “Kaminsky Bug” DNS vulnerability, has reportedly died at the age of 42.

It has been widely reported that Kaminsky’s niece confirmed his death from serious complications from his longstanding diabetes.

On Twitter, she rebutted emerging conspiracy theories that his death was linked to the coronavirus vaccine, which he had received April 12, saying her uncle would “laugh” at such views.

During his career as a white-hat hacker, Kaminsky worked for companies including Cisco, Avaya, and IOActive.

He occasionally spoke at ICANN meetings on security issues, and was since 2010 one of IANA’s seven Recovery Key Share Holders, individuals trusted to hold part of a cryptographic key that would be used to reboot root zone DNSSEC in the case of a massive disaster.

But he was best known for his 2008 discovery of a fundamental flaw in the DNS protocol that allowed cache poisoning, and therefore serious man-in-the middle attacks, across millions of name servers worldwide. He worked with DNS software vendors in private to help them with their patches before the problem was publicly disclosed.

His discoveries led in part to the ongoing push for DNSSEC deployment across the internet.

The vulnerability received widespread attention, even in the mainstream media, and quickly came to bear his name.

For me, my standout memory of Kaminsky is one of his series of annual “Black Ops” talks, at the Defcon 12 conference in Las Vegas in 2004, during which he demonstrated to a rapt audience of hackers how it was possible to stream live radio by caching small chunks of audio data in the TXT fields of DNS records and using DNS queries to quickly retrieve and play them in sequence.

As well as being a bit of a DNS genius, he knew how to work a stage: the crowd went mental and I grabbed him for an interview soon after his talk was over.

His death at such a young age is a big loss for the security community.

Fraud checks coming to .ch as SWITCH renews contract

Swiss ccTLD registry SWITCH has agreed to implement new security measures as part of its contract renewal with the government.

The company said Friday that it has extended its contract to run .ch names with the telecoms regulator OFCOM for five more years, bring it up to December 2026.

But as part of the renewal, SWITCH has agreed to “speed up the adoption and implementation of technical security standards”.

This will involved financial incentives for registrars to adopt DNSSEC, the registry said.

It will also introduce measures to combat fraud at the point of registration, with SWITCH saying “in the event of suspected fraudulent intent, newly registered domain names can be used only after an identity check.”

The policy appears similar to those at other ccTLDs, including .uk, where new regs are flagged under certain circumstances (such as containing coronavirus-related terms) and cannot resolve until further checks are carried out.