Russia blames DNSSEC, not Ukraine, for internet downtime

Another ccTLD has blamed DNSSEC after seeing hours of downtime affecting its country’s biggest web services yesterday.

This time it’s Russia’s ccTLD.ru, which confirmed today that it was responsible for the widely reported outages on Tuesday, which had sparked speculation that a cyber-attack related to the war in Ukraine might be the culprit.

It was rather a DNSSEC failure that affected both .ru and the Cyrillic .рф domains, the registry said. It was related to a cryptograpghic key rollover, the registry indicated.

“After the failure was detected, the updated keys were revoked, and the functionality of the .RU zone was fully restored, which took about two hours, including the distribution of data through the DNS system,” the registry said on its web site.

“The investigation into the incident is currently ongoing, but it is already clear that the main cause of the failure was the imperfection of the software used to create the encryption keys,” it added.

The explanation was echoed by Russian government officials on social media, and it’s sadly rather plausible. DNSSEC failures at ccTLDs, and to a lesser extent gTLDs, usually related to fluffed key rollovers, are rather common.

There have been similar outages reported in the last few years in Australia (twice), Namibia, Fiji, and Sweden. And those are just the ones reported on this blog. People who track this kind of thing more closely have recorded hundreds of incidents.

Another DNSSEC screw-up takes down thousands of .au domains

Australia’s ccTLD has become the latest to see a widespread outage that appears to be the result of a DNSSEC misconfiguration.

A reported 15,000 .au domains were affected, though some suspect it could have been more.

Registry overseer auDA said on Twitter that .au “experienced an error” that affected a “small number of domains” and that an investigation was underway.

Earlier today the .au domain experienced an error. We regret that a small number of domains were affected but are pleased to report the .au is back up and operating. An investigation as to the cause of this issue is underway.

— auDA (@auda) March 22, 2022

Donuts subsidiary Afilias, which runs the back-end for .au’s more that 3.4 million domains, has yet to publicly comment.

Network operators and DNS experts took to social media and mailing lists to observe that .au’s DNSSEC was broken, though it appears the problem was fixed rather quickly.

(For those following along from home, .AU DNSSEC key management was broken from 05:48 – 06:18 UTC. Quick repair by AuDA.)https://t.co/jmHAbZdWKo https://t.co/SzOi9U7Kwi

— Bill Woodcock (@woodyatpch) March 22, 2022

DNSSEC creates a chain of cryptographic keys all the way to the DNS root, and when that chain is broken by a misconfiguration such as a missing key, most DNSSEC-enabled resolvers treat the affected domains as if they simply don’t exist.

That means services such as web sites and email addresses stop working until the chain is reestablished. People not using DNSSEC resolvers wouldn’t have seen a problem.

It’s the third TLD to experience a significant outage due to DNSSEC in the last six weeks.

In February, thousands of domains in Sweden’s .se went dark for hours, and Fiji’s entire .fj zone disappeared for DNSSEC users less than two weeks ago.

The outage comes at a particularly unfortunate time in terms of public relations for auDA, which on Thursday will start making direct second-level .au registrations available for the first time.

It’s not immediately clear whether the DNSSEC fluff is related to the SLD launch.

Verisign saw MASSIVE query spike during Facebook outage

Verisign’s .com and .net name servers saw a huge spike in queries when Facebook went offline for hours last October, Verisign said this week.

Queries for facebook.com, instagram.com, and whatsapp.net peaked at over 900,000 per second during the outage, up from a normal rate of 7,000 per second, a more than 100x increase, the company said in a blog post.

The widely publicized Facebook outage was caused by its IP addresses, including the IP addresses of its DNS servers, being accidentally withdrawn from routing tables. At first it looked to outside observers like a DNS failure.

When computers worldwide failed to find Facebook on their recursive name servers, they went up the hierarchy to Verisign’s .com and .net servers to find out where they’d gone, which led to the spike in traffic to those zones.

Traffic from DNS resolver networks run by Google and Cloudflare grew by 7,000x and 2,000x respectively during the outage, Verisign said.

The company also revealed that the failure of .club and .hsbc TLDs a few days later had a similar effect on the DNS root servers that Verisign operates.

Queries for the two TLDs at the root went up 45x, from 80 to 3,700 queries per second, Verisign said.

While the company said its systems were not overloaded, it subtly criticized DNS resolver networks such as Google and Cloudflare for “unnecessarily aggressive” query-spamming, writing:

We believe it is important for the security, stability and resiliency of the internet’s DNS infrastructure that the implementers of recursive resolvers and public DNS services carefully consider how their systems behave in circumstances where none of a domain name’s authoritative name servers are providing responses, yet the parent zones are providing proper referrals. We feel it is difficult to rationalize the patterns that we are currently observing, such as hundreds of queries per second from individual recursive resolver sources. The global DNS would be better served by more appropriate rate limiting, and algorithms such as exponential backoff, to address these types of cases

Verisign said it is proposing updates to internet standards to address this problem.