Nightmare downtime weekend for some eNom and Google customers

Some eNom customers have experienced almost two days of downtime after a planned data center migration went titsup, leading to DNS failures hitting what users suspect must have been thousands of domains.

Social media has been filled with posts from customers complaining that their DNS was offline, meaning their web sites and email have been down. Some have complained of losing money to the downtime.

Affected domains include some registered directly with eNom, as well as some registered via resellers including Google Workspace.

The issue appears to have been caused by a scheduled data center migration, which was due to begin 1400 UTC on Saturday and last for 12 hours.

The Tucows-owned registrar said that during that time both reseller hub enom.com and retail site enomcentral.com would be unavailable. While this meant users would be unable to manage their domains, DNS was expected to resolve normally.

But before long, customers started reporting resolution problems, leading eNom to post:

We are receiving some reports of domains using our nameservers which are failing to resolve. Owing to the migration we are unable to research and fully address the issue until the migration is complete. This is not an expected outcome from the migration, and we are working to address it as a priority.

The maintenance window was then extended several times, by three to six hours each time, as eNom engineers struggled to fix problems caused by the migration. eNom posted several times on its status page:

The unexpected extension to the maintenance window was due to data migration delays. We also discovered resolution problems that impact a few hundred domains

eNom continued to post updates until it finally declared the crisis over at 0800 UTC this morning, meaning the total period of downtime was closer to 42 hours than the originally planned 12.

A great many posts on social media expressed frustration and anger with the outage, with some saying they were losing money and reputation and others promising to take their business elsewhere.

Help, it's Monday morning in the UK, I have a team who need to work. I have customers who want to place orders. I have a new partnership in progress which will be ruined at this rate. Please please let us know what is happening. #enom #enomdown @enomsupport

— Karen Burns (@cooper_karen) January 17, 2022

@enom @enomsupport What on the world is going on? Every 3 hours we get another boilerplate message saying “another 3 hours” This is completely unacceptable. How are you going to fix this and how are you going to make people whose work sites have been down all weekend whole?

— Andrew Pershing (@Sci_Officer) January 16, 2022

I almost have begrudging respect for how badly @enom has fucked the pooch with their migration. Blowing up your service so badly that your customers can’t even take their business elsewhere is some galaxy brain shit.

— Keith (@FilthySpecs) January 16, 2022

Same. I'm in a group with 6 other bloggers who also have the same DNS issue. Our sites have been down since yesterday. We're losing ad income & have readers who can't access the information they need.

— Elaina Newton (@TheRisingSpoon) January 16, 2022

Some said that they continued to experience problems after eNom had declared the maintenance over.

@enom your service status website states incorrect status for Google services? My email is still down!!! Also, getting error when logging in to enom domain management console!

— Tomislav Pavosevic (@tompavosevic) January 17, 2022

eNom primarily sells through its large reseller channel, so some customers were left having to explain the downtime in turn to their own clients. Google Workspace is one such reseller that acknowledged the problems on its Twitter feed.

Some customers questioned whether the problem really was just limited to just a few hundred domains, and eNom seemed to acknowledge that the actual number may have been higher.

Looks like one of the most severe internet outages I've seen in years is underway and has not hit MSM yet! @enom, "largest wholesaler of Internet domains" has screwed up a migration and taken possibly up to 2M domains offline.
Certainly my measly 100+ are dead.
See @enomsupport

— 😷Andy Stringer (@AndyStringer) January 17, 2022

At this point, we have been able to track a few hundred. If you have domains that are affected we encourage you to reach out to help@enom.com with the domain(s) in question.

— enomsupport (@enomsupport) January 16, 2022

I’m in contact with Tucows, eNom’s owner, and will provide an update when any additional information becomes available.

Form an orderly queue: New Zealand wants a new back-end

New Zealand is looking to possibly outsource its .nz ccTLD registry back-end for the first time, and has invited interested parties to get in touch.
Registry manager InternetNZ today published a request for expressions of interest in what it’s calling its “registry replacement project”.
It won’t be as straightforward as most registry migrations, as .nz is currently running essentially two different back-ends.
Today, about 65% of its registrations are based on an outdated custom Shared Registration System protocol, with the remainder on the industry standard Extensible Provisioning Protocol.
The proportion of registrars running SRS versus EPP is roughly the same, with about 65% on SRS, according to the REOI.
But the registry wants to get rid of SRS altogether, forcing all SRS-only registrars to adopt the EPP, and the new back-end provider will have to support this transition.
While registrars always have a bit of implementation work to do when a TLD changes back-ends, it’s not usually as complicated as adopting a completely different protocol with which they may not be unfamiliar.
So the risk of issues arising during the eventual handover — which will probably take a bit longer than usual — is probably a bit higher than usual.
But .nz is an attractive TLD. At the start of the month, it had 711,945 domains under management, a pretty good penetration on a per-capita basis when compared to the biggest ccTLDs.
It’s in the top 50 of the 1,338 TLDs for which I have data.
The deadline for responses to the REOI is November 29, a little over a month from now, InternetNZ said.
The registry is taking briefings at ICANN 66 in Montreal from November 2, and the following week in New Zealand.
UPDATE: This article originally stated that InternetNZ has decided to outsource its back end. In fact, outsourcing is just one of a number of options.

.blog registry handover did NOT go smoothly

The transition of .blog between registry back-end providers ended up taking six times longer than originally planned, due to “a series of unforeseen issues”.
Registry Knock Knock Whois There today told registrars that the move from Nominet to CentralNic took 18 hours to complete, far longer than the two to three hours anticipated.
An “unexpected database error” was blamed at one point for the delay, but KKWT said it is still conducting a post-mortem to figure out exactly what went wrong.
During the downtime, .blog registrations, renewals, transfers and general domain management at the registry level would not have been possible.
DNS resolution was not affected, so registrants of .blog domains would have been able to use their web sites and email as usual.
The migration, which covered roughly 200,000 domains, wrapped up at around 0800 UTC this morning. It seems engineers at the two back-end providers, both based in the UK, will have been working throughout the night to fix the issues.
KKWT reported the new CentralNIC EPP back-end functioning as expected but that several days of “post-migration clean-up” are to be expected.
Eighteen hours is more than the acceptable 14 hours of monthly downtime for EPP services under ICANN’s standard Registry Agreement, but below the 24 hours of weekly downtime at which emergency measures kick in.
CentralNic already handles very large TLDs, including .xyz, but I believe this is the largest incoming migration it’s handled to date.
KKWT is owned by Automattic, the same company as WordPress.com.

Mystery .vu registry revealed

Neustar has been selected as the back-end domain registry operator for the nation of Vanuatu.
The company, and the Telecommunications Radiocommunications and Broadcasting Regulator, announced the appointment, which came after a competitive tender process between nine competing back-end providers, last night.
The ccTLD is .vu.
It’s unrestricted, with no local presence requirements, and currently costs $50 per year if you buy directly from the registry, Telecom Vanuatu Ltd (TVL).
Unusually, if you show up at TVL’s office in Vanuatu capital Port Vila, you can buy a domain for cash. I’ve never heard of that kind of “retail” domain name option before.
A handful of international registrars also sell the domains marked up, generally to over the $80 mark.
TVL was originally the sponsor of the ccTLD, but ICANN redelegated it to TRBR in March after Vanuatu’s government passed a law in 2016 calling for redelegation.
Under the deal, Neustar will take over the registry function from TVL after its 24 years in charge, bringing the .vu option to hundreds of other registrars.
Most registrars are already plugged in to Neustar, due to its operation of .us, .biz and .co. It also recently took over India’s .in.
There’s no public data on the number of domains under management, but Vanuatu is likely to have a much smaller footprint that Neustar’s main ccTLD clients.
It’s quite a young country, gaining independence from France and the UK in 1980, a Pacific archipelago of roughly 272,000 people.
Neustar expects the transition to its back-end to be completed September 30.

Afilias gets Guinness record for .au migration

Afilias has got its recent takeover of .au recognized by Guinness as an official world record.
The company was given the title of “Largest Migration of an Internet Top-Level Domain in a Single Transition” at an event in New York today, according to a company press release.
It relates to its migration of .au from former registry provider Neustar to its own back-end a few months ago.
Australia’s .au ccTLD had about 3.1 million names under management at the time, about 400,000 names more than the previous record — the 2003 .org transition Afilias also handled.
I understand there’s a licensing fee due to Guinness for this kind of (let’s face it) shameless-but-effective PR stunt, but no guarantee the record will actually be printed in future editions of the annual Guinness Book of World Records.
I hope the fresh salt in Neustar’s wounds isn’t stinging too badly this evening.

Fight breaks out as Afilias eats Neustar’s Aussie baby

The transition of .au to Afilias’ registry platform over the weekend seems to have gone quite smoothly, but that hasn’t stopped Neustar and a former key executive from lashing back at what it says are the gaining company’s “misinformed” statements.
The war of words, which has got quite nasty, came as Afilias transferred all 3.1 million .au domains to its control, after 16 years with the former incumbent.
Neustar, which hadn’t said much about losing one of its most-lucrative TLD contracts, on Friday published a lengthy blog post in which it said it wanted to “set the record straight” about Afilias’ statements leading up to the switch.
Afilias, in a series of blog posts and press releases since it won the .au contract, has been bigging up its technical capabilities.
While it’s not directly criticized Neustar and predecessor AusRegistry (which Neustar acquired for $87 million), the implication of many of these statements is that Neustar was, by comparison, a bit shit.
In Neustar’s latest post, Aussie VP George Pongas takes issue with several of these claims.
Any implications that the company did not offer 24/7 registrar support were incorrect, he wrote. Likewise, the idea that it did not have a DNS node in Western Australia was not true, he wrote.
He also took issue with claims that Afilias would offer improved security and a broader feature set for registrars, writing:

We’ve raised a number of concerns directly with auDA about what we considered to be inaccurate remarks comparing Neustar’s systems with the new Registry and implying that the new Registry will include “all previous functionality plus enhanced security and authentication measures”, as stated in recent auDA Member communications. We questioned auDA about this and were informed that the statement is comparing the various testing phases of Afilias’ Registry – so the latest version has “all previous functionality” of the earlier versions. It doesn’t mean the Registry will have “all previous functionality” of Neustar’s platform – which we believe the statement implies. It is a fact that a number of the proprietary features and services that Neustar currently provides to Registrars will no longer be available under the new Registry system, and thus Registrars will likely notice a difference.

“We stand by our statements,” an Afilias spokesperson told DI today.
While Neustar’s corporate stance was fairly reserved, former AusRegistry boss Adrian Kinderis, never a shrinking violet, has been reacting in an almost presidential fashion, using Twitter to describe auDA CEO Cameron Boardman as “incompetent”, criticizing a reporter, and using the hashtag #crookedcameron.

#crookedcameron actually asked me to come work for @auda so I would be silent on all the bullshit he is peddling thinking I could be #paidoff. Happy to tell you all I gave him a polite #gofuckyourself does the @auda Board know he offered me the gig?

— Adrian Kinderis (@AdrianKinderis) July 1, 2018

Kinderis, who headed up AusRegistry for the whole of its 16-year run with .au, left Neustar in April, three years after the acquisition. He’s now running something called MadBarry Enterprises and is still associated with the new gTLD .film.
He reckons Neustar lost the .au contract purely for financial reasons.
While Neustar is believed to have lowered its registry fee expectations when pitching to continue as the back end, auDA will save itself about AUD 9 million a year ($7 million) under Afilias, compared to the old regime.
auDA is not expected to hand this saving on to registrars and registrants, though I hear registrars have been offered marketing rebates recently.
auDA has previously told us that Afilias scored highest on the technical evaluation of the nine bidders, and that it was not the bidder with the lowest fee.
Kinderis is also of the opinion that Afilias is among those helping auDA stack its membership with compliant stooges.
Last month, auDA announced a dramatic four-fold increase in its membership — getting 955 new membership applications in just a month.
auDA thanked Afilias for this growth in membership, alongside three of the largest .au registrars: Ventra IP, Arq Group (formerly Melbourne IT), and CrazyDomains owner Dreamscape Networks.
An Afilias spokesperson said that the company had offered its staff the option to become auDA members and about half — I estimate at roughly 150 people based on Afilias’ previously published headcounts — had taken it up on the offer.
It sounds rather like Afilias footed the AUD 22 per-person “Demand-class” membership application fees.
The rapid increase in membership at auDA has raised eyebrows in the .au community, with some accusing the registry of “branch stacking”.
That’s an Australian term used to describe the practice of signing up large numbers of members of a local branch of a political party in order to swing important votes.
The 955-plus new members will not be approved in time to influence the outcome of the vote to oust the auDA chair and others later this month.
But they will have voting rights by the time auDA’s annual general meeting comes around later this year. The AGM is when auDA will attempt to reform itself in light of a harsh government review of its practices.
As for the migration to Afilias itself, it seems to have gone relatively smoothly. I’m not aware of any reports of any serious technical issues, despite the fact that it was the largest TLD migration ever.
Some members have pointed out that most of .au’s ops are now off-shore, and old auDA Whois service is now hosted on a .ltd domain (hey, somebody’s got to use it) which is itself protected by Whois privacy.
I also noticed that the auDA web site, which used to have a hook into the registry that published an updated domain count every day, is no longer working.