Nightmare downtime weekend for some eNom and Google customers

Some eNom customers have experienced almost two days of downtime after a planned data center migration went titsup, leading to DNS failures hitting what users suspect must have been thousands of domains.

Social media has been filled with posts from customers complaining that their DNS was offline, meaning their web sites and email have been down. Some have complained of losing money to the downtime.

Affected domains include some registered directly with eNom, as well as some registered via resellers including Google Workspace.

The issue appears to have been caused by a scheduled data center migration, which was due to begin 1400 UTC on Saturday and last for 12 hours.

The Tucows-owned registrar said that during that time both reseller hub enom.com and retail site enomcentral.com would be unavailable. While this meant users would be unable to manage their domains, DNS was expected to resolve normally.

But before long, customers started reporting resolution problems, leading eNom to post:

We are receiving some reports of domains using our nameservers which are failing to resolve. Owing to the migration we are unable to research and fully address the issue until the migration is complete. This is not an expected outcome from the migration, and we are working to address it as a priority.

The maintenance window was then extended several times, by three to six hours each time, as eNom engineers struggled to fix problems caused by the migration. eNom posted several times on its status page:

The unexpected extension to the maintenance window was due to data migration delays. We also discovered resolution problems that impact a few hundred domains

eNom continued to post updates until it finally declared the crisis over at 0800 UTC this morning, meaning the total period of downtime was closer to 42 hours than the originally planned 12.

A great many posts on social media expressed frustration and anger with the outage, with some saying they were losing money and reputation and others promising to take their business elsewhere.

Help, it's Monday morning in the UK, I have a team who need to work. I have customers who want to place orders. I have a new partnership in progress which will be ruined at this rate. Please please let us know what is happening. #enom #enomdown @enomsupport

— Karen Burns (@cooper_karen) January 17, 2022

@enom @enomsupport What on the world is going on? Every 3 hours we get another boilerplate message saying “another 3 hours” This is completely unacceptable. How are you going to fix this and how are you going to make people whose work sites have been down all weekend whole?

— Andrew Pershing (@Sci_Officer) January 16, 2022

I almost have begrudging respect for how badly @enom has fucked the pooch with their migration. Blowing up your service so badly that your customers can’t even take their business elsewhere is some galaxy brain shit.

— Keith (@FilthySpecs) January 16, 2022

Same. I'm in a group with 6 other bloggers who also have the same DNS issue. Our sites have been down since yesterday. We're losing ad income & have readers who can't access the information they need.

— Elaina Newton (@TheRisingSpoon) January 16, 2022

Some said that they continued to experience problems after eNom had declared the maintenance over.

@enom your service status website states incorrect status for Google services? My email is still down!!! Also, getting error when logging in to enom domain management console!

— Tomislav Pavosevic (@tompavosevic) January 17, 2022

eNom primarily sells through its large reseller channel, so some customers were left having to explain the downtime in turn to their own clients. Google Workspace is one such reseller that acknowledged the problems on its Twitter feed.

Some customers questioned whether the problem really was just limited to just a few hundred domains, and eNom seemed to acknowledge that the actual number may have been higher.

Looks like one of the most severe internet outages I've seen in years is underway and has not hit MSM yet! @enom, "largest wholesaler of Internet domains" has screwed up a migration and taken possibly up to 2M domains offline.
Certainly my measly 100+ are dead.
See @enomsupport

— 😷Andy Stringer (@AndyStringer) January 17, 2022

At this point, we have been able to track a few hundred. If you have domains that are affected we encourage you to reach out to help@enom.com with the domain(s) in question.

— enomsupport (@enomsupport) January 16, 2022

I’m in contact with Tucows, eNom’s owner, and will provide an update when any additional information becomes available.

CentralNic makes another registrar acquisition

CentralNic said today it has bought another registrar, Chile-based NameAction, in a $1 million deal.

NameAction has been around since the late 1990s and specializes in ccTLDs in the Latin American region, including offering local presence services for foreign registrants.

It sells gTLD domains too, acting primarily in the brand protection space, but does not appear to be ICANN-accredited in its own right.

CentralNic said the deal will immediately add $2 million to its top line and $200,000 to profits.

CEO Ben Crawford said in a press release that the deal is small but of strategic importance, giving the company a beachhead from which to expand into Latin America.

It’s the fourth acquisition announcement from CentralNic, which describes itself as an industry consolidator, this year.

GoDaddy hack exposed a million customer passwords

GoDaddy’s systems got hacked recently, exposing up to 1.2 million customer emails and passwords.

The attack started on September 6 and targeted Managed WordPress users, the company’s chief information security officer Demetrius Comes disclosed in a blog post and regulatory filing this week.

The compromised data included email addresses and customer numbers, the original WordPress admin password, the FTP and database user names and passwords, and some SSL private keys.

In cases where the compromised passwords were still in use, the company said it has reset those passwords and informed its customers. The breached SSL certs are being replaced.

GoDaddy discovered the hack November 17 and disclosed it November 22.

It sounds rather like the attack may have been a result of a phishing attack against a GoDaddy employee. The company said the attacker used a “compromised password” to infiltrate its WordPress provisioning system.

Comes wrote in his blog post:

We are sincerely sorry for this incident and the concern it causes for our customers. We, GoDaddy leadership and employees, take our responsibility to protect our customers’ data very seriously and never want to let them down. We will learn from this incident and are already taking steps to strengthen our provisioning system with additional layers of protection

You may recall that GoDaddy came under fire last December for punking its employees with a fake email promising an end-of-year bonus, which turned out to be an “insensitive” component of an anti-phishing training program.

About 500 staff reportedly failed the test.

GoDaddy says it turned around Neustar, and .biz numbers seem to confirm that

GoDaddy is pleased with how its new registry division is doing, with CEO Aman Bhutani claiming last night that it’s managed to turn around the fortunes of Neustar, which became part of GoDaddy Registry a year ago.

Reporting a strong third quarter of domains revenue growth, Bhutani highlighted the secondary market and the registry as drivers. In prepared remarks, he said:

On Registry, we are continuing to prove our ability to acquire, integrate, and accelerate. A great example is the cohort performance within GoDaddy Registry. When we acquired Neustar’s registry assets in Q3 last year, its new cohorts were shrinking, with new unit registrations down 4% year over year. We are now one year into the acquisition, and we’re pleased to report that within that first year, we have been able to accelerate new business significantly. We are now seeing new unit registrations increase nearly 20% year over year — all organically.

If you’re wondering what a “cohort” is, it appears to refer to GoDaddy’s way of, for analysis purposes, slicing up its customers, how much they spend and how profitable they are, into tranches according to the years in which they became customers.

So GoDaddy’s saying here that Neustar’s number of new customers was going down, and it was selling 4% fewer new domains, at the time of the acquisition last year, but that that trend has now been reversed, with new regs up 20%.

The numbers are not really possible to verify. Neustar’s main three TLDs for volume purposes were .us, .co and .biz, and of those only .biz is contractually obliged to publish its zone file and registry numbers.

But look at .biz!

That’s .biz’s daily zone file numbers for the last two years, with the August 2020 acquisition highlighted by a subtle arrow. It’s only added about 50,000 net names since the deal, but it’s reversing an otherwise negative trend.

Monthly transaction reports show .biz had been on a general downward, if spiky, line since its early 2014 peak of 2.7 million names. It’s now at about 1.4 million.

When asked how the company achieved such a feat, Bhutani credited “execution” and left it at that. Perhaps this means something to financial analysts.

When asked by an analyst whether GoDaddy was giving its own TLDs preferential treatment, promoting its owned strings on the registrar in order to better compete with .com at the registry, Bhutani denied such frowned-upon behavior:

We don’t do that. All TLDs work on our registrar side in terms of their merit. It’s about value to the customer — whatever works best irrespective of whether we own the registry side or not. That’s what we’ll sell in front of the customer.

The company reported domains revenue up 17% at $453.2 million for the third quarter, with overall revenue up 14% at $964 million compared to year-ago numbers. Net income was up to $97.7 million from $65.1 million a year ago.

GoDaddy expects domains revenue to grow in the low double digits percent-wise in the current quarter.

Donuts shuts down 14 registrars, but it’s “not related to DropZone”

Donut has let 14 of its shell registrar accreditations expire, but told DI it’s not related to its recently approve drop-catching service, DropZone.

ICANN records show that the companies, with names such as Name118 Inc and Name104 Inc, all basically mini-clones of Name.com, recently had their registrar contracts terminated.

This kind of thing happens fairly regularly with companies resizing the networks they use for catching dropping domains. Donuts still has at least half a dozen active accreditations, records show.

But the move comes just weeks after ICANN approved a controversial new Donuts service called DropZone, which would see dropping domains across Donuts’ portfolio of 250+ gTLDs being handled by a dedicated parallel registry.

DropZone would reduce the need for owning vast numbers of shell accreditations in order to effectively drop-catch, but has faced criticism from rival DropCatch because a) Donuts may charge registrars for access and b) claims that Donuts-owned registrars would have an advantage.

But Donuts says the two things are unrelated. Name.com senior product marketing manager Ethan Conley said in an email:

We did recently let 14 ICANN registrar accreditations expire. These accreditations had become an administrative headache and a point of confusion for customers. This decision was not related to DropZone, and the domain drop business has not been a core focus of Name.com for quite some time.

It’s worth noting that cancelling registrar accreditations would also have an affect on the ability to catch names in other, unaffiliated gTLDs, including .com.

Most registrars did NOT “fail” abuse audit, ICANN says

Most registrars did not “fail” a recent abuse audit, despite what I wrote in my original coverage, according to ICANN.

“Referring to a certain blog, none of the registrars failed the audit,” ICANN senior audit manager Yan Agranonik said during a session of ICANN 72’s Prep Week last night.

He’s talking about ME! He’s talking about ME!

“Failure would mean that there’s an irreparable finding of deficiency that can not be corrected timely or it just goes against the registrar’s business model,” Agranonik said.

An accompanying presentation reads:

None of the registrars “failed” the audit. “Failure” means that the auditee did not acknowledge/remediate identified violations of the RAA or their business practices are not compatible with RAA.

At the risk of prolonging a tedious semantic debate, what I reported in August, when the results of the audit were announced, was: “The large majority of accredited registrars failed an abuse-related audit at the first pass, according to ICANN.”

A bunch of registrar employees, and now apparently ICANN’s own head auditor, disagreed with my characterization.

ICANN had issued a press release stating that of 126 audited registrars, it had identified 111 “that were not fully compliant with the RAA’s requirements related to the receiving and handling of DNS abuse reports.”

To me, if ICANN checks whether you’re doing a thing you should be doing and you’re not doing the thing, that’s a fail.

But to ICANN, if ICANN checks whether you’re doing a thing you should be doing and you’re not doing the thing, and it tells you you’re not doing the thing you should be doing, so you start doing the thing, that’s not a fail.

I think reasonable people could disagree on the definitions here.

But I did write that the registrars “failed… according to ICANN”, and that appears to be inaccurate, so I’m happy to correct the record today.

Alice’s Registry disappears down the rabbit hole

One of the oldest domain registrars appears to be on its way out.

San Francisco-based Alice’s Registry has been hit with a breach notice and termination warning by ICANN after apparently being incommunicado for over a year.

According to ICANN, they last spoke in August 2020, when AR indicated that it was thinking about “shutting down the registrar business”.

Since then, the web site has stopped working and ICANN can’t get through on the telephone.

The breach notice claims past-due fees and a failure to operate a working Whois service, and gives the registrar until November 1 to pay up or get its contract terminated.

Alice’s Registry is one of the oldest registrars, founded in 1999, but it’s never had more than a few thousand names under management. Its founder, Rick Wesson, has been involved in the ICANN community since pretty much the beginning.

Almost no security researchers asking for Whois records – Tucows

Security researchers are not asking for private Whois records in anywhere near the numbers you might have been led to believe, according to data released this week by Tucows.

The registrar revealed that it received just one request from the security community between September 2020 and the end of August 2021. That’s not even 1% of the total.

Over the same period, the “commercial litigators” category, presumably including intellectual property interests going after suspected cybersquatters, were behind 87% of requests.

About 9% of requests came from law enforcement agencies, Tucows said.

The company said that it disclosed private registrant data in 74% of cases. It denied the requests in 9% of cases. Other requests were incomplete or abandoned.

Tucows has been offering a Tiered Access service for its Whois records since the General Data Protection Regulation came into effect in May 2018. It has received 4,478 requests since then.

Cloudflare goes all-in with at-cost domains

Content delivery specialist Cloudflare has come out as a fully-fledged domain name registrar, promising to sell names in hundreds of TLDs with no markup on the registry wholesale price.

Since the company launched Cloudflare Registrar three years ago, it’s only been possible for customers to transfer in their domains from other registrars; no new regs were possible.

That’s now changed, with the company today announcing customers can buy their domains direct without having to faff about with transfers.

“It’s important to note that our registrar pricing is ‘at-cost.’ That means we charge our customers exactly what we pay the registry, plus any applicable ICANN transaction fees,” the company blogged.

A new .com costs $8.57, for example.

It appears that you need to be a Cloudflare CDN customer to use the registrar service, but it offers a free plan among its suite of offerings.

It’s not the only registrar to offer names at cost, but it’s certainly the one with the best brand recognition.

Cloudflare currently offers over 250 TLDs, and plans to launch 40 more next month. It just added .uk, and plans to add more ccTLDs, as well as premium-priced domains, in future.

The company had about 325,000 domains under management at the last count in May, having transferred a net of 118,000 names during the previous 12 months.

Make no mistake, Cloudflare will be losing money on this venture, despite what it said in September 2018, when it made out running a registrar was cost-free.

Could registrars get sued under new Texas abortion law?

Does the controversial new Texas state legislation effectively banning most abortions pose legal risks for domain name registries and registrars?

The so-called Texas Heartbeat Act, or SB 8, came into effect at the start of the month. It bans abortions in Texas when doctors can detect a heartbeat in the fetus, which is usually about six weeks after conception, when most women don’t know they’re pregnant.

In an apparent attempt to circumvent the US Supreme Court’s oversight, the enforcement of the law is left to civil actions — the cops won’t come to get you, but any US citizen will be allowed to file civil suits with a guaranteed payout of at least $10,000 if they win and no risk of paying court costs if they lose.

The ban extends not only to doctors who perform the procedure, but also those who “aid and abet”.

This part of the law has been written in such a way that it’s been broadly interpreted as even opening up taxi drivers who transport patients to abortion clinics to possible liability.

Taxi service giants Uber and Lyft have both already announced they will cover the costs of any legal representation their contractors need.

So if taxi drivers can get sued, why not also registrars and hosting companies? Clinics, counselling services and the like all need web sites, and web sites need domains.

It might be a stretch, and the law is worded in such a way that could give registrars a defense, saying liability is restricted to those who “knowingly engages in conduct that aids or abets the performance or inducement of an abortion”.

“Knowingly” is a key word. Taxi drivers dropping off a woman at a clinic know where they are driving. Registrars and hosting companies typically don’t know what is being hosted on their servers.

But what if they are told about pro-abortion content on their services, accompanied by a threat of litigation?

It seems that so far the registrar industry, even one company headed by a right-wing religious individual, are effectively, if not vocally, on the pro-choice side of the debate.

A “whistleblower” web site, run by Texas Right to Life at prolifewhistleblower.com, that was inviting users to essentially “doxx” abortion providers has been kicked off GoDaddy for violating its privacy rules, and even right-leaning Epik has asked the registrant to leave on similar grounds.