“We fell short” — Tucows says sorry for Enom downtime

Tucows has apologized to thousands of Enom customers who suffered days of downtime after a planned data center migration went badly wrong.

Showing true Canadian humility, the registrar posted the following statement this evening:

Beginning Saturday, January 15, 2022, Enom experienced a series of complications with a planned data center migration that caused significant disruptions for a subset of our customers.

We sincerely apologize to all of those impacted. We pride ourselves on being a reliable domain registration platform, and this weekend we fell short. We are committed to regaining your trust and to serving you better.

A full internal audit is underway and an incident report is forthcoming. This will include a summary of events and scope, learnings, and policy and process changes to mitigate future issues.

We reported on the downtime on Monday, as some customers were entering their third day of non-resolving DNS, which led to broken web sites and email.

At the time, Enom was saying it was tracking a “few hundred” affected domains. As customers suspected, that turned out to be a huge underestimate. The true number was closer to 350,000 domains, Tucows is now saying.

The company had been warning its customers about the planned maintenance for weeks, but it did not anticipate a “a bug in the new DNS provisioning system” that stopped customers’ domains resolving.

The migration started Saturday January 15 at 1400 UTC and was expected to last 12 hours. In the end, the DNS issue was not fully fixed until Monday January 17 at about 1845 UTC.

Nightmare downtime weekend for some eNom and Google customers

Some eNom customers have experienced almost two days of downtime after a planned data center migration went titsup, leading to DNS failures hitting what users suspect must have been thousands of domains.

Social media has been filled with posts from customers complaining that their DNS was offline, meaning their web sites and email have been down. Some have complained of losing money to the downtime.

Affected domains include some registered directly with eNom, as well as some registered via resellers including Google Workspace.

The issue appears to have been caused by a scheduled data center migration, which was due to begin 1400 UTC on Saturday and last for 12 hours.

The Tucows-owned registrar said that during that time both reseller hub enom.com and retail site enomcentral.com would be unavailable. While this meant users would be unable to manage their domains, DNS was expected to resolve normally.

But before long, customers started reporting resolution problems, leading eNom to post:

We are receiving some reports of domains using our nameservers which are failing to resolve. Owing to the migration we are unable to research and fully address the issue until the migration is complete. This is not an expected outcome from the migration, and we are working to address it as a priority.

The maintenance window was then extended several times, by three to six hours each time, as eNom engineers struggled to fix problems caused by the migration. eNom posted several times on its status page:

The unexpected extension to the maintenance window was due to data migration delays. We also discovered resolution problems that impact a few hundred domains

eNom continued to post updates until it finally declared the crisis over at 0800 UTC this morning, meaning the total period of downtime was closer to 42 hours than the originally planned 12.

A great many posts on social media expressed frustration and anger with the outage, with some saying they were losing money and reputation and others promising to take their business elsewhere.

Help, it's Monday morning in the UK, I have a team who need to work. I have customers who want to place orders. I have a new partnership in progress which will be ruined at this rate. Please please let us know what is happening. #enom #enomdown @enomsupport

— Karen Burns (@cooper_karen) January 17, 2022

@enom @enomsupport What on the world is going on? Every 3 hours we get another boilerplate message saying “another 3 hours” This is completely unacceptable. How are you going to fix this and how are you going to make people whose work sites have been down all weekend whole?

— Andrew Pershing (@Sci_Officer) January 16, 2022

I almost have begrudging respect for how badly @enom has fucked the pooch with their migration. Blowing up your service so badly that your customers can’t even take their business elsewhere is some galaxy brain shit.

— Keith (@FilthySpecs) January 16, 2022

Same. I'm in a group with 6 other bloggers who also have the same DNS issue. Our sites have been down since yesterday. We're losing ad income & have readers who can't access the information they need.

— Elaina Newton (@TheRisingSpoon) January 16, 2022

Some said that they continued to experience problems after eNom had declared the maintenance over.

@enom your service status website states incorrect status for Google services? My email is still down!!! Also, getting error when logging in to enom domain management console!

— Tomislav Pavosevic (@tompavosevic) January 17, 2022

eNom primarily sells through its large reseller channel, so some customers were left having to explain the downtime in turn to their own clients. Google Workspace is one such reseller that acknowledged the problems on its Twitter feed.

Some customers questioned whether the problem really was just limited to just a few hundred domains, and eNom seemed to acknowledge that the actual number may have been higher.

Looks like one of the most severe internet outages I've seen in years is underway and has not hit MSM yet! @enom, "largest wholesaler of Internet domains" has screwed up a migration and taken possibly up to 2M domains offline.
Certainly my measly 100+ are dead.
See @enomsupport

— 😷Andy Stringer (@AndyStringer) January 17, 2022

At this point, we have been able to track a few hundred. If you have domains that are affected we encourage you to reach out to help@enom.com with the domain(s) in question.

— enomsupport (@enomsupport) January 16, 2022

I’m in contact with Tucows, eNom’s owner, and will provide an update when any additional information becomes available.

Architect of Nominet boardroom bloodbath and Tucows backer win director seats

UK registry Nominet has announced the winners of its non-executive director election, with Simon Blackler securing a runaway victory. Ashley La Bolle of Tucows was also elected, with a strong share of the votes.

Blackler is the architect of the PublicBenefit.uk campaign, which was behind a boardroom bloodbath earlier this year, and La Bolle is director of domains at Tucows, the biggest registrar name to support that campaign.

According to Nominet, Blackler secured 1,285,370 of the 2,558,650 votes in the first-preference round of voting, a smidge over 50%. La Bolle got 750,447 votes, 29.3%, at the same stage, picking up the extra she needed after votes were transferred.

The other four candidates all received 7% or less of the votes in the first-preference ballot.

Voting was based on how many domain names members control, capped at 3% to avoid too much capture by the larger registrars.

Nominet said that turnout was 24.3% — 553 of the 2,276 eligible voters actually cast a ballot.

Blackler and La Bolle will join Nominet’s board at its next Annual General Meeting, which happens this Thursday.

They replace domain investor David Thornton, who had stood for reelection but received less than 6% of the first-round votes, and GoDaddy policy veep James Bladel, who did not stand.

Blackler, who runs the registrar Krystal Hosting, started the PublicBenefit.uk campaign earlier this year in protest at what was seen as Nominet’s unresponsiveness and lack of transparency towards its members.

He rallied a crowd of members upset with what they saw as the company’s diversification into non-core businesses, excessive director and executive compensation, and diminishing devotion to supporting public-benefit causes.

The campaign resulted in the forced resignation of the CEO, the ouster of the chair and almost half the directors, and a renewed focus on the .uk registry and charitable causes under a new chair.

Tucows was the biggest-name registrar to back the campaign, with La Bolle repeatedly blogging about how Nominet needed to be more transparent and engage better with its members.

“Humbled by the amount of support and looking forward to improving Nominet for ALL,” Blackler tweeted following the results announcement.

“I’m truly honoured to be appointed to Nominet’s Board as an NED and am grateful for the support and trust from my peers,” La Bolle said via email. “As well-stated throughout my campaign, I am committed to helping Nominet refocus on its core mandate and re-engage its members to better serve our entire community.”

Nominet rebels dominate directorship slate

Nominet has named the six people nominated for its two open non-executive director positions, and the slate is very much slanted towards the new postbellum reality of UK domain politics.

The PublicBenefit.uk campaign, which saw the CEO forced out and half the board fired at an EGM earlier this year, leading to a broad suite of proposed reforms, has a strong presence on the candidate list.

Simon Blackler of Krystal Hosting, who created and spearheaded the campaign, is standing for one of the so-called “NED” seats. He is also named as a proposer/seconder of two of his rival candidates.

This PublicBenefit slate includes Ashley La Bolle, recently promoted head of domains at Tucows, and consultant and former lawyer Jim Davies, both of whom have Blackler’s endorsement.

La Bolle’s candidacy statement focuses on the need for increased transparency and member engagement, while Davies stands on a platform of constitutional and financial reform.

In his endorsements, Blackler cites Tucows’ endorsement of the PublicBenefit campaign as a crucial turning point in its ultimate success, and Davies’ resignation from the Nominet board in 2009, in which he cited high executive pay and low transparency as reasons for his departure.

One incumbent NED is standing for re-election, David Thornton. Nominated by Michele Neylon and Jothan Frakes, Thornton has a platform based on governance and structural rebalancing.

Internet policy all-rounder Liz Williams is also standing, talking up her extensive experience in areas such as ICANN, privacy and security.

Then there’s Bulgaria-based Brit Stephen Yarrow, whose main policy concerns appear to be raising Nominet’s profile and distancing .uk from the EU’s General Data Protection Regulation.

Nominet members should already have been sent the election materials. Everyone else can read them here (pdf).

Votes will be cast November 18 at Nominet’s Annual General Meeting. There are two seats available.

Almost no security researchers asking for Whois records – Tucows

Security researchers are not asking for private Whois records in anywhere near the numbers you might have been led to believe, according to data released this week by Tucows.

The registrar revealed that it received just one request from the security community between September 2020 and the end of August 2021. That’s not even 1% of the total.

Over the same period, the “commercial litigators” category, presumably including intellectual property interests going after suspected cybersquatters, were behind 87% of requests.

About 9% of requests came from law enforcement agencies, Tucows said.

The company said that it disclosed private registrant data in 74% of cases. It denied the requests in 9% of cases. Other requests were incomplete or abandoned.

Tucows has been offering a Tiered Access service for its Whois records since the General Data Protection Regulation came into effect in May 2018. It has received 4,478 requests since then.

Disclosure: I sold Tucows shares

TL;DR — I made about $3,700 selling Tucows shares.

I’ve never been much of an investor in anything, largely because I’ve never really had the disposable income. However, back in 2006, when I lived in the US, I opened an online share-trading account and bought around $1,000 worth of shares in various technology companies.

The companies included what at the time I considered safe bets: Dell, AMD, Yahoo! and Adobe. Everyone still uses Yahoo, right?

But I also bought 50 shares of Tucows, the domain name registrar, for $3.75 a pop. I recall being inspired by a post from original ICANN blogger Bret Fausett, who coincidentally now works for Tucows, in which he touted the stock.

I left the US at the end of 2007 and went travelling for a while before returning to the UK.

At some point in 2009, when I tried to sell up and close the account, I was told that because I no longer had the US bank account I signed up with, I would be unable to access the funds.

So I pretty much chalked the experience down to an idiot tax and forgot all about it.

In 2010 I launched this web site.

Recently, I remembered the account and, trading platform policies having changed in the meantime, discovered I probably would be able to access the funds after all.

That $1,000 had turned into over $19,000 over the intervening 15 years, and the Tucows position had grown by almost 2,000%, a gain of over $3,700.

I’ve sold all my shares and am in the process of closing the account. After that, I won’t own any shares in any companies.

In short, I’m probably going to make a few grand by selling Tucows stock that I’ve owned for the last 15 years but which, until recently, I thought I’d lost forever.

Make of that what you will.

I’m disclosing it now not because I think I’ve had any market-moving impact on the stock over the years, but because it just seems like the kind of thing that needs to be disclosed.

Earnings reports: GoDaddy, Tucows and NameSilo report growth

Three of the industry’s largest registrars announced revenue growth in their latest reporting periods in recent days.

GoDaddy

Market-leading GoDaddy reported a whopping 18.8% year-over-year revenue growth from domains in its first quarter, with $422.7 million.

CEO Aman Bhutani told analysts that much of this growth is being driven by the company’s emerging strategy of acting as a secondary-market intermediary, making it easier for domainers to sell their domains quickly to end-users (what it calls “independent customers”) and vice-versa.

“Independent customers added over 200,000 domain names that had otherwise been passive into the aftermarket, spurring activity for domain investors,” Bhutani said.

It currently has over 20 million domains listed on its aftermarket platform, contributing 10% of total revenue, the first time it’s broken into double-digits, analysts were told.

Domains was the best-performing segment in growth terms by some margin.

Including its other segments, GoDaddy’s overall Q1 revenue was up 13.8% year over year, at $901.1 million. It had a net income of $10.8 million, compared to $43.2 million a year earlier.

Tucows

Tucows reported domain services revenue up 4%, from $59.5 million in Q1 2020 to $61.2 million, with adjusted EBIDTA of $13.8 million versus $11.5 million a year ago.

CEO Elliot Noss said in a statement that new domain registration growth was slowing following the “pandemic surge” it experienced in 2020, when lockdown-hit businesses flew online to keep afloat.

Including its non-domain segments, Tucows reported Q1 revenue of $70.9 million. That was down from $84 million a year earlier largely as a result of the sale of its Ting Mobile business to Dish Network.

Net income for the quarter was $2.1 million, down 24% compared to the year-ago period.

NameSilo

Fast-growing registrar NameSilo reported revenue for its full-year 2020 of $31 million, up 14.3% on 2019. That was primarily driven by domains growth and its newish add-on services, it said, but it does not break down its revenue by segment.

It had net income of $6.5 million in fiscal 2020, compared to a net loss of $4 million in 2019.

It added 235,347 net domains in gTLDs in 2020, according to reports filed with ICANN, ending 2020 with 3,663,090 names under management. NameSilo said that number is now around 3.9 million.

Stick a fork in Nominet’s leadership. Tucows votes to fire half the board

Tucows has become the latest registrar to say it will vote to fire five of Nominet’s 11 directors, including its CEO and chair, making the success of the ongoing member-driven coup pretty much inevitable.

The company said yesterday that it has already voted for the PublicBenefit.uk campaign’s motion, to be considered at the .uk registry’s Emergency General Meeting on Monday.

Tucows is Nominet’s fourth-largest registrar, with 381,468 domains under management. Its voting rights are capped at 3% of the total.

PublicBenefit.uk now says it has 29.1% of all votes backing its campaign, with 473 members signed up.

Because the threshold to pass its resolution is a simple majority of those who actually turn out to vote on the day, the likelihood of the five directors surviving the EGM are now surely negligible.

The first motion kicks out CEO Russell Haworth, chair Mark Wood, CFO Ben Hill, registry managing director Eleanor Bradley and appointed non-executive director Jane Tozer.

The second, which Nominet refused to put on the ballot, would have appointed two new directors: Sir Michael Lyons, who will serve as chair, and Axel Pawlik, deputy chair. Lyons is a former chair of the BBC Trust, who in 2015 oversaw a review into Nominet’s corporate governance. Pawlik is a former MD of European IP address registry RIPE-NCC.

Both have promised to refocus Nominet by abandoning its attempts to diversify into commercial areas such as cybersecurity, while also reducing .uk wholesale prices and donating more of its profits to public benefit causes.

In throwing its weight behind the resolution, Tucows’ director of domains Ashley La Bolle said in a blog post:

Most registries, but particularly country code registries are, or should be, very profitable operations. A country code TLD is also a public asset and an important component of a nation’s critical infrastructure. The registry should have a narrow and focused mandate, deliver a stable and secure service, operate in a risk-averse manner, and manage costs appropriately. As a public asset, surplus funds from the operation of a registry should be delivered to thoughtful and relevant public benefit initiatives, while also containing and reducing costs for the millions of businesses and consumers that use and rely on the domain names.

It’s the second of Nominet’s top 10 registrars to back PublicBenefit.uk, after #7 Namecheap, which has 201,355 .uk names under management.

The Internet Commerce Association, which represents the interest of domain investors but is not a Nominet member, said it took no position on the resolution, but broadly supported the overarching goals:

The ICA urges Nominet members to support efforts to restore Nominet’s core mission to operate the registry at cost as a not-for profit. Nominet’s management should never raise registration fees beyond what it takes to operate the registry in a prudent manner, with any excess revenue being directed to worthy causes and not to growing the breadth of Nominet’s limited mandate.

Those Nominet members who have pledged support for the board shakeup are being urged to give their voting proxy to Simon Blackler, who runs the registrar Krystal Hosting and initiated the PublicBenefit.uk campaign, before close of business UK time today.

Blackler says that almost all of these members have already voted.

It’s going to take an unprecedented turnout of Nominet’s remaining membership, with the vast majority opposed to the firings, to save these five directors at this point.

UPDATE: This article was updated shortly after its original posting to clarify that Nominet had refused to put the campaign’s second motion on the ballot.

The two biggest registrars knock it out of the park in Q2

GoDaddy and Tucows, the industry’s two largest registrars, both last week posted very strong second-quarter results due to the beneficial impact of the coronavirus lockdown.

Market-leader GoDaddy in particular seems to have knocked it out of the park, adding a ridiculous 400,000 net new customers during the April-June period, the strongest quarterly performance in the company’s 20-year history.

The company reported domains revenue of $369.6 million, up 10.5% on the year-ago quarter, its strongest-performing segment.

Tucows, meanwhile, reported domains revenue essentially flat at $60 million, but pointed to registration growth as an indicator of its showing.

Tucows CEO Eliot Noss said in prerecorded remarks that new registrations from its reseller channel were up 41% in the quarter, with overall wholesale registrations up 7% to 4.3 million.

In the retail channel, domains under management was up 9% year-over-year to 400,000, with new registrations up more than 20%.

The CEOs of both companies were unambiguous that the coronavirus pandemic could take credit for their results. Noss said:

As expected, in Q2 we saw the full effects of the pandemic that we began to experience toward the end of Q1, with businesses globally moving quickly online, and displaced workers turning to entrepreneurship as the next stage in their career paths. A large proportion of that new registration activity was those resellers who focus on helping small and micro-sized businesses and start-ups establish a web presence for the first time.

GoDaddy CEO Aman Bhutani characterized the virus as a catalyst for businesses stubbornly remaining offline to finally get a web presence, telling analysts:

COVID-19 has pushed a number of people past the point of inertia where they were not adopting digital… because people have no choice but to go digital to support their businesses, we’re seeing people experimenting with ideas. We’re seeing people come online, even though they had hesitated to do it in the past.

Overall, GoDaddy reported revenue up 9.4% at $806.4 million. Its net loss was $673.2 million, due mostly to a one-time tax-related payment.

Tucows overall revenue was $82.1 million, down from $84.1 million, largely due to the drag factor of its recently restructured Ting Mobile business. Net income was $157,000, down from $2.6 million.

Tucows sells off Ting business, retreats into the back-end

Tucows has sold its Ting Mobile brand and customer based to DISH Network, repositioning itself as a provider of white-label back-end mobile services.

The company which is also the second-largest domain registrar, has found success in recent years as a mobile virtual network operator (MVNO) with Ting. Following the DISH deal, it will become a mobile services enabler, or MSE.

It’s basically a move away from providing customer-facing mobile services. Instead, it will provide the back-end technology platform, and DISH is its first customer.

CEO Elliot Noss said in a prerecorded statement:

We still get asked about the connective tissue between Domains and the mobile business, and it all boils down to our competence in billing, provisioning, and customer service for underserved technology markets.

He added that Tucows has been approached by other potential MSE partners over the years.

DISH gets to use the Ting brand for two years, with an option to acquire it at the end. Tucows will continue to offer Ting wired broadband services, but will change its name if DISH exercises the buyout option.

All Ting mobile customers have been handed over to DISH as of Saturday, but no money has changed hands up-front. Instead, Tucows expects to see increased margins over time from the cost savings and monthly fees DISH will pay it. It will also be paid for transitioning the business over to DISH.

Tucows expects the deal to be “neutral to slightly negative” to its 2020 earnings.

DISH is primarily a satellite television provider, but it entered the mobile market a month ago with the acquisition of Boost Mobile.