Amazon has reversed, at least temporarily, its decision to yank its free list of the world’s most popular domains, after an outcry from researchers.
The daily Alexa list, which contains the company’s estimate of the world’s top 1 million domains by traffic, suddenly disappeared late last week.
The list was popular with researchers in fields such as internet security. Because it was free, it was widely used.
DI PRO uses the list every day to estimate the relative popularity of top-level domains.
After deleting the list, Amazon directed users to its Amazon Web Services portal, which had started offering the same data priced at $0.0025 per URL.
That’s not cheap. The cost of obtaining same data suddenly leaped from nothing to $912,500 per year, or $2,500 per day.
That’s beyond the wallets, I suspect, of almost every Alexa user, especially the many domain name tools providers (including yours truly) that relied on the data to estimate domain popularity.
Even scaling back usage to the top 100,000 URLs would be prohibitively expensive for most researchers.
While Amazon is of course free to price its data at whatever it thinks it is worth, no notice was given that the file was to be deleted, scuppering without warning goodness knows how many ongoing projects.
Some users spoke out on Twitter.
The quiet death of the @Alexa_Support top million sites is a grievous blow to internet researchers everywhere. $2500 per pull now.
— April King (@aprilmpls) November 21, 2016
Removing the top 1M list is a HUGE mistake. It was extremely useful to assess the impact of new security vulnerabilities. 🙁 @Alexa_Support
— Benjamin Beurdouche (@beurdouche) November 22, 2016
@Alexa_Support I'm disappointed, but I hope you reconsider. The Top 1M list is a standard reference in research. It's simply irreplaceable.
— Santiago Zanella (@xEFFFFFFF) November 22, 2016
I spent most of yesterday figuring out how to quickly rejigger DI PRO to cope with the new regime, but it seems I may have been wasting my time.
After an outcry from fellow researchers, Amazon has restored the free list. It said on Twitter:
Thanks to customer feedback, the top 1M sites is temporarily available again. We’ll provide notice before updating the file in the future
— Alexa Support (@Alexa_Support) November 22, 2016
It seems clear that the key word here is “temporarily”, and that the the restoration of the file may primarily be designed to give researchers more time to seek alternatives or wrap up their research.
There’s a new domain topping the charts as the most-visited new gTLD site.
A few days ago, namu.wiki replaced searchengines.guru in the top spot, the first time the leading position has changed hands since DI PRO first started tracking daily Alexa scores in July 2014.
namu.wiki appears to be a
Japanese Korean wiki site dedicated to some kind of manga/anime thing. It was registered in April.
searchengines.guru is a Russian forum devoted to discussions of search engine optimization.
Japanese Korean site has an Alexa rank of 1,875 today, compared to 1,994 for the SEO site. The highest score we’ve ever recorded for a new gTLD domain was 717.
Interestingly, only two of the site in the top 10 are in English. Two appear to be associated with spam.
The usual caveats about the reliability of Alexa data applies.
The .xyz and .country gTLDs are currently dominating the league table of most-popular new gTLDs, but massive pop-up advertising campaigns using junk domains can account for the majority of their leading sites.
Today, Amazon’s Alexa site popularity tool sees 2,425 new gTLD domains in its top one million. Of those, 163 are in the top 50,000 sites.
But almost two thirds of those 163 domains appear to be throwaways that receive traffic not because they’re attracting visitors, but because they’re used to serve pop-up advertising, in some cases via adware.
The trend has been visible for a few months now, restricted almost exclusively to .xyz, but over the last two weeks .country has also started to be used in this way.
That’s interesting because, unlike .xyz, .country is not a low-cost gTLD. Go Daddy currently sells it for $39.95 per year.
(UPDATE: As Andrew points out in the comments, Uniregistry is selling .country names for $1 for the first year, which almost certainly explains the .country bump.)
Almost 100 of the top 163 new gTLD domains comprise two unrelated dictionary words put together to make something nonsensical.
Domains such as iciclecellar.country, laborervolcano.country, classkitten.country, sweepstakesglove.country, rewardmen.country, installationdesk.country have recently joined have joined the likes of vasegiraffe.xyz, cactusstew.xyz, bedcrow.xyz, notebookwrist.xyz, wishgrass.xyz, pencilkite.xyz and basketriver.xyz on this list.
As far as I can tell, they’re all registered via Uniregistry and using its free Whois privacy service to mask the identities of the registrants.
Visiting these domains in your browser will either result in an error — where I suspect the site is checking the referrer before deciding whether to show a page — or will send you on a merry redirect chain that terminates in an affiliate marketing sign-up page.
Some of the domains have been discussed in online forums as serving up pop-up ads, which would account for large amounts of traffic and high popularity.
Some have alleged that they’ve seen adware serve up ads from some of these domains.
Pop-up ads may be annoying, but they’re legal and — unlike spam and malware — not usually a violation of gTLD registries’ terms of service.
Whether benefiting from adware would leave a registrant in violation of a registrar or registry’s ToS is also a fuzzy area.
But for the new gTLD industry, which is currently in a mindshare-building mode, this kind of use does not make for great optics. If internet users see new gTLDs most often in an unwanted context, it could impair their trust in the new gTLD environment.
Want to get a full daily list of which new gTLD domains have Alexa rank?
From today DI PRO subscribers can, with our new Popular New gTLD Domains feature.
Updated once a day, the report comprises a list of new gTLD domains that are used by the top one million web sites on the internet, according to data provided by Alexa.
The report currently has 635 domains, but it’s growing.
The report can be used to discover how early adopters are using new gTLDs and which TLDs are generating the most popular web sites.
Here’s a screen shot:
DI PRO subscribers can check it out here.
New gTLDs have been on the market for months now, and the slow process of building out sites is underway.
As regular readers and DI PRO subscribers know, one way DI tracks the popularity of domain names, and therefore their corresponding TLDs, is using Alexa rankings.
These scores are not perfect, but they’re a reasonable way to highlight which new gTLD domain names are getting traffic from internet users.
There are currently 635 new gTLD domains in Alexa’s top one million most-trafficked sites, up from just 10 when I checked almost six months ago, February 19.
Only 35 of those have a ranking better than 100,000.
I visited each in turn today to determine to what use the registrants have put their names.
In this top 35, I found two instances of apparent malware distribution and one instance of possible cybersquatting. Four returned errors. One (www.link) is a blocked name collision name.
Notably, controversial BitTorrent index The Pirate Bay, which has been TLD-hopping for many months and recently got kicked out of .guru, seems to have found a home in .uno.
Only one of the domains redirects to a domain in a different TLD.
One (gen.xyz) is a new gTLD registry’s official homepage.
The remainder represent a broad cross section of regular internet usage: blogs, tools, photos, sport, porn, get-rich-quick schemes, forums, file-hosting, and so on and so forth.
Varying degrees of professionalism can be found on these sites. Some are very pretty, others very ugly.
There’s even one site on the list that appears to be a legitimate corporate home page. On reflection, no it isn’t. It’s a Get-Rich-Quick site.
These are my results, make of them what you will.
|warriors.tips||36897||Y||Get Rich Quick||N|
|jid-company.trade||79730||Y||Get Rich Quick||N|
|iif.club||84736||Y||Get Rich Quick||N|
|bankcode.today||97414||Y||Get Rich Quick||N|