Amazon has reversed, at least temporarily, its decision to yank its free list of the world’s most popular domains, after an outcry from researchers.
The daily Alexa list, which contains the company’s estimate of the world’s top 1 million domains by traffic, suddenly disappeared late last week.
The list was popular with researchers in fields such as internet security. Because it was free, it was widely used.
DI PRO uses the list every day to estimate the relative popularity of top-level domains.
After deleting the list, Amazon directed users to its Amazon Web Services portal, which had started offering the same data priced at $0.0025 per URL.
That’s not cheap. The cost of obtaining same data suddenly leaped from nothing to $912,500 per year, or $2,500 per day.
That’s beyond the wallets, I suspect, of almost every Alexa user, especially the many domain name tools providers (including yours truly) that relied on the data to estimate domain popularity.
Even scaling back usage to the top 100,000 URLs would be prohibitively expensive for most researchers.
While Amazon is of course free to price its data at whatever it thinks it is worth, no notice was given that the file was to be deleted, scuppering without warning goodness knows how many ongoing projects.
Some users spoke out on Twitter.
The quiet death of the @Alexa_Support top million sites is a grievous blow to internet researchers everywhere. $2500 per pull now.
— April King (@aprilmpls) November 21, 2016
Removing the top 1M list is a HUGE mistake. It was extremely useful to assess the impact of new security vulnerabilities. 🙁 @Alexa_Support
— Benjamin Beurdouche (@beurdouche) November 22, 2016
@Alexa_Support I'm disappointed, but I hope you reconsider. The Top 1M list is a standard reference in research. It's simply irreplaceable.
— Santiago Zanella (@xEFFFFFFF) November 22, 2016
I spent most of yesterday figuring out how to quickly rejigger DI PRO to cope with the new regime, but it seems I may have been wasting my time.
After an outcry from fellow researchers, Amazon has restored the free list. It said on Twitter:
Thanks to customer feedback, the top 1M sites is temporarily available again. We’ll provide notice before updating the file in the future
— Alexa Support (@Alexa_Support) November 22, 2016
It seems clear that the key word here is “temporarily”, and that the the restoration of the file may primarily be designed to give researchers more time to seek alternatives or wrap up their research.
Verisign has revived its old name collisions security scare story, publishing this week a weighty research paper claiming millions are at risk of man-in-the-middle attacks.
It’s actually a study into how a well-known type of attack, first documented in the 1990s, might become easier due to the expansion of the DNS at the top level.
According to the paper there might be as many as 238,000 instances per day of query traffic intended for private networks leaking to the public DNS, where attackers could potentially exploit it to all manner of genuinely nasty things.
But Verisign has seen no evidence of the vulnerability being used by bad guys yet and it might not be as scary as it first appears.
You can read the paper here (pdf), but I’ll attempt to summarize.
The problem concerns a virtually ubiquitous protocol called WPAD, for Web Proxy Auto-Discovery.
It’s used by mostly by Windows clients to automatically download a web proxy configuration file that tells their browser how to connect to the web.
Organizations host these files on their local networks. The WPAD protocol tries to find the file using DHCP first, but fails over to DNS.
So, your browser might look for a wpad.dat file on wpad.example.com, depending on what domain your computer belongs to, using DNS.
The vulnerability arises because companies often use previously undelegated TLDs — such as .prod or .global — on their internal networks. Their PCs could belong to domains ending in .corp, even though .corp isn’t real TLD in the DNS root.
When these devices are roaming outside of their local network, they will still attempt to use the DNS to find their WPAD file. And if the TLD their company uses internally has actually been delegated by ICANN, their WPAD requests “leak” to registry or registrant.
A malicious attacker could register a domain name in a TLD that matches the domain the target company uses internally, allowing him to intercept and respond to the WPAD request and setting himself up as the roaming laptop’s web proxy.
That would basically allow the attacker to do pretty much whatever he wanted to the victim’s browsing experience.
Verisign says it saw 20 million WPAD leaks hit its two root servers every single day when it collected its data, and estimates that 6.6 million users are affected.
The paper says that of the 738 new gTLDs it looked at, 65.7% of them saw some degree of WPAD query leakage.
The ones with the most leaks, in order, were .global, .ads, .group, .network, .dev, .office, .prod, .hsbc, .win, .world, .one, .sap and .site.
It’s potentially quite scary, but there are some mitigating factors.
First, the problem is not limited to new gTLDs.
Yesterday I talked to Matt Larson, ICANN’s new vice president of research (who held the same post at Verisign’s until a few years ago).
He said ICANN has seen the same problem with .int, which was delegated in 1988. ICANN runs one of .int’s authoritative name servers.
“We did a really quick look at 24 hours of traffic and saw a million and a half queries for domain names of the form wpad.something.int, and that’s just one name server out of several in a 24-hour period,” he said.
“This is not a new problem, and it’s not a problem that’s specific to new gTLDs,” he said.
According to Verisign’s paper, only 2.3% of the WPAD query leaks hitting its root servers were related to new gTLDs. That’s about 238,000 queries every day.
With such a small percentage, you might wonder why new gTLDs are being highlighted as a problem.
I think it’s because organizations typically won’t own the new gTLD domain name that matches their internal domain, something that would eliminate the risk of an attacker exploiting a leak.
Verisign’s report also has limited visibility into the actual degree of risk organizations are experiencing today.
Its research methodology by necessity was limited to observing leaked WPAD queries hitting its two root servers before the new gTLDs in question were delegated.
The company only collected relevant NXDOMAIN traffic to its two root servers — DNS queries with answers typically get resolved closer to the user in the DNS hierarchy — so it has no visibility to whether the same level of leaks happen post-delegation.
Well aware of the name collisions problem, largely due to Verisign’s 11th-hour epiphany on the subject, ICANN forces all new gTLD registries to wildcard their zones for 90 days after they go live.
All collision names are pointed to 127.0.53.53, a reserved IP address picked in order to catch the attention of network administrators (DNS uses TCP/IP port 53).
Potentially, at-risk organizations could have fixed their collision problems shortly after the colliding gTLD was delegated, reducing the global impact of the vulnerability.
There’s no good data showing how many networks were reconfigured due to name collisions in the new gTLD program, but some anecdotal evidence of admins telling Google to go fuck itself when .prod got delegated.
A December 2015 report from JAS Advisors, which came up with the 127.0.53.53 idea, said the effects of name collisions have been rather limited.
ICANN’s Larson echoed the advice put out by security watchdog US-CERT this week, which among other things urges admins to use proper domain names that they actually control on their internal networks.
XYZ.com has settled a lawsuit filed against it against Verisign stemming from XYZ’s acquisition of .theatre, .security and .protection.
Verisign sued the new gTLD registry operator for “interfering” with its back-end contracts with the previous owners last August, as part of its campaign to compete against new gTLDs in the courtroom.
XYZ had acquired the .security and .protection ICANN contracts from security Symantec, and .theatre from a company called KBE Holdings.
As part of the transitions, all three applications were modified with ICANN to name CentralNic as the back-end registry services provider, replacing Verisign.
Verisign sued on the basis of tortious interference and business conspiracy. It was thrown out of court in November then amended and re-filed.
But the case appears to have now been settled.
Negari issued a grovelling not-quite-apology statement on his blog:
I am pleased to report that the recent case filed by Verisign against CentralNic, Ltd., XYZ and myself has been settled. After looking at the claims in dispute, we regret that as a result of our acquisition of the .theatre, .security and .protection extensions and our arrangement for CentralNic to serve as the backend service provider for these extensions, that Verisign was prevented from the opportunity to pursue monetization of those relationships. As ICANN’s new gTLD program continues to evolve, we would caution others who find themselves in similar situations to be mindful of the existing contracts extension owners may have with third parties.
Registries changing their minds about their back-end provider is not unheard of.
In this case, large portions of Verisign’s final amended complaint were redacted, suggesting some peculiarities to this particular switch.
If there was a monetary component to the settlement, it was not disclosed. The original Verisign complaint had demanded damages of over $2 million.
XYZ.com may be best known for its budget .xyz gTLD, but its portfolio is increasingly leaning toward the super-premium end of the industry price range.
The company entered Early Access Period with its .security, .protection and .theatre gTLDs today, and they ain’t cheap.
.security and .protection are expected to carry retail prices of $3,000 a year, when they hit general availability a week from now.
Today, they’re $65,000 apiece, with the price reducing to $35,000, $15,000, $8,750 and $5,000 over the coming days.
Meanwhile, .theatre starts at $64,000, going down to $32,000, $14,000, $7,000 and $4,000 before finally settling at the GA RRP of $750.
All three gTLDs were acquired by XYZ.com from other applicants.
That was also the case for .cars, .car and .auto, which XYZ runs in a joint venture with Uniregistry, where retail prices are roughly $2,500.
In terms of competition, .security and .protection are probably up against .trust, while .theatre may well find itself in competition with .tickets, which has made inroads in Broadway.
A US judge has dramatically reduced a $10 million ruling Afilias won against Architelos in a trade secrets case.
Architelos, which a jury decided had misappropriated trade secrets from Afilias in order to build its patented NameSentry domain security service, may even be thrown a lifeline enabling it to continue business.
A little over a week ago, the judge ordered (pdf) that the $10 million judgment originally imposed by the jury should be reduced to $2 million.
That won’t be finalized, however, until she’s ruled on an outstanding injunction demanded by Afilias.
The judge said in court that the original jury award had been based on inflated Architelos revenue projections.
The company has made only around $300,000 from NameSentry subscriptions since launch, and its sales pipeline dried up following the jury’s verdict in August.
The service enables TLD registries to track and remediate domain abuse. It was built in part by former Afilias employees.
Afilias has a similar in-house system, not available on the open market, used by clients of its registry back-end business.
Even a reduced $2 million judgment is a bit too rich for Architelos, which is desperately trying to avoid bankruptcy, according to court documents.
But the judge seems to be considering an injunction that would enable Architelos to continue to exist.
It may even be permitted to sell NameSentry, as long as it gives almost a third of the product’s revenue to Afilias for up to five years or until the $2 million is paid off.
The injunction might also grant joint ownership of the disputed patents to the two companies, allowing them to jointly profit from the technology.
This has all yet to be finalized, however, and Afilias can always appeal whatever injunction the judge comes up with.
It emerged in court earlier this month that Architelos offered to give full ownership of its patent, along with NameSentry itself, to Afilias in order to settle the suit, but that Afilias refused.
Afilias is also suing Architelos over the same matters in Canada, but that case is progressing much more slowly.