Twitter “completely unresponsive” on clickable domains

Elon Musk’s Twitter is “completely unresponsive” to outreach about Universal Acceptance of domain names, including problems such as the lack of linkification of new gTLD domains, according to an ICANN technologist.

Speaking at an ICANN 79 Prep Week session yesterday, senior UA technology manager Arnt Gulbrandsen said the Org has been attempting to work with major platforms such as Google’s Gmail and WordPress to encourage support for newer, longer gTLDs and internationalized domain names, but with mixed results.

“What we are doing is identifying the most important, the biggest actors… testing, reaching out or contributing changes,” he said. “We don’t work equally with all. If someone’s unresponsive, then we more or less stop talking to them and hope that they grow less important as time passes.”

“This means Twitter,” he said. “Twitter is completely unresponsive.”

Twitter and other platforms such as WhatsApp have been criticized recently by the people behind gTLDs including .music and .tube for failing to “linkify” their domains. When you tweet a .music domain without the http:// prefix it will not automatically become clickable, for example.

Twitter’s cut-off point for recognizing TLDs appears to be mid-2020. The three gTLDs delegated after that — .spa, .music and .kids — do not currently linkify.

Gulbrandsen said ICANN has been getting a more encouraging response from developers within the WordPress ecosystem, where ICANN discovered that UA support relies a great deal on just three software components maintained by volunteer developers — linkify-it, phpautolink and phpmailer.

“I’m really happy about the responses from some of these obscure, open-source maintainers,” he said. “They really want to do the best for the world, and they are volunteers mostly.”

Two of the identified components currently support UA and ICANN is working with phpmailer, he said. ICANN has also been contributing UA code even further down the stack, to programming languages such as Java, Python and Ruby, he said.

Gulbrandsen’s presentation came during the ICANN 79 Prep Week session on UA, which included contributions from members of various UA working groups and focused largely on IDN and email problems. You can listen to the session in full here.

.tube registry claims victory in linkification fight

Latin American Telecom, the company that runs the .tube gTLD, has claimed victory in its fight to get popular social media apps to “linkify” more than 400 TLDs that have gone live in the last eight years.

As I reported two weeks ago, CEO Rami Schwartz managed to figure out that any TLD that entered the root after November 2015 wasn’t being recognized by apps such as WhatsApp, the world’s most-popular messaging app.

This meant that any attempt to share a URL in .tube or 467 other TLDs (including major dot-brands and geo-gTLDs) would be frustrated by the fact that WhatsApp would not automatically turn the URL into a clickable link.

The root cause of the problem appeared to be a library used in the Android operating system, which had a hard-coded list of valid TLDs that had not been updated since November 2015.

In a press release today, the registry reported that the library in question was updated on September 11 (hey, that’s the same day I published my article!) with a brand-new list of TLDs.

So it seems the linkification issue will be solved, once the updated software actually makes it to affected devices.

There are not many TLDs in the pipeline for delegation for the next four years — maybe some contested 2012-round stragglers and the odd IDN ccTLD — so this particular issue is unlikely to cause much more upset for a while.

“This story exemplifies how the perseverance of a small company unearthed a Universal Acceptance issue of global significance, rallying the support of industry leaders and setting a precedent for cooperation that can positively impact billions of internet users,” the registry said in its press release.

Is this why WhatsApp hates some TLDs but not others?

Developers of major pieces of internet software, including the world’s most-popular messaging app, may be relying on seriously outdated lists of top-level domains.

That’s the picture that seems to be emerging from one new gTLD operator’s quest to discover why WhatsApp doesn’t recognize its TLD, and many others including major dot-brands, as valid.

And ICANN isn’t interested in helping, despite its declared focus on Universal Acceptance, the CEO of this registry claims.

When most social media apps detect the user has inputted a URL or domain name, they automatically “linkify” it so it can be easily clicked or tapped without the need for copy/paste.

But when Rami Schwartz of new gTLD .tube discovered that .tube URLs sent via WhatsApp, said to have two billion users, were not being linkified, despite the TLD being delegated by ICANN almost eight years ago, he set out to find out why.

Schwartz compiled a spreadsheet (.xlsx) listing which gTLDs are recognized by WhatsApp and which are not and discovered a rough cut-off point in November 2015. TLDs delegated before then are linkified, those delegated after were not.

According to my database, 468 TLDs have been delegated since December 2015, though not all are still in the root. That’s about a third of all TLDs.

This means that, for example, .microsoft domains linkify but .amazon and .apple domains do not; .asia domains linkify but .africa and .arab domains do not; .london works but .abudhabi doesn’t. Even .verisign missed the cut-off.

If WhatsApp users include a “www.” or “http://” then the app will linkify the domain, even if the specified TLD does not exist.

During the course of a discussion on the web site of the Public Suffix List — which maintains an open-source list of all TLDs and the levels at which names may be registered — it was discovered that the problem may be deeper rooted than the WhatsApp app.

It turns out a library in the Android operating system contains a hard-coded list of valid TLDs which hasn’t been updated since November 24, 2015.

Any app relying on Android to validate TLDs may therefore be susceptible to the same problem — any TLD younger than seven years won’t validate. Schwartz tells us he’s experienced the same issues with the Facebook app on Android devices.

The problem is of particular concern to Schwartz because he’s been planning to market .tube as a form of link-shortening service, and without full support among the most popular messaging apps such a service would be much less attractive.

“I can’t launch this now if it’s not going to work in WhatsApp, if it’s not going to work in Facebook,” he said.

While engineers from Facebook/WhatsApp parent Meta now seem to be looking into the problem, Schwartz says his complaints fell on deaf ears for a long time.

He additionally claims that “ICANN doesn’t really care about universal acceptance” and his attempts to get the Org to pay attention to the problem have been brushed off, despite ICANN making Universal Acceptance one of its key priorities.

Schwartz says ICANN is much more interested in UA when it comes to internationalized domain names (those in non-Latin scripts, such as Arabic or Chinese) and not the technical issues that underpin the functionality of all TLDs.

“I’ve no idea why ICANN makes the decisions it makes, but I think it has to do with inclusion, I think it has to do with diversity, I think it has to do with a lot of things — not technical,” he said. “But this is a technical issue.”

ICANN maintains a set of UA technical resources on its web site and supports the work of the independent Universal Acceptance Steering Group.

Universal unacceptance? ICANN lets XYZ dump languages from UNR gTLDs

Even as CEO Göran Marby was accepting an ambassadorship from the Universal Acceptance Steering Group last month, ICANN was quietly approving a registry’s plan to drop support for several languages, potentially putting dozens of domains at risk.

It seems portfolio registry XYZ.com was having problems migrating the 10 gTLDs it recently acquired in UNR’s firesale auction from the UNR back-end to long-time partner CentralNic, so it’s cutting off some language support to ease the transition.

The company told ICANN in a recent Registry Services Evaluation Process request (pdf) that internationalized domain names in Cyrillic, Chinese, Japanese and German were “causing issues with the [Registry System Testing] for the technical transition”.

“So, in order to move forward with the migration to CentralNic, we have no choice but to remove support for these IDNs. This will only impact fewer than 50 registrations in these TLDs,” the company told ICANN.

I asked both XYZ and CentralNic whether this means the IDN domains in question would be deleted but got no response from either.

Support for the four languages will be removed in .christmas, .guitars, .pics, .audio, .diet, .flowers, .game, .hosting, .lol, .mom according to contractual amendments that ICANN has subsequently approved.

The RSEP was published the same week ICANN signed a memorandum of understanding with .eu registry EURid, promising to collaborate on IDNs and universal acceptance.

The same week, Marby, who has stated publicly on several occasions his commitment to IDNs and UA, was named an honorary ambassador of the UASG to “help amplify the importance of UA work to enable a multilingual Internet”.

UPDATE July 24, 2022: CentralNic CTO Gavin Brown says:

I can confirm that no domains will be deleted or suspended due to the withdrawal of these IDN tables. The RSEP request template we provided to XYZ incorrectly stated that domains would be deleted, however, neither we nor XYZ have any plans to delete or suspend any domains, and we hope to re-enable the IDN tables in the near future.

Universal Acceptance – making the internet work for everyone [Guest Post]

Editor’s note: this is a guest post written by Aman Masjide, head of compliance at new gTLD registry Radix.

Back in 2014, to foster innovation and to better the choice in domain names, ICANN introduced new generic top-level domains through its New gTLD Program. It was a monumental move that enabled businesses, individuals, and communities across the globe to mark their presence on the internet.

Allowing users to be present digitally in their chosen language (non-ASCII characters and scripts) gave opportunities to local businesses, civil societies, and governments to better serve their communities.

Analysys Mason conservatively estimates that there is scope of $9.8 billion growth in potential revenue from both; existing users who are using new domain names and from new internet users coming online through Internationalized Domain Names (IDNs).

To achieve this, Universal Acceptance of new gTLDs and IDNs is critical in making the Internet more accessible to the next billion users. Founded in February 2015, the Universal Acceptance Steering Group (UASG) undertakes activities to promote Universal Acceptance of all valid domain names and email addresses.

Through its ambassadorship and local Initiative programs, UASG promotes Universal Acceptance globally. Their efforts are divided and executed through five working groups that include:

• Technology Working Group
• Email Address Internationalization Working Group
• Communications Working Group
• Measurement Working Group
• Local Initiatives Working Group

Before we get into the acceptance of new domain extensions (nTLDs), we must first understand what acceptance means and how it’s measured.

The Universal Acceptance Steering Group’s mission sums up acceptance in one short statement: “All domain names and all email addresses work in all software applications.”

While this is a simple understanding of the concept, for an end user of an nTLD, this statement further branches out into multiple questions such as:

• Will my domain name work on all platforms/applications–online or offline?
• Will my email address on a new domain extension get accepted on all websites/platforms and pass all the validation tests?
• Will my emails on new domain extensions, once accepted, stop going into the junk folder?
• Will I be able to use all the features of a website/platform irrespective of my domain extensions? For example, will a social media platform accept a new domain extension in the bio, comments, posts, messenger, etc, and process it exactly like any other legacy TLD?

The Universal Acceptance (UA) of all domain names and email addresses requires that every piece of software is able to accept, validate, process, store, and display them correctly and consistently.

As a new domains registry, it was critical for us to understand what the gaps were and how to close them so that the internet operates the same for nTLD users as it does for the legacy TLD users.

Initial research concluded that UA readiness issues occur when applications are not able to handle the following categories of a domains name or email addresses:

Domain Names

• New short top-level domain names: example.fun, example.site
• New long top-level domain names: example.berlin, example.space
• Internationalized Domain Names: παράδειγμα.ευ

Email Addresses

• ASCII@ASCII; new short or long TLD: ekrem@misal.istanbul
• ASCII@IDN: john@société.org
• Unicode@ASCII: 测试@example.com
• Unicode@IDN: ईमेल@उदाहरण.भारत
• Unicode@IDN; right to left scripts: لیم@لاثم.عقوم ای

For Universal Acceptance to succeed, it needs to be examined holistically.

Over the years, UASG working group members have conducted several gap analysis on programming languages and frameworks, networking command-line tools, web browsers, websites, and have made great strides in acceptance of new domain extensions.

According to UASG’s FY 2020 report, tests conducted on top websites showed that

• The acceptance rate of emails on short nTLDs has increased from 91% in 2017 to 98.3% in 2020.
• The acceptance rate of emails on long nTLDs has increased from 78% in 2017 to 84.8% in 2020.

Note: The table above compares the 2020 results to the earlier 2017 and 2019 testing results.

Two important caveats should be remembered in this case:

• Different email addresses were tested (but they were of the same type).
• The websites tested in 2020 were different from previous ones as they were the 50 most popular in the 20 countries rather than the 1,000 most popular globally.

However, these results may still be used to compare overall trends.

Universal Acceptance Readiness Report 2020 (pdf) also segregated test websites as per different categories such as eCommerce, government, education, etc and the results were promising.

Such studies help UASG ambassadors and advocates to identify and focus on websites of a specific category that require immediate attention. We conducted a similar study at Radix where we analysed top websites belonging to different categories. These were the results (click to enlarge):

While the acceptance rates for new short and new long cases is more than 80% under most categories, we see a drastic dip when a domain is on an IDN TLD. Such comparisons highlight problem areas and provide direction to ambassadors and members who are advocating for Universal Acceptance.

Radix’s contribution to UASG

UA is something that affects nTLD users the most. This is why it’s crucial to focus on the feedback that we receive from them. At Radix, we work closely with our users to ensure we have the first hand information on any UA related issues faced by the customer.

The feedback could be about linkification, validation or acceptance of emails on nTLDs on different websites and platforms. Radix also actively invests its resources in gap analysis by testing various websites and social media platforms. We are also part of the ambassadorship program promoting and supporting local and global UA initiatives.

Here are some of the UASG initiatives that Radix is part of:

• Participating in the Technical And Measurement working group
• Participating in workshops on Universal Acceptance such as:
  – India Internet Week 2020 Online Edition
  – FICCI – Bhashantara – Panel Discussions
• Sharing gaps and issues reported by customers to UASG

At Radix, our objective is to ensure that nTLDs are accepted across websites and platforms. To achieve this, we actively work with UASG and share as many issues and gaps noticed and reported by customers.

Contribution by other registries

A key objective for most registries is to ensure great customer experience when it comes to their nTLDs and I’ve always admired it when registry operators have actively taken initiative and participated in the five UASG groups mentioned above.

One of the ways to do this is to capture all the queries and complaints reported by their customers/registrar partners and share it with UASG. This will help their support team direct their resources in solving the problems and encouraging those websites to become UA compliant.

Contribution by registrars

When it comes to UA-related issues, registrars are the first in chain to receive a complaint or feedback from the user. Therefore, it’s crucial that their support teams have all the necessary information needed on how to best handle such complaints.

For now, they can:

• Inform the customer about the potential UA issue and raise a request on behalf of the customer with UASG. Issues can be logged at – https://uasg.tech/global-support-center/
• Report these instances to the Registry Operator so that they can connect and follow up with UASG.
• Join any of the five working groups and participate.

The path ahead

The UASG is consistently compiling and sharing all the important information needed for organizations and developers to become UA ready. This is not only about ensuring the readiness of a system to accept certain TLDs or emails, but also about realising the full potential of an organization by connecting with people and businesses that might not be even on it’s radar.

Every successful step taken by an organization towards UA readiness is also a step towards equality and inclusiveness on the internet.

Guest poster Aman Masjide leads compliance and abuse mitigation at Radix.

New gTLDs still a crappy choice for email — study

New gTLDs may not be the best choice of domain for a primary email address, judging by new research.
Over 20% of the most-popular web sites do not fully understand email addresses containing long TLDs, and Arabic email addresses are supported by fewer than one in 10 sites, a study by the Universal Acceptance Steering Group has found.
Twitter, IBM and the Financial Times are among those sites highlighted as having only partial support for today’s wide variety of possible email addresses.
Only 7% of the sites tested were able to support all types of email address.
The study, carried out by Donuts and ICANN staff, looked at 749 websites (in the top 1,000 or so as ranked by Alexa) that have forms for filling in email addresses.
On each site, seven different email addresses were input, to see whether the site would accept them as valid.
The emails used different combinations of ASCII and Unicode before the dot and mixes of internationalized domain name and ASCII at the second and top levels.
These were the results (click to enlarge or download the PDF of the report here):

The problem with these numbers, it seems to me, is the lack of a control. There’s no real baseline to judge the numbers against.
There’s no mention in the paper about testing addresses that use .com or decades-old ccTLDs, which would have highlighted web sites that with broken scripts that reject all emails.
But if we assume, as the paper appears to, that all the tested web sites were 100% compliant for .com domains, the scores for new gTLDs are not great.
There are currently over 800 TLDs over four characters in length, but according to the UASG research 22% of web sites will not recognize them.
There are 150 IDN TLDs, but a maximum of 30% of sites will accept them in email addresses.
When it comes to right-to-left scripts, such as Arabic, the vast majority of sites are totally hopeless.
UASG dug into the code of the tested sites when it could and found that most of them use client-side code — JavaScript processing a regular expression — to verify addresses.
A regular expression is complex bit of code that can look something like this: /^.+@(?:[^.]+\.)+(?:[^.]{2,})$
It’s not every coder’s cup of tea, but it can get the job done with minimal client-side resource overheads. Most coders, the UASG concludes, copy regex they found on a forum and maybe tweak it a bit.
This should not be shocking news to anyone. I’ve known about it since 2009 or earlier when I first started ripping code from StackOverflow.
However, the UASG seems to be have been working on the assumption that more sites are using off-the-shelf software libraries, which would have allowed the problem to be fixed in a more centralized fashion.
It concludes in its paper that much greater “awareness raising” needs to happen before universal acceptance comes closer to reality.

Companies losing $10 BILLION by ignoring new gTLDs — report

The world economy is “conservatively” losing out on almost $10 billion of annual revenue due to a lack of support for new gTLDs and internationalized domain names, according to an ICANN-commissioned research report.
The report, conducted by Analysys Mason for the semi-independent Universal Acceptance Steering Group, calculated that patchy new gTLD support means $3.6 billion of activity is lost, with lack of IDN support costing $6.2 billion.
Despite “new” gTLDs being around for a decade and a half, there are still plenty of web sites and apps that incorrectly assume that all TLDs are either two or three characters. Others don’t support non-Latin scripts.
This leads to internet users abandoning transactions, the report says, when their email addresses are rejected as invalid.
Mason calculated the $3.6 billion number by multiplying the estimated number of email addresses using new gTLD domains (152 million) by the estimated average annual revenue generated per email address ($360), then calculating what portion of these transactions cannot happen due to incomplete TLD support.
Earlier research by .CLUB Domains suggests that 13% of sites do not support new gTLDs, so that’s the number Mason used. The researchers then cut the number in half, to account for the 50% of people it reckons would simply switch to an email address in a legacy TLD name.
That gets you to $3.6 billion of potential revenue lost for want of gTLD support.
Another, more cynical way to spin this would be to say that new gTLDs are causing $3.6 billion of economic damage. After all, if everyone were to use legacy TLDs there would be no problem.
For the IDN number, Mason calculated how many users of five major language groups (Russian, Chinese, Arabic, Vietnamese and Indian languages) are not currently online, then estimated how much revenue would be generated if just 5% of these users (17 million people) were persuaded online by the existences of IDN TLDs.
The report was commissioned in order to raise awareness of the financial benefits of universal acceptance.
The UASG has spent most of its efforts so far focusing on UA as a “bug fix” to be communicated to engineers, so the report is intended to broaden its message to catch the attention of the money people too.
The report, which goes into much more detail about how the numbers were arrived at, can be downloaded here.

.sexy may be blocked in Iran

Some networks in Iran appear to be systematically blocking Uniregistry’s .sexy gTLD.
That’s one of the conclusions of a slightly odd experiment commissioned by ICANN.
The newly published An Analysis of New gTLD Universal Acceptance was conducted by APNIC Labs. The idea was to figure out whether there are any issues with new gTLDs on the internet’s DNS infrastructure.
It concluded that there is not — new gTLDs work just fine on the internet’s plumbing.
However, the survey — which comprised over 100 million DNS resolution attempts — showed “One country, Iran, shows some evidence of a piecemeal block of Web names within the .sexy gTLD.”
The sample size for Iranian attempts to access .sexy was just 30 attempts. In most cases, users were able to resolve the names with DNS, but HTTP responses appeared to be blocked.
The survey did not test .porn or .adult names, but it might be safe to assume similar behavior in those gTLDs.
APNIC also concluded that Israel’s .il ccTLD, included in the report as a known example of TLD blocking at the national level, is indeed blocked in Iran and Syria.
The study also found that there may be issues with Adobe’s Flash software, when used in Internet Explorer, when it comes to resolving internationalized domain names.
That conclusion seems to have been reached largely because the test’s methodology saw a Flash advertisement discretely fetching URLs in the background of web pages using Google Ads.
When the experimenters used HTML 5 to run their scripts instead, there was no problem resolving the names.
The study did not look at some of the perhaps more pressing UA issues, such as the ability for registrants and others to use new gTLD domain names in web applications.

Site names and shames shoddy TLD support

A self-professed geek from Australia is running a campaign to raise awareness of new gTLDs by naming and shaming big companies that don’t provide comprehensive TLD support on their web sites.
SupportTheNew.domains, run by university coder Stuart Ryan, has been around since last June and currently indexes support problems at dozens of web sites.
The likes of Facebook, Amazon, Adobe and Apple are among those whose sites are said to offer incomplete support for new gTLDs.
It’s the first attempt I’m aware of to list “universal acceptance” failures in any kind of structured way.
Ryan says on the site that he set up the campaign after running into problems signing up for services using his new .email email address.
The site relies on submissions from users and seems to be updated whenever named companies respond to support tickets.
Universal acceptance is a hot topic in the new gTLD space, with ICANN recently creating a steering group to promote blanket TLD support across the internet.
Often, sites rely on outdated lists of TLDs or regular expressions that think TLDs are limited to three characters when they attempt to verify domains in email addresses or URLs.

Group forms to stop new gTLDs breaking stuff

A little over a year into the live phase of the new gTLD program, a group of domain industry companies are getting together to make sure the expansion is supported across the whole internet.
A new Universal Acceptance Steering Group has formed, with the support of ICANN and the Domain Name Association, to help fix many of the compatibility problems facing new gTLD registrants today.
“The basic problem is that these new types of domains and email addresses just break stuff,” Google’s Brent London said during a UASG meeting at the ICANN meeting in Singapore last week.
“You try to use an internationalized domain or a long new gTLD, or even a short new gTLD, or certainly an internationalized email address and you’re likely to run into problems,” he said. “What we’re doing is going around asking developers to make their products work.”
Universal acceptance is a long-understood problem. Even 15 years after the approval of .info there are still web sites that validate email addresses by ensuring the TLD is no longer than three characters in length.
But the 2012 new gTLD round has brought the issue into sharper focus, particularly given the introduction of internationalized domain names, IDNs, which use non-Latin scripts.
Over the last year we’ve seen scattered examples of popular software — including browsers, instant messaging and social media apps — not recognizing new gTLD domains as domains. The problems I’ve seen are usually fixed quite quickly.
While I’ve not seen any deal-breakers that would prevent me registering a new gTLD domain, I gather that IDN email addresses are often basically unusable, due to the chain of dependencies involved in sending an email.
In my experience as a programmer, supporting all TLDs is not a particularly challenging problem when you’re coding something afresh.
However, when bad practices have been coded in to large, sprawling, interdependent systems over decades, it could be likened to the Y2K problem — the so-called Millennium Bug that caused developer headaches worldwide at the end of the last century.
There’s also a tonne of bad advice on the web, with coders telling other coders to validate domains in ways that do not support an expanding root.
UASG members think the problem is large-scale and that it’s a long-term project — 10 years or more — to fix it satisfactorily.
Members include Donuts, Google, Microsoft, Go Daddy and Afilias.
The DNA has started creating a repository of information for developers, with the aim of describing the problem in plain English and providing code samples. Along with other UASG members, there’s a plan to conduct outreach to make more people aware of the acceptance issue.
You can check out the repository in its unfinished state here.
ICANN is getting involved in a coordination role. After the UASG’s inaugural meeting in Washington DC a few weeks ago, ICANN hosted a session during ICANN 52.
It’s also hosting a mailing list and the group’s first conference call, which will take place tomorrow at 1600 UTC.