TAS down for at least another week

If you’re just joining us, welcome to the ICANN community.
The TLD Application System will be offline for another week, possibly more, as ICANN struggles to deal with the fallout from its embarrassing data leakage bug.
ICANN had promised an update today on the timing of the reopening of TAS, which was taken offline April 12 just 12 hours before the new gTLD application filing deadline arrived.
But what applicants got instead was a promise to provide another timing update a week from now.
Chief operating officer Akram Atallah said in a statement:

identifying which applicants may have been affected by the technical glitch, and determining who may have been able to see someone else’s data, require extensive analysis of a very large data set. This is a time-consuming task, but it is essential to ensure that all potentially affected applicants are accurately identified and notified.
Until that process is complete, we are unable to provide a specific date for reopening the application system.
In order to give all applicants notice and an opportunity to review and complete their applications, upon reopening the system we will keep it open for at least five business days.
No later than 27 April 2012 we will provide an update on the reopening of the system and the publication of the applied-for new domain names.

So the best-case scenario, if these dates hold up, would see TAS coming back online Monday, April 30 and closing Friday, May 4.
The April 30 target date for the Big Reveal is clearly no longer possible.
ICANN has stated previously that it expects to take two weeks between the closing of the application window and the revelation of the list of gTLDs being applied for.
The Big Reveal could therefore be postponed until mid-May, almost a month from now.
Any applicant who has already booked flights and hotels in order to attend one of the various reveal events currently being planned by third parties may find themselves out of pocket.
Regular ICANN participants are of course accustomed to delay.
ICANN’s image problem now is rather with the hundreds of companies interfacing with the organization for the first time, applying for new gTLDs, which may be wondering whether this kind of thing is par for the course.
Well, yes, frankly, it is.
That said, the time to avoid this problem was during testing, before the application window opened in January.
Now that the bug has manifested, it’s probably in most people’s best interests for ICANN to fully understand went wrong and what impact it could have had on which applicants. This takes time.

ICANN vows to fight TAS bug “monkey business”

ICANN chief security officer Jeff Moss has pledged to fully disclose what new gTLD application data was leaked to which users via the TLD Application System security bug.
Talking to ICANN media chief Brad White in a video interview, Moss said:

We’re putting everyone on notice: we know what file names and user names were displayed to what people who were logged in and when. We want to do this very publicly because we want to prevent any monkey business. We are able to reconstruct what file names and user names were displayed.

ICANN has been going through its logs and will know “very specifically” what data was visible to which TAS users, he said.
The bug, he confirmed, was related to file deletions:

Under certain circumstances that were hard to replicate users that had previously deleted files could end up seeing file names of users that had uploaded a file… Certain data was being revealed to users that were not seeking data, it was just showing up on their screen.

The actual contents of the files uploaded to TAS were not visible to unauthorized users, he confirmed. There are also no reasons to believe any outside attacks occurred, he said.
He refused to reveal how many applicants were affected by the vulnerability, saying that ICANN has to first double-check its data in order to verify the full extent of the problem.
The interview reveals that the bug could manifest itself in a number of different ways. Moss said:

The problem has several ways it can express itself… we would solve it one way and it would appear another way, we would solve it another way and it would appear a third way. At some point we were just uncomfortable that we understood the core issue and that’s when we took the system offline.

TAS was taken down April 12, just 12 hours before the new gTLD application window closed.
ICANN has been providing daily updates ever since, and has promised to reveal tonight when TAS will reopen for business, for how long, and whether April 30 Big Reveal day has been postponed.
Applicants first reported the bug March 19, but ICANN did not realize the extent of the problem until later, Moss said.

In hindsight now we realized the 19th was the first expression of this problem, but at the time the information displayed made no sense to the applicant, it was just random numbers… at that point there were no dots to connect.

Here’s the video:

First TAS security bug details revealed

The data leakage bug in ICANN’s TLD Application System was caused when applicants attempted to delete files they had uploaded, the organization has revealed.
In his latest daily update into the six-day-old TAS downtime, chief operating officer Akram Atallah wrote this morning:

ICANN’s review of the technical glitch that resulted in the TLD application system being taken offline indicates that the issue stems from a problem in the way the system handled interrupted deletions of file attachments. This resulted in some applicants being able to see some other applicants’ file names and user names.

This sounds rather like an applicant’s file names may have become visible to others if the applicant attempted to delete the file (perhaps in order to upload a revised version) and the deletion process was cut off.
Speculating further, this also sounds like exactly the kind of problem that would have been exacerbated by the heavy load TAS was under on April 12, as lots of applicants simultaneously scrambled to get their gTLD bids finalized to deadline.
Rather than being a straightforward web app, TAS is accessed via Citrix XenApp virtual machine software, which provides users with an encrypted tunnel into a Windows box running the application itself.
As you might expect with this set-up, performance issues have been observed for weeks. Every applicant logged into TAS last Thursday reported that it was running even more slowly than usual.
A security bug that only emerged under user load would have been relatively tricky to test for, compared to regular penetration testing.
But ICANN had some good news for applicants this morning: it thinks it will be able to figure out not only whose file names were leaked, but also who they were leaked to. Atallah wrote:

We are also conducting research to determine which applicants’ file names and user names were potentially viewable, as well as which applicants had the ability to see them.

This kind of disclosure would obviously be beneficial to applicants whose data was compromised.
It may also prove surprising and discomfiting to some applicants who were unwittingly on the receiving end of this confidential data but didn’t notice the rogue files on their screens at the time.
ICANN still plans to provide an update on when TAS will reopen for business this Friday. It will also confirm at the time whether it is still targeting April 30 for the Big Reveal.

New gTLD filing deadline delayed again

It looks like new gTLD applicants are in for more delays after ICANN announced that it will not reopen its TLD Application System tomorrow as planned.
In a statement tonight, chief operating officer Akram Atallah said that the recently discovered data leakage vulnerability has been fixed, but the fix is still being tested.

We believe that we have fixed the glitch, and we are testing it to make sure.
ICANN is committed to reopening the application system as soon as we can confirm that the problem has been resolved and we have had proper time for testing.
We also want to inform all applicants, before we reopen, whether they have been affected by the glitch. We are still gathering information so we can do that.
Accordingly, the application system will not reopen tomorrow.

ICANN shut down TAS last Thursday, just 12 hours before the new gTLD application filing deadline, after discovering a persistent bug that allowed some applicants to see the names of files uploaded by other applicants.
It had planned to open TAS again tomorrow and close it on Friday. However, that’s looking increasingly unlikely.
Atallah said that ICANN “will provide an update on the timing of the reopening no later than Friday, 20 April at 23.59 UTC.”
While ICANN said yesterday that it was still targeting April 30 for its Big Reveal event, subject to change, that’s now looking like an ambitious goal.

ICANN will alert gTLD security bug victims

ICANN plans to inform each new top-level domain applicant whether they were affected by the security vulnerability in its TLD Application System, according to its latest update.
The organization has also confirmed that it is still targeting April 30 for the Big Reveal day, when it publishes (deliberately) the gTLDs being applied for and the names of the applicants.
This morning’s TAS status update, penned by chief operating officer Akram Atallah, does not add much that we did not already know about the data leakage bug. It states:

An intensive review has produced no evidence that any data beyond the file names and user names could be accessed by other users.
We are currently reviewing the data to confirm which applicants were affected. As soon as the data is confirmed, we will inform all applicants whether they were affected.

ICANN staff and outside consultants have been working all weekend to figure out what went wrong, who it affected, and how it can be fixed.
The organization still intends to announce tonight whether it has fixed the problem to the point where it’s happy to reopen TAS to registered users tomorrow. It’s also sticking to is Friday extended submission deadline.

ICANN knew about TAS security bug last week

ICANN has known about the data leakage vulnerability in its TLD Application System since at least last week, according to one new top-level domain applicant.
The applicant, speaking to DI on the condition of anonymity today, said he first noticed another applicant’s files attached to his gTLD application in TAS last Friday, April 6.
“I could infer the applicant/string… based on the name of the file,” said the applicant.
He immediately notified ICANN and was told the bug was being looked at.
ICANN revealed today that TAS has a vulnerability that, in the words of COO Akram Atallah, “allowed a limited number of users to view some other users’ file names and user names in certain scenarios.”
The actual contents of the files are not believed to have been visible.
But other applicants, also not wishing to be identified, today confirmed that they had uploaded files to TAS using file names containing the gTLD strings they were applying for.
It’s not yet known how many TAS users were able to see files belonging to others, or for how long the vulnerability was present on the system.
However, it now does not appear to be something that was accidentally introduced during yesterday’s scheduled TAS maintenance.
This kind of data leakage could prove problematic — and possibly expensive — if it alerted applicants to the existence of competing bids, or caused new competing bids to be created.
ICANN shut down TAS yesterday and does not expect to bring it back online until Tuesday.
The window for filing applications, which had been due to close yesterday, has been extended until 2359 UTC next Friday night.
April 14 Update
ICANN today released a statement that said in part:

we are sifting through the thousands of customer service inquiries received since the opening of the application submission period. This preliminary review has identified a user report on 19 March that appears to be the first report related to this technical issue.
Although we believed the issues identified in the initial and subsequent reports had been addressed, on 12 April we confirmed that there was a continuing unresolved issue and we shut down the system.

It’s worse than you thought: TAS security bug leaked new gTLD applicant data

The bug that brought down ICANN’s TLD Application System yesterday was actually a security hole that leaked data about new gTLD applications.
The vulnerability enabled TAS users to view the file names and user names of other applicants, ICANN said this morning.
COO Akram Atallah said in a statement:

We have learned of a possible glitch in the TLD application system software that has allowed a limited number of users to view some other users’ file names and user names in certain scenarios.
Out of an abundance of caution, we took the system offline to protect applicant data. We are examining how this issue occurred and considering appropriate steps forward.

Given the level of secrecy surrounding the new gTLD application process, this vulnerability ranks pretty highly on the This Is Exactly What We Didn’t Want To Happen scale.
It’s not difficult to imagine scenarios in which a TAS user name or file name contains the gTLD string being applied for.
This is important, competition-sensitive data. If it’s been leaked, serious questions are raised about the integrity of the new gTLD program.
How long was this vulnerability present in TAS? Which applicants were able to look at which other applicants’ data? Did any applicants then act on this inside knowledge by filing competing bids?
If it transpires that any company filed a gTLD application specifically in order to shake down applicants whose data was revealed by this vulnerability, ICANN is in for a world of hurt.

Verisign: our DNS was not hacked

Verisign today reiterated that the recently revealed 2010 security breaches on its corporate network did not affect its production domain name system services.
In a statement, Verisign said:

After a thorough analysis of the attacks, Verisign stated in 2011, and reaffirms, that we do not believe that the operational integrity of the Domain Name System (DNS) was compromised.
We have a number of security mechanisms deployed in our network to ensure the integrity of the zone files we publish. In 2005, Verisign engineered real-time validation systems that were designed to detect and mitigate both internal and external attacks that might attempt to compromise the integrity of the DNS.

The statement followed several news reports that covered the hacks and speculated about the mayhem that could ensue if Verisign’s root or .com zone systems were ever breached.
The information the company has released so far suggests that the attacks were probably against back-office targets, such as user desktops, rather than its sensitive network operations centers.

Hackers stole data from Verisign, Blacknight

Hackers broke into Verisign’s corporate network and made out with sensitive data, it emerged today.
The attacks happened in 2010 and the company does not believe its all-important domain name infrastructure – which supports .com and several other top-level domains – was compromised.
Reuters broke the news today, but the attack was actually revealed in a Securities and Exchange Commission filing last October. The filing said:

In 2010, the Company faced several successful attacks against its corporate network in which access was gained to information on a small portion of our computers and servers. We have investigated and do not believe these attacks breached the servers that support our Domain Name System (“DNS”) network. Information stored on the compromised corporate systems was exfiltrated.

The filing, which was required under recent SEC disclosure rules, goes on to say that the attacks were “not sufficiently reported to the Company’s management” until September 2011.
It adds that Verisign does not know whether the “exfilitrated” – ie, stolen – data was used by the attackers. The filing does not say what was taken.
Back in 2010, Verisign was still a security company. It did not sell off its SSL business to Symantec until August that year. The filing does not say whether SSL data was breached.
As one of the logical single points of failure on the internet, Verisign is of course the subject of regular attacks, mainly of the performance-degrading distributed denial of service variety.
The bigger worry, as Reuters rather breathlessly notes, is that if hackers could compromise the integrity of the DNS root or .com/.net zones, it could lead to mayhem.
In unrelated news, the domain name registrar Blacknight today revealed that it got hacked on Tuesday.
The attackers may have got away with contact information – including email addresses and telephone numbers – for up to 40,000 customers, the company said.
Financial information such as credit card numbers was not compromised, Blacknight said.
The company has contacted Irish data protection regulators and will also inform the police. Customers are advised to change their passwords.
If you’re a Blacknight customer you’ll also want to be on the lookout for “spear-phishing” attacks in the near future. When the bad guys know your name, it can lead to a more convincing phish.

High-security .bank spec published

BITS, the technology arm of the Financial Services Roundtable, has published a set of specifications for new “high-security” generic top-level domains such as .bank and .pay.
The wide-ranging spec covers 31 items such as registration and acceptable use policies, abusive conduct, law enforcement compliance, registrar relations and data security.
It would also ban Whois proxy/privacy services from financial gTLDs and oblige those registries to verify that all Whois records were fully accurate at least once every six months.
The measures could be voluntarily adopted by any new gTLD applicant, but BITS wants them made mandatory for gTLDs related to financial services, which it calls “fTLDs”.
A letter sent by BITS and the American Bankers Association to ICANN management in late December (pdf) is even a bit threatening on this point:

We strongly urge that ICANN accept the [Security Standards Working Group’s] proposed standards and require their use in the evaluation process. We request notification by 31 January 2012 that ICANN commits to use these fTLD standards in the evaluation of the appropriate gTLD applications. BITS, the American Bankers Association (ABA), and the organizations involved in this effort are firmly committed to ensuring fTLDs are operated in a responsible and secure manner and will take all necessary steps to ensure that occurs.

BITS, it should be pointed out, is preparing its own .bank bid (possibly also .invest and .insure) so the new specs give a pretty good indication of what its own gTLD applications will look like.
ICANN’s Applicant Guidebook does not currently mandate any security standard, but it does say that security practices should be commensurate with the level of trust expected from the gTLD string.
Efforts within ICANN to create a formal High Security Zone Top Level Domain (HSTLD) standard basically fizzled out in late 2010 after ICANN’s board said it would not endorse its results.
That said, any applicant that chooses to adopt the new spec and can demonstrate it has the wherewithal to live up to its very strict requirements stands a pretty good chance of scoring maximum points in the security section of the gTLD application.
Declining to implement these new standards, or something very similar, is likely to be a deal-breaker for any company currently thinking about applying for a financial services gTLD.
Even if ICANN does not formally endorse the BITS-led effort, it is virtually guaranteed that the Governmental Advisory Committee will be going through every financial gTLD with a fine-toothed comb when the applications are published May 1.
The US government, via NTIA chief Larry Strickling, said this week that the GAC plans to reopen the new gTLD trademark protection debate after the applications are published.
It’s very likely that any dodgy-looking gTLDs purporting to represent regulated industries will find themselves under the microscope at that time.
The new spec was published by BITS December 20. It is endorsed by 17 companies, mostly banks. Read it in PDF format here.