ICANN chief security officer Jeff Moss has pledged to fully disclose what new gTLD application data was leaked to which users via the TLD Application System security bug.
Talking to ICANN media chief Brad White in a video interview, Moss said:
We’re putting everyone on notice: we know what file names and user names were displayed to what people who were logged in and when. We want to do this very publicly because we want to prevent any monkey business. We are able to reconstruct what file names and user names were displayed.
ICANN has been going through its logs and will know “very specifically” what data was visible to which TAS users, he said.
The bug, he confirmed, was related to file deletions:
Under certain circumstances that were hard to replicate users that had previously deleted files could end up seeing file names of users that had uploaded a file… Certain data was being revealed to users that were not seeking data, it was just showing up on their screen.
The actual contents of the files uploaded to TAS were not visible to unauthorized users, he confirmed. There are also no reasons to believe any outside attacks occurred, he said.
He refused to reveal how many applicants were affected by the vulnerability, saying that ICANN has to first double-check its data in order to verify the full extent of the problem.
The interview reveals that the bug could manifest itself in a number of different ways. Moss said:
The problem has several ways it can express itself… we would solve it one way and it would appear another way, we would solve it another way and it would appear a third way. At some point we were just uncomfortable that we understood the core issue and that’s when we took the system offline.
TAS was taken down April 12, just 12 hours before the new gTLD application window closed.
ICANN has been providing daily updates ever since, and has promised to reveal tonight when TAS will reopen for business, for how long, and whether April 30 Big Reveal day has been postponed.
Applicants first reported the bug March 19, but ICANN did not realize the extent of the problem until later, Moss said.
In hindsight now we realized the 19th was the first expression of this problem, but at the time the information displayed made no sense to the applicant, it was just random numbers… at that point there were no dots to connect.
Here’s the video:
The data leakage bug in ICANN’s TLD Application System was caused when applicants attempted to delete files they had uploaded, the organization has revealed.
In his latest daily update into the six-day-old TAS downtime, chief operating officer Akram Atallah wrote this morning:
ICANN’s review of the technical glitch that resulted in the TLD application system being taken offline indicates that the issue stems from a problem in the way the system handled interrupted deletions of file attachments. This resulted in some applicants being able to see some other applicants’ file names and user names.
This sounds rather like an applicant’s file names may have become visible to others if the applicant attempted to delete the file (perhaps in order to upload a revised version) and the deletion process was cut off.
Speculating further, this also sounds like exactly the kind of problem that would have been exacerbated by the heavy load TAS was under on April 12, as lots of applicants simultaneously scrambled to get their gTLD bids finalized to deadline.
Rather than being a straightforward web app, TAS is accessed via Citrix XenApp virtual machine software, which provides users with an encrypted tunnel into a Windows box running the application itself.
As you might expect with this set-up, performance issues have been observed for weeks. Every applicant logged into TAS last Thursday reported that it was running even more slowly than usual.
A security bug that only emerged under user load would have been relatively tricky to test for, compared to regular penetration testing.
But ICANN had some good news for applicants this morning: it thinks it will be able to figure out not only whose file names were leaked, but also who they were leaked to. Atallah wrote:
We are also conducting research to determine which applicants’ file names and user names were potentially viewable, as well as which applicants had the ability to see them.
This kind of disclosure would obviously be beneficial to applicants whose data was compromised.
It may also prove surprising and discomfiting to some applicants who were unwittingly on the receiving end of this confidential data but didn’t notice the rogue files on their screens at the time.
ICANN still plans to provide an update on when TAS will reopen for business this Friday. It will also confirm at the time whether it is still targeting April 30 for the Big Reveal.
It looks like new gTLD applicants are in for more delays after ICANN announced that it will not reopen its TLD Application System tomorrow as planned.
We believe that we have fixed the glitch, and we are testing it to make sure.
ICANN is committed to reopening the application system as soon as we can confirm that the problem has been resolved and we have had proper time for testing.
We also want to inform all applicants, before we reopen, whether they have been affected by the glitch. We are still gathering information so we can do that.
Accordingly, the application system will not reopen tomorrow.
ICANN shut down TAS last Thursday, just 12 hours before the new gTLD application filing deadline, after discovering a persistent bug that allowed some applicants to see the names of files uploaded by other applicants.
It had planned to open TAS again tomorrow and close it on Friday. However, that’s looking increasingly unlikely.
Atallah said that ICANN “will provide an update on the timing of the reopening no later than Friday, 20 April at 23.59 UTC.”
While ICANN said yesterday that it was still targeting April 30 for its Big Reveal event, subject to change, that’s now looking like an ambitious goal.
ICANN plans to inform each new top-level domain applicant whether they were affected by the security vulnerability in its TLD Application System, according to its latest update.
The organization has also confirmed that it is still targeting April 30 for the Big Reveal day, when it publishes (deliberately) the gTLDs being applied for and the names of the applicants.
An intensive review has produced no evidence that any data beyond the file names and user names could be accessed by other users.
We are currently reviewing the data to confirm which applicants were affected. As soon as the data is confirmed, we will inform all applicants whether they were affected.
ICANN staff and outside consultants have been working all weekend to figure out what went wrong, who it affected, and how it can be fixed.
The organization still intends to announce tonight whether it has fixed the problem to the point where it’s happy to reopen TAS to registered users tomorrow. It’s also sticking to is Friday extended submission deadline.
ICANN has known about the data leakage vulnerability in its TLD Application System since at least last week, according to one new top-level domain applicant.
The applicant, speaking to DI on the condition of anonymity today, said he first noticed another applicant’s files attached to his gTLD application in TAS last Friday, April 6.
“I could infer the applicant/string… based on the name of the file,” said the applicant.
He immediately notified ICANN and was told the bug was being looked at.
ICANN revealed today that TAS has a vulnerability that, in the words of COO Akram Atallah, “allowed a limited number of users to view some other users’ file names and user names in certain scenarios.”
The actual contents of the files are not believed to have been visible.
But other applicants, also not wishing to be identified, today confirmed that they had uploaded files to TAS using file names containing the gTLD strings they were applying for.
It’s not yet known how many TAS users were able to see files belonging to others, or for how long the vulnerability was present on the system.
However, it now does not appear to be something that was accidentally introduced during yesterday’s scheduled TAS maintenance.
This kind of data leakage could prove problematic — and possibly expensive — if it alerted applicants to the existence of competing bids, or caused new competing bids to be created.
ICANN shut down TAS yesterday and does not expect to bring it back online until Tuesday.
The window for filing applications, which had been due to close yesterday, has been extended until 2359 UTC next Friday night.
April 14 Update
ICANN today released a statement that said in part:
we are sifting through the thousands of customer service inquiries received since the opening of the application submission period. This preliminary review has identified a user report on 19 March that appears to be the first report related to this technical issue.
Although we believed the issues identified in the initial and subsequent reports had been addressed, on 12 April we confirmed that there was a continuing unresolved issue and we shut down the system.