ICANN not done with TAS bug analysis

Despite sending out hundreds of notifications to new gTLD applicants today, it looks rather like ICANN’s analysis of the TLD Application System bug is not yet complete.
(MAY 10 UPDATE — in a statement today, ICANN provided significantly more information about the notification process, rendering much of the speculation originally in this post moot. Read it here.)

TAS to reopen May 22. Big Reveal on for Prague?

ICANN’s bug-plagued TLD Application System will reopen on May 22 and close on May 30, according to a statement just issued by chief operating officer Akram Atallah.
The dates, which are only “targets”, strongly suggest that that the Big Reveal of all new gTLD applications is going to happen during the public meeting in Prague in late June.
If ICANN still needs two weeks to collate its application data before the reveal, we’re looking at June 14, or thereabouts, as the earliest possible reveal date.
But that’s just ten days before ICANN 44 officially kicks off, and I think it’s pretty unlikely ICANN will want to be distracted by a special one-off event while it’s busy preparing for Prague.
For the Big Reveal, my money is on June 25.
Atallah also said this morning that all new gTLD applicants have now been notified whether they were affected by the TAS bug, meaning ICANN has “met our commitment to provide notice to all users on or before 8 May”.
That said, some applicants I spoke to this morning, hours after it was already May 9 in California, said they had not received the promised notifications. But who’s counting?
The results of ICANN’s analysis of the bug appear to show that no nefarious activity was going on.
“We have seen no evidence that any TAS user intentionally did anything wrong in order to be able to see other users’ information,” Atallah said.
ICANN has also discovered another affected TAS user, in addition to the 50 already disclosed, according to Atallah’s statement.

Beckstrom breaks TAS bug silence, says Big Reveal could be as late as Prague

ICANN may not reveal its list of new generic top-level domain applications until as late as the last week of June, according to CEO Rod Beckstrom.
In his first interview since ICANN took its TLD Application System offline due to a security bug, Beckstrom told DI that he “hopes” to host the Big Reveal before he steps down as ICANN’s CEO.
He said he expects to have the new gTLD program back on track before he hands the reins to the organization over to his successor at the end of the ICANN 44 meeting in Prague, June 29:

I’d like to see us obviously get the technical issues resolved, notify applicants, reopen the window and publish the strings before I pass the baton in Prague. That’s not a commitment at this point in time, it’s an indication as CEO that it’s absolutely my intention to push for a timely resolution of this issue… If we can get things done sooner, then the sooner the better.

That’s two months away, a full month later than anyone was expecting.
The Big Reveal was originally scheduled for today. However, the TAS delays made this impossible. Following an ICANN update on Friday, a late-May date for the Big Reveal was looking more probable.
But Beckstrom would not commit even to the Prague date. He said:

That’s my hope as a CEO, to get these issues resolved by that time-frame and have the string reveal in that time-frame. I haven’t committed the organization, I’m indicating to you volitionally my desire as CEO and the person who’s running the organization.

He framed the issue as a blip on a nine-year process (six years of policy development, one year of outreach and application filing, and up to two years of evaluation). He said:

In the context of nine-year program, a delay of between here and Prague of a few months is undesirable, it’s not what we want to have happen, but the quality of this program is more important to everyone involved than the specific date and time. We’re all focused on quality here and not just doing things in a hurry. This program is too important.

He said he is “sympathetic” to applicants that are burning through start-up funding waiting for ICANN to sort this out, but he noted that the same concerns have been raised over the years whenever the program has previously missed a launch deadline.

We know that some parties have been very patient and we know it’s got to be frustrating right now to see any delay in the program. At the same time, I’m sure that those parties are very concerned that this be done well and that the program be reopened and administered successfully.

Beckstrom reaffirmed ICANN’s promise to notify all applicants whether or not they were affected by the TAS bug – which revealed user names and file names to other TAS users – by May 8.
But TAS will not, it seems, reopen immediately after the notifications have been sent. As well as the log audit, ICANN is also working on performance upgrades.
While Beckstrom confirmed that the plan is to open TAS for five business days, to give applicants a chance to finish uploading their applications and confirm that their data has not been corrupted, he would not say when this window is due to open.

We’re going to share more precise dates when we have them. What I can tell you precisely right now is that the key thing we’re working on is combing through this large data set we have so that the parties that were affected are notified within the seven days. When we have clarity on the next milestone in the process we’ll communicate that openly.
…
We’re still doing system testing, we’re still looking at some of the performance issues. We have a whole set of things to do and feel comfortable that we’re ready and have full internal sign off. We’ll notify you and other parties when we have that clarity. Right now we have the clarity that we’re going to get the notification done in seven days – that’s the key dating item at this time.
…
We have very strong reason to believe we understand the bug and we’ve fixed the bug, but every day that we continue to test we gain a higher level of confidence in the system that this specific issue will not reappear.

While the first report of the bug was received March 19, it was not until April 12 that ICANN managed to “connect the dots” and figure out that the problem was serious and recurring, Beckstrom said.
ICANN saw the bug show up again repeatedly on April 12, as many TAS users logged in to finish off their applications, which was why it chose to take the system down with just 12 hours to go before the filing deadline.
ICANN is currently analyzing a 500GB log containing a record of every data packet that went into and out of the TAS between January 12 and April 12, to reconstruct every user session and determine who could see what and when, Beckstrom said.
He refused to comment on whether this analysis has revealed any attempts by TAS users to deliberately exploit the bug for competitive intelligence on other applicants.
He also declined to comment on whether ICANN has discovered instances of data leakage between two applicants for the same gTLD string.
The full packet capture system was introduced following a third-party security audit of the system conducted late last year, he said.
That audit, of course, did not reveal the data leakage vulnerability that continues to delayed the program.
When I put it to him that this is precisely the kind of problem ICANN wanted to avoid, due to the confidentiality of the applications, Beckstrom played down the seriousness of the bug.

Let’s be clear here: some user names and file names were visible, not the contents of applications and not the contents of those files. I think that if that had occurred it would be an even more undesirable situation and we have no indication that that occurred.
I wouldn’t call this a security issue, I’d call this… every major software system we use has bugs in it or bugs that are discovered over time. Whether that’s our operating systems or desktop applications or specific applications, you conduct the best tests you can. You assemble a testing suite, you assemble testers, you take various methods, but there’s never a guarantee that software is bug-free. The issue is that if and when bugs are encountered you deal with them appropriately, and that’s what we’re doing right now.

But Beckstrom admitted that the problem is embarrassing for ICANN, adding that sorting out the mess is currently the top priority.

Obviously any time you have a software problem or technical problem with any program you come under enhanced scrutiny and criticism, and I think that’s understandable, that’s fair. What we’re focused on is resolving this successfully and I think ICANN has dealt with many challenges in its past successfully and we’re committed to resolve this issue professionally.
…
I should tell you that this is our top priority right now internally right now. The resolution of this issue is our number one priority, the number one issue for me as CEO, number one for most members of the executive management team and for a large part of the organization. We’re extremely focused on this.

ICANN plans to reveal how many applicants were affected by the bug at the same time as it notifies applicants, Beckstrom said. It will not publish information about who could see what, he said.
Unfortunately for applicants, it seems they will have to wait well into next week before they have any more clarity on the timetable for TAS coming back online and the application window finally closing.
With Prague now emerged as a potential deadline for the reveal, the delays could in fact be much worse than anyone was expecting.
DI PRO subscribers can read a full transcript of the 30-minute interview.

ANA demands TAS bug probe

Never one to miss the chance for a bit of trouble-making, the Association of National Advertisers has demanded a full independent probe into ICANN’s TLD Application System bug.
Writing to ICANN today, ANA president Bob Liodice has pointed to the TAS outage – now in its 13th day – as an example of why the new gTLD program needs to be scaled back.
“Doesn’t this situation demonstrate the need for a pilot project/test roll-out of the new Top Level Domain process to resolve any such problems before a major roll-out?” he asks.
In a press release, he added:

We are urgently requesting that the Department of Commerce and its National Telecommunications and Information Administration (NTIA) exercise their oversight of ICANN and encourage ICANN to engage an independent IT expert to fully investigate this serious and inadequately explained vulnerability.

The ANA has of course been the loudest objector to the program, forming the Coalition For Responsible Internet Domain Oversight last year to lobby against the gTLD expansion.
Liodice’s latest letter puts 10 questions to ICANN, several quite sensible and precisely the kinds of things I plan to ask just as soon as ICANN changes its mind about doing media interviews.
But it also asks for the release of information ICANN has already provided or has said it intends to provide, such as the number of affected TAS users or the date of the first reported incident.
The ANA also does not appear to be aware that the ICANN board new gTLD subcommittee recently passed a resolution calling for more work on the defensive registration problem.
Liodice notes that ICANN has not responded to its demands for a “Do Not Sell” list that would enable brand owners to block others from registering their trademarks in the DNS.
You can read the letter in PDF format here.
ICANN currently plans to provide its next big update on the TAS outage before the end of Friday.

TAS down for at least another week

If you’re just joining us, welcome to the ICANN community.
The TLD Application System will be offline for another week, possibly more, as ICANN struggles to deal with the fallout from its embarrassing data leakage bug.
ICANN had promised an update today on the timing of the reopening of TAS, which was taken offline April 12 just 12 hours before the new gTLD application filing deadline arrived.
But what applicants got instead was a promise to provide another timing update a week from now.
Chief operating officer Akram Atallah said in a statement:

identifying which applicants may have been affected by the technical glitch, and determining who may have been able to see someone else’s data, require extensive analysis of a very large data set. This is a time-consuming task, but it is essential to ensure that all potentially affected applicants are accurately identified and notified.
Until that process is complete, we are unable to provide a specific date for reopening the application system.
In order to give all applicants notice and an opportunity to review and complete their applications, upon reopening the system we will keep it open for at least five business days.
No later than 27 April 2012 we will provide an update on the reopening of the system and the publication of the applied-for new domain names.

So the best-case scenario, if these dates hold up, would see TAS coming back online Monday, April 30 and closing Friday, May 4.
The April 30 target date for the Big Reveal is clearly no longer possible.
ICANN has stated previously that it expects to take two weeks between the closing of the application window and the revelation of the list of gTLDs being applied for.
The Big Reveal could therefore be postponed until mid-May, almost a month from now.
Any applicant who has already booked flights and hotels in order to attend one of the various reveal events currently being planned by third parties may find themselves out of pocket.
Regular ICANN participants are of course accustomed to delay.
ICANN’s image problem now is rather with the hundreds of companies interfacing with the organization for the first time, applying for new gTLDs, which may be wondering whether this kind of thing is par for the course.
Well, yes, frankly, it is.
That said, the time to avoid this problem was during testing, before the application window opened in January.
Now that the bug has manifested, it’s probably in most people’s best interests for ICANN to fully understand went wrong and what impact it could have had on which applicants. This takes time.

ICANN vows to fight TAS bug “monkey business”

ICANN chief security officer Jeff Moss has pledged to fully disclose what new gTLD application data was leaked to which users via the TLD Application System security bug.
Talking to ICANN media chief Brad White in a video interview, Moss said:

We’re putting everyone on notice: we know what file names and user names were displayed to what people who were logged in and when. We want to do this very publicly because we want to prevent any monkey business. We are able to reconstruct what file names and user names were displayed.

ICANN has been going through its logs and will know “very specifically” what data was visible to which TAS users, he said.
The bug, he confirmed, was related to file deletions:

Under certain circumstances that were hard to replicate users that had previously deleted files could end up seeing file names of users that had uploaded a file… Certain data was being revealed to users that were not seeking data, it was just showing up on their screen.

The actual contents of the files uploaded to TAS were not visible to unauthorized users, he confirmed. There are also no reasons to believe any outside attacks occurred, he said.
He refused to reveal how many applicants were affected by the vulnerability, saying that ICANN has to first double-check its data in order to verify the full extent of the problem.
The interview reveals that the bug could manifest itself in a number of different ways. Moss said:

The problem has several ways it can express itself… we would solve it one way and it would appear another way, we would solve it another way and it would appear a third way. At some point we were just uncomfortable that we understood the core issue and that’s when we took the system offline.

TAS was taken down April 12, just 12 hours before the new gTLD application window closed.
ICANN has been providing daily updates ever since, and has promised to reveal tonight when TAS will reopen for business, for how long, and whether April 30 Big Reveal day has been postponed.
Applicants first reported the bug March 19, but ICANN did not realize the extent of the problem until later, Moss said.

In hindsight now we realized the 19th was the first expression of this problem, but at the time the information displayed made no sense to the applicant, it was just random numbers… at that point there were no dots to connect.

Here’s the video:

First TAS security bug details revealed

The data leakage bug in ICANN’s TLD Application System was caused when applicants attempted to delete files they had uploaded, the organization has revealed.
In his latest daily update into the six-day-old TAS downtime, chief operating officer Akram Atallah wrote this morning:

ICANN’s review of the technical glitch that resulted in the TLD application system being taken offline indicates that the issue stems from a problem in the way the system handled interrupted deletions of file attachments. This resulted in some applicants being able to see some other applicants’ file names and user names.

This sounds rather like an applicant’s file names may have become visible to others if the applicant attempted to delete the file (perhaps in order to upload a revised version) and the deletion process was cut off.
Speculating further, this also sounds like exactly the kind of problem that would have been exacerbated by the heavy load TAS was under on April 12, as lots of applicants simultaneously scrambled to get their gTLD bids finalized to deadline.
Rather than being a straightforward web app, TAS is accessed via Citrix XenApp virtual machine software, which provides users with an encrypted tunnel into a Windows box running the application itself.
As you might expect with this set-up, performance issues have been observed for weeks. Every applicant logged into TAS last Thursday reported that it was running even more slowly than usual.
A security bug that only emerged under user load would have been relatively tricky to test for, compared to regular penetration testing.
But ICANN had some good news for applicants this morning: it thinks it will be able to figure out not only whose file names were leaked, but also who they were leaked to. Atallah wrote:

We are also conducting research to determine which applicants’ file names and user names were potentially viewable, as well as which applicants had the ability to see them.

This kind of disclosure would obviously be beneficial to applicants whose data was compromised.
It may also prove surprising and discomfiting to some applicants who were unwittingly on the receiving end of this confidential data but didn’t notice the rogue files on their screens at the time.
ICANN still plans to provide an update on when TAS will reopen for business this Friday. It will also confirm at the time whether it is still targeting April 30 for the Big Reveal.

New gTLD filing deadline delayed again

It looks like new gTLD applicants are in for more delays after ICANN announced that it will not reopen its TLD Application System tomorrow as planned.
In a statement tonight, chief operating officer Akram Atallah said that the recently discovered data leakage vulnerability has been fixed, but the fix is still being tested.

We believe that we have fixed the glitch, and we are testing it to make sure.
ICANN is committed to reopening the application system as soon as we can confirm that the problem has been resolved and we have had proper time for testing.
We also want to inform all applicants, before we reopen, whether they have been affected by the glitch. We are still gathering information so we can do that.
Accordingly, the application system will not reopen tomorrow.

ICANN shut down TAS last Thursday, just 12 hours before the new gTLD application filing deadline, after discovering a persistent bug that allowed some applicants to see the names of files uploaded by other applicants.
It had planned to open TAS again tomorrow and close it on Friday. However, that’s looking increasingly unlikely.
Atallah said that ICANN “will provide an update on the timing of the reopening no later than Friday, 20 April at 23.59 UTC.”
While ICANN said yesterday that it was still targeting April 30 for its Big Reveal event, subject to change, that’s now looking like an ambitious goal.

ICANN will alert gTLD security bug victims

ICANN plans to inform each new top-level domain applicant whether they were affected by the security vulnerability in its TLD Application System, according to its latest update.
The organization has also confirmed that it is still targeting April 30 for the Big Reveal day, when it publishes (deliberately) the gTLDs being applied for and the names of the applicants.
This morning’s TAS status update, penned by chief operating officer Akram Atallah, does not add much that we did not already know about the data leakage bug. It states:

An intensive review has produced no evidence that any data beyond the file names and user names could be accessed by other users.
We are currently reviewing the data to confirm which applicants were affected. As soon as the data is confirmed, we will inform all applicants whether they were affected.

ICANN staff and outside consultants have been working all weekend to figure out what went wrong, who it affected, and how it can be fixed.
The organization still intends to announce tonight whether it has fixed the problem to the point where it’s happy to reopen TAS to registered users tomorrow. It’s also sticking to is Friday extended submission deadline.

ICANN knew about TAS security bug last week

ICANN has known about the data leakage vulnerability in its TLD Application System since at least last week, according to one new top-level domain applicant.
The applicant, speaking to DI on the condition of anonymity today, said he first noticed another applicant’s files attached to his gTLD application in TAS last Friday, April 6.
“I could infer the applicant/string… based on the name of the file,” said the applicant.
He immediately notified ICANN and was told the bug was being looked at.
ICANN revealed today that TAS has a vulnerability that, in the words of COO Akram Atallah, “allowed a limited number of users to view some other users’ file names and user names in certain scenarios.”
The actual contents of the files are not believed to have been visible.
But other applicants, also not wishing to be identified, today confirmed that they had uploaded files to TAS using file names containing the gTLD strings they were applying for.
It’s not yet known how many TAS users were able to see files belonging to others, or for how long the vulnerability was present on the system.
However, it now does not appear to be something that was accidentally introduced during yesterday’s scheduled TAS maintenance.
This kind of data leakage could prove problematic — and possibly expensive — if it alerted applicants to the existence of competing bids, or caused new competing bids to be created.
ICANN shut down TAS yesterday and does not expect to bring it back online until Tuesday.
The window for filing applications, which had been due to close yesterday, has been extended until 2359 UTC next Friday night.
April 14 Update
ICANN today released a statement that said in part:

we are sifting through the thousands of customer service inquiries received since the opening of the application submission period. This preliminary review has identified a user report on 19 March that appears to be the first report related to this technical issue.
Although we believed the issues identified in the initial and subsequent reports had been addressed, on 12 April we confirmed that there was a continuing unresolved issue and we shut down the system.