ICANN fixes new gTLD portal bugs

ICANN has brought its new gTLD program customer service portal back online after about five days of patching-related downtime.

In a notice posted late last night, ICANN said the delay was due to the wait for a vendor patch. ICANN said:

A recent, proactive review of the CSC system identified potential vulnerabilities. To address these vulnerabilities, the CSC portal was taken offline while vendor-provided patches were applied. There have been no known compromises to any data.

New gTLD applicants will now have to log in to their TLD Application System accounts, which use the Citrix remote terminal software, to use their customer service tools.
Non-applicants will be able to ask customer service questions via email.
The Knowledge Base — essentially a program FAQ — is still offline, but ICANN said it hopes to bring it back up within a few days.

ICANN shuts down new gTLD portal after finding more security bugs

ICANN has closed down part of its new generic top-level domain portal after finding “potential vulnerabilities” that put “confidential applicant information” at risk.
The shutdown — which has been going on for at least 30 hours — affects the Customer Service and Knowledge Base parts of the site, but ICANN said it is so far not aware of any attacks against the system.
While it’s waiting for a patch, ICANN has decided to move the affected areas behind the unpopular Citrix remote terminal software used previously in the TLD Application System.
This notice was posted on the site:

ICANN performs ongoing monitoring and analysis of our systems, including the Customer Service system. As part of this work, we recently identified potential vulnerabilities in the system used for Customer Service and the Knowledge Base (containing new gTLD articles and information).
Patches are being provided to ICANN to address these issues.
In the mean time, given that use of the Customer Service system was recently expanded, and now includes confidential applicant information, the decision was taken to move the system behind Citrix. This will provide for additional security for applicant information.
We are now testing the installation. This should be completed in the next few days. This decision is a proactive measure. There have been no known compromises to the data, attacks or other actions by third parties (other than our own analysis).

Off the top of my head — and I may be under-counting — this is the fifth significant technical glitch to hit the new gTLD program since April.
There was the notorious TAS bug, which took the system offline entirely for six weeks while ICANN fixed a data leakage vulnerability and upgraded its system capacity.
There was the Reveal Day screw-up, during which Arab community members noticed that all the applied-for Arabic gTLDs were broadcast back-to-front in a presentation.
Then ICANN accidentally published the home addresses of many applicants’ officers and directors, something it had promised not to do. This was probably human error and it has since apologized.
Then the “digital archery” batching system was yanked, after it emerged that TAS performance still wasn’t up to the task and that the scoring results were unreliable.
Former new gTLD program director Michael Salazar resigned a month ago; it is widely believed that he was taking the fall for the gTLD system bugs to that point.
While the latest bug appears — so far — to have not compromised any data, some applicants have nevertheless been frustrated by the fact that the customer service portal has been offline for over a day.

Digital archery suspended, surely doomed

ICANN has turned off its unpopular “digital archery” system after new gTLD applicants and independent testing reported “unexpected results”.
As delegates continue to hit the tarmac here in Prague for ICANN 44, at which batching may well be hottest topic in town, digital archery is now surely doomed.
ICANN said in a statement this morning:

The primary reason is that applicants have reported that the timestamp system returns unexpected results depending on circumstances. Independent analysis also confirmed the variances, some as a result of network latency, others as a result of how the timestamp system responds under differing circumstances.

While that’s pretty vague, it could partly refer to the kind of geographic randomness reported by ARI Registry Services, following testing, earlier this week.
It could also refer to the kind of erratic results reported by Top Level Domain Holdings two weeks ago, which were initially dismissed as a minor display-layer error.
TLDH has also claimed that the number of opportunistic third-party digital archery services calibrating their systems against the live site had caused latency spikes.
Several applicants also said earlier this week that the TLD Application System had been inaccessible for long periods, apparently due to a Citrix overloading problem.
Only 20% of applications had so far registered their archery timestamp, according to ICANN, despite the fact that the system was due to close down on June 28.
Make no mistake, this is another technical humiliation for ICANN, one which casts the resignation of new gTLD program director Michael Salazar on Thursday in a new light.
For applicants, ICANN said evaluations were still proceeding according to plan, but that the batching problem is now open for face-to-face community discussion:

The evaluation process will continue to be executed as designed. Independent firms are already performing test evaluations to promote consistent application of evaluation criteria. The time it takes to delegate TLDs will depend on the number and timing of batches
…
The information gathered from community input to date and here in Prague will be weighed by the New gTLD Committee of the Board. The Committee will work to ensure that community sentiment is fully understood and to avoid disruption to the evaluation schedule.

Expect ICANN staff to take a community beating over these latest developments as ICANN 44 kicks off here in Prague.
There’s light support for batching, and even less for digital archery. It’s looking increasingly likely that neither will survive the meeting.

Digital Archery lessons from tonight’s tweet-up

ICANN held a Twitter session tonight during which executives answered questions about the new gTLD program in that notoriously restrictive 140-character format.
Unsurprisingly, in light of the frustration borne out of ongoing delays, most of the questions were about timing.
New gTLD applicants wanted to know when ICANN plans to host its Big Reveal event, when the Digital Archery application batching system will open, and when the batches will be confirmed.
The only specific date applicants were given was May 29, which is when ICANN plans to publish its updated program timetable.

A15: Additional schedule details to be posted by 29 May. – @rodbeckstrom #newgTLD #ICANN #ICANNchat

— ICANN (@ICANN) May 22, 2012

But @ICANN gave away enough information to make a broad estimate about the date digital archery will commence.
First, ICANN confirmed that the Big Reveal will be before its public meeting in Prague kicks off on June 23.

A8: Update on timeline to be published shortly. Target reveal date still before Prague. – @rodbeckstrom #newgTLD #ICANN #ICANNchat

— ICANN (@ICANN) May 22, 2012

ICANN also said that the digital archery process will begin before the reveal day and finish after.

A1: Secondary timestamp (digarchery) scheduled 2 open b4 reveal day & close aftr. Designed for reliability & timeliness #ICANNchat #newgTLD

— ICANN (@ICANN) May 22, 2012

A6: #ICANN will publish batches after the reveal, after the digital archery process is complete. – @rodbeckstrom #ICANNchat #newgTLD

— ICANN (@ICANN) May 22, 2012

The archery window will be open for about three weeks, we learned.

A12: The secondary timestamp process will be open for approx 3 weeks, details later. – @rodbeckstrom #newgTLD #ICANN #ICANNchat

— ICANN (@ICANN) May 22, 2012

We can draw some broad conclusions from this information.
The latest possible date for the Big Reveal, given what ICANN said tonight, is June 22 (the Friday before Prague), so the latest possible date for the digital archery window opening is June 21.
In that case, digital archery would run June 21 – July 12, or thereabouts.
Because the archery can’t start before the applications are all submitted, the earliest window would be May 31 – June 20.
My estimates err towards the lower end. I think we’re looking at archery starting within a week of the application window closing and ending immediately before or during Prague.
If ICANN decides that it wants the archery out of the way before the meeting begins, the window could have to open as early as May 31.
If it wants the window to close post-Prague, we’re looking at it opening around June 11.

TAS reopens after humiliating 40 days

Forty days after it was taken offline for a bug fix, ICANN has reopened its TLD Application System, giving new gTLD applicants a week to finish off their applications.
TAS will now close May 30 at 2359 UTC, which is 1559 in California next Wednesday afternoon.
But applicants are being warned that waiting until the final day “may not provide sufficient time to complete all submission steps before the submission period closes.”
The date of the Big Reveal of applications, which I’m now expecting to come at some point before the Prague meeting at the end of June, is likely to be confirmed in the next day or so.
As well as fixing the bug – a data leakage vulnerability that enabled applicants to see each others’ file names, affecting over 150 users – ICANN has made system performance improvements and cleaned up its HTML preview function, in response to user complaints.
Repairing the vulnerability has cost ICANN “hundreds of thousands of dollars” since TAS was taken offline April 12, chief operating officer Akram Atallah estimated last Thursday.
The fact that the system has reopened half a day ahead of the most recently scheduled deadline – it was due to open at 1900 UTC tonight – is unlikely to win ICANN many plaudits.
If the opinions of the opinionated are any guide, the TAS outage has left ICANN with a severe dent in its already patchy reputation, even among fervent supporters.
Atallah and senior vice president Kurt Pritz came in for a pummeling during an ICANN summit attended by registrars and registries, many of them gTLD applicants, late last week.
Several outspoken long-time community members made it clear that their confidence in ICANN’s ability to hit deadlines is at an all-time low.
Expectations of professionalism have increased, as AusRegistry CEO Adrian Kinderis told Atallah, now that ICANN has $350 million of applicant cash in its bank account.
The bug itself may have been as unavoidable and understandable as any bug in new software, but ICANN’s tardiness resolving the problem has left applicant trust in many cases shattered.

ICANN not done with TAS bug analysis

Despite sending out hundreds of notifications to new gTLD applicants today, it looks rather like ICANN’s analysis of the TLD Application System bug is not yet complete.
(MAY 10 UPDATE — in a statement today, ICANN provided significantly more information about the notification process, rendering much of the speculation originally in this post moot. Read it here.)

TAS to reopen May 22. Big Reveal on for Prague?

ICANN’s bug-plagued TLD Application System will reopen on May 22 and close on May 30, according to a statement just issued by chief operating officer Akram Atallah.
The dates, which are only “targets”, strongly suggest that that the Big Reveal of all new gTLD applications is going to happen during the public meeting in Prague in late June.
If ICANN still needs two weeks to collate its application data before the reveal, we’re looking at June 14, or thereabouts, as the earliest possible reveal date.
But that’s just ten days before ICANN 44 officially kicks off, and I think it’s pretty unlikely ICANN will want to be distracted by a special one-off event while it’s busy preparing for Prague.
For the Big Reveal, my money is on June 25.
Atallah also said this morning that all new gTLD applicants have now been notified whether they were affected by the TAS bug, meaning ICANN has “met our commitment to provide notice to all users on or before 8 May”.
That said, some applicants I spoke to this morning, hours after it was already May 9 in California, said they had not received the promised notifications. But who’s counting?
The results of ICANN’s analysis of the bug appear to show that no nefarious activity was going on.
“We have seen no evidence that any TAS user intentionally did anything wrong in order to be able to see other users’ information,” Atallah said.
ICANN has also discovered another affected TAS user, in addition to the 50 already disclosed, according to Atallah’s statement.

ICANN affirms full refunds for pissed-off gTLD applicants, silent on new CEO

ICANN’s board of directors has approved full refunds for any new gTLD applicant that asks for one – something that the organization has already been offering for over a month.
At its two-day retreat in Amsterdam this weekend, the board’s New gTLD Program Committee resolved:

to offer to applicants a full refund of the New gTLD Application fee actually paid to ICANN if the applicant wishes to withdraw its application prior to the date that ICANN publicly posts the identification of all TLD applications.

The date of the Big Reveal, when the names of every applicant and every applied-for gTLD will be publicly posted and the refunds will no longer be available, has not yet been set.
While the resolution refers to the TLD Application System data leakage bug, the refund does not appear to be restricted to directly affected applicants. Anyone can claim it.
However, as regular DI readers know, ICANN had been offering full refunds to applicants that withdraw before the Big Reveal for weeks before the TAS bug emerged.
ICANN customer services reps told DI and at least one gTLD applicant in March that: “Applications withdrawn prior to the posting of the applied-for strings are qualified for a $180000 refund”.
ICANN said in a statement today:

We recognize that this represents an increase of only US $5000 over the refund that withdrawing applicants would otherwise receive, but we believe it is an important part of fulfilling our commitment to treat applicants fairly.

Under the terms of the Applicant Guidebook, the maximum refund available after the Reveal is $148,000.
In other news from Amsterdam…
The ICANN board has decided to let director Thomas Narten join the New gTLD Program Committee, which comprises all of the board members without new gTLD conflicts of interest.
Narten had been barred from the recently formed committee because he worked for IBM, which planned to apply for one or more new gTLDs.
But the board said he has now “mitigated the previously-identified conflict of interest with respect to the New gTLD Program”, so he gets to join the committee as a non-voting liaison.
It’s not clear from the weekend’s resolution why Narten is no longer conflicted. Two obvious possibilities spring to mind.
There was no news from Amsterdam on ICANN’s CEO hunt.
Incumbent Rod Beckstrom intends to “hand the baton” to his successor at the Prague meeting in late June, and the board already has a favored candidate lined up to replace him.
I understand that this candidate did attend the Amsterdam board retreat, albeit under a veil of secrecy lest his or her identity leak out before official confirmation.
But I also understand that the board has decided to move super-cautiously on the CEO decision, in order to avoid repeating the mistakes of the past.

ICANN expects up to 2,305 new gTLD applications

After months of speculation, ICANN has finally revealed how many new generic top-level domain applications it expects to receive.
The lowest amount appears to be 2,091.
That’s the number of applications in the TLD Application System when it was taken offline due to the data leakage bug on April 12, ICANN said.
Another 214 applications had been registered but not yet paid for.
That’s a potential total of 2,305 applications.
ICANN has $350 million in application fees in the bank as a result.
How many of the unpaid bids convert to full applications will be key in deciding how many evaluation batches the first gTLD round will have.
Closer to 2,091, and it’s likely to be four batches. Closer to 2,305, and we may see a fifth batch.
With Initial Evaluation expected to take five months per batch, with a possible 11 months after that for the final Extended Evaluations and string contention resolution, it could be June 2015 before the first new gTLD round is completely processed.
It remains to be seen how many unique strings have been applied for, and how many applications will be successful, but with ICANN only planning to delegate 200 to 300 new gTLDs per year, the first round is likely going to last a loooong time.

TAS bug hit over 100 new gTLD applicants

It just keeps getting worse.
ICANN’s TLD Application System security bug could have revealed file names belonging to 105 new gTLD applicants to 50 other applicants on 451 occasions, according to the organization.
With 1,268 applicants in the system, those numbers certainly fit with the “a minority of applicants” description previously given, but it still shows that the bug was widespread.
The supplied numbers are “approximate”, but ICANN said it is “continuing to review system logs and packet-level traffic to confirm how many viewings actually did occur.”
The latest news means, for example, that 50 new gTLD applicants may have had the ability to see information belonging to other applicants on average nine times each.
While the new data may not strongly suggest that the bug was deliberately exploited by any applicant(s), it’s not inconsistent with that scenario.
It could mean that one applicant saw the details of 56 others (suggesting exploitation), but it could also mean that 50 applicants saw about two third-party file names each (suggesting accidental viewing).
Without further information, it’s impossible to know.
ICANN has not revealed, and is unlikely to reveal in the short term, whether any applicant was able to view the metadata of another applicant for the same gTLD.
The organization has however started to notify affected applicants whether they were affected as victim or beneficiary, according to the latest update from chief operating officer Akram Atallah.
Atallah also revealed that TAS had 95,000 file attachments in the system when it was taken down April 12.
At an average of 75 files per TAS account, this would support the idea that, on average, each TAS account was being used to file more than one application.
ICANN still plans to wrap up the notification process before next Tuesday, May 8, but there’s no word yet on when TAS will reopen for the final five days of the application window.