Seems to be a bit of confusion about this. From the same article:

Meanwhile, Chilling Effects founder, Wendy Seltzer, seems to insist that this was an implementation mistake and that the team never meant to remove the whole domain:

I've seen this happen more than once... a development site has indexing blocked, and then gets pushed live without anyone noticing. (If the second comment in the article is correct.)

<meta name="ROBOTS" content="NOODP" />

I took my best guess about the domain name and then looked at some random pages.

Psst! ChillingEffects! HTML doesn't require a closing slash.

Looking at the robots.txt I wouldn't call this an implementation mistake by some developers. Everything except the /pages subtree where the about page resides is deliberately blocked for all bots. It's easy to reverse but they didn't do it until now, so they probably want it this way.

Anyone knows what the Google-Legal-Removals bot is doing BTW? Is that bot used to scrape the DMCA notices from the chilling effects site?

User-agent: Google-Legal-Removals
Disallow:

User-agent: Googlebot
Noindex: /
Allow: /pages

User-agent: *
Disallow: /
Allow: /pages

No need to guess at the domain name, it is linked to from here in the Charter: [webmasterworld.com...]

Idly wondering if their content is being "blocked" from any other crawlers?

User-agent: Googlebot
Noindex: /
Allow: /pages

wtf? Does Allow: also mean "override a Noindex: directive"? Does anyone other than Google use this?

Everything except the /pages subtree

... and that's only if the crawler understands the Allow: formulation. They don't have to; so far there's no robots.txt 2.0 standard.

When a robots.txt file mentions user-agents other than yourself or * doesn't it tend to mean that everyone sees the same file?

User-agent: Googlebot
Noindex: /
Allow: /pages

wtf? Does Allow: also mean "override a Noindex: directive"? Does anyone other than Google use this?

I have used "Allow: some-url-pattern" before. It is Google specific: Block URLs with robots.txt [support.google.com]

Noindex: /

Never heard of this within robots.txt. As far as I know, noindex is a directive that is declared only in meta robots or is returned within X-Robots-Tag as part of HTTP response headers.

So if they were to block everything apart from /pages then I would expect their robots.txt to look like this instead:

User-agent: Googlebot
Disallow: /
Allow: /pages

But would not expect to see Noindex: / syntax. Have I missed something new?

The Noindex: / directive in a robots.txt is meaningless - unless it is a recent secret change not mentioned anywhere - it does nothing. Most of that file would only confuse Googlebot imho.

Official standards haven't been updated since 1997 (for HTML 4.01) but Google and Bing both recognize and follow the Allow:

I have read that they use the subsequent length of the text string that follows both Disallow: and Allow: to decide whether to pay attention.. sort of iffy.

at one time google recognized the Noindex: robots.txt directive as an undocumented feature.

Noindex: validates in Google's robots.txt checker, so it should be OK.