While I appreciate the information, Jim, that's actually all the more reason for me to want to block the entire net range. :)

you may

-- or you may not. Since this thread will be read by many, I thought that background info "might" be useful.

Jim

"Requesting my robots.txt leads to a site-wide ban."

I'm curious as to how you do that, and also why?

You could do it by generating the robots.txt dynamically. You hook on the 404 handler, examine the query and then synthesize the robots.txt on the fly (as you set it so you have no physical robots.txt file). You can compare the UA for a url like string, which would imply the visitor wants to enter as a spider.

Is it good to ban IPs based on it? No it's not. Or maybe it is, for the competitors. Countermeasures include, if they ever figure that out, the expose of urls via links or images inside the page HTML towards the robots.txt file or disallowed directories on the other server, triggering the later to ban real visitors and popular spiders. I wouldn't want to go down that path.

For the original issue with the amazonaws bots, they pretend to have a search engine via the UA but they're not. They have no public search page at least from the urls they advertise so I block them by host and redirect them to a blackhole. I don't even want to waste b/w for 403 or 404 content.

@enigma1: I may be missing your point but I'm unaware of any of the 50-plus AWS UAs to date "pretend[ing] to have a search engine via the UA."

AmazonAWS [aws.amazon.com] is a server cloud, née farm, not a public search engine per se. (A9.com [a9.com], another Amazon-owned site, is search-related.) It's anybody's guess who/what the amazonaws.com-based bots are crawling for, literally, because the AWS cloud is a cloak.

And now, courtesy of Yet Another AWSbot, the domain's first logspam! Oh, joy...

ec2-[yada-yada].compute-1.amazonaws.com
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
robots.txt? NO

Referer: FAKE; Education-related portal in Mexico

[edited by: Pfui at 10:29 pm (utc) on Mar. 15, 2009]

ec2-[yada-yada].compute-1.amazonaws.com
IIITBOT/1.1 (Indian Language Web Search Engine; [webkhoj.iiit.net;...] pvvpr at iiit dot ac dot in)

robots.txt? YES

The included URL goes to a page saying "503 Service Temporarily Unavailable". Otherwise a long-standing domain for an Indian research institute. Maybe that's why they checked for robots.txt. Still going to be unlucky here, though.

Ironically, given another current thread, the domain registration suggests contacting a certain .org site for more info. Good ole netsol! :)

At least NetSol is a legitimate registrar. :)

Alexa/Internet Archiver uses Elastic Cloud Compute services.

So you may want to allow those sub-ranges of ECS.

Jim,

I think your first sentence negates your second sentence because blocking Alexa and Internet Archiver are an added bonus IMO.

netsol legit? Hmm. Legally, I suppose but they've "made some bad choices" over the years. :)

v0.5 on page 1 of this thread. Up a notch now, w/ same behavior:

ec2-[yada-yada].compute-1.amazonaws.com
Twitturly / v0.6
robots.txt? NO

I usually see Twitturly coming in from .algx.net or .a2webhosting.com

@pfui, well, they do pretend to have something like a search engine. Here is an eg from my logs:

174.129.111.#*$! - - [10/Apr/2009:19:49:26 -0400] "GET /robots.txt HTTP/1.0" 200 0 "-" "linkdexbot/Nutch-1.0-dev (http://www.example.com/; crawl at bla dot com)"

IP resolves to ec2-#*$!.compute-1.amazonaws.com and states "crawl"

But none of the recorded "crawl" links doesn't offer some public search facility. Also since I don't want to go around in circles for UAs and the various host names and how AWS changes the ips every time, I block the ultradns from the dns records and everyone else who comes from it.

PS:I altered the urls of the entry.

I've yet to see anything even remotely related to a 'real person in real time' come from amazonaws.com, presumably because it's basically Just Another Server Farm. So I block on the host and move on. If only they would move on.

Too bad 403s don't stop bad bots and UAs, ditto zombies, for good.

Too bad 403s don't stop bad bots and UAs, ditto zombies, for good.

Yes 403s don't, so instead what I do is redirect them to another place unloading the traffic. Something like a localhost address will do or another blackhole. Load their own servers instead.

Ditto. Good ol': [127.0.0.1...]

ec2-[yada-yada].compute-1.amazonaws.com
rdfbot/1.0 (Indian Language Web Search Engine; Rediff.com; rdfbotsupport AT rediffmailpro DOT com)

robots.txt? YES

-----
Related, from 11/08:

ec2-[yada-yada].compute-1.amazonaws.com
rdfbot/Nutch-1.0-dev

robots.txt? YES

-----
P.S. Bot bits:

host-202-137-236-nn.rediffdns.com
rdfbot/Nutch-1.0-dev

robots.txt? YES

host-202-137-237-nnn.rediffdns.com
IIITBOT/1.1 (Indian Language Web Search Engine; [webkhoj.iiit.net;...] pvvpr at iiit dot ac dot in)

robots.txt? YES

Well here they are again:

67.202.42.--- - - [--/May/2009:--:--:-- -0400] "GET / HTTP/1.1" 301 20 "-" "AISearchBot (Email: aisearchbot@gmail.com; If your web site doesn't want to be crawled, please send us a email.)"

In the same IP range as Pfui mentioned at the beginning of the thread.

If you check the DNS records they all point to ultradns. So instead of chasing around the various IP ranges block based on the dns. I found it to be more effective.

ec2-[yada-yada].compute-1.amazonaws.com
Mozilla/5.0 (compatible; heritrix/1.14.2 yptrino +http://www.buddybuzz.net/yptrino)

robots.txt? YES

ec2-[yada-yada].compute-1.amazonaws.com
Caliperbot/1.0 (+http://www.conductor.com/caliperbot)

robots.txt? NO

ec2-[yada-yada].compute-1.amazonaws.com
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.5) Gecko/2008120121 Firefox/3.0.5

robots.txt? NO

Worth checking the response of the ips when you see the amazonaws ptr. I run across one entry in particular that is strange

72.44.61.194 - - [19/May/2009:22:14:44 -0400] "GET / HTTP/1.1" 301 5 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729) RPT-HTTPClient/0.3-3"

Checking if the ip responds to http requests (port 80) it loads up something like the main google search page. Does anyone have info about it?

Verrrrry interesting about the IP. WHOIS sez Amazon (Amazon Web Services, Elastic Compute Cloud, EC2) but files are certainly Google's. From .js files ('/csi?v=3&s=webhp&action=' etc.) to images:

[72.44.61.194...]
[72.44.61.194...]

Check out the 404 for this dir:

[72.44.61.194...]

Looks like all sorts of companies/hosts hide behind AWS. I wonder if your discovery might have anything to do with Bill's "safebrowsing diagnostic spidering possibly going on at Google that may not be the standard Googlebot [webmasterworld.com]" topic?

If nothing else, that "RPT-HTTPClient [webmasterworld.com]" UA appendage is uncool -- and seriously ancient.

Well, at least the Google-on-AmazonAWS IP didn't show me as logged in to Google...

yes you see? but what do we know. Lots of strange things. In fact you can start checking IP ranges, subdomains even from google itself. Take a look at this one:
209.85.227.141
Now that points to google alright and the port 80 responds to the google home page isn't it? In fact it translates to the google appspot. "Run your web applications on Google's infrastructure" And then you search for subdomains of the appspot all hosted by google and all kinds of stuff appear. Looks to me like hacker's paradise.

And that's just one range while we remember discussions about strange google visits that do not appear to be from googlebot. So to me AWS looks tiny in comparison.

PS: Here is a related thread
[webmasterworld.com...]

ec2-[yada-yada].compute-1.amazonaws.com
LinkbackPlugin/0.1 Laconica/0.7.4

robots.txt? NO

(microblogger link-checker, apparently.)

ec2-[yada-yada].compute-1.amazonaws.com
Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)

robots.txt? NO

ec2-[yada-yada].compute-1.amazonaws.com
Mozilla/5.0 (compatible; woriobot +http://worio.com)

robots.txt? YES

[edited by: Pfui at 1:10 am (utc) on June 21, 2009]

ec2-[yada-yada].compute-1.amazonaws.com
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6

robots.txt? NO

(Interesting cloak -- an old-old FF localized for British English.)

ec2-174-129-91-58.compute-1.amazonaws.com
deva/deva-1.0 (Deva Fetcher; devarajaswami at yahoo dot com)

robots.txt - Yes, but promptly ignored it by requesting the home page.

After all these months of AmazonAWS bot-running others' cloaked UAs, I guess this one was inevitable -- whether it's real or fake:

ec2-[yada-yada].compute-1.amazonaws.com
Mozilla/5.0 (compatible; Googlebot/2.1; [google.com...]

robots.txt? NO

If you're blocking amazonaws by IP you may not be aware of the EU / Irish range. Found it today when something triggered a trap.

AMAZON-EU-AWS
Amazon Web Services, Elastic Compute Cloud, EC2, EU
79.125.0.0 - 79.125.63.255

Full range of Amazon on that block:
IE-AMAZON-20070824
Amazon Data Services Ireland Ltd
79.125.0.0 - 79.125.127.255

Full range now blocked here.

Anyone know of any other ex-USA ranges?