CrawlDaddy abot Site Harvester

Forum Moderators: open

Message Too Old, No Replies

CrawlDaddy abot Site Harvester

WarmGlow

7:00 pm on Sep 28, 2013 (gmt 0)

Did not request robots.txt file.
Banned itself by disobeying robots.txt standard directives.

Sample access log entry:

64.202.161.41 - - [27/Sep/2013:23:59:14 -0400] "GET / HTTP/1.1" 403 362 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; CrawlDaddy v0.3.0 abot v1.2.0.0 http*://code.google.com/p/abot)"

Note: Asterisk (*) added by me to disable hot linking the above URL in this message.

Came from two remote host IPs:

64.202.161.41 
64.202.161.46

The remote host IPs are owned by GoDaddy.com, LLC (GODAD).

The user of the two remote host IPs also attempted to find WordPress vulnerabilities. Here is a sample access log entry:

64.202.161.41 - - [27/Sep/2013:23:59:12 -0400] "GET /wp-login.php HTTP/1.1" 403 366 "-" "User-Agent\tMozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)"

HTH . . .

Dennis

not2easy

10:56 pm on Sep 28, 2013 (gmt 0)

GoDaddy is a Hosting company, they can never send any visitors as they don't provide Internet Services like, say, Verizon does so it is generally safe to block Hosting companies' full range unless some script services you use are hosted there.

I do have them blocked across all sites. From your observations on what they were doing, I'd just block the whole 64.202.160.0/19 GODaddy range.

keyplyr

11:37 pm on Sep 28, 2013 (gmt 0)

Report their behavior to GoDaddy.