Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Bots trying to find phpMyAdmin
5 SEOPTI 11:12 pm Sep 22, 2007
Mediapartners-Google from non-Google IP
Transparent spoof of a Googlebot
2 jdMorgan 9:27 pm Sep 22, 2007
Sogou
chinese bot
3 Bewenched 12:56 am Sep 22, 2007
Mozilla/4.0 (compatible; MSIE 6.0; Windows 98)
5 SEOPTI 2:41 pm Sep 19, 2007
Yeti
4 keyplyr 1:37 pm Sep 19, 2007
Help identifying -who are these guys
identifies itself as Danger
4 Bewenched 2:30 am Sep 13, 2007
rv:1.8.1 - voila bot?
2 jake66 1:06 am Sep 11, 2007
Who have Spider Stats? I Need Thanks~~
2 hostsoftware 8:28 pm Sep 4, 2007
Mediapartners-Google
Has Google changed the name to eliminate the version #?
12 GaryK 6:33 am Sep 1, 2007
Why don't bots send a special header?!
8 mikomido 11:39 pm Aug 30, 2007
EchO! Spider Agent
2 jshpik1 2:46 pm Aug 28, 2007
Ask with no user agent
lots of hits
5 Bewenched 1:36 pm Aug 24, 2007
PageFetcher-Google-CoOp;
6 keyplyr 8:57 pm Aug 22, 2007
ASK with no reverse DNS and no UA
13 Romeo 3:29 am Aug 21, 2007
Spider running from Qwest
7 btherl 1:27 am Aug 21, 2007
compute.amazonaws.com
sucked down entire site - maybe twice
2 amznVibe 9:31 pm Aug 17, 2007
ozzie
3 wilderness 1:47 pm Aug 17, 2007
France Telecom?
but really resolves to Performance Systems International Inc.
11 Bewenched 7:23 am Aug 6, 2007
Gigabot
not identifying itself?
2 Bewenched 5:41 pm Aug 4, 2007
Is this googlebot or mediapartners?
Same IP, different host resolution?
3 johnnie 5:41 pm Aug 4, 2007
http://127.0.0.1:4664/preview?event id=
5 frontpage 12:29 am Aug 3, 2007
HTMLParser/1.6
from research27.corp.sp1.yahoo.com
4 blend27 8:43 pm Aug 2, 2007
What am I doing wrong guys to block IP,s
rDNSbot help re - IP,s
2 whatsdoin 2:05 pm Aug 2, 2007
IBM Supercomputer : Blueice
Scraping my site
5 RayWatson 6:24 pm Jul 30, 2007
Ban All Nutch Variants?
2 keyplyr 5:34 pm Jul 29, 2007