Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
TencentTraveler
6 GaryK 1:34 am Oct 6, 2009
S2Bot/1.0
5 keyplyr 12:46 am Oct 6, 2009
parchmenthill
13 Pfui 12:34 am Oct 6, 2009
Will the Real Googlebot-Mobile Please setup forward.
Googlebot-Mobile
3 Ocean10000 10:24 pm Oct 4, 2009
LinkExaminer/1.01 (Windows)
2 GaryK 10:07 pm Oct 4, 2009
Yahoo! SearchMonkey
3 wilderness 10:03 pm Oct 4, 2009
taptubot
9 Pfui 5:19 pm Oct 4, 2009
Micro trend is goofing my system
Micro trend is goofing my system
14 schnee 5:28 pm Oct 3, 2009
Outlook 2007 RSS User-Agent
2 Ocean10000 6:05 am Oct 1, 2009
Dot TK - spider
4 keyplyr 5:31 am Sep 30, 2009
FlickySearchBot/1.0
3 keyplyr 9:23 am Sep 25, 2009
IP Banning Primer
29 httpwebwitch 10:37 pm Sep 23, 2009
YahooCacheSystem
6 Pfui 12:23 am Sep 23, 2009
LargeSmall Crawler
3 Pfui 12:20 am Sep 23, 2009
Yahoo
16 wilderness 7:11 pm Sep 21, 2009
Domain Tools Now Using GurujiBot
IP is from Name Intelligence
4 incrediBILL 7:27 pm Sep 20, 2009
MSN's cloak-crawling again: Twitter / Tweets
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)
3 Pfui 6:32 pm Sep 20, 2009
Fake Googlebot requests on the rise
Botnet, competitive intelligence, or semi-legit?
6 jdMorgan 6:00 pm Sep 20, 2009
Gist Server
2 Pfui 4:17 pm Sep 20, 2009
msnbot/2 snapshots?
9 keyplyr 6:44 am Sep 20, 2009
Wget/1.10.2
3 outland88 9:24 pm Sep 19, 2009
Cuil in stealth mode?
4 keyplyr 10:05 am Sep 19, 2009
Mozilla/4.0
MSN strikes (out) again.
11 Pfui 9:30 am Sep 19, 2009
Cat & Dog
Chinese UA
3 dstiles 1:41 am Sep 19, 2009
Zeta Interactive?
Keep getting hit by bots from 64.40.121.nnn
7 madmatt69 5:53 pm Sep 18, 2009