Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Bot from IANA
6 nativenewyorker 5:12 pm Sep 3, 2009
Viralheat Bot
11 Pfui 11:50 pm Sep 1, 2009
SindiceFetcher/sig.ma
2 keyplyr 9:20 am Aug 30, 2009
arste.info bot
2 Pfui 8:35 am Aug 30, 2009
jc unknown var cloak.daily-2009-08-25-62-00.vars.referer
5 1EightT 10:22 pm Aug 26, 2009
GingerCrawler/1.0
12 Mokita 4:18 pm Aug 23, 2009
FunWebProducts, is it more than just a browser add-on?
4 JS_Harris 3:50 am Aug 23, 2009
Is returning a blank page to known search crawlers acceptable?
Filtering search crawler clicks to affiliate links
3 JS_Harris 12:26 am Aug 23, 2009
Google
Relative links
6 wilderness 7:11 am Aug 21, 2009
Toata dragostea mea pentru diavola
All my love is for the devil
3 Ocean10000 4:34 pm Aug 20, 2009
Windows 3.
3 wilderness 4:28 pm Aug 20, 2009
Eurobot
2 keyplyr 4:25 pm Aug 20, 2009
MS weirdness
5 keyplyr 3:12 pm Aug 19, 2009
baypup/1.1 (Baypup; http://www.baypup.com/; jason@baypup.com)
3 GaryK 3:28 pm Aug 17, 2009
Thunderstone
2 wilderness 6:23 am Aug 17, 2009
Website Explorer/0.9.9.7
2 GaryK 11:51 pm Aug 16, 2009
Sosospider
Another Rude Spider
3 Ocean10000 8:22 pm Aug 16, 2009
What is User Agent "U"
found user agent or bot in our logfiles, dientified by U
8 Poolcenter 7:11 pm Aug 15, 2009
atraxbot/0.2
16 keyplyr 6:46 pm Aug 11, 2009
Google proxy junk
3 dstiles 7:04 pm Aug 10, 2009
Rewrite ranges
12 wilderness 2:40 am Aug 10, 2009
pigmej, wykop/0.3
18 Pfui 10:05 pm Aug 9, 2009
http://www.amagit.com/
distributed search engine posing as a "new web" browser
3 GaryK 3:43 pm Aug 9, 2009
Mozilla/5.0 (compatible; Firefox Addon; Windows XP 5.1)
2 GaryK 3:35 pm Aug 9, 2009
southwebventures.com
2 keyplyr 2:55 pm Aug 7, 2009