Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Timestamp in useragent string?
8 trintragula 11:19 pm Jan 28, 2014
Anybody have a clue
10 wilderness 3:57 am Jan 27, 2014
msnbot-media/1.1 generating a ton of 404s
NONE existent JPEGs and GIFs
29 blend27 9:41 pm Jan 18, 2014
Does IP Range Block Via WHM Blacklist Block Same IP Ranges at cPanel
Does IP Block at WHM BruteForce Level Filter Down to Unique Domains on VPS?
12 Webwork 10:40 pm Jan 14, 2014
BRIC nations
Eating Bricks
4 Angonasec 7:27 am Jan 13, 2014
sputnik
6 lucy24 7:52 pm Jan 12, 2014
Zeerch
9 Angonasec 6:30 pm Jan 11, 2014
Rampant of Redmond
Shivering my Timbers
7 Angonasec 12:52 pm Jan 10, 2014
112.0.0.0/8
7 keyplyr 9:43 pm Jan 7, 2014
UNTRUSTED in Nokia User Agent
[2] ( 1 2 )
45 lucy24 11:41 am Jan 4, 2014
thunderstone
yes, them again
3 lucy24 6:28 pm Jan 1, 2014
Come on Yahoo. Really?
7 bobothecat2 8:38 pm Dec 27, 2013
archive.org bot
bad bot
7 keyplyr 7:20 am Dec 20, 2013
Google Test-Bot: Google-Test2
11 doc_z 8:28 pm Dec 19, 2013
Google? Is that you?
13 lucy24 10:33 pm Dec 18, 2013
Is there a Browser. . .
10 wilderness 12:53 am Dec 16, 2013
TECH UPDATE: Javascript Makes Dumb Bots Obsolete
Javascript Must Be Crawled For Scrapers To Survive
3 incrediBILL 10:12 pm Dec 15, 2013
the plainclothes bingbot
8 lucy24 8:03 pm Dec 14, 2013
Yahoo:LinkExpander:Slingstone
7 keyplyr 2:06 pm Dec 13, 2013
Google Translate
7 wilderness 2:43 am Dec 11, 2013
YaBrowser
15 lucy24 10:04 am Dec 10, 2013
New stupid referer spammer
Referer spam via agent string (mistyped)
9 yaimapitu 2:58 pm Dec 8, 2013
rarely used
12 lucy24 9:22 am Dec 1, 2013
New Scraping Tool
9 keyplyr 9:07 am Nov 28, 2013
Yahoo phone home
19 lucy24 8:07 pm Nov 23, 2013