Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Web Download Tool
Something else to ban
12 carfac 6:39 pm Oct 2, 2002
We weirdest UA Ever!
5 carfac 7:23 pm Oct 1, 2002
New Fast IP
3 volatilegx 5:12 pm Oct 1, 2002
Another new one!
Lickity_Split
6 carfac 3:23 am Sep 30, 2002
What is w.x.y.z.nombres.ttd.es
What engine is this a spider from?
3 bnhall 1:21 pm Sep 28, 2002
Scooter/3.2.SF0
2 Scooter24 8:13 pm Sep 27, 2002
www.go2-yellowpages.com
Deep crawling for awhile now, and incorrectly
4 savvy1 8:11 pm Sep 27, 2002
Spiders/Bots and Javascript
Spiders without javascript indexing our "javascript required" page
6 Advantex 8:46 pm Sep 25, 2002
Caught another "Trademark Search" robot
3 carfac 6:44 pm Sep 25, 2002
antibot-V1.1.5/i586-linux-2.2
belongs to AntiSearch
8 volatilegx 5:32 pm Sep 25, 2002
NetMechanic V3.0
Seems to be spidering?
2 Nick_W 8:56 am Sep 24, 2002
I am getting hit really hard...
ia_archiver
8 mack 4:27 pm Sep 23, 2002
sherlock/1.3 httpget/1.3
2 frontpage 6:57 pm Sep 22, 2002
Scrubby 2.2
Is this a new spider?
2 Nick_W 4:37 pm Sep 22, 2002
V5 HTTP/S Component
Who is that?
3 frontpage 3:57 pm Sep 22, 2002
Spider from multiple IP's all at once
7 carfac 2:07 am Sep 22, 2002
63.215.240.27 (unknown.level3.net)
Scooter/2.0
6 Key_Master 3:45 pm Sep 20, 2002
av
4 PeteU 7:21 pm Sep 19, 2002
128.242.197.101
I am paranoid but . . .
4 rogerRabbit 10:54 am Sep 19, 2002
MSIE 5.01 requesting robots.txt?
anyone who knows this?
3 bull 3:37 pm Sep 18, 2002
searching for formmail
another candidate to ban?
2 bull 9:43 am Sep 18, 2002
The number of googles spiders
The number of googles spiders what does it stands for
2 Ove 7:14 am Sep 18, 2002
minibot
any ideas?
2 Finder 11:10 pm Sep 17, 2002
Mysterious Spider
5 tesla 7:44 pm Sep 17, 2002
Ask Jeeves
Ask Jeeves disregarding robots.txt
16 Scooter24 5:00 pm Sep 17, 2002