Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
209.83.114.254
3 wilderness 7:26 pm Jun 6, 2003
Altavista
Your Thoughts
18 cmjohnston 3:37 pm Jun 5, 2003
AltaVista
Heads Up
3 Key_Master 2:46 pm Jun 5, 2003
Fluffy the spider
Ahhh.....thats sweet
14 Skylo 2:32 pm Jun 5, 2003
Experience withthis spider
Fast
3 cmjohnston 1:22 pm Jun 5, 2003
freeze
5 DavidT 12:25 pm Jun 5, 2003
whats libfetch/2.0"
2 indiandomain 4:17 am Jun 5, 2003
68.184.115.214
Has a taste for 403's...
2 WitchLars 2:55 pm Jun 4, 2003
EasyDL
4 wilderness 10:45 pm Jun 3, 2003
weird robot
216.231.54.187 66.14.117.87 & C
26 berno72 11:59 am Jun 3, 2003
Microcomputers Etc.
new bot?
2 leifwessman 9:40 am Jun 3, 2003
franklin locator
what is it and where to get it?
7 wkitty42 5:33 am Jun 3, 2003
When do I need Options +FollowSymLinks?
in htaccess
8 guillermo5000 2:08 am Jun 3, 2003
Brand New IP gets hits!
An interesting experiment
6 carfac 8:07 pm Jun 2, 2003
Denying innocent bystanders
3 wilderness 5:43 am Jun 1, 2003
boitho.com-robot/1.0
6 sarahk 9:38 pm May 31, 2003
Need Spider Info Help
2 Porkchop 12:38 am May 31, 2003
Is this new?
NetNose-Crawler 2.0
2 jim_w 1:11 pm May 30, 2003
Need help identifying a spider
what spider is this?
3 tim3562 11:06 pm May 29, 2003
ETS v5.1 what is it?
7 bobmark 4:33 pm May 29, 2003
More Cogeco
Outdated thread
8 wilderness 4:24 am May 29, 2003
Microsofts .NET hitting my site like crazy
Need help to stop it
10 DrOliver 2:12 pm May 28, 2003
Halo - the magical bot
2 jimmykav 9:38 am May 28, 2003
Here is a reason NOT to block blank U-As
It just might be this website checking for valid code
9 Wizcrafts 6:21 pm May 27, 2003
need help with 403 custom page not working in .htaccess
I included an ErrorDocument 403 custom page but I get error message
9 Wizcrafts 5:34 pm May 26, 2003