Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
how to stop people from spoofing google ips?
3 jake66 5:25 pm Apr 22, 2007
What to do with libwww-perl/5.79?
Netscape or NetScrape?
2 blend27 9:10 pm Apr 21, 2007
cgi bad-bot script
interesting ip
15 dolcevita 4:21 am Apr 19, 2007
Tips to detect what are bad bots in your bot trap
And what are humans clicking bot trap
21 silverbytes 2:02 pm Apr 18, 2007
New Yahoo bot.
Yahoo-Test/4.0
4 bobothecat 2:11 am Apr 18, 2007
Faking to be Googlebot
2 sami 2:57 am Apr 17, 2007
Banning Microsoft URL Control
Best method?
18 bouncybunny 10:13 am Apr 14, 2007
Jeeves
What they up to?
3 wilderness 1:53 am Apr 12, 2007
K-Meleon
13 wilderness 11:11 pm Apr 9, 2007
someone spoofing?
these are some wierd UAs
5 wkitty42 2:11 am Apr 9, 2007
Songbird/0.1
Botnet, Controlled?
4 blend27 10:44 pm Apr 8, 2007
RufusBot
"Selects and repackages" web site pages?
15 bouncybunny 6:25 pm Apr 8, 2007
Human or Bot?
6 Mokita 5:54 pm Apr 8, 2007
RufusBot
2 writer51 3:05 am Mar 31, 2007
Microsoft Office Existence Discovery
Appears to be human powered but not sure.
3 GaryK 7:27 pm Mar 26, 2007
EmeraldShield.com
robots rules don't apply to them
3 koan 11:03 pm Mar 25, 2007
Deny Morfues [expletive deleted] Scanner
2 TheDuDe 7:32 pm Mar 23, 2007
guglibot
Anyone heard of this one?
3 malachite 10:24 am Mar 22, 2007
CazoodleBot
formerly nutch
10 keyplyr 5:21 am Mar 20, 2007
New MS Search UA?
3 GaryK 1:55 pm Mar 13, 2007
fembot
Is this new?
3 Mokita 11:01 pm Mar 12, 2007
Strange user agent
2 fischermx 10:33 pm Mar 12, 2007
Is it possible ;)
13 wilderness 5:34 pm Mar 6, 2007
MSN IP's?
8 Brett_Tabke 2:20 pm Mar 6, 2007
why dont we see spiders in Live chat softwares
sorry for this foolish query
3 benevolent001 3:15 am Mar 5, 2007