Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Major Intel.com Spidering Activity - Multiple IP's
3 Brett_Tabke 11:04 pm May 5, 2007
Project Honey Pot files $1B+ Lawsuit Against Spammers and Spambots
spam harvesters about to be harvested in court
24 incrediBILL 2:06 pm May 1, 2007
Open-I-Search Unveiled
Interview with the BotMaster
2 yodokame 4:48 pm Apr 29, 2007
Referer is Google with no search string
http://www.google.com/ - that's the whole URL
5 Umbra 8:06 pm Apr 24, 2007
how to stop people from spoofing google ips?
3 jake66 5:25 pm Apr 22, 2007
What to do with libwww-perl/5.79?
Netscape or NetScrape?
2 blend27 9:10 pm Apr 21, 2007
cgi bad-bot script
interesting ip
15 dolcevita 4:21 am Apr 19, 2007
Tips to detect what are bad bots in your bot trap
And what are humans clicking bot trap
21 silverbytes 2:02 pm Apr 18, 2007
New Yahoo bot.
Yahoo-Test/4.0
4 bobothecat 2:11 am Apr 18, 2007
Faking to be Googlebot
2 sami 2:57 am Apr 17, 2007
Banning Microsoft URL Control
Best method?
18 bouncybunny 10:13 am Apr 14, 2007
Jeeves
What they up to?
3 wilderness 1:53 am Apr 12, 2007
K-Meleon
13 wilderness 11:11 pm Apr 9, 2007
someone spoofing?
these are some wierd UAs
5 wkitty42 2:11 am Apr 9, 2007
Songbird/0.1
Botnet, Controlled?
4 blend27 10:44 pm Apr 8, 2007
RufusBot
"Selects and repackages" web site pages?
15 bouncybunny 6:25 pm Apr 8, 2007
Human or Bot?
6 Mokita 5:54 pm Apr 8, 2007
RufusBot
2 writer51 3:05 am Mar 31, 2007
Microsoft Office Existence Discovery
Appears to be human powered but not sure.
3 GaryK 7:27 pm Mar 26, 2007
EmeraldShield.com
robots rules don't apply to them
3 koan 11:03 pm Mar 25, 2007
Deny Morfues [expletive deleted] Scanner
2 TheDuDe 7:32 pm Mar 23, 2007
guglibot
Anyone heard of this one?
3 malachite 10:24 am Mar 22, 2007
CazoodleBot
formerly nutch
10 keyplyr 5:21 am Mar 20, 2007
New MS Search UA?
3 GaryK 1:55 pm Mar 13, 2007
fembot
Is this new?
3 Mokita 11:01 pm Mar 12, 2007