Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Grafix-is
6 keyplyr 8:05 pm Sep 20, 2011
Level 3 redux
6 Pfui 3:25 pm Sep 19, 2011
Resolving "urlresolver"
Google IPs repeat no-robots runs
30 Pfui 4:55 am Sep 19, 2011
Netscape 5.0 is driving me nuts
Netscape 5.0 showing on logs
9 walkman 9:00 pm Sep 16, 2011
Does everyone use bespoke scripts for spider identification
6 scooterdude 9:47 pm Sep 14, 2011
Sitebuilder name in results-spammers thru a backdoor?
Sitebuilder in search results-block "backdoor"
5 KaraScene1 4:24 pm Sep 14, 2011
Traffic from Malicious IPs and Adwords ads for my website
I get visitors from adwords when I never started a campaign
5 Makis77 8:45 pm Sep 13, 2011
isc header collector
2 Pfui 9:30 pm Sep 9, 2011
Iparadigms
8 keyplyr 7:01 pm Sep 3, 2011
Why is this agent fetching my images?
7 aristotle 4:58 pm Sep 3, 2011
PostPost
2 Pfui 1:52 pm Sep 3, 2011
pflab
3 keyplyr 12:50 am Sep 3, 2011
Netvigator
5 keyplyr 11:31 pm Sep 1, 2011
ScooperBot
3 Pfui 2:46 pm Sep 1, 2011
who is Covario and what do they want from me?
3 lucy24 7:27 pm Aug 31, 2011
74.125.52.93 in my logs today
6 apauto 9:11 pm Aug 30, 2011
AIM7 WinHTTP/1.0
2 Pfui 7:00 pm Aug 29, 2011
YodaoBot
7 Pfui 1:16 am Aug 29, 2011
BOTW Spider
Jurgle LLC in New York
2 grandma_genie 6:50 pm Aug 24, 2011
jikespider
Chinese bot shape-shifts on the fly.
4 Pfui 7:29 pm Aug 23, 2011
Google IPs but MSIE UA
Huh?
2 Pfui 4:47 pm Aug 22, 2011
NerdByNature.Bot
13 keyplyr 10:18 pm Aug 21, 2011
Level3 and more
5 wilderness 8:51 pm Aug 21, 2011
AhrefsBot
née SiteBot
5 Pfui 4:57 pm Aug 21, 2011
Butterfly
2 Pfui 7:30 pm Aug 20, 2011