Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
50.nu
4 Pfui 9:09 am Aug 7, 2009
Colo's
9 wilderness 3:26 am Aug 7, 2009
Dav
9 wilderness 1:50 pm Aug 6, 2009
Sogou web robot
New UA format
2 GaryK 7:29 am Aug 6, 2009
YahooMobile/1.0 (Resource; Server; 1.0.0)
5 GaryK 10:06 pm Aug 5, 2009
Mozilla/4.0 (compatible; MSIE 4.01; Windows CE; PPC)
Is this SoftLayer?
7 GaryK 11:26 pm Aug 3, 2009
your-server.de hosts bad bots --
findfiles.net, heritrix, Mr. X, GrubNG, Eurobot, etc.
10 Pfui 10:53 pm Aug 3, 2009
nu tch
9 wilderness 7:41 pm Aug 3, 2009
DoubleVerify Crawler
3 GaryK 4:27 pm Aug 3, 2009
WebsiteWorth v1.0
Website worth calculator
3 GaryK 3:35 pm Aug 3, 2009
grabber.exe
Screenshot creator
2 GaryK 8:17 am Aug 3, 2009
SaladSpoon/ShopSalad
(née YowedoBot)
4 Pfui 4:07 pm Aug 2, 2009
Computer Sciences Corporation
Cloud Computing
2 keyplyr 4:05 pm Aug 2, 2009
Huasai
Another Vrtservers variation on a larbin _viewer theme?
3 Pfui 3:45 pm Aug 2, 2009
mobile/; pda/ /iphone; . What does this bot want?
bot looking for directories like mobi/; m/; /images/ixcp.css
3 knonymouse 7:30 pm Jul 31, 2009
Microsoft and Yahoo Search Deal Agreement
How will it affect search bot user agents?
2 GaryK 8:02 am Jul 30, 2009
IBM Research's: "SAI Crawler"
2 Pfui 8:41 am Jul 28, 2009
Google-Site-Verification/1.0
17 GaryK 8:37 pm Jul 27, 2009
bitlybot
AWS again
4 GaryK 7:24 am Jul 27, 2009
u n d e t e c t a b l e 4 u!
2 GaryK 7:43 pm Jul 26, 2009
EvriStreamMiner/0.1b
3 keyplyr 12:15 am Jul 25, 2009
jrCrawler/1.0b
5 Pfui 8:40 pm Jul 24, 2009
Microsoft and Danger
no pun
3 wilderness 7:30 pm Jul 24, 2009
MSN fakes many things
in addition to referrers...
6 caribguy 9:59 pm Jul 22, 2009
Bare Fruit
404 redirector affilliate
5 dstiles 9:42 am Jul 22, 2009