Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| WordPress support page bot? |
5 | GaryK | 4:19 am Jun 22, 2009 | ||||
| msnbot using IE UA should i suspect screenshots? |
3 | GaryK | 11:21 pm Jun 21, 2009 | ||||
| acont.de's ACONTBOT |
15 | Pfui | 11:04 pm Jun 21, 2009 | ||||
| Baidu Behaving Badly: Goes undercover w/ cloaked UA; omits robots.txt "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)" |
8 | Pfui | 9:47 pm Jun 21, 2009 | ||||
| Mahalobot/1.0 |
3 | keyplyr | 5:51 pm Jun 20, 2009 | ||||
| googlebot head request anyone seen googlebot performing head requests? |
16 | enigma1 | 2:18 pm Jun 18, 2009 | ||||
| MSIndianWebcrawl |
8 | Pfui | 7:11 am Jun 16, 2009 | ||||
| SpiderMan/1.0 |
5 | Pfui | 4:07 am Jun 16, 2009 | ||||
| Something new from Yandex? VisualParser |
8 | GaryK | 4:57 am Jun 14, 2009 | ||||
| AppEngine-Google |
4 | dstiles | 10:24 pm Jun 13, 2009 | ||||
| Servage Robot |
2 | caribguy | 3:11 pm Jun 8, 2009 | ||||
| renlifangbot MSN China |
9 | keyplyr | 12:33 am Jun 5, 2009 | ||||
| FollowSite dot com What is this thing? |
15 | tangor | 12:26 am Jun 5, 2009 | ||||
| AppEngine-Google: New Google UA to watch for |
14 | GaryK | 8:29 am Jun 4, 2009 | ||||
| Bing user agent? Does Bing use Msnbot or not? |
5 | ning | 7:40 am Jun 3, 2009 | ||||
| msnbot/2.0b New crawler from MS in the works?[2] ( 1 2 ) |
41 | GaryK | 6:20 am Jun 2, 2009 | ||||
| Hunting Google Safebrowsing Diagnostic Spidering Anyone seen anything? |
6 | incrediBILL | 2:56 am May 31, 2009 | ||||
| livebot-searchsense/0.1; +http://search.msn.com/msnbot.htm MSN strikes again |
12 | Pfui | 12:27 am May 31, 2009 | ||||
| Topsy.com / labs.topsy.com / butterfly.topsy.com Butterfly/1.0 *and* libwww-perl |
2 | Pfui | 7:37 pm May 27, 2009 | ||||
| How often does googlebot hit a large new website? How many reqs/min? |
12 | DiscoStu | 9:56 pm May 25, 2009 | ||||
| is there a data centre ip list |
7 | topr8 | 4:46 am May 25, 2009 | ||||
| Total Domain Data Corp Just got a total eclipse of the heart... |
7 | caribguy | 4:31 am May 25, 2009 | ||||
| searchers.co.uk crawling from 213.122.174.x |
6 | Hobbs | 4:56 am May 23, 2009 | ||||
| Google IP Information Request |
4 | hugh | 8:46 pm May 21, 2009 | ||||
| flatlandbot/allspark |
13 | wilderness | 8:23 am May 20, 2009 |
