Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Twitterbot |
2 | Pfui | 11:15 pm Apr 8, 2010 | ||||
| MetaURI |
3 | Pfui | 9:04 pm Apr 8, 2010 | ||||
| Search17Bot |
5 | Staffa | 11:02 pm Mar 31, 2010 | ||||
| Kroger.com 'webcrawlers' |
3 | Pfui | 9:46 pm Mar 29, 2010 | ||||
| Why Google uses this? |
5 | smallcompany | 6:33 pm Mar 23, 2010 | ||||
| InternetDevels |
3 | keyplyr | 10:56 pm Mar 22, 2010 | ||||
| spbot |
17 | Pfui | 10:12 pm Mar 22, 2010 | ||||
| Mozilla/5.0 (compatible; Purebot/1.1; http://www.puritysearch.net/) |
6 | GaryK | 6:27 pm Mar 21, 2010 | ||||
| iisbot A DIY crawler? |
3 | dstiles | 10:06 pm Mar 20, 2010 | ||||
| Reasons for using Googlebot user agent? |
5 | DiscoStu | 9:25 am Mar 20, 2010 | ||||
| SnookBot Spider for Small Business Advertising Network |
5 | incrediBILL | 6:21 pm Mar 17, 2010 | ||||
| Banning spiders except for a few I want, via .htaccess .htaccess earch engine spider control |
3 | revrob | 7:41 am Mar 17, 2010 | ||||
| Netfront and Kindle? What is this? |
3 | tangor | 5:31 am Mar 17, 2010 | ||||
| mAgent |
4 | smallcompany | 6:36 am Mar 16, 2010 | ||||
| NjuiceBot |
2 | Pfui | 11:02 pm Mar 14, 2010 | ||||
| spreadia |
2 | Pfui | 7:08 pm Mar 14, 2010 | ||||
| LegalX Deep Analyzer (née WideCircles Analysis Agent) |
10 | Pfui | 6:46 pm Mar 12, 2010 | ||||
| Facebook? Search engine, what? |
11 | jake66 | 4:01 am Mar 12, 2010 | ||||
| Multiple sites showing similar crawl spikes |
4 | DiscoStu | 1:52 am Mar 11, 2010 | ||||
| 'Rippers 0' user agent from corporate networks UA looks like a ripper, but netork and behaviour do not |
11 | graeme_p | 6:44 am Mar 9, 2010 | ||||
| Speedy Spider New UA |
4 | keyplyr | 6:14 pm Mar 8, 2010 | ||||
| Shopwiki bot IPs |
6 | dstiles | 6:29 pm Mar 7, 2010 | ||||
| CSimpleSpider Robot |
3 | keyplyr | 1:31 am Mar 7, 2010 | ||||
| Wtabot |
2 | Pfui | 8:41 pm Mar 3, 2010 | ||||
| Host as IP Illegal access to sites without domain in URL |
5 | dstiles | 2:44 am Feb 28, 2010 |