Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Yellow Pages (heritrix) |
2 | Pfui | 9:48 pm Apr 21, 2010 | ||||
| blocking user agents - 403 UA continues request |
5 | smallcompany | 3:04 am Apr 21, 2010 | ||||
| Anyone know the Basefarm user agent name? |
3 | internetheaven | 9:22 pm Apr 20, 2010 | ||||
| OpenX Ad Server Script Causing 404s On Server Some Odd User Agents Requesting URIs |
5 | incrediBILL | 9:07 pm Apr 20, 2010 | ||||
| dotnetdotcom or DotBot block or not? |
7 | smallcompany | 12:55 pm Apr 20, 2010 | ||||
| Creating a category specific crawler crawler, search engine |
2 | rodriguez1804 | 10:02 am Apr 20, 2010 | ||||
| RatePoint Scraping About Us pages |
4 | caribguy | 7:08 pm Apr 14, 2010 | ||||
| moeenbot |
2 | Pfui | 9:47 pm Apr 11, 2010 | ||||
| Facebook Sues Data Scraper |
18 | Brett_Tabke | 7:55 pm Apr 9, 2010 | ||||
| blocked regular IE6 by mistake |
7 | smallcompany | 4:10 am Apr 9, 2010 | ||||
| Wanted: Crawler Quality Assurance Engineer For MSNbot/2.0 |
8 | jdMorgan | 3:04 am Apr 9, 2010 | ||||
| Twitterbot |
2 | Pfui | 11:15 pm Apr 8, 2010 | ||||
| MetaURI |
3 | Pfui | 9:04 pm Apr 8, 2010 | ||||
| Search17Bot |
5 | Staffa | 11:02 pm Mar 31, 2010 | ||||
| Kroger.com 'webcrawlers' |
3 | Pfui | 9:46 pm Mar 29, 2010 | ||||
| Why Google uses this? |
5 | smallcompany | 6:33 pm Mar 23, 2010 | ||||
| InternetDevels |
3 | keyplyr | 10:56 pm Mar 22, 2010 | ||||
| spbot |
17 | Pfui | 10:12 pm Mar 22, 2010 | ||||
| Mozilla/5.0 (compatible; Purebot/1.1; http://www.puritysearch.net/) |
6 | GaryK | 6:27 pm Mar 21, 2010 | ||||
| iisbot A DIY crawler? |
3 | dstiles | 10:06 pm Mar 20, 2010 | ||||
| Reasons for using Googlebot user agent? |
5 | DiscoStu | 9:25 am Mar 20, 2010 | ||||
| SnookBot Spider for Small Business Advertising Network |
5 | incrediBILL | 6:21 pm Mar 17, 2010 | ||||
| Banning spiders except for a few I want, via .htaccess .htaccess earch engine spider control |
3 | revrob | 7:41 am Mar 17, 2010 | ||||
| Netfront and Kindle? What is this? |
3 | tangor | 5:31 am Mar 17, 2010 | ||||
| mAgent |
4 | smallcompany | 6:36 am Mar 16, 2010 |
