Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Proxy IT |
2 | keyplyr | 5:41 pm May 11, 2010 | ||||
| Twitter Chasing Bots How many bots are chasing your tweets? |
10 | incrediBILL | 5:52 am May 9, 2010 | ||||
| SERPAnalytics yet another SEO scraper |
4 | incrediBILL | 10:25 pm May 8, 2010 | ||||
| SmartViper another SEO scraper |
5 | incrediBILL | 12:43 am May 8, 2010 | ||||
| TweetmemeBot |
2 | incrediBILL | 10:28 pm May 7, 2010 | ||||
| 80legs on the crawl |
5 | incrediBILL | 6:15 am May 6, 2010 | ||||
| Strange Requests from Googlebot |
9 | aristotle | 8:23 pm May 5, 2010 | ||||
| downforeveryoneorjustme Revisited (incl. AppEngine-Google) |
4 | Pfui | 12:24 am May 5, 2010 | ||||
| Blocking Cuil bots |
8 | Asia_Expat | 4:40 am May 4, 2010 | ||||
| Apple iPad UA ID |
3 | Pfui | 5:09 am Apr 28, 2010 | ||||
| MSN's many cloaked bots. Mass undocumented activity in search.msn.com ranges[2] ( 1 2 ) |
42 | Pfui | 8:28 am Apr 27, 2010 | ||||
| Protoype Cloaked somethingorother from .us.ibm.com |
5 | Pfui | 3:09 pm Apr 26, 2010 | ||||
| Interesting Google-bot Image Encounter |
3 | caribguy | 3:08 pm Apr 25, 2010 | ||||
| Blank User agent string Blank User agent string query |
4 | baiwan | 6:50 pm Apr 24, 2010 | ||||
| Yellow Pages (heritrix) |
2 | Pfui | 9:48 pm Apr 21, 2010 | ||||
| blocking user agents - 403 UA continues request |
5 | smallcompany | 3:04 am Apr 21, 2010 | ||||
| Anyone know the Basefarm user agent name? |
3 | internetheaven | 9:22 pm Apr 20, 2010 | ||||
| OpenX Ad Server Script Causing 404s On Server Some Odd User Agents Requesting URIs |
5 | incrediBILL | 9:07 pm Apr 20, 2010 | ||||
| dotnetdotcom or DotBot block or not? |
7 | smallcompany | 12:55 pm Apr 20, 2010 | ||||
| Creating a category specific crawler crawler, search engine |
2 | rodriguez1804 | 10:02 am Apr 20, 2010 | ||||
| RatePoint Scraping About Us pages |
4 | caribguy | 7:08 pm Apr 14, 2010 | ||||
| moeenbot |
2 | Pfui | 9:47 pm Apr 11, 2010 | ||||
| Facebook Sues Data Scraper |
18 | Brett_Tabke | 7:55 pm Apr 9, 2010 | ||||
| blocked regular IE6 by mistake |
7 | smallcompany | 4:10 am Apr 9, 2010 | ||||
| Wanted: Crawler Quality Assurance Engineer For MSNbot/2.0 |
8 | jdMorgan | 3:04 am Apr 9, 2010 |