Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| WebAir proxies WebAir using compromised machines as proxies |
5 | dstiles | 10:11 pm Feb 1, 2012 | ||||
| CMS Crawler |
20 | keyplyr | 10:37 pm Jan 31, 2012 | ||||
| LinkSaver and LinkParser wikimedia bots? |
9 | dstiles | 9:58 pm Jan 31, 2012 | ||||
| Pagepeeker.com bad bot |
9 | cyberdyne | 11:41 am Jan 31, 2012 | ||||
| ldspider code.google.com |
3 | Pfui | 11:53 pm Jan 27, 2012 | ||||
| WBSearchBot |
8 | Pfui | 10:45 pm Jan 27, 2012 | ||||
| HTTwaddle |
6 | lucy24 | 11:22 am Jan 25, 2012 | ||||
| ucsb seclab crawler |
8 | Pfui | 9:02 pm Jan 22, 2012 | ||||
| Vancouver 0.9 |
3 | MxAngel | 10:45 am Jan 20, 2012 | ||||
| SpiderLing |
2 | Pfui | 6:58 am Jan 18, 2012 | ||||
| Gigabot Revisited |
4 | Pfui | 5:53 am Jan 18, 2012 | ||||
| obot IBM iss.net IPs |
5 | Pfui | 4:55 am Jan 16, 2012 | ||||
| EuripBot |
3 | keyplyr | 8:46 pm Jan 14, 2012 | ||||
| MS Crawler Hiding as a Browser |
14 | incrediBILL | 4:19 am Jan 11, 2012 | ||||
| Is this a valid user agent? and how do I block it? Is this a valid user agent? and how do I block it? |
12 | spiritualseo | 4:59 am Jan 10, 2012 | ||||
| Nook |
2 | Pfui | 7:22 pm Jan 8, 2012 | ||||
| Servers using open proxies |
4 | dstiles | 3:21 pm Jan 7, 2012 | ||||
| boia.org tools request, ignore robots.txt |
2 | Pfui | 7:06 am Jan 7, 2012 | ||||
| Googlebot-richsnippets |
2 | Pfui | 2:37 am Jan 7, 2012 | ||||
| MFE expand |
3 | Pfui | 12:44 am Jan 7, 2012 | ||||
| Vocus Hocus Pocus |
2 | Pfui | 11:17 pm Jan 6, 2012 | ||||
| Kindle & Amazon IPs? |
28 | keyplyr | 8:45 pm Jan 5, 2012 | ||||
| Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1) |
9 | Pfui | 8:41 pm Jan 5, 2012 | ||||
| Automatically detect crawlers / bots |
13 | Globetrotter | 3:03 pm Jan 5, 2012 | ||||
| Semrushbot/0.9 |
5 | tangor | 11:12 pm Jan 4, 2012 |
