Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| How To Block Possible Click Bombing |
20 | EmptyRoom | 5:03 am Apr 15, 2014 | ||||
| Can the Googlebot read the blocking rules inside .htaccess? |
5 | Andy500 | 8:55 pm Apr 14, 2014 | ||||
| Can Cloudfare stop content scrapers? |
4 | Andy500 | 5:12 pm Apr 14, 2014 | ||||
| Should I block msnbot-media |
5 | shamrock | 8:26 pm Apr 9, 2014 | ||||
| Previous Server Farms Threads |
4 | incrediBILL | 4:46 pm Apr 9, 2014 | ||||
| Should blank user agents be blocked or not? |
5 | aristotle | 7:47 pm Apr 7, 2014 | ||||
| Server Farms - Feb. 2014 Ongoing Hosting Data Center Discussion[4] ( 1 2 3 4 ) |
109 | incrediBILL | 5:43 pm Apr 7, 2014 | ||||
| Is this googlebot or not? Seems this in GBot's Range but doesn't Validate |
14 | webcentric | 2:36 pm Apr 7, 2014 | ||||
| How to block googleusercontent.com blocking google and googleusercontent in htaccess |
8 | rogerroger | 5:33 am Apr 7, 2014 | ||||
| Bots Crawling Pages Blocked by Robots.txt |
7 | incrediBILL | 6:52 am Apr 5, 2014 | ||||
| Is there a list of user-agents to block as of today? I'm launching a new website soon and want to avoid scrapers |
6 | shamrock | 6:06 am Apr 5, 2014 | ||||
| Hack attempt from a MS range |
21 | keyplyr | 2:32 am Mar 25, 2014 | ||||
| What is Project 5 Research, Inc. |
12 | HollyWMarshall | 2:29 am Mar 25, 2014 | ||||
| Strange crawl |
8 | wilderness | 11:22 am Mar 23, 2014 | ||||
| Semalt Referrer spamming gone mad. |
23 | blend27 | 2:10 am Mar 22, 2014 | ||||
| seegnifybot user agent and IP data |
4 | keyplyr | 11:03 pm Mar 20, 2014 | ||||
| Best Practices for blocking in Windows Server, IIS, .NET environment? |
8 | webcentric | 3:11 am Mar 20, 2014 | ||||
| Threatglass Barracuda's web site threats database |
4 | dstiles | 12:07 am Mar 20, 2014 | ||||
| Google preview |
5 | lucy24 | 10:01 pm Mar 17, 2014 | ||||
| anyone seen the kimono scraper tool yet? |
23 | londrum | 2:00 am Mar 15, 2014 | ||||
| What about your Parked domains. And bots?[2] ( 1 2 ) |
44 | Angonasec | 1:15 pm Mar 13, 2014 | ||||
| Stealth crawler out of Cogent / PSINet 38.99.82.191 |
3 | carson63000 | 3:00 am Mar 5, 2014 | ||||
| .NET Framework Test Client HttpWebRequest.UserAgent |
4 | blend27 | 7:11 pm Mar 2, 2014 | ||||
| Google-HTTP-Java-Client/1.17.0-rc (gzip) Anyone seeing this? |
4 | physics | 4:17 pm Feb 23, 2014 | ||||
| Announcing find spiders.pl Wow. |
5 | physics | 4:14 pm Feb 23, 2014 |