Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| 65.24.37.57 - toto toto? - no referrer, no ua |
6 | misosoph | 4:32 am Jun 4, 2002 | ||||
| Unwanted Spiders - how to limit/kill access simple ways to code, spiders list etc. |
3 | lak12 | 7:56 pm Jun 3, 2002 | ||||
| scooter/1.0 (netscan.fr) Is this Alta Vista? |
3 | volatilegx | 5:51 pm Jun 3, 2002 | ||||
| Monitoring bot activity with a script? |
8 | Bluestreak | 1:00 am Jun 3, 2002 | ||||
| autoemailspider email harvester or poorly named spider? |
3 | zem | 12:48 am Jun 2, 2002 | ||||
| DiaGem Another bot to ban |
6 | Son_House | 6:03 am Jun 1, 2002 | ||||
| 216.200.130.207 Ask Jeeves Spider? |
3 | Key_Master | 7:15 pm May 31, 2002 | ||||
| new google spider IP? 64.95.79.195 |
5 | chiefmonkey | 5:44 pm May 31, 2002 | ||||
| NetResearchServer/2.2 Another new name for IncyWincy |
2 | mbauser2 | 7:04 am May 31, 2002 | ||||
| Alexa impersonator? Is someone faking an Alexa spider? |
6 | Ralph_Slate | 10:20 am May 30, 2002 | ||||
| new FAST IPs just found these |
7 | volatilegx | 9:10 am May 30, 2002 | ||||
| Is this a spider messes up my logfiles |
4 | MarkArt | 4:01 am May 30, 2002 | ||||
| Using mod_rewrite to stop bad spiders Where is it? |
5 | transistor | 12:47 am May 30, 2002 | ||||
| What is too much... |
2 | opulyst | 11:55 pm May 29, 2002 | ||||
| Legit Spider Usage? Are there ways I can use spiders that don't cause problems for site owners? |
2 | OpeRabbit | 8:40 pm May 29, 2002 | ||||
| Identifying These IP's Totally Skewed my Log Report |
16 | JanCounselor | 6:23 pm May 29, 2002 | ||||
| Atspider Spam spider gone mad |
2 | estaquieto | 2:57 pm May 28, 2002 | ||||
| New Spider? Gigabot/1.0 |
2 | dcheney | 9:15 am May 25, 2002 | ||||
| Win32; WinHttp.WinHttpRequest.5 67.113.225.66 |
4 | wilderness | 12:39 am May 25, 2002 | ||||
| Block and Complain success Sometimes they do clean up their act |
5 | jdMorgan | 5:31 am May 24, 2002 | ||||
| Interesting spiders from AV & Mercator |
17 | Air | 3:47 pm May 23, 2002 | ||||
| Cruizer IV Does anyone know it |
3 | olwen | 8:38 am May 22, 2002 | ||||
| What is this user agent? lwp-request/2.01 |
3 | korkus2000 | 9:15 pm May 21, 2002 | ||||
| Zealbot LookSmart Spider |
2 | Key_Master | 6:43 am May 21, 2002 | ||||
| How do I disallow spiders? I want to know what code to put on a page to not allow spiders to crawl. |
2 | Mitch | 5:03 pm May 20, 2002 |