Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Who's looking at my pages Is this a search engine? |
2 | Roscoe | 9:46 pm Mar 23, 2003 | ||||
| New Heavy Use Spider? 1900 pages + |
2 | Blade | 10:29 pm Mar 22, 2003 | ||||
| I need a new program to parse out my logs. Not satisfied with the reports I am getting |
7 | juniperwasting | 11:18 pm Mar 21, 2003 | ||||
| NetRatings / 64.124.34.46 Generating a lot of 404s |
4 | pallo | 11:13 pm Mar 21, 2003 | ||||
| Intelliseek brand protection robot |
7 | ScottMcG | 5:54 pm Mar 21, 2003 | ||||
| Black Widow Web Crawler Something to watch out for? |
5 | MarieC | 2:09 pm Mar 21, 2003 | ||||
| AmfibiBOT what is it? |
7 | khillman | 6:08 pm Mar 20, 2003 | ||||
| Current verified SPAM-bots User-agents and IP's |
4 | Orange_XL | 9:23 am Mar 20, 2003 | ||||
| 202.130.245.216 what's it? a new spider? |
4 | Chris_1977 | 10:08 pm Mar 19, 2003 | ||||
| Freshbot and Robots.txt |
7 | DavidT | 6:58 pm Mar 19, 2003 | ||||
| Eversion Avenger/37.17 (Chorus/MiX 3.2; 4-bit) No luck with Google {SIGH} -- HELP :) |
8 | xxX_RIPP_Xxx | 7:10 am Mar 19, 2003 | ||||
| Some quaint and unique (?) UAs *lol* |
17 | thermoman | 5:13 am Mar 19, 2003 | ||||
| Mozilla/3.0 (compatible) NOT Mozilla/3.0 (compatible;) |
3 | pendanticist | 11:39 pm Mar 18, 2003 | ||||
| My Spider report showing bots getting error and leaving Unknown error is keeping bots out of my site. |
12 | juniperwasting | 9:18 pm Mar 18, 2003 | ||||
| could "netscape 5" be a spider? |
4 | BillM | 7:02 am Mar 18, 2003 | ||||
| How to Ban Bots which come without http_user_agent info? It's real problem, I don't know how to set it up with rewrite mod |
11 | Chris_1977 | 2:52 am Mar 18, 2003 | ||||
| aol.com crawler = googlebot.com? |
9 | damiengazier | 1:42 am Mar 18, 2003 | ||||
| Lite Bot 0616B Ver numbers decreasing? |
3 | MarieC | 6:54 pm Mar 17, 2003 | ||||
| 216.191.20.26 |
3 | Brett_Tabke | 6:23 pm Mar 17, 2003 | ||||
| lachesis spider from Japan (Nara province) lachesis spider from Japan (Nara province) |
5 | misslulu | 2:10 pm Mar 17, 2003 | ||||
| PlantyNet_WebRobot_V1.9 Has anyone seen this? |
2 | MarieC | 12:26 pm Mar 16, 2003 | ||||
| obot spidered by 5 day old site |
3 | coffeelover | 9:11 am Mar 16, 2003 | ||||
| T-h-u-n-d-e-r-s-t-o-n-e Anyone know about this one? |
11 | pendanticist | 5:14 am Mar 16, 2003 | ||||
| 4 | StanTheMan | 7:28 pm Mar 15, 2003 | |||||
| Anyone know about UA: Datum/0.1 coming from IPs with a web server |
2 | acronym | 6:40 pm Mar 15, 2003 |