Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| What's this? not a spider but it's "pulling" everything |
17 | SuzyUK | 3:32 pm Sep 11, 2003 | ||||
| New Fast/Overture UA Yet another new one from FAST |
7 | jdMorgan | 10:09 am Sep 11, 2003 | ||||
| Sleipnir what is this? |
11 | keyplyr | 7:17 am Sep 11, 2003 | ||||
| Need help with IP block range RegExp for RewriteCond I want to block 211.104.0.0 - 211.119.255.255 |
9 | Wizcrafts | 8:16 pm Sep 10, 2003 | ||||
| Googlebot's schedule? When does Googlebot visit? |
3 | Widestrides | 11:11 pm Sep 9, 2003 | ||||
| Spiders Identification |
5 | irvseykat | 9:13 pm Sep 8, 2003 | ||||
| More Bad Bots? |
3 | BlueSky | 8:39 pm Sep 7, 2003 | ||||
| AOLserver-Tcl/3.5.6 what is it? |
4 | coyote | 7:24 pm Sep 7, 2003 | ||||
| 64.40.105.197 Look.com |
4 | wilderness | 6:03 pm Sep 7, 2003 | ||||
| PlantyNet_WebRobot_V1.9 a bot and its two humans |
2 | coyote | 2:17 am Sep 6, 2003 | ||||
| Exalead -- it comes for your .mov files Why should I let this sucker on my system? |
4 | berli | 12:36 am Sep 6, 2003 | ||||
| 68.166.235.57 changed UA |
5 | coyote | 3:51 am Sep 5, 2003 | ||||
| Illinois State Tech Labs from a french IP |
3 | coyote | 2:28 am Sep 5, 2003 | ||||
| Feedster Crawler |
2 | frontpage | 10:52 pm Sep 4, 2003 | ||||
| Is slurp friendly? doesn't follow 301's and ignores robots.txt |
3 | wingslevel | 7:09 pm Sep 4, 2003 | ||||
| aaatravelermags.com (12.10.127.130) new spider? |
5 | gibEll | 12:23 am Sep 4, 2003 | ||||
| new bot: http://www.almaden.ibm.com/cs/crawler [c01] IBM Web Fountain |
9 | claus | 11:46 pm Sep 2, 2003 | ||||
| eseek-crawler-larbin-2.63 crawler@exactseek.com exactseek.com - 2 UA's |
4 | claus | 11:53 pm Sep 1, 2003 | ||||
| managednetworks.com Grabbed / from a bunch of sites |
6 | seindal | 10:29 pm Sep 1, 2003 | ||||
| What isthe purpose of "HEAD / HTTP/1.1" or "GET / HTTP/1.1"? I see these entries often in my web logs |
7 | Wizcrafts | 8:26 pm Sep 1, 2003 | ||||
| suchtop-bot-1.14 Can't find anything about it |
4 | Elmar | 8:41 am Sep 1, 2003 | ||||
| 216.33.229.163 Can I block this IP? |
2 | salson | 1:04 pm Aug 30, 2003 | ||||
| Is 63.225.216.82 a bot? It's slamming pages that don't exist |
4 | malexandrou | 12:25 am Aug 30, 2003 | ||||
| 216.217.36.130 stupid user or bot? |
5 | coyote | 10:49 pm Aug 29, 2003 | ||||
| From China with Spam heads up on a few IPs |
2 | coyote | 7:12 am Aug 28, 2003 |