Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Is slurp friendly? doesn't follow 301's and ignores robots.txt |
3 | wingslevel | 7:09 pm Sep 4, 2003 | ||||
| aaatravelermags.com (12.10.127.130) new spider? |
5 | gibEll | 12:23 am Sep 4, 2003 | ||||
| new bot: http://www.almaden.ibm.com/cs/crawler [c01] IBM Web Fountain |
9 | claus | 11:46 pm Sep 2, 2003 | ||||
| eseek-crawler-larbin-2.63 crawler@exactseek.com exactseek.com - 2 UA's |
4 | claus | 11:53 pm Sep 1, 2003 | ||||
| managednetworks.com Grabbed / from a bunch of sites |
6 | seindal | 10:29 pm Sep 1, 2003 | ||||
| What isthe purpose of "HEAD / HTTP/1.1" or "GET / HTTP/1.1"? I see these entries often in my web logs |
7 | Wizcrafts | 8:26 pm Sep 1, 2003 | ||||
| suchtop-bot-1.14 Can't find anything about it |
4 | Elmar | 8:41 am Sep 1, 2003 | ||||
| 216.33.229.163 Can I block this IP? |
2 | salson | 1:04 pm Aug 30, 2003 | ||||
| Is 63.225.216.82 a bot? It's slamming pages that don't exist |
4 | malexandrou | 12:25 am Aug 30, 2003 | ||||
| 216.217.36.130 stupid user or bot? |
5 | coyote | 10:49 pm Aug 29, 2003 | ||||
| From China with Spam heads up on a few IPs |
2 | coyote | 7:12 am Aug 28, 2003 | ||||
| Tandem scooter? (Scooter/3.3 & Scooter/3.3_SF) two UA's now |
4 | claus | 6:46 pm Aug 27, 2003 | ||||
| New UA signature for FAST |
2 | rbs10025 | 6:41 pm Aug 27, 2003 | ||||
| Mozilla/4.0 (compatible; GoogleToolbar 1.1.60-deleon; Windows 98 SE 4. wot? |
2 | claus | 6:07 pm Aug 27, 2003 | ||||
| Will this work Trying to get my robots.txt file right |
3 | le_gber | 2:28 pm Aug 27, 2003 | ||||
| Update on boitho.com-robot/1.1 Possible UA fake |
7 | cyberkat | 1:11 pm Aug 27, 2003 | ||||
| .net Clr 1.1.4322 Is this ok or not? |
4 | JadedJane | 4:57 am Aug 27, 2003 | ||||
| 64.37.103.2 (bundy.infomaxinc.com) |
5 | WarmGlow | 7:27 pm Aug 26, 2003 | ||||
| Mister Pix II 2.10 User-Agent Mister Pix II 2.10 |
3 | tnlnyc | 2:16 pm Aug 26, 2003 | ||||
| Taco Bell |
2 | wilderness | 11:58 pm Aug 25, 2003 | ||||
| Propfind Konqueror/3.1; Linux? |
5 | jim_w | 8:50 pm Aug 25, 2003 | ||||
| 69.10.144.45 Terrar-UK_Search robot@terrar.co.uk |
2 | volatilegx | 11:32 am Aug 25, 2003 | ||||
| User_Agent "MARTINI" - from Looksmart? Anybody heard of this UA? |
3 | Friday | 7:00 am Aug 25, 2003 | ||||
| Hit-and-run spidering? |
6 | jomaxx | 9:19 pm Aug 24, 2003 | ||||
| Mozilla/4.0 (compatible; MSIE 5.00; Windows 98 maxmp3 - chinese bot? |
3 | claus | 1:19 pm Aug 24, 2003 |
