Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Pete-Spider Light |
2 | keyplyr | 11:24 pm Mar 20, 2008 | ||||
| TekSavvy |
8 | wilderness | 12:20 pm Mar 20, 2008 | ||||
| gnu-classpath Cute |
3 | wilderness | 9:27 pm Mar 19, 2008 | ||||
| zermelo |
3 | zerillos | 8:34 pm Mar 19, 2008 | ||||
| odd access log entry access log get form entry |
2 | esme | 10:03 pm Mar 18, 2008 | ||||
| anyone know this UA? all the spaces are +'s :? |
11 | wkitty42 | 4:07 pm Mar 18, 2008 | ||||
| agodar |
3 | Mokita | 10:28 pm Mar 17, 2008 | ||||
| User Agent "Firefox" |
8 | marodhum | 10:17 pm Mar 17, 2008 | ||||
| Ban spider by connection: close Header, (idea) I got an idea by ban spider by connection: close header |
15 | Eric | 9:04 am Mar 17, 2008 | ||||
| libwww-perl/5.53 archive.org |
10 | react | 7:32 am Mar 17, 2008 | ||||
| Quintura-Crw/0.1 Quintura-Crw/0.1 from IP 208.82.204.zz |
4 | Eric | 1:46 am Mar 17, 2008 | ||||
| New RSS Reader - Spider? Asa/1.0.1 |
2 | Ocean10000 | 1:07 am Mar 17, 2008 | ||||
| Jeanie 2008 bot No robots request, fell straight into spider trap |
2 | Receptional_Andy | 9:57 pm Mar 16, 2008 | ||||
| Yahoo Slurp China htaccess block Slurp China htaccess block |
13 | cyberdyne | 9:02 am Mar 8, 2008 | ||||
| strange assignment |
2 | wilderness | 10:47 am Mar 4, 2008 | ||||
| LTI/LemurProject another Nutch pest |
8 | keyplyr | 6:10 am Mar 3, 2008 | ||||
| T-h-u-n-d-e-r-s-t-o-n-e |
3 | wilderness | 6:20 pm Feb 29, 2008 | ||||
| Defining good bad bots |
4 | smallcompany | 7:08 am Feb 29, 2008 | ||||
| Home brewed spider? 216.237.31.*** scraping my site |
6 | Megaclinium | 2:17 am Feb 29, 2008 | ||||
| spider analysis software |
3 | gdawg | 12:31 am Feb 28, 2008 | ||||
| Exalead = Abovenet Communications, Inc. 193.47.80.xx and now 64.124.148.xx |
5 | Bewenched | 11:34 pm Feb 26, 2008 | ||||
| Rogue bot from microsoft Msrbot |
5 | marodhum | 7:35 pm Feb 25, 2008 | ||||
| Baidu Image Spider Is it valid? |
5 | GaryK | 11:02 am Feb 24, 2008 | ||||
| Teemer from NetSeer |
20 | keyplyr | 10:00 am Feb 24, 2008 | ||||
| Return of Yahoo! Slurp/3.0 Last seen from Inktomi |
6 | jdMorgan | 1:42 pm Feb 23, 2008 |