Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| I am getting hit really hard... ia_archiver |
8 | mack | 4:27 pm Sep 23, 2002 | ||||
| sherlock/1.3 httpget/1.3 |
2 | frontpage | 6:57 pm Sep 22, 2002 | ||||
| Scrubby 2.2 Is this a new spider? |
2 | Nick_W | 4:37 pm Sep 22, 2002 | ||||
| V5 HTTP/S Component Who is that? |
3 | frontpage | 3:57 pm Sep 22, 2002 | ||||
| Spider from multiple IP's all at once |
7 | carfac | 2:07 am Sep 22, 2002 | ||||
| 63.215.240.27 (unknown.level3.net) Scooter/2.0 |
6 | Key_Master | 3:45 pm Sep 20, 2002 | ||||
| av |
4 | PeteU | 7:21 pm Sep 19, 2002 | ||||
| 128.242.197.101 I am paranoid but . . . |
4 | rogerRabbit | 10:54 am Sep 19, 2002 | ||||
| MSIE 5.01 requesting robots.txt? anyone who knows this? |
3 | bull | 3:37 pm Sep 18, 2002 | ||||
| searching for formmail another candidate to ban? |
2 | bull | 9:43 am Sep 18, 2002 | ||||
| The number of googles spiders The number of googles spiders what does it stands for |
2 | Ove | 7:14 am Sep 18, 2002 | ||||
| minibot any ideas? |
2 | Finder | 11:10 pm Sep 17, 2002 | ||||
| Mysterious Spider |
5 | tesla | 7:44 pm Sep 17, 2002 | ||||
| Ask Jeeves Ask Jeeves disregarding robots.txt |
16 | Scooter24 | 5:00 pm Sep 17, 2002 | ||||
| Ill-behaved libwww-perl UA Reads and ignores robots.txt |
3 | jdMorgan | 4:15 pm Sep 17, 2002 | ||||
| Scooter ignoring one page entry in robots how can i get it to not index this page? |
8 | semick | 7:48 pm Sep 16, 2002 | ||||
| Is this a user... or a bunch of people? |
2 | carfac | 6:57 pm Sep 14, 2002 | ||||
| MFC Foundation Class Library 4.0 |
3 | Scooter24 | 2:21 pm Sep 14, 2002 | ||||
| Polybot Old News? |
3 | brotherhood_of_LAN | 11:14 pm Sep 12, 2002 | ||||
| Where are these coming from? VERY bad spiders attacking my site... |
9 | irock | 11:11 pm Sep 12, 2002 | ||||
| New Marketing Campaign Bot won't go away... |
2 | fiestagirl | 6:17 am Sep 12, 2002 | ||||
| Inktomi poking around 209.131.51.166 |
4 | fiestagirl | 11:42 pm Sep 10, 2002 | ||||
| ec009.inktomi.com |
7 | littleman | 7:56 pm Sep 10, 2002 | ||||
| Is there a list of user-agents from robots |
9 | martin | 8:20 am Sep 10, 2002 | ||||
| dloader(NaverRobot)/1.0 Anyone know what this is? |
4 | Finder | 10:23 pm Sep 9, 2002 |