Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| 66.207.120.227 almost bringing down server |
6 | lazerzubb | 11:09 pm Feb 1, 2003 | ||||
| Monitoring Spiders |
14 | glenv | 1:13 pm Feb 1, 2003 | ||||
| Zelig/0.4 alpha2 Yet a new bot? |
4 | FineWare | 4:26 am Feb 1, 2003 | ||||
| Spider Behavior To hammer or not to hammer |
5 | tkarade | 5:22 am Jan 31, 2003 | ||||
| [OWR_Crawler] New bot? |
7 | Orange_XL | 11:08 pm Jan 29, 2003 | ||||
| obot is back and not being very nice... ..asking for tons of files not even there. |
7 | pendanticist | 1:41 pm Jan 29, 2003 | ||||
| mod_rewrite and httpd.conf how do they fit together? |
5 | NickH | 1:08 pm Jan 29, 2003 | ||||
| Python-urllib/2.0a1 |
5 | wilderness | 3:28 pm Jan 27, 2003 | ||||
| LinkSweeper/1.1 One of them Warez thangs. |
5 | pendanticist | 9:26 pm Jan 26, 2003 | ||||
| NPBot-1/2.0 crawler1.crawler918.com |
3 | pendanticist | 11:49 am Jan 26, 2003 | ||||
| Newbie Spidering Question How does that work? |
8 | lanesharon | 5:20 am Jan 26, 2003 | ||||
| BaiDuSpider |
29 | Steph | 3:39 am Jan 26, 2003 | ||||
| UdmSearch 3.0.3 |
2 | delpino | 5:37 am Jan 25, 2003 | ||||
| spiders behavior on 301 redirect |
6 | pardo | 5:02 am Jan 25, 2003 | ||||
| Watchfire WebXM 1.0 Anyone Recognize? |
6 | pendanticist | 1:50 am Jan 25, 2003 | ||||
| Newbie QUESTION(s) about bots What is the typical % of bot traffic for any given site? |
2 | sys700 | 7:41 pm Jan 24, 2003 | ||||
| Where to put an Allow/Deny in httpd.conf Putting in a list of banned IP's |
6 | carfac | 5:48 pm Jan 24, 2003 | ||||
| top 10 list of useful bots to allow on your site? instead of a trap how about only the good ones |
5 | amznVibe | 4:27 pm Jan 24, 2003 | ||||
| pavuk/0.9pl28 |
3 | wilderness | 8:48 am Jan 24, 2003 | ||||
| Scooter Doing Double Duty? Scooter hits my site from Spain? |
3 | carfac | 5:07 am Jan 24, 2003 | ||||
| W3CRobot/5.4.0 Does anyone else think this is a fake? |
4 | Dreamquick | 1:23 am Jan 24, 2003 | ||||
| potbot 1.0 Unknown robot from everyone's internet |
9 | jdMorgan | 4:57 pm Jan 23, 2003 | ||||
| IAArchiver-1.0 & robots.txt Okay how much do I need to pay to take a big stick to the archive.org bot? |
4 | Dreamquick | 1:32 pm Jan 23, 2003 | ||||
| 65.102.17.89 just tripped a site trapdoor this IP appears in a removed message? |
15 | amznVibe | 4:44 am Jan 23, 2003 | ||||
| Lacnic IP ranges |
2 | wilderness | 11:48 pm Jan 22, 2003 |