Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Just to let you know of a new web crawling robot. Kiwi robot |
4 | Dave_A | 5:05 am Jan 5, 2005 | ||||
| Majestic-12 distributed search engine/spider |
3 | volatilegx | 10:05 pm Jan 3, 2005 | ||||
| 63.251.132.118 - chewed up 7.4GB of bandwidth on my site |
4 | merrick_lozano | 9:31 pm Jan 3, 2005 | ||||
| "SURF" user agent A more detailed look... |
2 | JAB_Creations | 1:35 am Jan 3, 2005 | ||||
| Fast |
5 | wilderness | 5:06 am Jan 2, 2005 | ||||
| User agent: KMHTTP Is it Excite Japan, HP or something else? |
3 | GaryK | 5:40 pm Dec 31, 2004 | ||||
| Blaiz-Bee/1.0 New Cluster Engine? |
3 | pendanticist | 9:16 pm Dec 30, 2004 | ||||
| Anyone recognize this? getting crawled like crazy |
2 | Easy_Coder | 4:28 pm Dec 30, 2004 | ||||
| e-collector Spider |
4 | joeychgo | 10:08 pm Dec 27, 2004 | ||||
| speedy.net.pe who is |
5 | kdawg | 8:36 pm Dec 23, 2004 | ||||
| Faxobot/1.0 coming soon per home page |
10 | Busynut | 6:50 am Dec 22, 2004 | ||||
| New Exava bot? BecomeBot@exava.com |
18 | Reflection | 5:57 am Dec 22, 2004 | ||||
| Microsoft spider? had site hit from multiple IP's one of which assigned to Microsoft |
9 | ecomagic | 12:51 am Dec 17, 2004 | ||||
| PlantyNet_WebRobot_V1.9 'Nuther one with an e-mail contact addy... |
2 | pendanticist | 1:01 pm Dec 16, 2004 | ||||
| htaccess regex syntax |
7 | mattie | 7:12 pm Dec 15, 2004 | ||||
| StanleyWebSpider/1.0 |
4 | Ankheg | 12:34 am Dec 15, 2004 | ||||
| Netcraft Web Server Survey What's this one? |
9 | Rollo | 5:28 am Dec 12, 2004 | ||||
| MAXOMObot Video Search rides the 'Net. ( Another Nutch. ) |
4 | pendanticist | 2:50 am Dec 11, 2004 | ||||
| sna-0.0.1 |
9 | wilderness | 2:44 am Dec 11, 2004 | ||||
| Mozilla/5.0 (Sonar) Java 100k hits yesterday |
4 | petez1 | 11:19 pm Dec 10, 2004 | ||||
| Slurp Yahoo is spidering on new Class Cs |
2 | volatilegx | 8:19 pm Dec 10, 2004 | ||||
| 7 | Strange | 3:00 pm Dec 10, 2004 | |||||
| William Lu |
4 | wilderness | 1:19 pm Dec 10, 2004 | ||||
| UA's with email addresses |
5 | JAB_Creations | 12:17 am Dec 8, 2004 | ||||
| StackRambler/2.0 From then, 'till now. |
3 | pendanticist | 9:51 pm Dec 7, 2004 |