Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| 216.32.64.10 ? |
5 | mark roach | 8:23 pm Sep 21, 2000 | ||||
| Brett's IP lists maybe some corrections needed.. |
9 | PeteU | 4:36 am Sep 19, 2000 | ||||
| Strange Request from inktomi2-not.server.ntl.com |
2 | shri | 7:03 am Sep 18, 2000 | ||||
| NEW Spider?? Anybody seen this guy before |
3 | drbill | 1:16 am Sep 13, 2000 | ||||
| infoseek spiders? |
2 | roscoepico | 9:04 pm Sep 11, 2000 | ||||
| AskJeeves Crawler? Was I sleeping, and missed some topic on this? :) |
3 | redzone | 2:11 pm Sep 8, 2000 | ||||
| 1x1 pixel links on index page |
8 | knoxy | 11:28 pm Sep 7, 2000 | ||||
| Help needed |
7 | adamcmoran | 1:46 am Sep 7, 2000 | ||||
| Directhit is crawling hard. Mozilla/4.06 [en] (Direct Hit Grabber) |
11 | littleman | 7:25 pm Sep 6, 2000 | ||||
| Indeed it is |
2 | adamcmoran | 2:51 pm Sep 6, 2000 | ||||
| Houston5 |
3 | bluesriff | 11:43 pm Sep 3, 2000 | ||||
| who owns this IP? 24.165.231.85 |
4 | symmetry | 6:10 pm Sep 1, 2000 | ||||
| AOL HTTP Requests w/ No User Agent? Anyone else seeing this in their logs? |
2 | redzone | 3:18 pm Aug 31, 2000 | ||||
| crawler0-complaints-to-admin.webresearch.pa-x.dec.c Have I been sprung? |
5 | han chung | 8:35 pm Aug 27, 2000 | ||||
| av? |
7 | roscoepico | 7:31 am Aug 26, 2000 | ||||
| "Kite-Hourly" Intomi spider ?? |
3 | grnidone | 3:07 pm Aug 23, 2000 | ||||
| BEWARE... might be a sniffer.... |
5 | Kamikaze | 11:07 pm Aug 21, 2000 | ||||
| Aranha? Who, what, where and why? |
3 | makemetop | 9:32 am Aug 20, 2000 | ||||
| AOL's Spiders? They have been visiting my site for 1 week. |
2 | iambbq | 6:56 am Aug 16, 2000 | ||||
| Oxxbot1, should I be happy, or worried? Found this while wandering the web... |
3 | han solo | 7:29 pm Aug 13, 2000 | ||||
| Is Big Brother Lurking? Microsoft URL Control! I think they found me... |
5 | metaman | 4:54 am Aug 12, 2000 | ||||
| New Engine? |
5 | scott | 9:35 pm Aug 11, 2000 | ||||
| Infoseek Are we having fun...yet? |
2 | Dumpy | 4:17 pm Aug 10, 2000 | ||||
| inktomi1-car.server.ntl.com |
8 | stcrim | 9:33 am Aug 10, 2000 | ||||
| AV spider Do I want to know what this is?! |
5 | jilly | 4:24 pm Aug 9, 2000 |