Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Need to block a specific spider Help with robots txt |
7 | coosblues | 9:38 pm May 25, 2003 | ||||
| Visits from 'mercury.hiddenservers.com' Is this a Google bot? |
5 | salson | 1:37 pm May 25, 2003 | ||||
| kulokobot backweave |
3 | DavidT | 8:07 pm May 24, 2003 | ||||
| Why is deepbot using freshbot ip? |
2 | requiem | 7:44 pm May 24, 2003 | ||||
| wizard.dataforce.net kore wa, nani? (= what's this?) |
4 | berli | 7:43 pm May 23, 2003 | ||||
| freshbot instead ob deepbot 216.x.x.x 64.x.x.x classification still true? |
3 | plasma | 5:23 pm May 23, 2003 | ||||
| Is this Googlebot or an imposter? New type of entry in my log files |
4 | birdstuff | 9:03 pm May 22, 2003 | ||||
| Googlebot? 216.239.39.x |
10 | Stefan | 8:37 pm May 22, 2003 | ||||
| 207.42.75.170 |
8 | DavidT | 4:44 am May 22, 2003 | ||||
| How do I set up a spider trap? badwidth theft |
3 | Digger | 11:55 pm May 21, 2003 | ||||
| 218.63.199.179 The China Farm Bank QUJING Branch |
3 | carfac | 3:45 pm May 21, 2003 | ||||
| ElectricSurfMaster::CHTMLTagGenSearcher Webwasher.com GmbH |
3 | jim_w | 2:42 pm May 21, 2003 | ||||
| MixCat Out Crawling or Log Spamming? |
3 | EliteWeb | 2:02 pm May 21, 2003 | ||||
| How to exclude linko/0.1 libwww-perl/5.66 |
5 | ovtech | 4:45 am May 21, 2003 | ||||
| 212.150.53.131 |
4 | ircgeeks | 6:01 am May 20, 2003 | ||||
| Qango.com Web Directory http://www.qango.com/ |
2 | DavidT | 6:47 pm May 19, 2003 | ||||
| 139.81.162.6 |
2 | wilderness | 6:42 pm May 19, 2003 | ||||
| 209.49.118.22 |
2 | wilderness | 6:42 pm May 19, 2003 | ||||
| New robot? |
3 | fandangood | 6:00 pm May 19, 2003 | ||||
| Java Script and DHTML Spider Detection Is text detected by the spiders |
6 | jgold454 | 8:34 pm May 17, 2003 | ||||
| CJ Spider ? |
16 | DavidT | 4:40 pm May 17, 2003 | ||||
| 65.94.226.152 new spider? |
11 | werty | 6:37 am May 17, 2003 | ||||
| AOL search query results in single hit to page requests only html file w/o menu, css, background, etc. |
7 | nativenewyorker | 8:42 pm May 16, 2003 | ||||
| Here's a new one... Microsoft Internet Browser |
3 | volatilegx | 5:15 pm May 16, 2003 | ||||
| New Websense IP? 64.156.198.69 |
4 | WitchLars | 3:58 am May 16, 2003 |