Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Girafabot does it disobey robots.txt? and a question... |
5 | davelms | 8:22 am Aug 7, 2003 | ||||
| SpiderKU/0.9 158.108.35.123 |
4 | volatilegx | 2:57 pm Aug 6, 2003 | ||||
| libwww-perl/5.53 |
2 | Raymond | 6:31 pm Aug 5, 2003 | ||||
| WebRACE/1.1 (University of Cyprus, Distributed Crawler) |
2 | jdMorgan | 6:20 pm Aug 5, 2003 | ||||
| about pay how much do i get,if i am CCNA,CCIE |
3 | nighteye | 4:21 pm Aug 5, 2003 | ||||
| newskies.net Bad bot? |
3 | dorward | 3:03 pm Aug 5, 2003 | ||||
| Who is Savvis.net with IP 216.88.158.142? |
2 | akogo | 3:15 pm Aug 4, 2003 | ||||
| Linkman Referrer spam bot? |
3 | dorward | 12:59 pm Aug 4, 2003 | ||||
| How To Block trek6.sv.av.com from my site trek6.sv.av.com takes up too much bandwidth |
6 | akogo | 8:59 pm Aug 2, 2003 | ||||
| Exalead NG/MimeLive Client (convert/http/0.123) New bot or former NG/1.0? |
2 | claus | 10:21 pm Aug 1, 2003 | ||||
| WebSearchBench no robots either |
2 | bull | 6:30 pm Aug 1, 2003 | ||||
| Zibie Spider 0.1 Java/1.4.2 no robots. |
2 | bull | 5:38 pm Aug 1, 2003 | ||||
| Good guy spiders and the Bad ones! Is there a list |
7 | atlinx | 3:50 pm Aug 1, 2003 | ||||
| My list is outdated |
2 | rainborick | 2:21 pm Aug 1, 2003 | ||||
| Ano-Kato Greek SE |
3 | volatilegx | 2:18 pm Aug 1, 2003 | ||||
| zeus 41852 webster pro v2.9 win32? zeus 41852 webster pro v2.9 win32 bot |
4 | mcavill | 10:11 am Aug 1, 2003 | ||||
| DigExt |
4 | Tonearm | 12:35 am Aug 1, 2003 | ||||
| QuePasaCreep v0.9.13 from ipa 69.28.130.229 new semi-bogus spider? |
18 | rodent | 9:01 am Jul 31, 2003 | ||||
| Anyone recognise this? |
2 | davelms | 7:40 pm Jul 30, 2003 | ||||
| New Exalead UA string? |
3 | jrobbio | 12:44 am Jul 30, 2003 | ||||
| Spoofing LookSmart? Fell into Spider Trap |
2 | frontpage | 11:50 pm Jul 29, 2003 | ||||
| Anybody recognise this ip? Just checking the log's when I found this |
13 | Glovebox | 10:55 pm Jul 29, 2003 | ||||
| allowing 403 page to denied IPs |
5 | nancyb | 10:40 pm Jul 29, 2003 | ||||
| AOLserver |
2 | wilderness | 4:40 pm Jul 29, 2003 | ||||
| Dolly/1.0 Anyone else see this? |
8 | Finder | 9:41 pm Jul 28, 2003 |