Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| ccubee/9.0 |
2 | GaryK | 10:56 pm Nov 10, 2006 | ||||
| Is there an opt in script for bots? What bots should be allowed and how can it be done? |
14 | Gimp | 7:59 pm Nov 9, 2006 | ||||
| FAST Data Search Document Retriever |
2 | referer | 7:50 pm Nov 6, 2006 | ||||
| Bluecoat DRTR |
5 | GaryK | 7:29 pm Nov 2, 2006 | ||||
| Accelobot created by "masterminds" |
5 | Umbra | 12:43 am Nov 2, 2006 | ||||
| CobWeb/1.0 (auto classifier) |
3 | GaryK | 9:52 pm Nov 1, 2006 | ||||
| dynamic or static? how do I tell if an IP address is static |
6 | the_nerd | 5:34 am Oct 31, 2006 | ||||
| Analog, Webalizer, AWStats.which one identifies spiders? |
4 | johnnstacy | 2:30 am Oct 31, 2006 | ||||
| 209.2.34.114 cloaking or what? Banned bot keeps coming |
3 | simplicity | 11:15 am Oct 30, 2006 | ||||
| Search Spider to search inside pdf files? Any sfree pider to search pdf files of a website? |
2 | financialhost | 3:13 pm Oct 28, 2006 | ||||
| Make identification easier with cache control? Theoretically ... |
4 | Umbra | 2:56 pm Oct 26, 2006 | ||||
| msnbot/1.0 from CN real or fake? |
5 | thetrasher | 1:48 pm Oct 25, 2006 | ||||
| Spoofing as GoogleBot |
4 | keyplyr | 8:12 pm Oct 24, 2006 | ||||
| NetIQ - what are they looking for? |
16 | Mokita | 6:50 pm Oct 22, 2006 | ||||
| `ائ Yahoo! or Open Proxy? |
2 | GaryK | 7:08 am Oct 22, 2006 | ||||
| Hitwise Spider How do you block |
13 | RichTC | 4:26 am Oct 21, 2006 | ||||
| Simple syntax errors Can be difficult to see |
2 | wilderness | 1:14 pm Oct 19, 2006 | ||||
| Cks Google UA? |
3 | keyplyr | 1:04 pm Oct 19, 2006 | ||||
| Requests threw a 406 error User agent: "Mozilla/5.0" |
4 | Mokita | 10:47 pm Oct 18, 2006 | ||||
| Mozilla/4.0; (Compitable; MSIE 6.0; Windows NT 5.1;) MS is doing sneaky stuff |
8 | GaryK | 8:32 pm Oct 16, 2006 | ||||
| Fotochecker Except it didn't check any photos! |
2 | GaryK | 4:27 pm Oct 16, 2006 | ||||
| BlockNote.Net HTML editor. Ripped an entire site. |
3 | GaryK | 3:35 pm Oct 16, 2006 | ||||
| Spider coming out of 208.17.184.* Mystery spider coming out of Verisign |
8 | moocow | 3:25 am Oct 16, 2006 | ||||
| Edacious & Intelligent Web Crawler |
12 | volatilegx | 7:38 pm Oct 15, 2006 | ||||
| MSRBOT With Abovenet IP address Is this legit |
18 | dataguy | 3:32 am Oct 14, 2006 |