Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| AlphaBot |
2 | jonasjacek | 7:59 pm Nov 12, 2017 | ||||
| damn404 broken link checker |
6 | keyplyr | 12:49 am Nov 12, 2017 | ||||
| Ant.com beta |
3 | keyplyr | 6:12 am Nov 7, 2017 | ||||
| MuckRack |
9 | keyplyr | 7:20 pm Nov 6, 2017 | ||||
| humans.txt file We Are Humans, Not Machines! |
12 | keyplyr | 5:59 pm Oct 27, 2017 | ||||
| Google referers Not HTTPS |
27 | dstiles | 6:34 pm Oct 24, 2017 | ||||
| um-LN |
8 | keyplyr | 6:30 pm Oct 24, 2017 | ||||
| check http python,bot |
6 | TorontoBoy | 2:42 pm Oct 24, 2017 | ||||
| AhrefsBot new assigned crawl range |
7 | keyplyr | 6:13 pm Oct 22, 2017 | ||||
| Dataprovider.com new UA |
6 | keyplyr | 4:48 pm Oct 20, 2017 | ||||
| Gowikibot |
8 | keyplyr | 9:39 am Oct 18, 2017 | ||||
| Blank header fields |
7 | dstiles | 6:44 pm Oct 12, 2017 | ||||
| What is this IP doing |
3 | born2run | 6:50 pm Oct 7, 2017 | ||||
| DarcyRipper website ripper/website downloader |
5 | jonasjacek | 8:22 pm Oct 5, 2017 | ||||
| Magic Browser still with us |
4 | lucy24 | 4:49 am Oct 4, 2017 | ||||
| shrinktheweb new UA, new range |
9 | keyplyr | 8:34 pm Oct 1, 2017 | ||||
| So I am blocking some bots via UA please help. |
6 | born2run | 2:33 am Sep 29, 2017 | ||||
| Chrome/50: fully human(oid) |
5 | lucy24 | 1:56 am Sep 27, 2017 | ||||
| elefent new UA, new host |
4 | keyplyr | 11:27 am Sep 26, 2017 | ||||
| Konqueror |
17 | keyplyr | 4:52 pm Sep 25, 2017 | ||||
| Blueno New Naver UA |
5 | keyplyr | 7:36 pm Sep 17, 2017 | ||||
| SISTRIX Crawler |
5 | keyplyr | 2:58 am Sep 16, 2017 | ||||
| PrintFriendly |
4 | lucy24 | 6:06 am Sep 13, 2017 | ||||
| Pearltrees |
4 | keyplyr | 5:47 am Sep 12, 2017 | ||||
| Ubuntu/dapper-security |
6 | keyplyr | 7:43 am Sep 11, 2017 |