Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| msn & js |
13 | wilderness | 6:55 pm Mar 31, 2020 | ||||
| okhttp/3.8.1 |
3 | tangor | 12:31 am Mar 30, 2020 | ||||
| another day, another botnet |
5 | lucy24 | 5:11 pm Mar 21, 2020 | ||||
| DomainStatsBot He is back. |
6 | Martin_Potter | 12:35 am Mar 12, 2020 | ||||
| Don't shoot the hostages - sparing DuckDuckGo on AWS Poking holes in AWS ranges |
16 | JamesSC | 8:58 am Mar 8, 2020 | ||||
| What are these +1200 USA datacenters bots doing to my site ? |
24 | adrianTNT | 4:02 pm Feb 26, 2020 | ||||
| node-superagent/3.8.3 |
13 | hekasi | 10:51 am Feb 26, 2020 | ||||
| reqwest/0.9.19 All it wants is images |
3 | tangor | 11:17 pm Feb 20, 2020 | ||||
| Not sure but the wall works for now.. |
9 | blend27 | 4:01 pm Feb 15, 2020 | ||||
| msnbot still active? |
6 | JorgeV | 12:40 pm Feb 10, 2020 | ||||
| Yandexbot |
3 | JorgeV | 12:29 pm Feb 10, 2020 | ||||
| Some resources for some. |
6 | Dimitri | 11:47 am Feb 10, 2020 | ||||
| Marketing related robots? |
3 | JorgeV | 11:37 am Feb 10, 2020 | ||||
| Thinking of blocking 40.64.0.0/10 and 51.140.0.0/14 (MSFT) Seen some bot stuff recently |
5 | SumGuy | 8:01 pm Feb 4, 2020 | ||||
| Anchorage crawler |
3 | ClosedForLunch | 6:55 am Feb 4, 2020 | ||||
| Phasing out User Agents What will they look like and how will they affect us |
15 | dstiles | 12:19 pm Jan 23, 2020 | ||||
| referer fbapp://nnnnnnnnnnnnn/unknown |
8 | tangor | 1:37 am Jan 19, 2020 | ||||
| Server Farms - July 2018 Continuing discussion of hosting company IP ranges[5] ( 1 2 3 4 5 ) |
139 | keyplyr | 5:54 pm Jan 17, 2020 | ||||
| Info.com crawler |
3 | ClosedForLunch | 8:55 pm Jan 10, 2020 | ||||
| UnChaos does not follow robots.txt |
2 | notriddle | 6:14 pm Jan 9, 2020 | ||||
| fake googlebot? from a comcast IP address |
3 | SumGuy | 1:26 am Jan 9, 2020 | ||||
| RDDocuments |
2 | Pfui | 6:17 am Jan 8, 2020 | ||||
| BuiltWith |
6 | lucy24 | 2:24 am Dec 29, 2019 | ||||
| Status 101 |
2 | tangor | 8:40 pm Dec 28, 2019 | ||||
| woorank |
2 | lucy24 | 2:29 am Dec 10, 2019 |