have attempted to block them through cloudflare which dont seem to be working

This is where we will need details. *Exactly* what have you tried, and what if anything is the result? (I don't personally speak cloudflare, but plenty of others do, so be specific.)

some free services at our site that are limited by hits, before they are not free anymore

Does it go by total requests, or only by successful requests? In some cases, blocking a request doesn't really save your server much work, unless the pages themselves are quite hefty. I'm assuming your robots, like most, are only requesting pages, no supporting files--or at least, no images.

<tangent>
In recent months, I too have been much vexed by robots requesting the same page over and over--generally clusters of requests from all different IPs and UAs, rather than multiple requests from the same source. Short of getting stricter about oldish browsers, and blocking the rare IP that is a server farm/colo rather than human ISP, there's not a ### thing I can do.
</tangent>

Not sure if they are. The all have the same thing in common, Oculus Networks, which is a proxy I think,and the browser specifics. I have gone into cloudlflare and blocked individual ip adresses, (ip.src eq 40.223.131.239 and ip.src eq 172.121.149.193 and ip.src eq 40.223.203.229 and ip.src eq 40.223.213.113 and ip.src eq 40.223.212.140 and ip.src eq 172.252.237.166 and ip.src eq 40.223.127.209 and ip.src eq 40.223.219.108) but they just keep bouncing around the ips. so maybe be question would be how to configure a complete range?

how to configure a complete range?

Well, that would depend on Cloudflare. Do they recognize IPv4 blocks in the form 162.121.128.0/19, or wider ranges in the form 172.252 ? (i.e. the same format you would use in your htaccess / config / IIS equivalent.)

Anything involving proxies or ISP ranges means you also risk blocking legitimate humans, so there's a tradeoff. If your robots are anything like mine, there won't be any obvious header deficits. Do you have the option of blocking by User-Agent? This too is risky, since some humans will be stuck in older browsers.

Just picking out one of those - 40.223.0.0 - 40.223.255.255 = CIDR: 40.223.0.0/16 which was an Eli Lily range but WHOIS shows it as reallocated to IXPO and used as a proxy of Oculus Networks.

172.121.149.0 - 172.121.149.255 = CIDR: 172.121.149.0/24 (also IXPO and used as a proxy of Oculus Networks. )
Same thing with 172.252.237.0 - 172.252.237.255 = CIDR: 172.252.237.0/24

As mentioned, it does not help much blocking a single IP as the proxy can use another IP in the same range. You're stuck blocking every user of the proxy range or playing constant whack-a-mole. I don't use Cloudflare so I'm of no help there, but there are many CF users around who might have actual useful suggestions.

Yes, they recognize they recognize IPv4 blocks in the form 162.121.128.0/19 It can be configured to pose a captcha rather than a block, making me really wonder now if by the capta choice, if it is still actually connecting to the site, making a "hit". Again, my only issue is the bounce rate , and the annoyance of statistics being all trown out of wack.

Thank you. Yes these are the IP ranges of the offender. I will see if I can use Cloudflare to block the complete range.

Thanks. I went into cloudflare and made a security rule to block those CIDR numbers and a few more, I also blocked the associated ASN. Ill know tomorrow . I learned what all that means today! thanx again.

So nice! Im seeing normal hits again, instead of Oculus hitting the same page with different IPs all day. I have seen this before, and when your sites bounce rate goes up, seems google ranking goes down. I hope this fixes the issue. Thanks all, and especially to not2easy!

Peter, I am seeing the same sort of visits to my website (stand-alone, not on Cloudflare). There must have been several hundred requests over a period of a couple of hours, asking for the same page over and over again (html only, none of the images there), from apparently random IP addresses, but including a small number in the 40.223.0.0/16 range that you noticed. Some of the requests got a "404" because I had already blocked that particular IP address range, but many others were fulfilled (200), for what purpose I know not. All of this action has just about doubled my normal bandwidth usage for the today!

All of this action has just about doubled my normal bandwidth usage for the today!

As a rough-and-dirty metric, I can estimate traffic by the overall size of an Apache log file. For my main site it used to run about 1MB daily; lately it’s been 2 or 3, up to 4. And about half the total log size used to be 403-or-similar (418, 429, 400 ... gosh, what a lot of ways to send a bad request); lately it's more like 2/3 or 3/4. Which does mean I'm succeeding in blocking a decent proportion of them, but how I wish they'd go away.

On the other hand, the number of requests is nowhere near so vast that I'd suspect a DDoS attack--and in any case, why me? (Or, I guess, why my host's server?)

@ Martin Potter
I Thought cloudflares fix did it...but it didnt. I added the following code to my .htaccess file and it now blocks those ranges...
#BEGIN Blacklist
<RequireAll>
Require all granted
Require not ip 40.223.0.0/16
Require not ip 172.121.0.0/16
Require not ip 172.252.0.0/16
</RequireAll>
# END Blacklist

I think it is someone from a free plugin that offers so many free visits and then wants to charge you for a premium. What it seems to be is someone using a meta refresh on an Iframe containing a page on my site, and then going through a proxy to hide their location.

I don't know if the format is the same as it was for the older Apache version (your blacklist is in the Apache 2.4 format) but there should be an equivalent available for blocking cross-site XFrame efforts. Something lucy24 would know, something for

Header append X-FRAME-OPTIONS "deny"

to prevent others from using Iframe to show your content.
_{Sorry, I'm on my old laptop today and it would take too long to find newer references.}

Nope, not a header I'm familiar with, but it can't hurt. (And I'm pretty sure directives involving Header haven't changed between 2.2 and 2.4.)

:: brief side trip to Apache docs, followed by longer side trip to developer.mozilla.org ::

Possibly we want "SAMEORIGIN" rather than "DENY", on the off chance that something on the site uses internal framing. Disclaimer: This header wouldn't prevent a visitor from downloading the material; it would only prevent them from viewing it. And I’m not sure that makes a difference when there's a robot behind it all instead of a human browser.

:: business with htaccess to confirm that neither value of this header affects display of analytics opt-out box, which lives on a different site ::

Oculus (AS398781) is similar to Oxylabs (AS396319)... Their business model is to provide bots with services to make them hard to detect.

This is why blocking single CIDRs doesn't work. These guys BRAG about the fact that they spread their clients' activity around, to make bad activity harder to detect & block. They do it automatically. It's the very service they provide. Heck, if you pay more, they can make you look like you're a normal US residential IP address.

Lookup their FAQ, on their website... On their pricing page... A few direct quotes:

Why do I need proxies:
Proxies are a necessity for any bot or tool when scraping or trying to purchase something from a website. Proxies allow you to run more tasks, decrease the odds of being banned, and have a faster connection to most websites.

What is an ISP Proxy?
This is the highest quality Residential proxies leased from Internet Service Providers (ISP) in various countries that you can purchase. This plan works great on eCommerce websites with bot protection such as Shoppy, Walmart, Target, Best Buy, Amazon, and more.

What is a ISP Premium Proxy?
This is the highest quality Residential proxies leased from US Internet Service Providers (ISP). This plan works great on Google related and key social media websites allowing for unblocked access from US Residential IP addresses.

What is an Datacenter Proxy?
Oculus Data Center Proxies is a shared DC plan which comes at a very affordable cost. This product's main purpose is for web scraping.

:: wanders off sobbing brokenly ::

They even offer special services for automating event ticket purchases & sneaker purchases.

I am in 'block all datacenters' mode. I help out on a small site, but man do they want our data.

I have no problem with humans visiting the site, of course. And search engines that actually generate traffic.

For all its faults, our traffic appears to be mainly generated from Google and links from other sites. (Search engine breakdown is 81% Google, 5% Bing, 14% misc.) We somehow still get pretty good ranking on Google, I think because we're kinda niche. We even show up *above* sponsored results on certain queries, mainly, for queries regarding bugs/issues/failures/problems, which the sponsors don't want to be associated with. And that's one of the main things folks come to us for, the stuff the vendors won't fess up to...

In an ideal world, we would find a simple rule to block everything that isn't - truly - a residential IP. Allow Google/Bing/Duck thru. And that's it.

I see no reason to allow any traffic from any datacenter that supports commercial-only customers.

Today, I block by ASN. The bad behavior is at the org level. Blocking by CIDRs is a waste of time. You have to block ALL the CIDRs for orgs that provide services and safe haven to these bots.

My host uses mindmax, and recently exposed ASNs as server vars that I can use in .htaccess. Folks like Oculus & Oxylabs take ONE line each to block now.

Before my host provided this, I used the freebie DBIP datasets, and generated a list of all CIDRS for my list of bad ASNs. I wrote a utility that combines and cleans the list of CIDRS, eliminating overlaps & consecutive ranges, and produces the cleanest, shortest list of CIDRs mathematically possible, ipv4 & ipv6.

It'd be much nicer to have a residential-only whitelist, though... More thorough, less whack-a-mole... That's my grail quest, I guess...

In theory... My understanding is that hosts are REQUIRED to segregate residential-only, and report this to ARIN. In that simple (likely naive) reading of the rules, that data is out there. I believe in the old days this was to make it easier to identify email spammers.

And... In theory... My understanding is that this is what the Spamhaus PBL is - residential-only IPs; the list is generated to make it easier to block suspicious email sources.

I'm going to try to get my hands on the PBL... Do some analysis & see if it might help... It'd be good to know if it takes a bite out of the problem.

Because, even blocking all of Oculus, they still offer that premium service where they're using other ISPs' IPs...

How are you blocking residential proxies?