Yep. We need to find a strategy where the bots can only crawl each other...

I quite agree.
Most of my htaccess file is now rules for blocking, instead of rules to make my site more user-friendly. Where will it all end?

A MILLION? Holy ### that's a lot. My record is
:: shuffling through raw logs ::
28000, set this past 14 July, making me think longingly of guillotining selected robots.

Something needs to happen here, this can't continue.

It probably won't, at that. Most of mine seem to have got bored and gone away; a couple of days recently I've even had logs topping out at less than 1MB. (For comparison purposes, that decapitation-worthy file was 5.8MB, or six times as big as the site is accustomed to.)

It actually went to 1.5 million over 24h period since I posted that, 17.3 requests blocked per second. It's down to 600K range today which is a little more normal. That's just what is blocked, it's much smaller amount but I still have a lot getting through. The primary issue now is US based, I have blanket policy to challenge everyone else.

I'm popular presumably because it's niche forum content, I also have over 100k unique images and other attachments that are quite popular with the bots.

Whatever the case this is insanity, I'm on a small VPS but even with that my site would be unusable if I wasn't actively blocking all these requests. Who knows what it would be if I wasn't turning so many of them away. I'm getting to the point I'm just going to block entire world except US and CA. That won;t solve problem complety but it will certainly solve some of it.

I don't think it's going to go away. In fact, it's certainly going to get much worse.

What do you think all these new datacenters are going to be doing? Where are they gonna get their answers? What are they going to train on? (Stage whisper: us!)

Seriously. This is just the tip of the iceberg.

I'm just niche forum content also... But I've recently seen bursts of 400K requests in a 2-hour period.

I allow AI crawlers on my site as long as I know there is potential benefit. Really don't have a choice with Google and Bing anyway. My complaint, and I'm sure it's the complaint of most site owners is the random IP's from all over the place that don't identify themselves.

I'm down to 170K blocked over past 24h and just 50K in the past 12h.

I allow AI crawlers on my site as long as I know there is potential benefit

What kind of benefit, and to whom? (Or, if you want to be pedantic, cui bono with nifty double dative.)

As noted elsewhere, I allow AI crawlers because it adds to the probability that the information they give will be wrong or at least irrelevant. Heh, heh.

The benefit to me is my site is almost always referenced for any question about the main topic at least once. I know this is double edged sword but not sure what other choice I have with the way things are going.

Up to 1.3 million blocked requests over past 24 hours. <sigh>

Not getting blocked was a lot of requests for post link, viewtopic.php?p=123456 where 123456 is random number. No referrer, no cookie, multiple IP's, multiple networks, multiple countries and most are residential.

The multiple countries is disconcerting because that means they are getting by Cloudflare challenge. I have blanket policy that issues challenge for anyone outside of US and CA.