They've said that if you block their AI bots completely then the site will be dropped from search index as well...

As far as I understand, they don't have separate bot for AI scraping. They are using data gathered by Googlebot, which I haven't blocked of course. I have blocked Google-Extended for more than a year so that shouldn't cause any problems.

I will need to investigate after few days where is this steep drop coming from and maybe it is from "snippet ban" or something like that.

Meanwhile, today I have received over 200000 HTTP requests from ChatGPT. This number has been in 6 figures for a while and continues to rise.

I randomly checked some of the IP addresses and they are within OpenAI published range for the ChatGPT-User user-agent. It is now confirmed that they ignore robots.txt and burden websites with millions of requests as they see fit.

And why all this attention to my adjudicated "Spam Site" anyway?

New month, new thread: [webmasterworld.com...]

[edited by: not2easy at 3:32 pm (utc) on Sep 1, 2025]
[edit reason] New month, new thread [/edit]