Forum Moderators: Robert Charlton & goodroi

Should I unblock AI bots (Google, GPTBot) to gain citations?

Zero-Click is crushing my geo-directory

         

guarriman3

4:03 pm on Feb 27, 2026 (gmt 0)

10+ Year Member Top Contributors Of The Month



Hi everyone,

I run a massive directory website covering tens of thousands of towns, villages, and local businesses for a specific country. Recently, the Zero-Click effect has become brutal. For exact-match location queries, I’m ranking #1 to #3, but my CTR is sitting between 1% and 3%. The Google Local Pack and maps are simply eating all the clicks before users even scroll.

Right now, I have GPTBot, ClaudeBot, and Google-Extended blocked in my robots.txt because I didn't want them scraping my massive structured database for free to train their models.

However, I'm reconsidering this strategy. Since Google Maps is already answering the simple "where is this town" queries, I'm wondering if I should "open the tap" and let the AI bots in. My theory is that by allowing LLMs to crawl my deep structured data (full lists of regions and towns inside the regions), my site might become a cited source (with a link) in ChatGPT, Claude, or SearchGPT when users ask complex, long-tail questions that Google Maps fails to answer.

Has anyone here opened their robots.txt to AI bots specifically to chase citation traffic? Is the referral traffic from ChatGPT/SearchGPT actually moving the needle for you, or am I just giving away my database for free?

Would love to hear your thoughts.

tangor

3:23 am on Mar 5, 2026 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Might want to rethink this. Data, in and of itself, cannot be copyrighted. Database STRUCTURE can be copyrighted.

It is doubtful that any LLM is interested in your structure, only the information contained.

MEANWHILE, I personally don't allow AI bots, not that that has actually stopped them! One reason why is the site has been scrapped, duplicated, replicated, etc. etc. etc. so many times in so many countries (it is a small niche!) that horse left the barn a few decades ago!

Juniya

10:25 am on Mar 5, 2026 (gmt 0)

10+ Year Member Top Contributors Of The Month



At this point, just unblock them unless you have super duper unique content that isn't found anywhere else in this 'specific' country. A lot of AI companies pay 3rd party bots that simply ignore the robots.txt and sell the data back to the AI companies. This isn't surprising I am sure even for you, so my point is, try to unblock them and then compare traffic and data. You need a minimum 6-8 months of data and see if your site was better off without AI bots.

I get a pretty good amount from AI chatbots now but it was a gradual thing not instant.