Forum Moderators: open

Why does ClaudeBot / Anthropic do this?

Repeatedly ask for robots and sitemap

         

SumGuy

11:18 pm on Oct 12, 2025 (gmt 0)

5+ Year Member Top Contributors Of The Month



Besides pinging my web-server's IP addresses a few dozen times per day (something no other bot or scraper does) why do I see this in my logs?

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

5:04:06 216.73.216.152 GET /robots.txt
5:04:06 216.73.216.152 GET /sitemap.xml
5:57:45 216.73.216.152 GET /sitemap.xml
6:21:02 216.73.216.152 GET /robots.txt
6:21:02 216.73.216.152 GET /sitemap.xml
8:25:43 216.73.216.152 GET /robots.txt
8:25:43 216.73.216.152 GET /sitemap.xml
11:15:33 216.73.216.152 GET /robots.txt
11:15:33 216.73.216.152 GET /sitemap.xml
11:25:19 216.73.216.152 GET /sitemap.xml
12:26:11 216.73.216.152 GET /robots.txt
12:26:11 216.73.216.152 GET /sitemap.xml
12:42:14 216.73.216.152 GET /sitemap.xml
13:17:35 216.73.216.152 GET /sitemap.xml

Over time, Claude has retreived all the files on my site, some only a few times, some a few dozen times. Today it's only focused on robots and sitemap. Historically 51% of all of Claude's requests is for those 2 files.

(it's funny, or strange, that googlebot never asks for sitemap).

And btw, I only see claudebot from 216.73.216.0/24. If you've seen it from other (IPv4) IP's please let me know.

lucy24

6:10 am on Oct 13, 2025 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If you've seen it from other (IPv4) IP's
In the first part of the year--I don't see any later than May--there were a great many from The Usual Suspects, mainly 3, 18, and 52. (I count 480 /24s, so we'll stipulate that you don't need an exact list.) Also a couple from 2604:ed40:1000:1711:f094:37ff:fe2b: on an ipv6 site. They seem to have settled on 216.73.216 in June.