Bot user-agent: HTTP or HTTPS

Forum Moderators: open

Message Too Old, No Replies

Bot user-agent: HTTP or HTTPS

Things change

dstiles

7:36 am on Aug 3, 2022 (gmt 0)

Just a heads-up. I was wondering why Yeti kept getting rejected. I discovered that they recently changed the URL in the link from HTTP to HTTPS.

If you use a significant portion of a user-agent for allowing bots, check they haven't changed. I now have https?:// as the URL part of the Yeti test.

lucy24

3:10 pm on Aug 3, 2022 (gmt 0)

You're more detailed than I am. With rare exceptions I only use the smallest distinct portion of the UA. Aside from G### and--hahaha--Baidu, there simply aren't that many recognized bots that attract fakers. I do check every year or so for blocked robots, just-in-case. Usually it turns out to be because they've changed a header.

Sometimes, admittedly, this can turn around and bite you. Only yesterday (really) I figured out that the reason, er, “Botname” behaved so unpredictably is that there are actually two robots: “BotnameBot”, which is compliant, and “Botname Crawler”, which isn’t. The first UA string includes the element “https://bot.Botname.com”; the second includes “http://www.Botname.com”. This particular robot--it appears to be a search engine based in Germany--crawls from a sprawling range of IPs.

:: detour for closer study of archived logs ::

Oh. I guess I needn't kick myself so vigorously. The compliant version only showed up about six weeks ago, replacing the non-compliant version. If you're changing your behavior--I've known it to happen--it does help to tweak your name concurrently.