Gobyus Search Engine (gobyus.com) in Slovakia. But even though they label themselves a SE, their homepage looks more like an amateur link directory. Why their bot is crawling from an ISP is puzzling unless this is a fledgling project.

So just for clarification for those who cut'n paste...
There are 2 UAs:
UA: gobyus
UA: Gobyus

And none that say:
UA: Gobyus (or gobyus)

Also, the ranges looks like:
Host: SK-IMAFEX
188.123.100.0 - 188.123.103.255
188.123.100.0/22
Parent: IMAFEX
188.123.96.0 - 188.123.111.255
188.123.96.0/20

then requested "HEAD /robots.txt" and got another 403

Goodness. How did they manage to get 403d on robots.txt? I think it's theoretically possible on my site for robots.txt requests to get 418, because of the way mod_security works--not that anyone claiming to be any of mod_security's target UAs ever does ask for robotst.txt--but a 403 would take some doing.

:: detour to look up ::

Huh. There must have been a time back around 2011 when I didn't have the <Files> "Allow from all" envelope for robots.txt. Anything more recent can only be a technical booboo on my part. (My test site racks up a lot of 500s, but after all, that's what it's there for!)

Years ago when I first moved from server-side work to user-side (building webpages) I intuitively blocked all HEAD requests. I thought all HEAD requests were snooping to find vulnerabilities... probably because I was doing it :)

Then through experience dealing with numerous bots and utilities, I learned that I needed to allow HEAD requests to accomplish many benneficial tasks and if I had a comprehensive defense stragedy in place, it was perfectly safe to give access to this information.

Now of course there are a multitude of tools and features to view server set-up info so the problem is moot.

I don't control all the sites that I provide reports for. I share unusual findings. I don't have server access in some cases and in this case I can only guess. I won't see logs from that site again for 2 weeks. I try to stick to what's requested and not expect explanations. This is basic level sorting, not advice. They may well be blocking all HEAD requests.