I see it all the time, always have. At least once a week.

... and, under the heading of “D’oh!” ...

I had entirely forgotten that they have been denied* since who-knows-when. So what they're doing is walking past the door over and over again to see if it still says "No Admittance" the way it did an hour ago-- but they're not brazen enough to rattle the doorknob. Not that it would do them any good, but they haven't even tried.

Along with checking robots.txt I double-checked logs to confirm that nobody else has been coming around from the same IP. Some entities do use a different user-agent for robots.txt requests than for their "real" requests.


* I'm trying to get in the habit of distinguishing between "deny" (robots.txt) and "block" (htaccess/config). It's the difference between putting up a sign that says Employees Only, and deadbolting the door.

I look at every 403... and 404.

I look at every 403... and 404.

I look at human 403s (request for errorstyles.css) and any 404, though most of the latter are the Googlebot testing for soft 404s. Or obvious typos. The rest of the time I just can't be bothered, unless I'm getting ready for an installment of At Home With The Robots.

It helps to have a hobby.

Follow-up:

Over the course of several days, they asked for robots.txt more than fifty times. So I decided to let them in and see what happens. Of course they were wild with excitement and ran around requesting pages that haven't been on that site in 2½ years--sometimes longer. Nothing in roboted-out directories, and definitely not a fresh crawl; they had an old shopping list and were going to follow it. The robots.txt requests turn out to be an inherent behavior, continuing at a rate of 10-15 a day. (Hm. Could they be related to bing?)

In spite of the colossal number of redirects they picked up, to date there have been no requests at all on the new site (the one all the redirects point to).

:: detour to raw logs ::

Looks as if they decided to become robots.txt compliant on or about 8 February 2015. That's the last time they made a (403) page request.

An earlier thread [webmasterworld.com] postulated a relationship between DotBot and ezooms, but I haven't seen them around either.

There may be various posers, but DotBot belongs to Moz [moz.com...] and is used to gather data for their marketing tools; good if you publish ads.

DotBot (which I allow) came 'round today and requested robots.txt then ignored disallow directive on one directory/page. I also use the on-page meta robots noindex.

The data we collect through DotBot is surfaced on this site, in Moz tools, and is also available via the Mozscape API.

I used their Site Explorer tool and found about a 3rd of my pages listed. I did not find the disallowed page from today, however it is likely too early to see that result.

After letting them in, I sat back and watched.

Access permitted in robots.txt: 2 June
Reaction at original site: instant
First appearance at new site (target of redirects from old site): 9 June
First request on new site for page that didn't exist on old site: 15 June
Last sighting on new site: 24 June (after requesting material from four pages or directories that didn't exist at the old site-- but not a comprehensive crawl of all new material, though they do seem fond of one new directory*).

They're still coming by regularly at the old site, picking up redirects in a quite bing'esque fashion, probably averaging out to 1 request per page per week. Also bing-like is their continuing appetite for robots.txt, about ¼ of total requests. No requests for pages in roboted-out directories on either site.

Interesting quirk: One group of pages used to have URLs in the form /dir/subdir/FileName.html where /subdir/ only contained one file. Long after the site move, I changed all these to /dir/subdir/ alone. But the DotBot has repeatedly asked for /dir/subdir/ at the old site-- an URL that never existed, though it's obviously deducible from the URL I did use. Happily the redirect target is the same either way so, hey, whatever rocks their boat.


* Edit: I checked more carefully. They asked for the directory index page twice, and each named page within that directory exactly once. So, yeah, at least that bit is a comprehensive crawl. I doubt they're following someone else's link, since it's a brand-new directory.

Those crawl-to-results times are helpful, thanks.

"Interesting quirk..." IMO that only shows the behavior is vertical, and not a linear crawl.

In a way it makes sense - I'm seeing the DotBot from Wowrack:
216.244.64.0 - 216.244.95.255
216.244.64.0/19
checking some of their IPs. I am seeing that they come from where I used to get ezooms visits.
from old records:
208.115.113.80 - 208.115.113.95 dotnetdotcom.org – Ezooms
208.115.96.0/19Wowrack.com208.115.96.0 - 208.115.127.255Ezooms Bot

DotBot from Wowrack

A quick search will show the UA is used (faked) by many IPs at hosting companies (example: wowrack.com)

I only allow DotBot from from Moz :)

Moz, No Moz - no DotBot.

Moz's DotBot respectfully obeys robots.txt