Marvin, right?

I think it's some kind of housekeeping. For the past several weeks I've seen a big spike in visits to one site they've got no reason to crawl at all, at least not more than every month or so, because nothing on it ever changes. Looking more closely, there's a whole lot of this pattern:
/directory/
/directory/index.html
/directory
where the second form hasn't returned content in, oh, four or five years at least, and the third form has never returned content.

In addition they've been requesting pages that were redirected anywhere from two to five years ago-- not just the occasional one or two as a reminder of "Don't think we've forgotten this URL!" but massive lists.

Since all of this leads to more 301 responses, they've also been doing a fair amount of soft-404 testing (where they request some garbage URL like djhrubvghuyvcmhdf.html to ensure that it gets a 404). I've always assumed this was programmatically triggered any time they pass some number, or some percentage, of redirects.

So they're doing something, but I really don't think it's anything nefarious. Maybe they're tweaking the algorithm to give more-- or less!--weight to "technical quality" issues.

They definitely seem to be in a pre-algo update mode, maybe using a deep crawl to get to what they consider a "clean" database, to be used as a reference as they start calibrating what I've been assuming will be refined engagement and UX metrics for the long tail.