Forum Moderators: open
Lots of requests were made with a root as a referrer to inner pages that were never linked from the root, that was a big help. Lots of requests were made to URIs that are long been rewritten and had no external kinks pointing to them either.Another good one is extensionless URLs if your site doesn't use them: either a request for extensionless giving anything as referer, or a request for an otherwise-valid URL giving extensionless as referer. Nice try, robot.
Here is one of the old ones for IIS hosted Sites:Many years ago, I accidentally created some internal links with // mid-URL (had to do with a cluster of now-defunct php-generated pages). From this I learned that Apache simply ignores any duplicate // in the URL. In order to redirect to the single-/ version, I had to use a RewriteCond; double // in a RewriteRule pattern isn’t recognized.
Double // in URI
RewriteCond %{REQUEST_URI} /paintings//+(.*)
RewriteRule ^paintings https://example.com/paintings/%1 [R=301,L]