They are real people, like active housewives blogs

Or the Norwegian dentist I mentioned above. I looked up the site and found everything perfectly legit, no mysterious Brazilian addresses among the contact information, hosted in Norway and so on.

But infecting real sites is obviously a possibility-- especially if it's someone who only has a website because someone told them you're supposed to, and nobody ever pays any attention to it. In fact easy infectability is one of the things those random robots (nearby thread) are looking for.

look at a common header that is not all it should be

This by itself was worth it, because I found one that made me laugh heartily:

Accept-Language: en,en-US;q=0.8,it;q=0.6,it-IT;q=0.4,bg;q=0.2,de;q=0.2,de-DE;q=0.2,fr;q=0.2,fr-FR;q=0.2,en-GB;q=0.2

or even better

Accept-Language: en-US,en;q=0.8,eo;q=0.6,es;q=0.4,fr;q=0.2,it;q=0.2,ja;q=0.2,ko;q=0.2,th;q=0.2,vi;q=0.2,zh-CN;q=0.2,zh;q=0.2,ru;q=0.2,fi;q=0.2,zh-TW;q=0.2

Honestly, I had to double-check the IP to make sure it wasn't me* in a different browser :) (Amusingly, both of these were asking for the same page-- and it's exactly the page you would expect from this kind of header.)

Wonder if I'd gain anything by blocking

Accept-Language: zh

(no closing anchor) Probably most are from barred IPs already, so why put the server to extra work.


* Kidding. My own list is equally long, but there's almost no overlap in languages.

Fr0mCha0s:
As I said earlier, some may be "genuine" log spam - gullibles who are paying money to fraudsters who tell them this is a good promo method. But a lot of what I'm seeing is one-year registrations, some with gmail registration addresses.

Lucy:
Not the header I was thinking of, though. :)

A lot of languages for this are brief and mostly (but not all) PT in various forms with EN as a secondary one.

Not the header I was thinking of

I was pretty sure it wasn't, or I wouldn't have posted about it ;) In fact I got so distracted I never did get back to looking at other headers. Will do so now.

Anyway, pt-br is what you'd expect from an infected Brazilian machine. If I'd found pt-br from obviously non-Brazilian IPs, that would have been cause for further study, since I have no Portuguese-language material.

:: returning from investigation ::

Hm, now that's interesting. In my own admittedly very small sample pool, the only people who ever send the header

Blah-Blah: blah

(just one word, not a comma-delimited string)

are referer spammers and a handful of robots* that would get blocked anyway. (Notably the Baiduspider-- both the real one and a faker from Chinanet. What on earth does

If-None-Match: "@NULL@"

mean?)

MSIE 6 really doesn't like the Accept-Language header, does it? If it's present at all, it just says * and nothing else.

Matter of fact, I could probably block a final * in the Accept-Language header. Yandex and mail.ru use it non-finally; can't find anyone using it in final position. (What would be the point? Surely no server in the world is going to say "Nope, I haven't got the page in any of those languages, so I'm just not sending anything at all.")


* Can't remember if I know anything about the meanpathbot. But it crawls from OVH.

Pretty much my own results and speculations, Lucy. There are a few gotchas using just that test but I'm sure you'll work it out. :)

Continuation of the same: I eventually had one of those belated "D'oh!" moments as I noticed that

Blah-Blah: blah

isn't the only option. Sometimes the header is missing entirely-- and I don't think this can ever happen with human visitors. Well, maybe if they were using a browser dating from 1997, but most humans don't. So the pattern is really

^(blah)?$

or possibly

^(blah|-)?$

(logs say "-" when a header is absent, "" when it's empty, but I don't know which form the server sees when processing the request).

Currently the only header I absolutely insist on is User-Agent. Clearly I should be looking at more stuff.

I check several headers and reject in part on combinations. It took me a while to work out which combinations, mind. :)

Especially when you're looking at negatives. The minimalist robotic visit goes

IP: 12.34.56.78
Host: example.com
Connection: close

and-that's-all ... but by this time I'm sure everyone operating on this level has been banned by IP. The combination of Referer header with complete absence of {certain others} probably merits sudden death, though.

:: idly wondering if anyone knows why the googlebot sometimes sends "If-Modified-Since" and sometimes not ::

My findings show that several phone/tablet thingies and proxies can often be valid exceptions to most rules, including the ones we suspect. I have had to make several white-holes for such things.

I wonder if the if-modified test is only applied to "updates" and that periodically gbot goes hunting for pages that do not return the modified status condition (eg I think some (old?) web servers are not set up to return such things).