Robots don't seem to be very good at learning from experience. (In practical terms, it is probably less trouble to try a locked door than to keep stopping to consult the piece of paper that tells you which doors will be locked.) And if you trim your htaccess because some particular robot hasn't been seen in six months, you can bet they will show up next week.

Forget the IPs and just ban 'em by UA. (Whois is quite amazingly uncommunicative about the domain name-- but I'm glad I stopped by, because it reminds me there was someone else I was going to look up :))

I found this :

domain: lycosa.se
created: 2010-01-12
expires: 2012-01-12
nserver: ns2.binero.se
nserver: ns1.binero.se
registrar: Binero AB

and Binero is a hosting company.
Nothing further. Seems like a lost cause unless the host is willing to do something about it.

Yes, that's all it would tell me too. Wonder what AB is? I initially misread it as the expected A/S. Not important, just curious. I guess paying for two years in one initial lump means they have long-range robotic plans ;)

They are blocked by user-agent in .htaccess at this point.

My trap logs them and blocks by IP because often things falling into the trap use an agent similar to a real browser. I logged in this morning, saw hundreds of IPs banned with this agent, added the agent to .htaccess and removed the IPs.

Either way it its banned it still grows the Apache logs with needless fluff.

My trap makes note of hits, including those outside of the trap once an IP or UA is banned. If they still exist after a period of time writes out an firewall rule to a database. The rules have to be manually approved to activate them in the firewall. I'd much rather move long term hitters to the firewall verses .htaccess :)

Russian. Uses proxies to get around IP blocking. You need to block on the UA or ideally on the X_FORWARDED_FOR

current IP 109.173.122.122

but others possible.

Thanks. That's one rule (X_FORWARD_FOR) I hadn't implemented - now I have. :-)

I have the actual binero IP range blocked - 195.74.36.0 - 195.74.39.255

If they are using a LOT of IPs then consider it may be using a botnet. This is suggested anyway because the IP 109.173.122.122 looks as if it's a dynamic DSL IP.

I have the actual binero IP range blocked

As do I

Blocking Binero will do you no good whatsoever in this case and you're about to find out why I roll my own custom log files so I can track PROXY DATA!

@Frank_Rizzo, Yup, I was hit over 1K times by Lycosa so far it's always via a proxy forwarding from Russia.

An abbreviated sample log entry for Lycosa:

46.45.108.189 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; LYCOSA; http://lycosa.se)" VIA=1.1 192.168.7.254 (Mikrotik HttpProxy) FORWARD=109.173.122.122 CONNECT=

Also recently forwarded from 89.178.228.105 on 11/09/2011, 89.178.174.20 on 11/06/2001, some say "FORWARD=unknown", "FORWARD=127.0.0.1", etc. so the only fool proof blocking method appears to be the user agent at this time plus some known forward IPs to monitor in case it changes tactics.

The forwarded IPs involved come from nationalcablenetworks.ru. and broadband.corbina.ru. with some old ones from slingshothost.com which is now a parked domain. Remember, track the proxy data, it's very useful, esp. when some of the so-called anonymous proxies are leaky and share too much data :)

FWIW, If it's not known proxies, it could be a rented botnet, but either way I think adding a dump of these IPs into a block list and see what else hits it might be quite revealing.

Some countries use proxies for all outbound traffic which is why I added the ability just to track and block specific forwarded IPs instead of penalizing an entire country (or host) just because of one idiot. I think moving forward in the future, especially with rapid growth of mobile, including scrapers using unlimited data plans, that you'll find the proxy forward way more important to implement than it is today.

I've been undecided on how to deal with proxies forever it seems. Trouble is, there's a hell of a lot of them with different dynamics.

Anyway, I had Binero blocked for other reasons but also had "lycosa" blocked by UA. Can't remember but obviously they caused havoc at some point.

FWIW, if bots like this go stealth with browser UAs using tracking links [webmasterworld.com] may be the only way to stop them which is why I've been looking at deploying such tactics.

Worse yet, when IPV6 becomes the norm and IPs are being handed out like candy, I think we'll see so many bots coming from so many places, like Lycosa is doing, that blocking IP ranges will possibly become obsolete as the lists will be potentially too large to manage or process in real-time.

I found a site a couple of days ago that lists current (and to an extent past) proxy IPs in a variety of ways, including by country - if mods will allow the link...

freeproxylist.free-webmaster-resources.org

I've begun cross-checking against what I already have and adding some of the proxies into my database. In some cases I've had bad activity on an IP and now find it's a known proxy; in other cases the proxy is new to me (it has either never been used against me or has been so brief and conventionally "browser" that I haven't noticed it). Checking each IP's "owner" also indicates some server farms I was unaware of, some of which I've tagged as static (and even dynamic) broadband, and I'm gradually adding those in as I go.

It's a lengthy process - so far I've "completed" UA and am working through a larger CN list. I may get fed up before completing the lot but if I go by country, picking up on known "bad" countries first, at least I'll have some of them listed.

Of course, new proxies come up daily and the lists on that site are to some extent dynamic rather than server-based, but at least I stand a chance of confirming new proxy accesses as they arise.

I take Bill's point re: valid proxies (I see this particularly from SA but also (eg) yahoo mobile) and already have a monitored "accept" filter. But I would say that one of the worst sources of evil currently is compromised servers, usually individual but sometimes on a farm basis.

It's worth noting that on several occasions I see (eg) a CN proxy IP used by a US server to get past my server farm blocks. The techniques they use almost never succeed.

IPV6 - agreed: that is going to be a nightmare! :(

The forwarded IPs involved come from nationalcablenetworks.ru. and broadband.corbina.ru

... and that's really all the information we need isn't it. Sigh.

FWIW: This just in, from Bulgaria --

85.187.44.227 [projecthoneypot.org...]
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; LYCOSA; http://lycosa.se)

Went for one semi-obscure html page; took botbait.

SERPs say the IP's an anonymous proxy.