Just a tip for OSX users - the Network Utilities App included with the OS offers you a desktop Whois for ARIN, RIPE and APNIC. It has been so handy when finding out who is where and what to block.

Opera mini is probably generating preview for its start page.

Just be careful with all your blocking that you don't cut your nose off to spite your face. I've found that some of my tools don't work on my sites with the entire AWS range blocked, so I went back to just blocking the IP ranges that were hitting my sites. I also don't want to accidentally block real users (I've done that before).

Hey, Thanks Guys. This is extremely useful and exactly what I needed to know. I really appreciate the help and precise guidance in the matter.

Ran across this range recently for Opera Software ASA for those who were looking it.

141.0.8.0/21

Just be careful with all your blocking that you don't cut your nose off to spite your face. I've found that some of my tools don't work on my sites with the entire AWS range blocked, so I went back to just blocking the IP ranges that were hitting my sites. I also don't want to accidentally block real users (I've done that before).

Yes! Blocking by CIDR can easily have unintended consequences. Make sure you understand what you are doing.

As far as AWS is concerned, I just created an AWS instance for a client and am in the process of trying to get Ubuntu Server going so they can more easily support email shared among desktops, smartphones, tablets and - for all I know - soup cans on a string.

So far it's murky going... But at least they tell me it will be free for a while.

WDR

I think this thread should be made sticky; if only for some of the CIDR information.

WDR

I think you should remove advertise on the site a few days

I got click bombed on two days in march.

Despite all kinds of page flood control and what-not, adsense reported $17k revenue in just one day and an average RPM of $70 so someone ran something to flood clicks in.

There is nothing that I can do to defend against this, and it is the first time I've seen it reflected in the monthly reports -- if not the actual payments.

Actually, there are a bunch of ideas in this post for defending against this; you should read the whole thing. The main thing is, get familiar with your logfiles and stat programs.

Actually, there are a bunch of ideas in this post for defending against this; you should read the whole thing. The main thing is, get familiar with your logfiles and stat programs.

Which is why I'd like to see this thread made "sticky".

WDR

Which is why I'd like to see this thread made "sticky".

And why I agree although what is probably needed is a good summary rather than the complete discussion.

A good summary would have a better chance of staying as a sticky, the specific current IPs mentioned would have to be left out as those change over time and really need to be determined case by case to avoid locking out legitimate traffic. Stats and analysis of traffic is an ongoing part of defending any site and bad players are a moving target.

actually if you think you can defend against this you haven't been bombed by anyone good yet. in my case the traffic came from a network of ips and was not a spike on either urls or in total for the day.
existing filters for excessive or out of the ordinary requests stop dumb attacks but since we don't get to filter how the google servers receive clicks there is no way to detect a pattern that looks like ordinary traffic but clicks ads at a rate 20x more often than a regular user. my ecpm went from 4 bucks to 80. That's only 80 to 160 clicks per 1000 pages and so the traffic bump was lost in the noise.
of course if you have a Wordpress blog or something that doesn't get a ton of traffic spikes from click bombing are easier to block.

the key part is; you don't know if a page request resulted in an ad click and detecting that would be violating policy. If you get 1000 page requests from 10000 ips to random urls that all click an ad only google can see and defend against it.

actually if you think you can defend against this you haven't been bombed by anyone good yet. in my case the traffic came from a network of ips and was not a spike on either urls or in total for the day.

I actually have had one of these going on with one of my sites for a couple years now. Slow drip bot attack. They appear to be infected Windows machines. But I'm pretty sure they're not clicking ads - there's no AdSense spike on that site at all (and it's not my top earner by a long shot)

The way we're defending against it is that we're blocking a couple countries in full, and then once a week or so I harvest a bunch of IP numbers (from all the other countries) out of my Piwik analytics that fit the profile of this bot, and deny them in htaccess. And at the same time, we run a little PERL script my host gave us to remove all the IP blocks from htaccess that haven't attempted to hit the site in 30 days (because after a while, they tend to give up) That way my .htaccess stays more or less manageable.

But again, this one is not related to AdSense.

Thanks for mentioning that technique netmeg (even if not directly related to clickbombing). I'll also just reiterate another downside to blocking AWS in it's entirety. I recently decided to put some Pinterest widgets on my site and found I had to open up 4 or 5 different AWS ranges to get things working properly. A maintenance app for IP blocking would be quite useful and more practical than shutting off whole segments of the Internet to stop a few bad apples. Having said that, I use country blocking as well and that's been one of the most effective pieces of bot protection in general that I've ever implemented.

I am convinced that the March AWS clickbot attacks was a one-month test from Google's new acquisition to determine whether they could identify a certain pattern of clicks and then remove them. It's actually something I would do given the ease with which one can activate an "instance" on AWS and have them from various parts of the globe.

It is also not an unreasonable idea that some very profitable web sites that are now suddenly not so profitable were regular targets of C&C clickfraud bots. Now that G can identify them...

Netmeg's visitors are interesting. What could be the motivation if not clickfraud? It's likely that these are all over the place; Netmeg seems to keep a more careful watch over her sites than most. I wonder if these are C&C bots on infected PCs whose controlling servers have been taken down for one reason or another; they're still following a patttern but not being commanded to hit any specific sites.

My technique is to use analytics to find numerous instances of bounces; 98% to 100% bounces on a regular basis tells me something is up. Then I look more closely using the Piwik toolset.

WDR

I don't think the AWS thing was from Google. Too dicey, if not for publishers than for advertisers.

The first time I had that type of bot attack, it lasted for two months on the nose - I figured at the time it might be some kind of test. Then when it came back, it just kept growing. Pretty sure there's a list somewhere, and just that one site (of mine) is on it.

I don't think the AWS thing was from Google. Too dicey, if not for publishers than for advertisers.

The only way to know if a system is performing properly and identifying (and removing) invalid clicks is to generate invalid clicks and see if they get removed. No issue for advertisers because Google can just create its own set of ads (and Adwords accounts).

The March click issue for most of us was constant take-backs. The clicks looked to be generated overhwhelmingly by AWS visitors.

The only flaw in my argument is that if you make all your clickfraud bots come from one system (AWS) it becomes too easy to identify them by account and deny access to the entire CIDR; which, of course, is what most of us did.

So there must have been some others in the mix. :)

WDR

I've been down the blocking road before. You can get pretty obsessed with it and have over a thousand blocks in your firewall or htaccess. I found out the hard way that it's not really worth the time keeping the list updated unless you're losing a boatload of money, in which case you can hire out the work and save yourself the headache.

If you find yourself bored you can look up many of these bots with a segment of Session Duration < 10 seconds and a resolution of 1024x768. If you want to find them really quickly you can look at Languages and select C as a language. It's almost as if the developers of whatever script they are using wanted the signature to stick out like a sore thumb.

Some scripts are setup to only execute AdSense and not stats, so they won't show up in your JavaScript based stats such as G Analytics and statcounter etc. To get the real data you have to have something that analyzes the server side access log files and compiles a very detailed and flexible report. The problem is that due to the success of Google Analytic, many of the great access log analytic vendors have gone under or got bought out and phased out. So you're left with outdated software or cheap/free software that doesn't provide you with the flexibility you need to detect these buggers. Going through logs all day it not something you want to waste your time on anyway. You could get a system coded to detect bots but it's gonna be pricey, it will need to be monitored and updated and in the end it will never detect at 100%.

So, 99.98 percent of webmasters shouldn't really concern themselves with blocking IPs unless they really really need to.

Unless you run AdSense, in which case, it's a good idea to pay attention.

So, 99.98 percent of webmasters shouldn't really concern themselves with blocking IPs unless they really really need to.

A lot of us really needed to in March when revenue numbers were bouncing all over the place and CTRs were through the roof.