hammered by some kind of bot linked from Google News - Crawler, Spider, and User Agent ID forum at WebmasterWorld - WebmasterWorld

Forum Moderators: open

Message Too Old, No Replies

hammered by some kind of bot linked from Google News

Shaun Calhoun TR

8:32 pm on Mar 13, 2012 (gmt 0)

10+ Year Member

I run the website for a technology news magazine. One of our stories today made it to the front page of news.google.com and for about an hour we got slammed by some kind of bot hitting the URL of that story from many different IP addresses. Essentially a DDoS, but I'm not sure if the DDoS was intentionally malicious or not.

The hits all come from different ip addresses and the few I've check have all been regular residential addresses in the US (from Comcast, for example). They all have the same user agent and have no referrer url. They're all just hitting the page, they're not hitting any of the CSS, images, or JavaScript that would come with a normal hit.

The user agent is:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10 (.NET CLR 3.5.30729)

I've searched google for that user agent and found no usable info.

After about an hour we dropped off the Google News front page and the DDoS subsided.

Something similar happened to our site a few years ago. It turned out it was AVG antivirus hitting all the URLs on the google front page to see if they contained malware. All the traffic from everyone that had AVG installed was enough to bring the site down. But AVG doesn't do this anymore.

Anyone have any idea?

keyplyr

1:05 am on Mar 14, 2012 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Top Contributors Of The Month

Anyone have any ideas?

Same UA, different IPs sounds like a botnet.... infected machines.

wilderness

2:12 am on Mar 14, 2012 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Top Contributors Of The Month

It's been a long while since there's been a newbie in this forum.

As is the tradition, welcome to Webmaster World.

Frequently when these types of attacks occur, you find a band-aid to stop the bleeding.
An eye-popper is "ends with 30729", course you'd need to remove that after the attack ceased, as it's a fairly common UA ending (i. e.,NET updates).
The "ends with 30729" may not even have slowed a botnet down as it could readily change UA's after eating a few 403's, however in desperation, almost anything is worth a try.

Additionally the use of two sets of parentheses within the the UA is a weakness as well, although it's fairly common today.

There are some long threads on the previous-AVG weakness you mentioned.
In fact, on a site that had been down for 2.5 years I had a recent Amazon request for that AVG redirect.

iamzippy

10:12 am on Mar 14, 2012 (gmt 0)

10+ Year Member

I've only seen this precise UA twice, back in Mar/11 coming from a Honolulu Road Runner Holdco host and a Mexican mail server. Both visits linked to Feedburner.

Three scraping hosts from Seoul Korea Telecom (1) and Nournet in Saudi (2) drove by on Jun/18/11 and Jun/19/11 using a variant (3.0.10 without the .NET CLR update). Around 50 rapid-fire hits apiece.

From Mar/11 to Dec/11 I found 450 hits from FF 3.0.x of which ~70% were bounced for bad intent. Around 25% were linked to Spinn3r (news aggregator), TweetmemeBot, Birubot/1.0 and Butterfly/1.0.

I haven't seen any FF 3.0.x hits so far this year. I've been denying FF versions prior to 4.0 for months, but I still see the dents their knuckles leave in the door.

Legacy Firefox is nowadays just as likely as legacy MSIE to be twisting your melon.

dstiles

9:07 pm on Mar 14, 2012 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Top Contributors Of The Month

Agreed. FF 3.0 is obsolete. I permit 3.6 in some circumstances (some linux machines still not upgraded - it's a distro thing!). I also log warnings on any FF version less than 10 - there are still quite a few about. :(

Personally I think MS's instrusion of .NET onto FF was a bad move. It only lasted a short while but the legacy is still there. Anyone with .NET enabled in FF should examine why it's there and unless it's really required, get rid of it completely.

lucy24

10:10 pm on Mar 14, 2012 (gmt 0)

WebmasterWorld Senior Member

10+ Year Member

Top Contributors Of The Month

I permit 3.6 in some circumstances (some linux machines still not upgraded - it's a distro thing!).

The one thing Camino never upgrades is the piece of the UA string that says "(like Firefox 3.6.nn)". It may have been useful once upon a time but can lead to unwanted consequences now.

I have no idea why any Mac user would stick with MSIE-- last release, 5.2 in 2001 give or take a year-- but they've got an exemption too.

fips

9:54 am on Mar 18, 2012 (gmt 0)

10+ Year Member

@Shaun Calhoun TR

I have exactly the same problem - I posted a message about it here:
[serverfault.com...]

I use akamai in front of my website and I consider to block traffic by UA. Akamai should support that.

But, it would be great to better understand the source of the traffic. I don't think it's a botnet or a DDOS (it just requests valid URLs and is always the same UA).

I assume AVG antivirus fixed the problem, so it's unlikely that it caused the issue we're seeing in the last couple months.

I looked a bit closer at the IPs that use this UA. The only other UA I get for those IPs indicate a mobile device (android and iphone). So currently my best guess is some news aggregator on a mobile device (which are a pain anyways). So far I was not able to track down the traffic source to a specific mobile app.