Stop DOS attack from MSN bot

Forum Moderators: mack

Message Too Old, No Replies

Stop DOS attack from MSN bot

newbies

3:52 am on Dec 8, 2012 (gmt 0)

Recently MSN bot keeps crawling my site almost all day long and sends the loading to the roof. It is really a DOS attack. I googled it and found many people have the same problem. How can I stop this from happening? Using robots.txt? I also heard the MSN bot does not follow robots.txt and heaves really bad.

BTW, whenever i see the word bing, I want to vomit.

ZydoSEO

3:46 am on Dec 15, 2012 (gmt 0)

Did you determine it was MSNbot by the User Agent or IP?

dstiles

11:22 pm on Dec 15, 2012 (gmt 0)

Newbies - I suggest for a better over-view you read the Search Engine Spider and User Agent Identification forum - lots of recent stuff there about MSN/bing.

From my own viewpoint I see no more over-browsing than from googlebot.

If it's not a bot IP (check rDNS to see that) AND it's not a bot UA (there is only one I accept but there are two or three valid ones) then you should be rejecting access.

jskrewson

1:49 am on Dec 16, 2012 (gmt 0)

msnbot (and bingbot) will obey the crawl-delay directive from the robots.txt file. This will slow it down and prevent the DOS. It has worked wonders for me.

eeek

9:26 pm on Apr 20, 2013 (gmt 0)

Did you determine it was MSNbot by the User Agent or IP?

I've been seeing the same thing lately and it's most certainly MSNbot and Bingbot from both IP addresses (do they really need so many?) and the UA. The also send a no-cache header which really messes up my squid front end.

amznVibe

4:36 pm on Sep 4, 2013 (gmt 0)

We observe 1000+ parallel connections by msnbot. It's insane and unacceptable.

And yes, it really is msnbot based on ip

RhinoFish

6:15 pm on Sep 4, 2013 (gmt 0)

Free load testing, aren't they nice!