Crawl Rate Settings?

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Crawl Rate Settings?

nrep

9:45 am on Apr 27, 2012 (gmt 0)

Perhaps it's something that I've not noticed before, but in WMT the crawl rate settings now say:

Limit Google's maximum crawl rate

I'm fairly sure that this setting didn't use to be a "limit". Am I correct in thinking that I'll get faster crawling by picking the 1st option?

I've currently got it set to a crawl rate of 10/sec, as the server can easily handle this rate.

sunnyujjawal

9:57 am on Apr 27, 2012 (gmt 0)

Googlebot employs sophisticated algorithms that determine how much to crawl each site it visits. For a vast majority of sites, it's probably best to choose the "Let Google determine my crawl rate" option, which is the default. However, if you're an advanced user or if you're facing bandwidth issues with your server, you can customize your crawl rate to the speed most optimal for your web server(s). The custom crawl rate option allows you to provide Googlebot insight to the maximum number of requests per second and the number of seconds between requests that you feel are best for your environment.