Forum Moderators: Robert Charlton & goodroi
29. Your server has limited bandwidth resources and you would like Google only to crawl at night. What should you do?
* a) Send an email to info@google.com
* b) Add "Allow: 11pm-7am" to your robots.txt file
* c) Have your server respond with HTTP result code 503 to all users when it's bogged down
* d) Dynamically change your robots.txt to disallow crawling during the day
Has anyone used the option d?
<crawl_rate from =08:00UTC to=17:00UTC>medium</crawl_rate> Google Webmaster Tools Patent on Crawl Rates
[seobythesea.com...]
Has anyone used the option d?
I do believe that would be the Kiss of Death. ;)
To me, none of the answers appear correct. If I had to choose, it would be b) based on the information available.
* c) Have your server respond with HTTP result code 503 to all users when it's bogged down.
Isn't that the default behavior? I think it's a trick question. :)
This quiz provides guidelines and is not an exhaustive list of issues/causes webmasters may encounter. Discussion of specific cases is encouraged on the Webmaster Help Forum.Keep in mind your username may be posted on the Google Webmaster Central Blog if you are a top scorer.
10.5.4 503 Service UnavailableThe server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.
RFC2616: Hypertext Transfer Protocol -- HTTP/1.1 [w3.org] (section 10.5.4)
See that "Retry-After" header they mention? There's the direct answer to the question posed in the subject line of this thread. However, I can't comment on whether this approach is advisable, having never tried it.
Jim