Forum Moderators: open
Do Googlebot and Slurp pay attention to the Expires (or perhaps another http header) so that we can tell it that the page expires in a week so it won't crawl it multiple times per day?
My hope is that by slowing down the crawl on unnecessarily heavily hit pages we'll speed up the crawl on less frequently visited pages.
Thanks for any advice!
It won't solve all of your bot problems and I'm not sure how effective it is but may be worth a shot if bots sucking bandwidth is a real issue for you.