Hey, that's pretty slick!

Bewenched,

On the bottom of the page is this:

The rate at which Googlebot crawls is based on many factors. At this time, crawl rate is not a factor in your site's crawl. If it becomes a factor, the Faster option below will become available.

Very interesting.

On one of my sites I could select "faster" it currently crawls about 800 a day which is enough for me. I won't change it as I don't want the server under any extra strain.

Intressting!

I can only select slow or normal.

Any opinions or actual views on when you can select fast?

Yeah, pretty cool...
For one of my larger sites I have:

Pages: Max/6310 - Avg/1796
Kb: Max/334995 - Avg/79489 (80Mb per day!)
Mil sec: Max/3063 - Avg/1749

and for this site I do have the option to go "Faster"
I wonder how much faster can it go...
I'll switch it to faster and will keep you guys posted...

TBW - looking at the graph I've had big spike end of Aug and right now...

Makes me wonder how much "faster" is, faster would be nice for sure, im tempted, but then again I don't want google hammering my server too much when it's not really needed.

Yea .. I don't think i'd want it much faster... I can only select normal or slow. We have a very large site.

Anybody else seeing this message?:

"We've detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn't use too much of your server's resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below."

I wonder if the choise 'Faster' will have an effect and if it will speed up the proces for me (I'm still waiting to see more pages from my new site -- other than my root page -- in the SERPs).

My only concern about selecting faster (If I had that choice) is if google finds too many new pages too fast that it will throw some sort of spam filter on the site.

My only concern about selecting faster (If I had that choice) is if google finds too many new pages too fast that it will throw some sort of spam filter on the site.

For the time beeing my website only have about 60 pages that can be indexed.

[edited by: OutdoorMan at 10:42 pm (utc) on Oct. 17, 2006]

---

System: The following message was spliced on to this thread from: http://www.webmasterworld.com/google/3124862.htm [webmasterworld.com] by tedster - 3:47 am on Oct. 18, 2006 (EDT -4)

---

Absolutely fantastic guys - congratulations on introducing this crawl rate control into WebmasterCentral.

[googlewebmastercentral.blogspot.com...]

How will this folks out there on WebmasterWorld?

I wonder how quickly will it kick into action and if this mean pages will get cached quicker on all DC's?

Now all i ask is for G to identify the problems that cause a site to be filtered, so that webmasters can see if they can fix or send a reinclusion request legitimately [ but at least this is a big step ]

[edited by: tedster at 7:34 am (utc) on Oct. 18, 2006]

I've just set it to go faster. This option has been there all week for me by the way but now I'm also getting some nice graphs. I have on average 15K pages indexed daily. I never realized G was so active on my site.

In my account, all of the sites where the 'faster crawl' option was offered are database driven, and I'm a little wary of enabling them all for faster crawling until I see how much impact it's going to have on database queries. So I'm just taking one site at a time, starting with the smallest (one of my personal sites)

That said, all the new tools are pretty cool; I appreciate all the improvements they keep making to Webmaster Central. Now, if they'd just make it so that I don't have to scroll through four pages to see all my sites...

Why don't you use caching? (e.g. smarty for php sites)
I always use caching for my sites to offload my server.

This is pretty sweet. We have the faster option available. Oddly it only engages for 3 months then will revert to normal. I wonder if it will remind me to approve the faster speed.

LOL, this "Crawl Rate selector" reminds me of the "TURBO" button on my old computer.

Hmm, hang on, that button didn't actually do anything.

But hey, the placebo effect works for me!

Maybe we could also get some "go-fast" stripes for Webmaster Tools as well!

Very interesting.
Nice services.

So what would an ideal graph look like for a growing site - continuing in an upward trend, a straight line or general fluctuations (and could you presume that sites with graphs which don't match this trend will be suffering in Google SERPS)?

[edited by: Idris at 9:02 am (utc) on Oct. 20, 2006]

What would be good is to set a time window for G to visit. Here in UK maybe we could choose 1am to 6am GMT etc.

That would be a cool feature as G could then run at fast speed for all sites and we wouldn't have to worry about server hammering.

The graphs are great...

I actually identified a problem (my bad coding) with a site of mine that I suspected, but didn't realize how bad it was until I saw that since the beginning of Oct(when I made the code change), the number of pages Googlebot was crawling daily drastically dropped, while at the same time, the time it was taking the bot to download the pages, drastically increased.

Why don't you use caching? (e.g. smarty for php sites)
I always use caching for my sites to offload my server.

We do for some sites, but it's not practical for everything.

Has anyone narrowed down how often the crawl stats page updates? It seemed to update for me last Wed and Thurs, but hasn't updated since.

I changed the speed of crawling to "faster" two weeks ago, but I don't see any difference?!

Anyone noticed any improvements after changing it?

Anyone noticed any improvements after changing it?

I have a small (in pages, but heavy on content) website, probably smaller than the rest of you guys' websites -- 60 pages in all.

Before the choise of changing crawl rate was available, I only had 13 pages indexed. Now I have 58 pages in G's index.

Maybe 58 pages is a small step for mankind, but it's a huge step for me ;)

and you can see the change on number of pages crawled per day?

and you can see the change on number of pages crawled per day?

I can see a change in how often Googlebot crawls my site:

October (31 days - the last 13 days october are with the "faster" option toggled on):
Googlebot 384+31 4.02 MB

(I believe that approx. 50-60% of the 384 hits came after I've changed the crawl rate to "Faster".)

November (7 days so far):
Googlebot 122+3 1.45 MB

That's the statistics I have (AW Stats).

[edited by: OutdoorMan at 2:59 pm (utc) on Nov. 7, 2006]