Google publishes guide for crawling large sites

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Google publishes guide for crawling large sites

NickMNS

8:54 pm on Jan 29, 2021 (gmt 0)

[developers.google.com...]
This a new in depth guide to everything you need to know about how Googlebot crawls large websites and how crawl budget is allocated. It is targeted to webmasters of large and frequently updated sites but in my opinion contains valuable information for everyone. This is definitely worth a read

There is also this video regarding crawling released 2 days ago:
[youtube.com...]

JorgeV

10:24 pm on Jan 30, 2021 (gmt 0)

Hello, and thank you for sharing,

However, all this is just a compilation of common sense information. But I guess it's always good to have it repeated.

robzilla

11:04 pm on Jan 30, 2021 (gmt 0)

Interesting read, thanks. It's good to be reminded that Googlebot does not just crawl the Web indiscriminately. Here, too, page quality and performance are important.