Rendering pages with Fetch as Google

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Rendering pages with Fetch as Google

phranque

11:03 pm on May 27, 2014 (gmt 0)

google announced today on the webmaster central blog that the Fetch as Google feature of Webmaster Tools will now show how Googlebot would render the page.

http://googlewebmastercentral.blogspot.com/2014/05/rendering-pages-with-fetch-as-google.html [googlewebmastercentral.blogspot.com]:

In order to render the page, Googlebot will try to find all the external files involved, and fetch them as well. Those files frequently include images, CSS and JavaScript files, as well as other files that might be indirectly embedded through the CSS or JavaScript. These are then used to render a preview image that shows Googlebot's view of the page.

since this feature respects robots.txt, you will see the effects of any images, javascript and css that are excluded from crawling, such as broken navigation.

goodroi

3:39 pm on May 28, 2014 (gmt 0)

About time Google made it easier for me to test my spammy cloaked pages :)

keyplyr

8:28 pm on May 28, 2014 (gmt 0)

When it comes to Google's so called *rendering* tools, my dilemma has always been them not including the images I disallow in robots.txt. But if I allow these image directories, Google will include my image files in their Image-Search, thus inviting every hobby page and summer-vacation forum on the www to hot-link.

Browsers (whether desk-top or mobile) have no issue, only Google's rendering tools... so what's the point?

EditorialGuy

10:14 pm on May 28, 2014 (gmt 0)

What's the point? To let you see what Googlebot "sees." Some people will find the tool useful or reassuring; others will ignore it.

goodroi

10:39 pm on May 28, 2014 (gmt 0)

A free tool is better than no tool. I can see this being helpful for educating clients that don't understand best practices or how crawling works.

As much as I joke about Google, I don't forget that Google needs to address a wide range of webmasters from ignorant and/or stupid to web geniuses. This tool might not be for everyone but I do think it can be helpful for many.

motorhaven

10:41 pm on May 28, 2014 (gmt 0)

You can allow Google a fetch of images yet still keep it out of the index. Allow it in robots.txt, then send this header with the image:

Header set X-Robots-Tag "noindex"

keyplyr

12:25 am on May 29, 2014 (gmt 0)

You can allow Google a fetch of images yet still keep it out of the index. Allow it in robots.txt, then send this header with the image:

Header set X-Robots-Tag "noindex"

Thanks for the suggestion, but no... I've used the X-Robots header tag for years and still my images were scraped by both Bing and Google because at one point I didn't have the image directories disallowed in robots.txt. I have spent endless hours writing emails to them both, finally getting my images (10s of thousands) out of their image-search.

I'm not about to experiment again with this just so Google's little tool looks better.

derfmann

9:15 pm on May 30, 2014 (gmt 0)

big news for SEOs ... its now really easy to show customers how bad the website renders for google crawler, you can even check mobile device renderings.

@keyplyr why not prevent hotlinking with referer-check?

keyplyr

12:57 am on May 31, 2014 (gmt 0)

@derfmann - I do that and more.

keyplyr

10:31 am on May 31, 2014 (gmt 0)

Update on my above statement concerning image file indexing: I've just been in a discussion at Google Groups concerning X-Robots tags. I've been assured by a Google employee they now fully support this tag (not so years ago.)

phranque

11:10 am on May 31, 2014 (gmt 0)

thanks for following up on that, keyplyr.