Googlebot-Video/1.0"

Forum Moderators: open

Message Too Old, No Replies

Googlebot-Video/1.0"

audio & video files

keyplyr

10:04 am on May 10, 2015 (gmt 0)

66.249.73.223 - - [09/May/2015:02:55:42 -0700] "GET /file_name.mp3 HTTP/1.1" 304 200 "-" "Googlebot-Video/1.0"

First time I've seen this bot. Valid Google crawl range. Grabbed a couple mp3 audio & mp4 video files, but no request for robots.txt. The normal Googlebot had already read robots.txt several times earlier in the day, so I guess it's within reason.

crawl-66-249-73-223.googlebot.com
66.249.64.0/19
66.249.64.0 - 66.249.95.255

Pfui

4:42 pm on May 11, 2015 (gmt 0)

Do you allow G access to those mp files? Or did it violate robots.txt?

not2easy

6:43 pm on May 11, 2015 (gmt 0)

It is listed as one of their UAs: [support.google.com...]
I think if it is going to be getting large files I would not want it visiting. :(

keyplyr

9:49 pm on May 11, 2015 (gmt 0)

@Pfui - I allow Googlebot access. If I didn't, it would not get 'em :) IMO SEs need access to all files a browsers uses nowadays, at least for the purpose of my sites.

@not2easy - I'm not on a bandwidth budget, but unlike Bing, Googlebot doesn't DL the same file 10x each and every day.

lucy24

11:25 pm on May 11, 2015 (gmt 0)

Yup, I never saw it until some time last year when I added a few .wav files. But I looked up and found they've been around forever; it's just about file format.

:: detour to doublecheck ::

Oops, my bad, they're mp3. But I'm pretty sure it's the same UA for both.

All the different Googlebots seem to share robots.txt information. The only place I've seen where UA really makes a difference is in the If-Modified-Since header, where the time is based on the last visit by that same UA.