Mediapartners-Google

Forum Moderators: open

Message Too Old, No Replies

Mediapartners-Google

Has Google changed the name to eliminate the version #?

GaryK

5:30 pm on Aug 26, 2007 (gmt 0)

I had a visit from a UA named Mediapartners-Google. In the past I've always seen this bot with a version number at the end. This one didn't have that. And yet it does appear to have come from Google. Plus other well known and valid Google user agents were crawling the same site at the same time as Mediapartners.

None of my sites runs any kind of advertising at all.

So I'm wondering if this is another example of a bot taking advantage of Google's open proxies.

Thanks in advance.

keyplyr

6:24 pm on Aug 27, 2007 (gmt 0)

I no longer see a version number either. But I wasn't aware Google has *open* proxies?

Are you saying, non-Google entities can use IP addresses assigned to Google? If this is true, I'll need to change a lot of filters.

GaryK

7:00 pm on Aug 27, 2007 (gmt 0)

That's my understanding from reading Bill's blog. Perhaps I misunderstood what he meant.

volatilegx

12:33 am on Aug 28, 2007 (gmt 0)

Google has some open WAP "proxies" but when used they modify the user agent so as to be recognizable.

GaryK

1:35 am on Aug 28, 2007 (gmt 0)

I have seen user agents modified to indicate they're using Google's WAP proxy. ;)

Based on what keyplyr mentioned it looks like this is a valid bot from Google. I guess I need to update my database and publish new files. :)

Has anyone ever seen three different bots from Google on one website for two days? In all my years doing this it's the first time it's happened to me!

keyplyr

7:22 am on Aug 28, 2007 (gmt 0)

Has anyone ever seen three different bots from Google on one website for two days?

Yeah Gary, each and every day I see:

Googlebot
Googlebot-Image
Mediapartnes-Google

...in addition to the WAP crawls. Sometimes they don't request much, but they're always there it seems.

I get the attention because I have Adsence on 300+ static HTML pages and edit several pages each day, each time updating my sitemap.xml.

GaryK

3:24 am on Aug 30, 2007 (gmt 0)

Thanks. :)

We seem to be in a similar situation with the sitemaps. I update mine hourly because there's significant content (text and photos) being added regularly.

I'm still unsure why a site with no ads would get crawled so often by Mediapartners, but I've already taken care of that!

keyplyr

9:59 am on Aug 30, 2007 (gmt 0)

As of the last year or so, I've suspected that Mediapartnes-Google is being used for more than just Adsence/Adwords.

Just as Googlebot will hit a page seconds after a toolbar user visits, I've seen Mediapartnes-Google do similar behavior on sites that I have no ads. It may be following the user from a site using ads and profiling my site as relevant in some fashion. That has to be a good thing.

That's why I haven't banned it from those sites, but it's only a hunch.

wilderness

3:13 pm on Aug 30, 2007 (gmt 0)

At one time I had some crawling from G-Media.
May have it documented some place.

Never been a part of their ad program.

Have "begins" with Media denied. Not sure if that's google or another bot.

I did have some problems last year with the Google image bot (even thoug it had been in my robots.text for ever) and after contacting google the crawling ceased.
I still kept the deny intact however.

The MSN bots are leaps and bounds ahead of google in stray procedures!

Don

incrediBILL

3:26 pm on Aug 30, 2007 (gmt 0)

Gary,

The version # has been dropped so that's really Google, assuming the IP can be verified using round trip DNS checking and contains ".googlebot.com" in the host name.

GaryK

5:38 am on Sep 1, 2007 (gmt 0)

Thanks, Bill.

Round trip DNS checking contains ".googlebot.com" in the host name.

I thought I saw something you wrote once that it's possible for other bots to appear to be coming from Google. I've been known to have wild hallucinations so maybe I imagined it. Oh well. :)

incrediBILL

6:33 am on Sep 1, 2007 (gmt 0)

Things can use some of Google's proxy servers, such as the language translator or web accelerator, but those don't have googlebot.com in their host name.