@keyplyr
Didn't you read the whole post on purpose?
"Some more reloads and ads are back again" - if it would be finished here, then yes it would be crawler.
But it says then "Then again no ads. Then half ads."
Read carefully please.

[edited by: sdksjdksjd at 3:27 am (utc) on Jan 16, 2018]

You could be seeing ads from 3rd party networks.

This doesn't explain why they were shown like this:
No ads - Ads - No No No No Ads - Ads (same) - No No ads - Half Ads (different)

@sdksjdksjd - what has been reported all through this discussion is...

Mediapartners-Google (the bot that validates your pages to allow Adsense ads) is re-crawling pages that publish Adsense. The official reason for this is unknown, but seemingly to evaluate each page for the upcoming Google Search Console Ad Experience Report.

Until each webpage has been re-crawled, Adsense ads have stopped displaying.

If you visit those pages missing the ads, Mediapartners-Google *may* follow and recrawl that page (especially if you are using the Chrome browser.) If this is successful, the ads will display again within a few hours, possibly the next day. Best practice is to purge cache, purge cookies, sign-out of your Google or Adsense account, close and relaunch your browser to see the updated page ads.

There is no official announcement, so we are discussing and comparing experience. All indication points to this.

[fix typo]

[edited by: keyplyr at 3:45 am (utc) on Jan 16, 2018]

Does Mediapartners bot disregard "noindex" tag ?

Mediapartners-Google is not indexing your page, so that tag is not relevant.

@keyplyr
Are you talking to yourself? Your post is absolutely irrelevant to my posts, so I don't see any reason you are addressing it to me.
Thanks for ... whatever, but your info is useless.

@dksjdksjd there is no need to be rude. You ask questions and I take my time to help out & offer the answers. If you are not interested, then don't ask.

Mediapartners-Google is not indexing your page, so that tag is not relevant.

@Keyplyr, think about this for a second....


1 second is up.
If there is no index, and given the new policy, they would never be able to serve ads to anyone, since they would never know what was on a page. Each crawl would be the first. So there must be an index somewhere. I agree that it certainly isn't the search index but an index must exist. [see link below]

Mediapartners-Google (the bot that validates your pages to allow Adsense ads) is re-crawling pages that publish Adsense.

I'm not sure what you mean by re-crawl. Mediapartners-Google continuously crawls pages each time (or possibly most times) it is visited. This has always been the case it is not new and documented by Adsense. The difference now is that if it hasn't seen the page previously now AdWords will not bid on the impressions. Here is the response I received from Adsense Support about this:

The launch restricts bidding by Adwords & DBM on uncrawled impressions from Adsense and Adx so that is why you are seeing more impressions from 3rd party ad networks.

@sdk AdSense is not blocking ads, AdWords is not bidding. But the 3rd party networks continue to bid on the inventory. But without AdWords there is very little demand, and no coverage. On occasion a 3rd party appears at the auction, wins by default (at a very low price) and an ad shows up on the page. But extremely low demand results in this sporadic appearance of ads. If you wait an hour and return to the page you will then see ads without interruption because AdWords will now be bidding on the inventory.

Does Mediapartners bot disregard "noindex" tag ?

I'm not sure about the no index, because this relates specifically to search but it appear to obey no-follow and it definitely obey robots.txt

This Adsense help page was linked in the original support email that made the rounds it discusses crawling and indexing.

One point I find very interesting is the final bullet (unrelated to this specific discussion but very relevant to the topic) and worth noting:

Re-crawling sites
At this time, we're unable to control how often our crawlers index the content on your site. Crawling is done automatically by our bots. If you make changes to a page, it may take up to 1 or 2 weeks before the changes are reflected in our index.

How I read this is that once a page is crawled, if the request for the page is made again the bot will not return for 1 or 2 weeks. Which suggest that the max life of a page is 2 weeks and min life is 1 week.

If there is no index, and given the new policy, they would never be able to serve ads to anyone, since they would never know what was on a page. Each crawl would be the first. So there must be an index somewhere. I agree that it certainly isn't the search index but an index must exist.

My comment was to the above question and was of course in relation to the Search Index. Google ad server would obviously have its own index. Not sure why this is even being brought up, but no matter. Same goes for re-crawl. Every time Mediapartners-Google visits a page it is a re-crawl from the last time it was there.

@keyplyr
There is no reason to delete my replies to you, if you're uncomfortable with them.
I can repeat. I din't ask a single question before "noindex" related. I don't know what are you replying at, posting your useless info with mentor attitude addressing to me. I was first registered here back in 2004 and perfectly know how and what to do here. You're not in the position to dictate me. Be happy somebody returns to this useless place, giving it another chance to exist.
You can delete this reply as well, but take a note before.

@NickMNS
I asked about "noindex", because there is "noindex, nofollow, noarchive" on the page I described above.
At moment I see all of the ad-slots are filled. Can we make conclusion, that these tags don't prevent Mediapartners from doing its job?

@sdksjdsjd Does Mediapartners bot disregard "noindex" tag ?

Both the Google and AdSense Mediapartners crawlers honor your robots.txt file. If your robot.txt file prohibits access to certain pages or directories, then they will not be crawled.

[support.google.com...]
all google bots obey noindex nofollow tag but different type statement you can check on here I have google robot no follow pages not mediapartnergoogle. there is no nofollow or such for mediapartners but you can block all directories or some prevent medipartners to index. I will remove googlebot nofollow page from my website all pages see what will happen in 2 weeks if I don't go broke till then. I am also heavily affected by this stupid update.

Interesting.
Then why do I see ads (at least sometime) on the page with "noindex, nofollow, noarchive" ....
But this page also carry "canonical", so probably ads are shown based on that.

Do your pages, where you see blank ad spots, carry "canonical" ?

I am definitely seeing adsense on pages with "noindex" (and not criteo ads etc). The pages do not have canonical tags and are not blocked in robots.txt
Danone, I think "noindex" will not stop adsense robot visiting a page. That is not the same as robots.txt, perhaps that does stop them.

@Rasputin I believe that "noindex" is a special case because it signals to Google and other not to show your pages in search, so I doubt it would have an impact. But "nofollow" tells the crawler not to crawl the page so that should block all crawlers that obey this directives. Moreover, Adsense says explicitly that even if you block Googlebot in the your robots.txt Mediapartner-Google will still crawl (assuming you show AdSense on those pages).

Also, some of the first reports of problems for this issue were sites that reported using "nofollow" tags.

As for the ads not being Criteo how do you make the distinction?

So, can we make conclusion, that ads should be shown no matter, what bots related tags carry the page ?

@nickmns, if I hover over the 'adchoices' in the corner of the ad, the link usually refers to google but occasionally refers to criteo

@rasputin interesting. I just tried it on some media.net ads and it showed Google. After a few refreshes and different ad unit it eventually revert to media.net. So it would appear that Google is buying impressions on Media.net (this was already known, but strange none the less given the current situation).

I'm seeing instances where a URL got visited 3 times in the last 2 weeks and they are still not displaying ads. Same URL had ads on it 100% before the update last month. Has anyone done any kind of test to see if a URL they visited had ads on it, then goes blank. Kinda like these crawls are not staying in some adsense cache for very long?

Or is anyone noticing that it may take 4-5 or more visits from visitors to a URL before the page is crawled and starts to display ads?

1. I used to have ads visible nearly 100% of the time. Even in pages that are visible for logged in user only. So definitely there were no any mediapartners index, because it couldn't crawl those pages. Now I see so many blanks that my eyes bleed.
2. My site's url structure is complicated, so a lot of unique urls for the users. And I feel that this is one of the reasons I'm being hit so hard by this new "rule" and some people don't even see the difference.
3. For the last four days my income has recovered to the everyday-good-level. So at the moment I feel good and bad at the same time. That makes me think that maybe Google has given up for a part of it's share of income just to keep us happy? Because if this was the level I would make every day with near 100% coverage, I would be buying a new Ferrari every year.

@rokrmane
Here this link should help you with the logged on content:
[support.google.com...]
It was included in the AdSense Support message I got. It explains how to set things up so that the crawler can crawl the login protected content.

Now I see where my income is going. Please return.
))

Thanks NickMNS, I've had that for years. Btw. statistics shows that the total time this adsense account has been logged in and crawled my site is over a year now. I mean for the last few years it's been crawling my site's non-visible parts like crazy. It spends like 1/2 of a day there.

@sdksjdksjd Do your pages, where you see blank ad spots, carry "canonical" ?
Yes my pages are canonical but google bot tags noindex nofollow noarchive should not affect behavior of media-partners google bot which adsense use. What I am hoping for is to remove nofollow tags as @NickMNS suggestion, if googlebot visit page it maybe trigger media-partner google bot to index my pages too.
@kelsheikh seen some urls got blank again this is also happen to me, some urls got blank again.

I guess this update is very new and adsense still understanding impact of some publishers and revert back some (although my revenue goes into deep $2.4 dollars only from $50-100 daily) not much but still income is income. I know somebody he is in shock his revenue dropped $40K-$60K per month to $3.5K go figure, I believe this update have impact at least 5% to 10% publisher and it is not only brand safety update but behind they changed algorithm too which unfortunately hard to guess on our end.

I can confirm this, they do this by PURPOSE.

The reason is they want to kill off any traffic which is not from their monopoly.

I see this happen on all long tail database driven sites across the board. The coverage dropped in December by 70%.

Just imagine a site getting Bing/Yahoo visitors, but NO google traffic and you can NOT monetize this site any more with adsense.

Hope this clears it up once and for all.

Don't ever rely on them, never ever. This is my advice, they will kill all of your sites in no time, even 1 day bevore Christmas.

@danone
I have pages with "noindex nofollow noarchive" perfectly showing ads.
And pages without any robot restrictions showing nothing.
I wouldn't expect too much removing "nofollow".

How many of you guys had policy violations warning before?
I'm curious why new policy is for everyone on the web, but not everyone suffers from it.
Probably G. wants to re-crawl some sort of websites? Big news mills are under control of the government authorities anyway, so G. shows ads there even on 2008, obviously not visited for a while, articles. News mills should suffer for sure. They are big. They have huge amount of articles with few visits. Why are they keeping silence? Or tripadvisor for example. No doubt it has a lot of rarely visited pages. Don't they notice? Don't they care?
Why just some suffer? What do we have in common?

Dear Publishers!

Now that you've spent your time and efforts to bring us to the top of the tops in the advertising industry - from the above mentioned top We want to tell you: Thank you everybody, we just don't need you anymore.

Sincerely,
G