I think what you may see is not necessarily what others, on different IPs, with different browser histories, with different language settings, etc... see.

@keyplyr I totally agree. But the goal was not see what everyone else was seeing, the goal was to test the bot's behavior. And I can confirm again the pages I visited a first (that were not crawled by mediapartners) did not show ads at first. After an hour or so I returned to the pages and the ads were showing up. I returned this morning and the pages I visited showed ads and those that I didn't visit did not have ads.

FWIW, I'm in the same position as @keyplyr, I've had 3 good months in a row with a significant improvement over the previous year.

My ad balancer is set to 100/76, and my coverage hovers around the 75% mark. I had a very slight dip to 72% on Dec 18, but it was back to 75% by Dec 22.

I implemented an ads.txt on Oct 28, 2017. I've also had 3 reports about "violations" on pages that have been up for 15 years (that I really don't understand), so I don't think it's that I'm super-high on their list of favorites or anything... but I do think I'm being scrutinized pretty hard.

@csdude55 are your page views concentrated on a few pages that are visited frequently?

Hi everyone. I am now aware of the issue, and have forwarded information on this thread to relevant folks within Google. As soon as they tell me something I will update all of you.

@pubpolicycomms Thanks!

You may also want to the checkout all the discussion going on at Januaray Adsense thread here:https://www.webmasterworld.com/google_adsense/4880340.htm

Thank you @pubpolicycomms for dropping by on this. It's good to know Google is looking out for potential issues.

ads only appear on home pages of all my sites, and some categories and old articles, but on new pages - blank ads

Not sure if it is in anyway related, but views and earnings started diverging for my YouTube channel on the 21st of December: views went up, but earning fell sharply.

I haven't seem any coverage drop on my websites, but January is always peak time for me with the most advertisers. Many advertisers in my sector only advertise in the first quarter, so I'd be very surprised to see a shortage of ads at this time of year.

The YouTube channel is a completely different category (more general interest), so maybe this issue applies to YouTube as well?

@NickMNS - I did find 2 pages on my personal site with extreme ad failure. Normally there should have been 3 tower ads on each page... there were none. These are pages that do not get much traffic, and they may have not received *any* since this issue beagan.

I reloaded these pages with a couple different browsers a few times, then returned 20 minutes later and ads were again displaying. Logs show Mediapartners-Google following me to each page, which is astonishing.

@csdude55 are your page views concentrated on a few pages that are visited frequently?

Umm, it's tough to say how Google would see it. Technically, yes, in the sense that the dominant PHP scripts would be:

example.com/board/index.php

But the actual links are Rewrite's in htaccess, so Google sees:

example.com/local/
example.com/recipes/
example.com/sports/

and:

example.com/local/this-is-the-subject/123456/
example.com/local/another-subject/123457/

and so on.

I THINK that Google would see it more like 300-400 pages a day, but I can't swear to that.

@csdude55 the answer would lie in your logs, go see what URL MediaPartners-Google is requesting.

What to do users type into to the browser to reach the desired pages? Do they type example.com/local/this-is-the-subject/123456/?

Assuming that this is a yes, then the question becomes how often are those page accessed by users? Once a day may be a problem. Whereas, with several hundred requests a day, your probably ok.


@keyplyr, what you describe seems to match the pattern.

The best solution I've come up with is to use a crawler that fully renders the page (executes javascript etc) and have it start crawling your site. Mediapartners bot does follow it around to an extent. You can see it in your logs. Often the next time you hit the page you get Adwords ads.

@pubpolicycomms If you really are a Google rep following this thread, my question to you guys is why not give us the benefit of the doubt or have something like a three strikes policy (or even one strike) where we are white-listed if there hasn't been policy violations. Use the new Ad Experiences rules you are rolling out for Chrome ad blocking. The 'brand protection' policy has essentially forced us to commit ad impression fraud just to get Adsense to do what it is supposed to do. Of course I guess this doesn't matter as much to Google because the initial ads displayed are often back fill from networks other than Adwords, so forcing us to pre-crawl pages just makes your competitors look like they're serving ineffective ads. Is this a clever use of Googles monopolistic position in the ad market or an unintended consequence?

End of day update. I blocked 3rd party networks yesterday (Tuesday), Wednesday was my first full day showing only AdWords ads. I had fewer impressions, for the same number of page views overall. But the blocking pushed up my total Adwords impressions by about 15% and I was able to nearly double my RPM on the day, it is now in the low end of the normal range. But, this is only one day of data, so lets not jump to conclusions.

@NickMNS
Very interesting.

So I asked my AdPushup manager about the blank ads thing. He told me that Google basically wants to make whether or not 'our content is user-friendly'.

I just got confirmation from my partner manager that I am being affected by this new AdWords & DBM crawling rule.

@NickMNS
How is your experiment going?

I never paid attention to AdSense in-depth details, because I considered it as an outsource, so I could spend more time creating. Have to pay attention now.
I'm surprised to discover, that AdWords is my lowest earner according to impressions RPM :/ 3rd party networks perform much better.

I've been reading in a few places that there may be a problem with the AdSense Bot failing to properly crawl pages.

Perhaps you could check the crawler access error tab which you'll find in “Status”

I've been reading in a few places that there may be a problem with the AdSense Bot failing to properly crawl pages.

You are not reading in a few places... You are reading in echo chamber where each person is repeating what the other is saying, and then using the number of reports as means of lending credibility to the report.

I think it should be obvious to any webmaster that when AdSense/AdWords says they wont display ads if the crawler hasn't crawled the page, that if a page is being blocked from the crawler then the ads wont appear. This is not news.

The issue is, was and remains that pages that allow full access to the crawler (Mediapartners-Google) are not showing ads. When I say allow, I mean for which there are logs to show the presence of the crawler on those pages. I will not repeat the details of what the problem is exactly as this has been done many times in this thread, January Adsense thread and in the AdSense Help Forum. It would be nice that sites such @RustyBricks' SearchEngineRoundTable and others would take the time to actually research an issue instead of simply repeating what the last person said. To note, what was reported by MarketingLand was not new, it hand been included in the initial email sent out Adsense Support:

We detected that your AdX /Adsense account is sending a significant number of ad requests from URLs that are not crawled. Below are some of the possible reasons why a URL might not be crawled:
You may be using complex parameters or encoded strings in your URLs that are unique for each visit, instead of sending us the canonical URL, which is easier to crawl
Your URL may represent newly available content which had not been crawled before you sent ad requests. This is transient as your URL will be crawled shortly after your first ad request
You may be sending an incorrect URL to us because you are manually sending an incorrectly formatted URL in your ad request
You may be sending the URL of an iframe with an ad instead of the URL of the content page that hosts the iframe. (This typically applies to larger publishers).
You may have limits on how often they can be crawled (trawler rejects our crawl requests)
To avoid a potential revenue impact from this change, please consider the following best practices for ensuring URLs can be properly crawled:

Let's not assume that everyone has a perfect site and has no problem with the bot. We can't be sure of that as everyone has a different site.

These are reports, not confirmations, as some have suggested. I chose my words carefully, and suggest "may" have a problem.

No one is assuming anything. If a site has a problem with crawler access, then the issue can be fixed. Instructions were provided by AdSense Support since the first day of implementation of this policy to anyone that reached out for it. So crawler access is not the issue. The issue is that there is a subset of sites that due to a specific design/traffic pattern are exposed to the full impact of this algo update (AdSense/Adwords algo not Google Search/Ranking), these sites allow full access to the crawler but still see few ads ads appearing.

The subset of sites with crawler access issues is likely much smaller (negligible) than those with design/traffic pattern issue. Moreover the design/traffic pattern is not a binary situation it applies in varying degrees, so some sites may be impacted severely whereas others may barely notice a difference.

I think it should be obvious to any webmaster that when AdSense/AdWords says they wont display ads if the crawler hasn't crawled the page, that if a page is being blocked from the crawler then the ads wont appear. This is not news.

hmmm, i've seen sites with specific problems on certain pages, while the rest of the site seems fine, so it's not always obvious to everyone. ;)

Back on topic.
I've asked Google for an update on the, if they have one.

I have same behaviour across multiple sites:

1 site with traffic generaly from Croatia
1 site with traffic generally from USA
1 site with traffic from english speaking countries (USA, India, Australia., Canada...)

All sites have implemented responsive units.

Ads are sometimes appearing and sometimes not, so if it is a mediapartners bot crawling issue then ads would not appear at all.
RPM dropped 50% on site with traffic from Croatia and 70% on site with traffic from USA.

Out of three ad units , when only one ad is displayed , it is often on different ad unit.

I think it is advertiser pressure issue (market not filled at optimum level yet).

If it is mediapartner bot not crawling issue -> ads would not display at all
If it is content quality or policy implementation -> ads would not display at all

Behaviour like I've described looks like lack of advertiser pressure or/and technical issues on Google side.

Do you have similar behaviour?

I am confused ...

old sites: ad coverage 82%
new sites: ad coverage 65%

old sites: ad earnings 50%
new sites: ad earnings 50%

old sites: total visitors 30%
new sites: total visitors 70%

old sites: uk/us visitors 20%
new sites: uk/us visitors 90%

logic

@dk385
Yes, this is what I see on my website and also can't understand.
If G. would be short of advertisers, then it wouldn't be so sudden. It would go down the hill slowly.
I saw huge amount of mistakes from 3rd party networks in Javascript console of the browser before NY. Now they are gone. So, looks like tech probs for me. But how G. still can't fix this blank ads situation is out of my understanding.
Probably coincidence of some factors.
The only thing I'm 200% sure about - there is no problems on my side.

I just saw the page was full of ads, but totally without ads after reload. Some more reloads and ads are back again. Then again no ads. Then half ads.
Don't tell me this is about the crawler.

I checked the archive of the website, that is subsidiary of NewYorkPost.
Pages from 2008 are showing Google ads perfectly. There is no chance so outdated news pages are visited and re-crawled accordingly.
Don't tell me this is about the crawler.

Some more reloads and ads are back again...
Don't tell me this is about the crawler.

That pretty much shows evidence it *is* about the crawler :)

Use a new browser, purge cookies, sign out of any logins and visit those pages again.

@sdksjdksjd yes you are right I guess they have some sort of level of advertiser and it affect us not for them. slideshare I looked a presentation 10 years old I am pretty sure no one looked that topic now guess what! ads are showing. and newly created page is also shown for some countries immediately and some country not. check my thread especially my last post . [webmasterworld.com...]