Hi yaashul, you don't mention how you use AdSense. For plain vanilla AdSense account you should allow Google's "Mediapartners-Google" bot.

If it is combined with other services, they should let you know what UAs to expect.

I am using Plain vanilla adsense. But in adsense there are bids from other network. These other network also verify ad click quality and they also crawl pages to check page quality as well.

E.g. one of the article get around 500-700 users a day from US/Canada. Everytime an ad is clicked 2 bot usually come other than adsense to check whether that click is genuine. One come from this ASN AS27257 - WEBAIR-INTERNET and one has admantx.com.

admantx

I wouldn't mind much about a company which doesn't know how to set up a TLS certificate for their own site ...

Is there any list of crawler which we should allow if we are using adsense to get better revenue?

By the way, this is also something I am wondering about, ... there were two attempts of discussions on this subject :

Marketing related robots?
[webmasterworld.com...]

Marketing / Ads related bots
[webmasterworld.com...]

However, I do block every bot, beside Mediapartners, and it does not seem to impact my earnings.

admantx aquired by Integral ad science. Which is an important ad verification company. The thing I really do not know and could know find information anywhere. How come these company know if an ad is clicked. (I have one analytics and adsense has 3rd party script on page). It must be google who provide this information and pass it on to these company. In this case is it relevant to allow these bots or not.

I definitely wouldn't block Admantx unless it is causing you specific issues. Many buyers will not bid on domains unless they get positive scores from verification companies. If you block them all you could find yourself with greatly reduced auction pressure.

>How come these company know if an ad is clicked. (I have one analytics and adsense has 3rd party script on page). It must be google who provide this information and pass it on to these company. In this case is it relevant to allow these bots or not.

The URL where ads are displayed is part of the standard clickstream data. Rightly so too. Would you want to advertise on sites without knowing where your ads appear?

Another curiosity regarding these ad verification bots:

Why do they visit a site with NO advertising of any kind? What prompts the visit? Why are they so persistent?

For that one site I routinely block all... but that does beg the question regarding the source impetus for the visits in the first place.

matbennett,

The problem is if we do not block bad bots google says you have more SIVT traffic, if we block them there is less auction pressure. But there is not a list given by google what to block and what not to block.

Regarding where my ad is appearing should be solved by google. Google should give this information via adword/analytics to advertiser.