Besides the various Google Adsense bots, so far I've discovered and now allow:

ConveraCrawler, ERACrawler, VSWCrawler, A6-Indexer, GrapeshotCrawler

Others?

I am not particularly pro bot, and there is no real reason for any bots other than search engine bots (or tools that I run or subscribe to) to be on my sites, so my answer as to important bots is - "none".

Bots are a bother... and none of them give back, even the mighty G, so my robots.txt is specific, my .htaccess nails it down and after 20 years on the web that seems to be working just fine.

I am curious as to I may be blocking some bots that contribute to advertisers' interest. Where did that come from? Any one know of BOTS that help advertisers?

@keyplyr -Are those bots you have mentioned part of specific ad-serving networks? Each network should be able to tell you if they have some UA to allow. All of them are blocked on my sites, though A6 is the only one I've seen trying. Did I miss some news?

OK, here's why I came to ask...

I've run a relatively high traffic set of related sites for over 15 years. I block thousands of ranges, dozens of UAs. Of course there's always some covert scraper that occasionally gets through the filters.

I have used Adsense for 10 years, each year with declining returns to where I'm barely seeing a monthly check now. My fault, I was asleep at the wheel.

In the last few days I've decided to be more liberal and allow many advertisers and categories I've previously blocked, allowing both text and display ads, pulling out all the stops.

By doing this I had expected to see a surge of new ads. However, except for a few, I am not seeing these ads appear. I'm still seeing the 20 to 30 ads I've always seen. I'm thinking that these advertisers are not producing ads for my sites, especially image & display ads.

I was hoping to see better results if I started allowing some of the "ad marketing" bots to crawl my content and make it available for advertisers to target.

My fault, I was asleep at the wheel.

I saw the same thing on some old sites where AdSense had been in place many years. I had removed ads there about 5 years ago and then put back the same original code. Nothing there, so I went in to see why. I noticed a few of my old units were shown as inactive because of an unchecked checkbox that wasn't part of the setup when those units were created. My fault for not checking first.

Not saying that is what happened to yours, but I updated a few ads and it started picking up again, mostly where I changed out old html for more responsive pages and fresh css.

Thanks not2easy, but not really what I'm saying.

Yes my Adsense code is old, but the web pages (several hundred) are always updated and fresh. Almost all pages contain content rich, fairly lengthy articles.

Up until now, I had always used plain text ads exclusively on all pages. They did well for years. Then a few years ago slowly started to decline in revenue. So a couple days ago I turned on Enhanced Text Ads and all the types of Display Ads. I also unblocked all advertisers (over 1500) and enabled all categories.

My point is that even after doing all this, I still see mostly just text ads, and the very same text ads I was seeing prior to the *upgrade* except since I enabled Enhanced Text Ads, I now see the ugly arrows. I *do* see a few display ads here and there but I would of thought I'd see a huge difference. I don't.

So I was considering alternative ways of getting advertisers interested in my sites, hence the interest in 3rd party ad bots.

What you see isn't what other people see. In 2014 for the first time, Interest Based Ads are *far* outstripping contextual ads on my sites, in impressions, clicks and earnings. Contextual won't ever go away, but IBAs are definitely the priority. (And it's not just Google - all ad networks are doing them, and advertisers love them because they convert much better. Maybe not on every site, but overall)

So if I were in your situation, I guess what I would do (rather than worry about bots or AdSense knobs and dials) is figure out what I could do to attract a lot more high quality traffic.

(And no, I'm not just talking about organic search)

@keyplyr

First, you've made a lot of changes so narrowing down the effect of any one of those changes is gonna be difficult.

Turning on Text and Display Ads doesn't mean you're going to get an equal proportion of each. It's still an auction. Normally, for me, I see mostly text ads with that option turned on. The ratio fluctuates during certain times of the day and from day to day but text is by far the dominant result.

Next, what you see is not necessarily what your visitors see because interest-based ads will follow you around. I look at my sites on other people's computers whenever possible to get a better idea of the diversity of ads.

I'm with netmeg on the "important bot" issue. Haven't seen many outside the SERPS and companies I work with.

I'd be paying close attention to what Enhanced text ads are or are not doing for you. Since enhanced text ads were introduced, normal text ads also lost their underlined hyperlink feature and if you didn't find a way to compensate for that, that might be a better place to start (working with color, font and position). I guess I'm saying that the ads you're seeing on your site are not what everybody is seeing and where relevance is a huge factor in my book, it's hard to gauge. I'd look at refining ad layout, use the Experiments feature in Adsense to see if you can get more production from the ad style (whether with enhanced text ads turned on or not).

If you want more display ads, try setting up some slots for display only and others for Text and Display or Text Only. This allows you to control where each type of ad appears much better and carefully locating each can have interesting effects on the way a view interacts with the page.

Good stuff, thanks all :)