1. Home
  2. Forums Index
  3. WebmasterWorld / Webmaster General 7:08 am May 21, 2026

Forum Moderators: phranque

Message Too Old, No Replies

Robots.txt files

         

Tom1961

8:31 am on Jun 21, 2018 (gmt 0)

5+ Year Member



I notice both my sites have a robots.txt file with this as the only entry:

User-agent: Mediapartners-Google
Disallow:

I don't remember having created this file on either site, and see that Adsense suggests this be removed.

[support.google.com...]


Is the robots.txt file actually hurting my revenue?

keyplyr

8:35 am on Jun 21, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



It is *not* hurting you site nor your Adsense account.

There is no value for Disallow so it is not doing anything.

[fix typo]

[edited by: keyplyr at 8:38 am (utc) on Jun 21, 2018]

engine

8:36 am on Jun 21, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



If that's all it has in the robots.txt file, you don't need it at all.
It shouldn't be hurting your revenue as all it does is specify Google media partners bot is ok to scan your site.
Take it away and it'll do that same.

phranque

8:40 am on Jun 21, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



this tells a Mediapartners-Google user agent it can crawl everything and puts no exclusions on this or any other user agents.

Tom1961

12:04 pm on Jun 21, 2018 (gmt 0)

5+ Year Member



Why does adsense say to remove it from the robots.txt file?

phranque

12:48 pm on Jun 21, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



the support page you referenced in the OP starts with:
If you’ve modified your site’s robots.txt file to disallow the AdSense crawler from indexing your pages

this is not the case with your robots.txt file, which has been stated 3 different ways above.

engine

1:20 pm on Jun 21, 2018 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Why does adsense say to remove it from the robots.txt file?

If that is all your robots.txt file says, just delete the file, you don't need it.

tangor

7:24 pm on Jun 21, 2018 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Delete the content of the file, keep the file itself. This avoids having 404s appear in your log files.

As for all the rest, learn more about what robots.txt actually does then make your own decision. As it is it merely says NOTHING IS HAPPENING AND THANKS FOR ALL THE FISH.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

  1. Home
  2. Forums Index
  3. WebmasterWorld / Webmaster General 7:08 am May 21, 2026