Forum Moderators: martinibuster
if they have nothing that they don't want indexedIt's important to remember that a robots.txt Disallow--which prevents compliant robots such as search engines from crawling a page--has nothing to do with Noindex. Pages that have never been crawled are still theoretically in the index, because the search engine has seen links to the pages. The only way for a search engine to see a "noindex" instruction (whether in a header or an on-page meta) is to let it crawl.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow:
Allow: / User-agent: *
Disallow: