Can a robots.txt or php header tell the browser to noindex but follow?

Forum Moderators: phranque

Message Too Old, No Replies

Can a robots.txt or php header tell the browser to noindex but follow?

ntbgl

1:52 am on May 22, 2009 (gmt 0)

I have all my outbound links on domain1 funneled through a PHP script on domain2 before the user ends up seamlessly on their destination page.

At heart, the PHP script on domain2 is something like:

<?php
$url=$_GET["url"];
header("HTTP/1.1 301 Moved Permanently");
header("Location: ".$url);
?>

At the root of domain2 I have a robots.txt file that looks like:

User-agent: *
Disallow: /

What I want is for domain2 to not show up in the search engines, but to be able to pass "link love" to the pages I link to.

Because I'm using a PHP redirect, which is what I want to keep, I can't use a meta tag to noindex,follow, but can I set my robots.txt file to something like that, or to send a PHP header that gives the same message?

Thanks

g1smd

2:05 am on May 22, 2009 (gmt 0)

URLs that redirect (especially 301 redirect) should not appear in the SERPs of most major search engines (exception: URLs that did once return "200 OK" and content, and then later change to "301" are retained and continue to be listed by Yahoo in SERPs).

Search engines generally like to index only URLs that directly return "200 OK".

Your robots.txt exclusion stops them seeing the redirect because they are disallowed from spidering the URLs that return the redirect code.

ntbgl

2:10 am on May 22, 2009 (gmt 0)

Awesome!

So I don't have to worry about messing coding. Thanks very much.

g1smd

2:18 am on May 22, 2009 (gmt 0)

Do test and look with Live HTTP Headers for Firefox to make sure that the http:// prefix is showing on the target of the redirect.

That is, if the http:// isn't on the front of the $url variable, then you'll need to add it to your code above.