...the SE's are using some of our URL's as the dynamic affiliate URL instead of our static URL

midoriweb - Can you clarify what you mean by "dynamic affiliate URL"?

Also, what's the inbound linking situation? Is anyone linking to your dynamic urls?

Hi Robert,

Dynamic Affiliate URL's are urls that are dynamic and include the tag at the end to track each affiliate (for example: &Click=1234).

An affiliate url would like like:

mysite.com/ProductDetails.asp?ProductCode=product1&click=1234

The actual URL I have on my site that I want spidered is:

mysite.com/Product_Name/productcode1.htm

Problem is... Yahoo, and now a little of google is indexing the affiliate URL instead of the URL located on my website.

So to answer your second question, yes, people are linking to the dynamic URL's. But I don't want those URL's showing in the SE's. I only want my static URL's being listed.

midoriweb - To clarify then... this is not a case of a dynamic site where you've used mod_rewrite, say, to clean up dynamic urls, and the dynamic and static pages are now each seen under two different urls.

This appears to be a situation where the urls of the inbound links with various tracking strings have caused the engines to index the page under multiple urls.

A page's content will be indexed under as many urls as there are different inbound tracking strings. These multiple urls are seen as dupe content by the engines, and the engines sort out dupes in various ways. Generally, on Google at least, the url with the highest PageRank wins.

Usually, a single affiliate link contributes only a small fraction of the PageRank of a page, so affiliate links generally don't show up in the serps. Yahoo seems to have more problems with tracking strings than the other engines do... but generally Yahoo also sorts this out.

A high PageRank inbound affiliate link, though, can be a source of your problem. You're also likely to have problems if you have no other external inbounds to the "correct" url of your page (ie, to what you would call your "static" url).

I'd recommend getting other non-affiliate inbound links to your affected pages, and/or to the site as a whole.

A PS to the above... I've seen Yahoo retain a url with a tracking string instead of the canonical url of a PR7 page. It took them a couple of months to figure it out.

Hi Robert,

Thank you for the reply. We're in the process of trying to get ISAPI_Rewrite to modify our URL's to allow affiliates to point to our static HTML pages and still record their affiliate tracking.

I guess some SE's will determine that if 200 sites point to a certain URL for a product but our site only links to a different URL, the one that people link to is most relevant and will be shown in the results. That makes sense.

I just wish SE's would follow our robots.txt file and NOT index the pages we have listed there. But I'm guessing robots.txt is only for spiders to be told where to go and where not to go. When it comes to indexing items in SE's, if people link to pages you have as dissallowed in your robots.txt file they will still show up in the search results.

midoriweb - If, say, the directory a page is in is disallowed by robots.txt, Google will still index references (ie, links) to the page that it finds elsewhere. If you don't want those urls to be indexed, you need to use the robots "noindex" meta tag in the head section of the page whose url you don't want indexed.

Here's the twist... If you do that, but also use robots.txt to keep the page itself from getting crawled, Googlebot won't crawl the page and therefore won't see the robots "noindex" meta tag, so the links will get indexed.

So the best thing would be to include all directories or files you don't want indexed in your robots.txt file. But... if people are going to link to certain directories or files you don't want indexed, then remove them from your robots.txt file and include the noindex metatag instead?

So... I'm going to remove ProductDetails.asp from my robots.txt file but include the following in my product pages:

If Request.ServerVariables("url") = "/ProductDetails.asp" Then
wl("<meta name=""robots"" content=""noindex,follow"">")
wl("<meta name=""googlebot"" content=""noindex,follow"">")
Else
wl("<meta name=""robots"" content=""index,follow"">")
wl("<meta name=""googlebot"" content=""index,follow"">")
End If

That way... if the ProductDetails.asp page is called robots will be told not to index. But if any other page is called (like my SE friendly URL's, it will index. This is the best way to get around my situation?

So the best thing would be to include all directories or files you don't want indexed in your robots.txt file. But... if people are going to link to certain directories or files you don't want indexed, then remove them from your robots.txt file and include the noindex metatag instead?

Your question is too general for me to answer. I still don't know, eg, whether you have dynamic pages that get dynamic affiliate links, and then you have static versions of these pages for general searchers to find.

If you're set up this way, you definitely wouldn't want to be blocking your static pages... but you might be able to set up all of your dynamic pages to use the meta noindex tag.

On the other hand, if all of your pages are static and you're talking about "dynamic" links to them, I'd think you'd run into SEO problems blocking them at all.

I do not have a clue how you'd manage a situation where affiliates might link to some pages and not others without also blocking search engine access. And you might need a full time person just to manage your robots.txt. ;)

Something I'd pursue in the robots.txt forum.

PS - I can't advise you about the use of server variables at all.

Thanks again Robert.

Our ProductDetails.asp page is dynamic and only linked to by affiliates. Our site uses ProductDetails.asp to display our static URL's as well (our store software will convert the static URL and process it with ProductDetails.asp (if that makes sense).

So... both pages show the exact same content... it's just the URL that's different. I made it so if the page is called from ProductDetails.asp it displays the Meta noindex tag (this will only display when affiliates link to the site). If the page is called by the static URL as shown throughout our site noindex will not be displayed in the headers.

Thank you for clarifing everything though. If you didn't explain the catch-22 I would have modified the metatag noindex and I would have left the robots.txt the same.

What I have done with affiliate links on my site is have them 301 redirect to the page the engines should be seeing. This is done AFTER the script has recorded the affiliate in a cookie ;-)

So... both pages show the exact same content... it's just the URL that's different.

midoriweb - In that case, I would dynamically insert the robots noindex meta into all of the dynamic pages... make sure it's not also on the static pages... and not use robots.txt to block the dynamic pages at all.

Thanks Robert,

But new question... I can do what xxcli#*$! said before your post. I can put a 301 redirect on my ProductDetails.asp page. So now, if a user views the dynamic page they will be redirected to the static page (and their code will still be still be tracked).

But... what does this do as far as links in the SE?

If someone links to the ProductDetails.asp page (dynamic) will the SE's pick up the redirection and list the page as the static URL? Or... will the SE's still continue to list the URL as a dynamic URL?

The plus side to doing this redirection is that all links and PR should go towards one page instead of multiple URL's for 1 single page. I'm thinking right now that a 301 redirect of dynamic URL's to a static URL is the best way to optimize my site. My only concern would be how SE's are dealing with my redirection. Will they still displays incoming dynamic links in the search results... or will they update their dynamic incoming links and start to display my static URL's?

midoriweb - Good question, and I don't know for sure, as I've only dealt with 301s on Apache, where the rewrite is done before the page is served. I don't exactly understand 301s in the code on the page.

If one takes the approach that a 301 is a 301, then the referring url should be rewritten, and should not be indexed. On the other hand, this might be the kind of 301 that people are complaining about staying on in the engines.

I think you're going to need someone who knows servers better than I do for this answer.

Thanks Robert,

I'll try posting this question in another forum to see what people think.