Yahoo! Slurp indexing non-canonical domain

Forum Moderators: open

Message Too Old, No Replies

Yahoo! Slurp indexing non-canonical domain

301 handlig appears to be broken again

jdMorgan

10:39 pm on Dec 5, 2008 (gmt 0)

I just found out that Yahoo! has indexed a non-canonical version of one of my domains, despite the fact that this non-canonical domain has been 301-redirected to the canonical domain since inception.

So, despite a 301 redirect from example.com to www.exmaple.com that actually predates the DNS going live ten years ago, Slurp has "inferred" this non-www domain and is now showing it and linking to it in their search results.

Thanks for the referrals, Yahoo -- But I sure wish that every one of them didn't result in a redirect because the domain you're linking to is incorrect!

If anyone else sees something similar, don't panic -- This is definitely something wrong on their end -- Either now or in the recent past (the Yahoo cache date on the page I looked at was Nov 19th, 2008).

Jim

bwnbwn

2:28 pm on Dec 8, 2008 (gmt 0)

jdmorgan don't panic I agree but be prepared to disapper from the serps,
[webmasterworld.com...] as I said it has been broken for at least a couple months now.

jdMorgan

3:31 pm on Dec 8, 2008 (gmt 0)

This non-profit, informational site has its own dedicated user-base, and will fulfill its main function even if it drops from all search engines. So I'm not particularly worried about that aspect.

I'm just amazed that Yahoo and MSN Live continue to tolerate flaws in their fundamental infrastructure that sabotage any chances they have of competing with Google -- It amazes me that they think they can produce great search results when their crawlers evidently do not understand and/or properly handle basic HTTP signaling... They are apparently oblivious that they're building their houses on swampy, unstable ground. :o

In this case, I believe the cause may be related to Yahoo's very-annoying habit of "inventing" and making requests for URLs even when there are no links on the Web pointing to those resources. An example is the directory-level URLs that Slurp attempts to fetch, even when those directory URLs have no inbound links and the server is configured to return a 403-Forbidden to directory requests when no index page exists in a directory -- a very common (and in many cases, default) server setting.

Jim

bwnbwn

6:37 pm on Dec 8, 2008 (gmt 0)

We have been having an issue with the yahoo slurp on most of our content management sites with the slurp trying to access a folder that needs to be filled out and submitted, it looks like the slurp is trying to do just that multiple times causing the DB to time out and send the slup a bad response. The slurp has as well crashed the sites from the multiple responses multiple times over and over.

I spoke to Yahoo at the PubCon about this and he said they would look into it. The slurp isn't quite as agressive as it was but still continues to try and access fill in request.

Just this all started with the august update along with the improper reponse of long established 301's.

jdMorgan

9:00 pm on Dec 9, 2008 (gmt 0)

This seems to be clearing up on its own.

I no longer see the non-canonical URL in the links of Yahoo search results; The link in the SERPs is now correct. However, I do still see a non-canonical home page listing in Yahoo Site Explorer when "Show pages from: Only this domain" is selected.

Jim