Forum Moderators: phranque

Message Too Old, No Replies

Redirecting / to /index.htm

         

castor_t

1:04 pm on Mar 6, 2010 (gmt 0)

10+ Year Member



I wanted to redirect / to /index.htm on my website. Since Google and other search engines treat / and /index.htm as separate URLs, I thought and (read somewhere) that it's wise to redirect both the urls to one single page.

But after I did a 301 redirect for all / requests to /index.htm, when I used Page Speed plugin, Gave the following errors.

1. Remove the following redirect chain if possible:

* http://www.example.com/
* http://www.example.com/index.htm

2. The following resources have identical contents, but are served from different URLs. Serve these resources from a consistent URL to save 1 request(s) and 11.1KiB.

* http://www.example.com/index.htm
* http://www.example.com/

Any suggestions in this regard?

[edited by: phranque at 1:41 pm (utc) on Mar 6, 2010]
[edit reason] exemplified urls [/edit]

phranque

1:47 pm on Mar 6, 2010 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



- the server should be configured so that index.htm(l) is the default directory index document.
- all requests for the default directory index document should be 301 redirected to the directory url. (trailing slash)

in this case (root directory) you should be doing the reverse of what you implemented - redirect /index.htm to / instead.

as an aside you should consistently refer to your home page in internal links as http://www.example.com/

rocknbil

8:22 pm on Mar 6, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



^ ^ Completely agree, and here is another "why" to that.

You refer people to your site. Do you indicate index.htm? No, just example.com. Most of the requests for the main page will be example.com. So it's a goose chase to try and always redirect to index.htm.

Second, if you ever change or alter your technology for the main page, the URL will change if you use the file name. If you start outputting something dynamic with php (index.php) or SSI (index.shtml) or whatever, you can do this immediately without worrying about the effect on S.E.'s . . . your URL will still be the same.

While you're doing this, pick one or the other, www version or non-www version, and rewrite/301 one to the other, this is also duplicate content.

castor_t

5:22 am on Mar 7, 2010 (gmt 0)

10+ Year Member



Ok. I get it. Now, here's my biggest problem.

I have around 700-1000 sub-directiories dynamically created. I have been 301 redirecting http://www.example.com/dir1/ to http://www.example.com/dir1/index.htm and so on for the last 6-7 months. Now, If i try to redirect backwards, I am worried that Goog might penalize me.

And I also observed from my logs that Google is always trying to spider http://www.example.com/dir1/ first, and then http://www.example.com/dir1/index.htm. I haven't linked to http://www.example.com/dir1/ anywhere in my site, and I don't know why google is trying to spider that.

[edited by: phranque at 6:26 am (utc) on Mar 7, 2010]
[edit reason] fixed urls [/edit]

castor_t

7:37 pm on Apr 26, 2010 (gmt 0)

10+ Year Member



Will google penalize me for doing the reverse 301 redirect now?
I have been 301 redirecting / to /index.htm for a long time now. I am worried I might lose my SERP if I did the reverse (/index.htm to /)/

Pls Advice.

lammert

9:01 pm on Apr 26, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



redirecting / to index.htm does not hurt you but Google may just ignore the redirect. I have a similar 301 redirect on some of my sites because of ad targeting reasons but google just shows the www.example.com/ link in the SERPs instead of www.example.com/index.html

jdMorgan

10:35 pm on Apr 26, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Remove the "/" to index redirects, then wait for a complete indexing cycle and see what Google does. If you start to see any of the "/" URLs re-appearing in the search results, that would be a strong indicator that Google "prefers" "/" over "/index.xyz".

After this, correct all on-site links to point to /dir/ instead of /dir/index.htm

After you've waited for another indexing cycle, it should then be safe to put the /dir/index.htm to /dir/ redirect in place. Basically, we're just trying to make sure that G has noticed that the original redirect is gone before putting the new (reversed) one in place.

As phranque observes above, "the server should be configured so that index.htm(l) is the default directory index document." This is done with an internal rewrite implemented by mod_dir's DirectoryIndex directive. DirectoryIndex rewrites the client-requested URL-path "/dir/" to the file-path "/dir/index.htm"

Because this is an internal rewrite, you cannot simply add code that redirects *all* requests for "/dir/index.htm" to "/dir/" because that would lead to an 'infinite' rewrite/redirect loop as a result of the DirectoryIndex and redirect directives repeatedly countermanding each other. Therefore, you need to make sure that the request for "/index.htm" is coming directly from the HTTP client, and is not being requested as a result of mod_dir's internal rewrite.

With mod_rewrite, this can generally be done in two ways, with equivalent results:

# Externally redirect direct client requests "/index.htm" in any directory to "/" in that directory
RewriteCond %{THE_REQUEST} ^[A-Z]+\ /([^/]+/)*index\.htm([?#][^\ ]*)?\ HTTP/
RewriteRule ^(([^/]+/)*)index\.htm$ http://www.example.com/$1 [R=301,L]

-or-

# Externally redirect direct client requests "/index.htm" in any directory to "/" in that directory
RewriteCond %{REDIRECT_STATUS} ^$
RewriteRule ^(([^/]+/)*)index\.htm$ http://www.example.com/$1 [R=301,L]

Jim

castor_t

3:20 am on Apr 27, 2010 (gmt 0)

10+ Year Member



Well, first of all, That's a good idea. Thank you. And, I generate the urls dyanamically. So, I have a code that can handle requests to /folder/ and /folder/index.htm. So redirecting is not a problem. I don't have to use mod-rewrite.

Now, I removed the 301 redirect, so I have two sets of pages that serve the same content. /folder/ and /folder/index.htm. As you said, I shall wait for a few days time to see if Google is indexing /folder/.

castor_t

4:28 am on Apr 27, 2010 (gmt 0)

10+ Year Member



By the way, just so you know, the links still point to /folder/index.htm. I just removed the redirect from /folder/ to /folder/index.htm

castor_t

1:24 pm on Apr 27, 2010 (gmt 0)

10+ Year Member



Update: I have now 46 out of 1560 urls with no /index.htm indexed in Goog. How long should I wait before I can 301 redirect /folder/index.htm to /folder/ ?

castor_t

3:18 pm on Apr 28, 2010 (gmt 0)

10+ Year Member



Update: 52 urls out of 1520 now with not having index.htm. (/folder/). When I did a "-.htm site:www.website.com", I noticed that there is a link with no .htm for one of my popular page. However, when I did a google search, the link with .htm is appearing. Also, when i searched for "term site:www.website.com" the one with .htm is appearing.

So, there are two copies of the same page indexed by google. Isn't that a problem. Can I start 301 redirecting /folder/index.htm to /folder/ yet?

Pls Advice.

jdMorgan

5:05 pm on Apr 28, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Make sure the links on your site go to the correct URLs now. That is step #2 after removing the old redirects.

Once you see the majority of old URLs dropping, put the new 301 in place (final step 3).

This is simply a matter of letting the search engines see your new/revised links and follow them "voluntarily" before you "force them" with a bunch of 301 redirects. Too many "wrong links" and resulting 301-redirects are a sign of poor site administration, and at some level, must count against the site in terms of "quality" and affect ranking.

The temporary duplicate-content is not a problem as long as all of your on-page links are correct. The search engines distinguish between on-site links that you control and off-site links that you cannot control; The latter won't be "counted against you."

Jim

castor_t

7:00 pm on Apr 28, 2010 (gmt 0)

10+ Year Member



Ok. Now, I updated all of my links in the site to point to the correct URLs (/folder/). I donot have /folder/index.htm anywhere in my site now and there is no 301 redirects either. So, I now have two copies of the same page /folder/index.htm and /folder/.

I have like 1300 links in Goog that points to /folder/index.htm and I will wait for them to update to the new linking structure and then as suggested by you, will 301 redirect. Hopefully, this will not affect my traffic and SERPS. I am kinda worried about it, but I took the step now. Hopefully it's for the best.

Thanks for your advice. I will post an update here later.

castor_t

4:07 pm on May 1, 2010 (gmt 0)

10+ Year Member



Update: My traffic has been down for the past 3 days by 25% and my revenue is down by 50%. I guess this is because of the link changes I made in the site. I am hoping the issue gets resolved soon and I will be back on track, but may be it might take some time.

When I did a search for /folder/ for my site using site: in Goog, It returned 113 results. 834 pages with index.htm .

castor_t

7:42 am on May 2, 2010 (gmt 0)

10+ Year Member



If i don't do a 301 redirect from /folder/index.htm to /folder/, won't Goog consider that as a duplicate content? I have now 108 with /folder/ and 801 with /folder/index.htm.

lammert

7:54 am on May 2, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



A 301 redirect is not the only way for the Google algorithm to know if two URLs point to the same content. The larger part of webmasters has never heard of 301 redirects and therefore on many sites /folder/index.php, /folder/index.htm or /folder/index.html point to the same content as /folder/. If the content on those pages doesn't change often and the Google algorithm can automatically determine that these different URLs are practically the same, they will be combined in the background, even if no 301 redirect was in place. But this algorithm is not flawless and doesn't detect and combine all occurrences. Once /folder/ and /folder/index.htm start to appear as independent entries in the index, it may influence your rankings.

jdMorgan

6:30 pm on May 2, 2010 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If your on-site links are all updated and you've got the 301 in place, then the best thing to do is to ignore this whole thing for a week or two, and go work on something else that will add value to your site. Really, there's nothing you can do while Google "does their thing" re-computing your site's link-graph, and there's no use wasting time getting stressed-out about it.

A temporary "hit" in rankings and traffic is normal and should be expected. However, since your site's "quality of administration" will now be visibly improved, you may also expect to come back stronger than before. Thinking about that may help take some of the 'sting' out of this temporary situation, and both the pain and the gain will remind you to be very careful about correct/precise linking in the future, so as to avoid a repeat.

Jim

Rachels178

3:48 am on May 3, 2010 (gmt 0)

10+ Year Member



Redirections may cause your site to lose rankings for a while but this is normal.

rajendradhakal

10:16 am on May 3, 2010 (gmt 0)

10+ Year Member



your hosting main documents not added index.htm or index.html file thats why it's saying sooo
you have to add documents at hosting panel then it'll work

phranque

4:09 am on May 4, 2010 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



welcome to WebmasterWorld [webmasterworld.com], rajendradhakal!

i'm not sure i understand what you're trying to describe.
also some hosting environments don't use a control panel (just a command line interface) so what would you suggest in that case?

castor_t

5:55 am on May 4, 2010 (gmt 0)

10+ Year Member



I started doing a 301 redirect from /folder/index.htm to /folder/ (Last step of jdMorgan's suggestion). I know it will take some time to get all things in place and in the mean while it affects my ranking and traffic. Just wanted to let you know that my traffic became better than yesterday and also did my revenue.

Currently, Goog has 172 /folder/ links and 884 /folder/index.htm links of our site. Also, my top pages still show /folder/index.htm.

Thanks for all your help guys. WebmasterWorld rocks! Will post my updates later.

castor_t

4:57 am on May 7, 2010 (gmt 0)

10+ Year Member



Update: 1470 without index.htm. 547 with index.htm. Most of the pages are indexed pretty much. However, the revenue impact is still there as I get 50% less than what I used to get normally. :(