Google is terrible at this. I had a similar 100,000 page dynamic site that took well over a year for Google to update from .asp pages to .html and I'm still getting http errors in my analytics from Google looking for the .asp pages that haven't existed for over two years. Expect a looong haul.

The strange thing is that Google DID in fact update the index for the non-dynamic pages, the ones that just end in .asp and were changed to .htm. This update happened right away.

Why would Google react in a different way when it comes to updating the dynamic-looking pages? Did this trigger some kind of filter?

Anyone else have experience with this, or have seen anything similar?

Hi, I was glad to see your post because this has been my major concern for a good while. My site currently ranks very well, but not as well as it did and we are in the process of redoing the entire site and makeing our dynamic .cfm urls that are SE friendly, what we hope, are even more friendly static html urls with the keywords in them. The current site is 5 years old.

I have been hesitant to make this overhaul primarily because of losing the SE traffic we get currently due to the 301 redirects on every page and changing the old url structure.

Obviously I hope it has no negative effect only positive, but am very curious to hear other responses and any specifics of what should and should not help this type of situation. Is it a mistake/

Thanks.
J

It's been a few months since I worked on a url rewrite project, so things may have changed a bit. But my experience was that, long term, it was always a positive step. Short term, a site with anything lower than a PR7 Home Page may experience a few bumps on the way to improvement. If the traffic matters a whole lot, have a PPC budget waiting in the wings, just in case, to soften the bumps. Once it's done, you're in great shape for the future. You can have urls that don't expose your underlying technology - no ".cfm" extensions!

It's essential to get the technology right in your rewrite scheme, and there are several pitfalls.

1) Poorly configured, you may open the door to duplicate urls for the same content. I've seen rewrite schemes where the url still keys off a product or category number as well as including the keyword. Problem was that as long as the record number was present, any old word or typo could replace the kyword in the url and the url would resolve, That's a time bomb.

2) Remember that there are two steps to address: a. get the new urls to resolve and b. 301 redirect the old urls

3) I found it more effective to do a ranking, traffic and backlink study and then only redirect the key urls, letting the rest go 404. That approach seemed to give the quickest "recovery" time in Google I ever achieved. In fact, that site never saw a dip in Google traffic, and then rankings improved quickly.

But definitely double check and triple check the technical set-up for the urls. Make sure the http headers have the correct status codes. Make sure you cannot deconstruct the intended urls and still see them resolve with a 200 OK.

Ted,

Thanks for your reply.

I have a hunch, based on the old addresses and the new ones, and the ones that Google dropped and the ones that they the updated, that the issue may be this:

Is a link in such a format penalized:

[te...] [snip] st.com/article.htm/aid/1234

The old version was:

[te...] [snip] st.com/article.asp?aid=1234

Meaning, does this trigger some kind of SEO flag to place these links on some kind of "tarpit"?

Anyone have experience with this?

As long as the old URL issues a proper 301 redirect to the new URL, it will sort itself out.

The old URLs may take a few months or more to drop out, but while they are listed in the SERPs they will still provide visitors to the site. That is not a problem.

Your measure of success is to be found in the number of new URLs that appear in the SERPs, and the speed at which they are added.

Using 301s may eventually allow google to sort things out, but the decline in traffic for the duration can be staggering.

We moved about 3000 pages in May, using 301s to redirect requests for the old URLs to their new equivalents. Within a couple days, inbound traffic from Google dropped to roughly half of its previous level. After 4 months, traffic levels have not recovered.

Old URLs were like: example.com/oldbrand/categoryname/123
New URLs are: example.com/newbrand/relevant-keywords-here.html

The site's homepage is PR6. The old pages were well-indexed and generally well-ranked, having been online for over five years. We did not do an extensive ranking and linking study as tedster suggests, and I regret that because we don't now have a clear picture of what "recovery" means aside from the gross traffic level.

I've read a lot on this board and elsewhere about the trouble with "toolbar pagerank," so although I don't put a lot of stock in it, I'll report that these best of these pages were PR3-PR5 before the move... for 3 months following the move, they showed no PR (i.e., the Firefox pagerank plugin shows 'n/a'), and then in mid- to late August they all came up as PR0 and have been ever since.

Getting back to tedster's recommendations, we discovered after the move that these 3000 pages were responsible for a disproportionate percentage of the whole website's traffic AND unique visitors. So although I'm relatively confident that over the long term, the changes we made (rebranding, better titles, better META descriptions, URLs that are more obviously relevant to search-engine users, the addition of social media features to the pages) will ultimately result in better rankings, in the medium term the URL change is viewed by the business folks as a big fat mistake.

Many of the 301 success stories we saw prior to the URL switch were answering the question "can the search-engine spiders find my new site through a 301," rather than the much more critical "will my pages retain their rankings?"

Subsequent to this debacle, we've been tasked with revamping another section of our website, which unfortunately has absolutely worse URLs than the section we moved in May. All our competitors for this traffic, who BTW mostly outrank us already, have short, relevant, keyword-dense URLs, whereas we're saddled with a URL scheme a designer came up with in 1999 in which the few relevant terms that are present in the filenames are squashedtogethernabbreviat'd. But rather than risk another 4+ month traffic loss we're keeping these stinky old filenames and attempting to build a CMS around them.

The strange thing is that Google DID in fact update the index for the non-dynamic pages, the ones that just end in .asp and were changed to .htm.

My guess is that was the simplest change for Google to figure out immediately. Compare the previous version vs the new, no change other than file extension, update immediately.

Why would Google react in a different way when it comes to updating the dynamic-looking pages? Did this trigger some kind of filter?

You changed the URI paths. In theory, the page has a brand new address and therefore needs to go through the process of recalculation.

Once that process has fully propagated, you should see most, if not all pre-rewrite traffic return. Yes, there is a process and time delay based on my experience with this. And, if you changed any of the click paths, the delay seems to be a bit longer as that is a rather major change and will take a bit of time to sort things out. Yes, PR is a determining factor in how long this process takes. On average, I'd say anywhere from 90, 120 to upwards of 180 days. Again, that will all depend on what changes took place and other external factors.

The above is why these days I am recommending a slow release of URI restructuring if it can be done. If your livelihood depends on organic search, DO NOT make any major URI changes in bulk. Test first, test again, and then if you feel comfortable that things are okay, test some more! By the time you are done testing, you will have released each new section slowly over time by default. :)

Of course there is always the chance that things are not 100% correct with the rewrite. Improper server headers can wreak havoc on a sites indexing. In fact, I've seen new sites launch with new URI structures and where there were server header issues, the indexing of those sections was challenging to say the least.

If you release a site and there are technical problems from the first indexing, you will be behind the eight ball from that point forward until you get things in order. Expect 30-60-90 day delays each time you make a major change to correct problems that were discovered after the fact. A Proactive approach to these types of technical challenges is the way to go. You sure don't want to be Reactive. At that point, the problems are already in existence and it takes a while to undo them.

Patience is key in this entire process. It isn't going to happen quickly. Any major changes like this should be done well in advance of any naturally occurring major spikes in traffic to your site such as the holidays! From experience, I'd say if you have plans on a major URI Rewrite, wait until after this holiday season and then do it immediately thereafter. You'll have a good 6-8 months of recovery time before the next holiday. :)

I have successfully redirected large numbers of URLs with little to no change in traffic / SERPs by leaving links to the old URLs in place and running through a redirect until the new URLs begin appearing in the SERPs.

This method has been effective for me, but 'flies in the face' of some advice given here on WebmasterWorld which indicates you should not run internal links through redirects and should make sure you change your links to point to the new URLs immediately, if not sooner.

I usually see changes take effect within 2 to 3 weeks.
My personal opinion is the key is in the rewrites, as noted by PageOne.
Get them right, including using only 1 redirect to arrive at the target.

IOW If you redirect non-www to www, and old-URL.asp to new-URL.htm both must happen in a single redirect to effectively pass inbound link weight, etc.

Justin

[edited by: jd01 at 2:54 pm (utc) on Sep. 21, 2007]

Google DID in fact update the index for the non-dynamic pages

It's true that Google now indexes dynamic URLs much more effectively than they did a few years back - but let's consider how they accomplished that. When googlebot sees a "?" in the url, there still is likely to be some more extensive logic triggered to protect googlebot, and the site's server, from getting trapped in loops from potentially infinite information spaces.

In other words, Google didn't just throw caution to the winds and start spidering URLs with "?" marks in exactly the same fashion as they do other URLs. And that extra processing logic still adds in its overhead.

>> If you redirect non-www to www, and old-URL.asp to new-URL.htm both must happen in a single redirect to effectively pass inbound link weight, etc. <<

A redirection chain is always best avoided.

Additionally, the file extension is completely irrelevant to indexing and ranking. You can have any extension you want.

I have worked on a site that moved over to using PHP, but the URLs still end in .asp as before.

Cool URIs don't change.

Hello

My site has 1000 pages and i 301-redirected 50pages for starters. These pages were crawled and indexed.

1. Should i change the internal links that pointed to the old pages, make them now point to the new pages, since they have been indexed?

Example:

Page A (301 redirect)--> Page B (new URL)

(Page B was indexed and ranks relatively well in google)

Page C links to page A. After how long should i make C link directly to page B?(that is , without page A being a "proxy page")

2. Since google doesnt like bulk changes, after how long should i redirect the next 50 pages? any suggestions? I am planning on redirecting most of the site in the near future.

Thank you.

As soon as the new urls are indexed, you can (and should, I think) change the links that point to it. In fact I've done both the 301 redirects and the link changes at the same time with good results - and that was sitewide over about 450 urls. However, I let all but the key urls go 404.

And for your second question, I think the next batch of redirects would be fine as soon as the first batch is ranking.

Tedster, Mcglynn and others,

After reading these intersting posts my situation sounds like mcglynns in that our old and new url structure is very similar and it's unsettling to be starting this transfer after reading these posts with limited vision of the best process.

1. I have concern that the home page which ranks very well will be hurt by having so many current internal urls go 404 which no longer will pass their strength back to the homepage.

2. Mcglynn, what would you do differently if you were starting your process over?

J

what would you do differently if you were starting your process over?

Well, considering that our traffic hasn't recovered, the short advice I would give is "don't change URLs." But our pages were performing really well before, although we didn't understand that until later.

What I would do differently:

I would research and document the current pages' value before making a move:
- how many click/month from google/yahoo do they generate per month?
- how many unique visitors is this worth per month?
- what keywords are the pages ranking for (especially: are these pages ranked #1 for any terms?)
- how well-linked are these pages from 3rd-party sites?
- do search-engine users click through from these pages to other areas of the site?

I would make sure the company management understands the risk of a 6-12 month dropoff in pageviews.

I would secure a budget for staff to pursue linkbuilding strategies following any URL change.

I will in the future be very cautious about comprehensive URL changes. For our next move, we're going to start with the pages that have no pagerank, no external links, and which drive little to no traffic. They have no traffic to lose, so our opportunity for gain is much greater.

I'm hearing some dramatically different experiences, ranging from "301 redirects are fine if done properly" to "don't change url's".

I am in the process of rebuilding a well-indexed ecommerce site from scratch. In the process, url structures will go from:

http://example.com/catid/120/subcatid/32/productid/230

to

http://example.com/product/widgets

There are about 200 products and categories that will change URL's. It will be quite impossible to only redirect a few at a time, as the site will be re-built on a completely different platform and database.

From reading all of the above posts, I have gathered the following strategies, maybe some of you can shed more light on this:

- put 301 redirects on a few key URL's, and allow the rest to go 404
- make sure that only one redirect occurs with each request

Is there anything else I can do to lessen the chances of prolonged indexing problems?

[edited by: tedster at 9:49 pm (utc) on Sep. 24, 2007]

[edited by: Robert_Charlton at 10:50 pm (utc) on Sep. 24, 2007]
[edit reason] switch to example.com - it can never be owned [/edit]

I'm about a week in with a site that has more than 100,000 pages. I've used the method that jd01 outlined above. So far no negative consequences, everything is proceeding smoothly. jd01 gave me good advice in years past on WebmasterWorld about ridirects. Tip-o-the-hat;)

SO, just to clarify jd01's advice, I should set up the new site in such a way that the link url's are the old ones, but 301 redirect to the new url's. Then, once the new ones are finally indexed, change the links to reflect the new ones?

I have successfully redirected large numbers of URLs with little to no change in traffic / SERPs by leaving links to the old URLs in place and running through a redirect until the new URLs begin appearing in the SERPs.

I'd also like to clarify jd01's method of redirection. 301? 302? How are you handling that?

guru5571:

Thanks.
Glad things are working for you.

codeman:

Put 301 redirects on a few key URL's, and allow the rest to go 404.

Means not all URLs were redirected...
Only the main pages were redirected and the rest were removed.
I'm guessing some pages were not redirected because there was not an 'equivalent' page to redirect to.

Make sure that only one redirect occurs with each request.

# If someone requests example.com send them to www.example.com
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

# If someone requests page.html send them to newpage.html
RewriteRule ^page\.html$ http://www.example.com/newpage.html [R=301,L]

In the above example if someone requests http://example.com/page.html there will be two redirects.

When the request is made the 'host' will not be www.example.com, so the first rule will redirect the visitor to www.example.com/page.html, then when page.html is requested the second rule will redirect page.html to newpage.html... There will be two redirects.

One way to help solve the problem is to make sure you redirect in the most efficient order. By changing the order of the rules and redirecting the page request first, you correct the host issue at the same time, so the ruleset below would only result in one redirect with same request from a visitor.

# If someone requests page.html send them to newpage.html
RewriteRule ^page\.html$ http://www.example.com/newpage.html [R=301,L]

# If someone requests example.com send them to www.example.com
RewriteCond %{HTTP_HOST} !^(www\.example\.com)?$
RewriteRule (.*) http://www.example.com/$1 [R=301,L]

This is just a simple example... from there it gets more complicated.

PageOne:
301, always, unless it is only a temporary redirect, then I use a 307 (or 302).

Justin

Yes. Avoid creating any sort of redirection chain for any of the redirects.

codeman:

I think we were posting at the same time...
Yes, your clarification is what I have been using.

I try to get the URLs changed over as soon as I can after the new URL(s) is in the index, and sometimes, I change a little before.

Examples:
If I have a section with 1000 pages in it, and I can see the main traffic pages have the new URLs indexed, I usually go ahead and switch over to the new URLs in the entire section.

If I have a section with 30 pages in it I usually run through the redirects until all pages are indexed at the new location.

Does it help?
Honestly, I don't know, because it has worked previously, and I usually try to not fix it if it's not broken.

new site

A new site is a completely different story.
If you are changing domain names, don't unless absolutely necessary.
(If you are, it is very similar to starting a new site, no history, from scratch.)

Internal redirects (within the same domain) are handled in a completely different way than redirects to a new domain.

Justin