is it harmful if Google is requesting lots of removed pages?

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

is it harmful if Google is requesting lots of removed pages?

graeme_p

5:13 pm on Aug 28, 2019 (gmt 0)

On a site with something like this:

Main pages for red widgets, blue widgets etc (across about 400 products)

Each of these has links more specific pages: small red widgets, large red widgets, luxury red widgets etc.

Those more specific pages are often unpublished (and URLs return a 404), usually because the product is not available (and will not be for some months, typically).

What I usually do in that situation is to show the page but clearly mark it unavailable (e.g and overlay or similar that cannot be missed). The decision to use 404s here was not mine (and predates my involvement in the site)!

The 404s are invariably seen by crawlers (Googlebot and Bing) and very rarely by human visitors so my main concern is whether its bad for SEO.

If so, would it be better to fix by:

1) marking unavailable as I usually do
2) redirecting to the main product page

Again, purely from an SEO point of view because of the lack of human visitors.

lucy24

6:54 pm on Aug 28, 2019 (gmt 0)

Those more specific pages are often unpublished

Are they linked from pages that are published? If yes, and it happens recurringly, it raises the spectre of “poor technical quality”, which is one of the 8000 ranking factors. If no, where are they learning about the URLs?

redirecting to the main product page

Now, you know Google doesn’t like it when you handle a 404 with some other response. (Edit: Your question is about SEO, but I shouldn’t think humans would like it either, especially if they got there by following a link on your own site. “Why are they sending me around in circles? This is where I started out!”)

Wilburforce

7:14 pm on Aug 28, 2019 (gmt 0)

Wouldn't it be better - for SEs, bots and humans - not to link to unavailable pages at all?

If you want the links to work later, commenting them out until the product page goes live would be one way to go (and for human benefit you could always format the text as a link and add a mouseover comment while waiting).

Lucy24's last paragraph is right on the money, too.

martinibuster

7:24 pm on Aug 28, 2019 (gmt 0)

Mueller has said that Google checks to see if a page has returned due to publisher error. He also said it's evidence that Google has extra crawl budget for your site so they use it to double back on missing pages.

So the answer is no, it's not harmful. It only means that Google has spare crawl budget for the site.

Just make sure there aren't incoming links to those page, both external and internal. It's a bad user experience.

If a page is truly missing, showing a 404 is the correct response. There is no negative ranking effect from showing a 404 response.

Redirecting page not found instances to the home page or some other page is not the right response and is not recommended. There is no ranking benefit from doing that.

tangor

4:45 am on Aug 29, 2019 (gmt 0)

I let the chips fall where they may. When I take out a page I give a 410. Any other 404s are everybody else's fault, not mine! It is not my worry that some coder messed up a cut and paste url from my site, typed it in incorrectly, or whatever. Nor do I worry about the batch bs hack attempts that return 404 (unless I have already identified and return 403).

As for g (or any other SE) wasting time on removed pages, not there pages, that's not my business, or worry. This is not an SEO question, or concern.

graeme_p

4:14 pm on Aug 30, 2019 (gmt 0)

Are they linked from pages that are published?

No. All internal links that 404 have been fixed.

Now, you know Google doesn’t like it when you handle a 404 with some other response. (Edit: Your question is about SEO, but I shouldn’t think humans would like it either, especially if they got there by following a link on your own site. “Why are they sending me around in circles?

That is not exactly what I mean. I mean the unpublished "small red widgets page" should redirect to the "red widgets" page, and in this case the that page will have very close substitutes for what they wanted. I have ensured there will be no internal links pointing at the "small red widgets page" too.

graeme_p

4:20 pm on Aug 30, 2019 (gmt 0)

@Wilburforce, sorry, I was unclear there. There are no internal links to these pages. I think Google are checking old pages.

@martinibuster, I am pretty confident all internal links are fine. Most of the links are generated by the system so assuming I have fixed all faulty code we are fine with regards internal links, and Google Webmaster Console would point out external links that 404? I would not normally consider redirecting but I would also not normally 404 a url that might come back.

The other thing I have probably not made clear is that some of these pages will come back when the product is available again. One thing that worries me is any external links might be lost in the meantime. I do not know how common or how high value links to these pages are. On the whole I think they are pretty rare as I have not seen 404s that would result from that.

Wilburforce

4:44 pm on Aug 30, 2019 (gmt 0)

@graeme_p, if they are old pages Google will keep checking them whatever you do (even returning a 410 - which ought to stop it - doesn't always work, particularly if there are still backlinks to them from anywhere).

In your case 410 wouldn't be right for any pages that might be resurrected, but 404 isn't really right for those either. Personally, I'd be inclined to return a 302 linking to a "Sorry, we don't know when this product will be available" page - generic, section-generic or page-specific as appropriate - for that purpose.

That won't stop Google checking them, but Google checks old pages idefinitely. Don't worry about it.

_{Edit - actually, a 302 will stop them showing up as "errors" in GSC.}

tangor

5:33 pm on Aug 30, 2019 (gmt 0)

Personally, for any product that is currently unavailable THAT statement is put on the page and allowed to stand (not removed, just marked "Out of Stock" or "Not presently available")

404s are not generally harmful at any time. All it means is "page not found" ... and that can mean all kinds of things, and all of them quite normal.

graeme_p

12:13 pm on Aug 31, 2019 (gmt 0)

@tangor, thanks. That is what I usually do from the start. In this case I came to the site to find it set up like this and was unsure which way to go from where we are now.

@Wilburforce, thanks for confirming a 404 is not right.

tangor

12:57 pm on Aug 31, 2019 (gmt 0)

Beauty of leaving the unavailable page active (so marked) is the opportunity to also include:

"See related products" with a list is links. The user is kept on the site and opportunities are offered to make a sale.

In most cases there's few reasons to take down a page, even if outdated. One always wants the hit and one more chance to move the user deeper into the site.

lucy24

4:53 pm on Aug 31, 2019 (gmt 0)

Unhappy thought: What happens, though, if the SERP snippet for one of those temporarily-gone pages says baldly “This product is currently unavailable”, because that happens to be what the search-engine spider has most recently seen? This doesn’t seem calculated to make searchers click through--which, after all, is the point of having all those interior pages indexed. Is this an argument in favor of a 302? (Assuming it doesn't continue for long enough that the search engine gives up and starts treating it as a 301.)

tangor

5:47 pm on Aug 31, 2019 (gmt 0)

Do search snippets actually do that? Pull a line "Product not available" in place of what has already been recorded?

Personally never seen that, but golly gee, it is possible, right?

tangor

5:52 pm on Aug 31, 2019 (gmt 0)

301, of course, is a necessary tool, and there are reasons when to use it, but for the most part outdated for un-updated pages with existing index serps are more valid as an entry point to NEW content via a SEE RELATED or UPDATES INFO HERE, etc. Make the old work while pushing new, and at the same time enjoy the old still delivering....

Wilburforce

7:43 pm on Aug 31, 2019 (gmt 0)

Hmmm. Not taking down a page would be OK if the page didn't have incorrect information on it (e.g advertising an unavailable product, or an old price), but a page with the wrong message is better removed than left alive.

A "not available" message should be OK if it isn't there too long (in my sector products - always those needed yesterday - can often be on back order for a couple of months), but not such a good idea if it is there long enough to give users the idea than nothing is ever in stock. On the other hand, a quick search for Product Not Available doesn't suggest you're going to rank well for that term even if you leave it on multiple pages for years.

But coming back to the original question, I'd say stick with option 1, and no, 404s are not in themselves a problem for SEO (although linking to a lot of 404s is another matter).

tangor

6:37 am on Sep 1, 2019 (gmt 0)

Obviously bad data needs to be removed. Things change over time and this happens. As for "not available" being a black mark, I doubt that is a problem. After all, things also go out of production over time, yet for historical purposes those pages are still valid for those that are doing research or actually have the product.

REMOVE pages when it makes sense. REDIRECT pages when it makes sense. Otherwise, KEEP the pages active as they are future LANDING pages from which you have the opportunity to move the visitor deeper into the site. Never miss a bet ... and since it has already been anted up, leave it on the table. :)

graeme_p

2:12 pm on Sep 4, 2019 (gmt 0)

Thanks for all the help, I think we will go with option 1.

Given the nature of the product it will be normal for anything unavailable to be available some months later so they are future landing pages.

@lucy the point of a 302 would be to redirect them to a page of close substitutes.