Is it finally time to do away with the xml sitemap?

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Is it finally time to do away with the xml sitemap?

Sgt_Kickaxe

2:43 pm on Jun 29, 2011 (gmt 0)

I've been contemplating the removal of my xml sitemap creating script for some time as it feels like its use is limited.

- Not much use to search engines if your pages are already indexed.
- No use to real visitors.
- A small performance gain can be had by removing plugins that create them.

To replace the sitemap, and to help visitors, I'm leaning towards creating an HTML sitemap instead, an archive of sorts on my site.

While I've used xml sitemaps to guide a search engine back to my content after implementing redirects or when indexing new content I don't see the value on an established and well crawled site anymore.

Question: Would removing a sitemap from a well established site cause a rankings effect of any kind?

tedster

7:55 pm on Jun 29, 2011 (gmt 0)

I think XML sitemaps have strong value for large sites with lots of churn. Other than that situation, it might be very worthwhile to look at the total resource required.

An HTML sitemap is a good idea even if you still have the xml sitemap. Make it more of a "guided tour" approach, rather than trying to link to every page on a mammoth site.

jerednel

8:15 pm on Jun 29, 2011 (gmt 0)

I think an XML sitemap definitely has its place on a site with hundreds of thousands of product pages. I feel like it can help them get indexed. I feel there is a small chance that with so many products, they will be indexed in any timely manner otherwise.

For a small website, yeah they can seem kind of pointless.

Demaestro

8:25 pm on Jun 29, 2011 (gmt 0)

I am not sure that removing it will affect your rankings but it is so hard to tell now.

I know I have removed them from sites with less than 10 pages and hadn't noticed any drops but the weren't in competitive keyword markets.

I find the XML sitemap very useful when submitting new sites to be indexed, but I agree once the all your pages are indexed and there isn't a lot of content turnover I don't see a benefit.

CainIV

11:30 pm on Jun 29, 2011 (gmt 0)

I find they work fantastic in at least getting Google to understand, quickly, that there is new content at url X.

We publish the content, then update the xml feed, then 24+ hours we syndicate.

g1smd

12:58 am on Jun 30, 2011 (gmt 0)

I've never used an XML sitemap.

A tight structure and sane URL hierarchy is enough on small sites.

Sgt_Kickaxe

3:11 am on Jun 30, 2011 (gmt 0)

I took the plunge and removed the sitemap/plugin from a wordpress site with roughly 4000 visitors a day. I removed mention of it in robots.txt and deleted it from webmaster tools (Bing and Google).

I'd like to make other speed related improvements to the site but will wait 2 weeks between changes to get a good idea of effect so for now... tic-toc.

deadsea

9:50 am on Jun 30, 2011 (gmt 0)

I do mega sites. Even on mega sites, the value of the xml sitemap is limited if your site has a crawl-able hierarchy.

Currently my XML sitemaps have every crawlable url in them. On some sites we use the priority field based on how good the page should be at ranking for keywords.

I'm thinking about trying something new. I'm thinking about only putting new pages and pages that have been updated since they were last crawled in the sitemap. With any luck this would let Googlebot focus its resources on the places that it actually needs to be crawling. I'd see boosts for QDF searches. Maybe Googlebot would even stop crawling some of my pages that rarely change multiple times a day.

AnkitMaheshwari

10:43 am on Jun 30, 2011 (gmt 0)

One of the use of XML sitemap of a very large site is that the sitemap section of webmaster tool gives number of URL's indexed in Google and any anomaly (say penalty to a section) might get detected early if you are looking at this number periodically.
I regularly keep track of it as site: operator give too vague numbers(at times)....

Sgt_Kickaxe

6:05 am on Jul 6, 2011 (gmt 0)

7 days after removing it traffic is completely unchanged.

There is no requirement to using a sitemap so I didn't expect a loss in traffic but since it was an existing sitemap... you never know. Well, now you do.

FranticFish

7:10 am on Jul 6, 2011 (gmt 0)

One of the use of XML sitemap of a very large site is that the sitemap section of webmaster tool gives number of URL's indexed in Google

I have always found it rather daft that Google won't give you indexing data otherwise!

g1smd

8:23 am on Jul 6, 2011 (gmt 0)

I would have said yes until I saw the announcement about On-demand Indexing for Google CSE data recently made over at code.google.com.

tangor

9:16 am on Jul 6, 2011 (gmt 0)

Never fed the gorg or any other SE a sitemap. I've been testing their crawl capability. If they can't find all the pages I made available I put them 403 in my .htaccess as persona non grata...

Almost hit SUBMIT then changed my mind... that's a bit of humor, but is EXACTLY the reason why I've never used a sitemap.

htdawg

1:25 pm on Jul 6, 2011 (gmt 0)

how big of a site (in pages) would you say might need a sitemap?
anything over 100?

Sgt_Kickaxe

4:14 am on Jul 21, 2011 (gmt 0)

I would have said yes until I saw the announcement about on-demand indexing for Google CSE data recently made over at code.google.com.

I saw that too but it only impacts results displayed on your site, not in Google's index. The two are not different for very long so the resources required may not be worth the benefit.

The added functionality of adding an expires tag to urls in your sitemap is great but again I'd probably upload a sitemap for the purpose of removing something on an as-needed basis.

Still no sign of any rankings loss btw. Slightly increased crawl rate but not out of the norm so it's likely not attributable to the sitemap removal.

MarvinH

11:47 am on Jul 21, 2011 (gmt 0)

Sgt_Kickaxe, thanks for bringing up this topic. I just removed my XML sitemap few moments ago. I do have an HTML one.

I run a small hobby site (forum). I'm curious to see if the removal of the XML sitemap impacts anything at all. I suspect (and hope) it will not. :-)

MarvinH

12:03 pm on Jul 21, 2011 (gmt 0)

The following quote is from Google Webmaster Tools > Site Configuration > Sitemaps

Submit a Sitemap to tell Google about pages on your site we might not otherwise discover.

None of the pages in my XML sitemap would not otherwise be discoverable, so if "discoverability" is the only benefit of XML sitemaps, my site should not be impacted by the removal of it.