Forum Moderators: phranque

Message Too Old, No Replies

xml sitemap issue

how to create sitemap automatically for huge site

         

Sachet

6:49 am on Mar 3, 2016 (gmt 0)

10+ Year Member



Hello,

I have started working on a website having numbers of folders containing huge numbers of pages. I want to create an index page of sitemap in root to link to all sitemaps in sub-folders.

How can I do it in the least time or what kind of tool I can use for this purpose ? or I should create all sitemaps manually ? Please help

Andy Langton

8:47 pm on Mar 10, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hi Sachet, why do you want to create so many sitemaps? Is there a problem with your site's link structure?

Sachet

4:30 am on Mar 11, 2016 (gmt 0)

10+ Year Member



Andy, I have created a sitemap containing all URLs but it is more than 4 mb in size & URLs are supposed to increase everyday, I want to create small sitemaps for each folder for optimal size.

Andy Langton

10:06 am on Mar 11, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Understood, but my question is about whether all your effort to create sitemaps is worth it? You could consider just submitting a sitemap of new pages, for instance. If you're just resubmitting pages that Google is already aware of, this is very unlikely to have any impact on performance.

Sachet

10:27 am on Mar 11, 2016 (gmt 0)

10+ Year Member



I don't think sitemap is only to let search engine know about my new pages, around 20% of total pages of my websites are yet to be indexed & pages are generally updated on time to time. I want all pages to index all pages quickly

Andy Langton

11:54 am on Mar 11, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I don't think sitemap is only to let search engine know about my new pages, around 20% of total pages of my websites are yet to be indexed & pages are generally updated on time to time. I want all pages to index all pages quickly


Sitemaps are about forcing Google to discover URLs, for instance:

- When you have a lot of URLs that Google isn't already aware of (e.g. you launch a new site or a large section of new content)
- When you need URLs processed more quickly than may occur naturally
- When you have a lot of changing content

If your content is not being indexed without a sitemap, then you likely have a bigger issue with how your pages are linked together and how many third parties link to you, which is not fixed by a sitemap. Rather, a sitemap will just conceal the underlying problem.

adder

7:10 pm on Mar 11, 2016 (gmt 0)

10+ Year Member Top Contributors Of The Month



There is a school of thought putting sitemaps on the shelf of "overrated SEO gimmicks"

I must admit I still do them by virtue of habit but I'm yet to be convinced that having a sitemap can improve the crawl stats

If it's a brand-new site, and judging by the OP it is, I'd agree with Andy Langton and only do a sitemap listing the key pages and the fresh pages.

Throwing 4MB of XML at Google won't speed it up. Make sure that each folder and page is linked to in a relevant and logical way and that visitors are discovering your content (not only your home page) via social media and natural links.

In all honesty, I see those full sitemaps as a helping hand to scrapers and content spinners.

Ok, something tells me this is not convincing :) If you want to go ahead with it, I hope you have some sort of a database behind it. The easiest way is to generate sitemaps dynamically from your database. There are plenty of free/cheap scripts out there, just search for "sql sitemap script" or similar...

Andy Langton

9:03 pm on Mar 11, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



"overrated SEO gimmicks"


I don't know if I would go quite so far, and for some sites sitemaps are extremely useful. But for a typical site, is generating sitemaps the most useful activity? I think this is unlikely.

tangor

10:20 pm on Mar 11, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Do remember that g does have limitations on the size of sitemaps:

50,000 urls per map
<50mb uncompressed per map

Break large sites into smaller sitemaps and use a master sitemap to submit those. In that regard you can have up to 1,000 sitemaps, thus 50,000,000 urls - 1,000

Sachet

3:29 pm on Mar 12, 2016 (gmt 0)

10+ Year Member



If we need to have sitemap with main pages only & rest would be taken cared by website’s internal link structure, then why Google (google.com/sitemap.xml) is using the sitemap structure as Tangor has described above ?

Andy Langton

5:33 pm on Mar 12, 2016 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



why Google (google.com/sitemap.xml) is using the sitemap structure as Tangor has described above ?


Because Google is one of the main proponents of sitemaps. They want to discover as much content as possible. They don't care about where the discovered content ranks.

Note that Google do not perform at all well in search results. They don't rank in the top three for "search engine" for instance, so I would not suggest using the Google website as a model for your own SEO efforts.