I use Pro-Sitemaps to crawl and submit my site. Less than 1/3 of my website has been indexed by Google. Does anyone know what is happening?
40,702 URLs submitted 12,911 URLs indexed
netmeg
5:22 pm on May 9, 2016 (gmt 0)
They don't typically index all of my sites, but the ratio is usually a little closer than that. They often won't index certain types of pages - are all those pages full content, unique pages? Have you kept out things like tags, or anything else that creates duplicate content?
azlinda
5:59 pm on May 9, 2016 (gmt 0)
Yes, Netmeg. I have no duplicate content. They are all unique pages.
trebuchet
7:43 pm on May 9, 2016 (gmt 0)
Have you done the usual stuff like checked robots.txt? Checked WWW/non-WWW duplication? Checked WMT for crawl errors? Fetched as Google?
azlinda
7:48 pm on May 9, 2016 (gmt 0)
Yes, I've done all that, trebuchet. I'm really stymied.
netmeg
7:54 pm on May 9, 2016 (gmt 0)
One thing is that from what I can tell, that number of pages indexed that GSC reports isn't necessarily accurate. I have my sitemap broken out into various categories (event and taxonomy pages) , and the number of pages I get when I do a site: command for each category is usually higher (and sometimes substantially higher) than what's reported in Search Console.
This stuff is actually probably more for the Google SEO forum, but another thing you might take a look at in Search Console is any changes in your crawling history.
Andy Langton
8:13 pm on May 9, 2016 (gmt 0)
Most sitemap crawlers are highly aggressive in terms of what's submitted - I'd go as far as saying they almost always submit undesirable URLs.
The key question is which URLs are not indexed. The only good way to answer that is by comparing indexed URLs with submitted ones. This is not straightforward without specialist tools, however, if your website has distinct top-level directories, one "manual" way is to compare counts for site:example.com/directory with the count of URLs in your sitemap for that directory, and keep going for the biggest directories. For directories with big discrepancies, you can then drill down further to find the missing pages.
tangor
3:20 am on May 10, 2016 (gmt 0)
Is this a CMS or a hand coded site? How dynamic is it?
buckworks
6:13 am on May 10, 2016 (gmt 0)
How does Bing compare? More pages indexed, fewer, or about the same?
azlinda
12:25 am on May 13, 2016 (gmt 0)
It's a hand coded site. Bing has more pages indexed.
lucy24
8:30 pm on May 13, 2016 (gmt 0)
40,702 URLs submitted .... It's a hand coded site.
Perhaps "hand coded" was the wrong either-or to use.
azlinda
5:01 pm on May 14, 2016 (gmt 0)
In fact, Bing has ALL my pages indexed!
Andy Langton
5:12 pm on May 14, 2016 (gmt 0)
If you're looking at the count for a site search, beware. Bing's estimated numbers are even less reliable than Google.
Aside from this, I would usually assume that a much higher number of Bing URLs vs. Google indicates filtering of (perceived) lower quality content that Bing does not do very well at all.
tangor
7:31 pm on May 14, 2016 (gmt 0)
Or, Bing does it better than g ... pretty sure the folks with (perceived) lower quality content websites would be happy with that. :)
Now the question is if Bing has them all, why does G not?