I have older articles on my site about 3000 of them
For some reason the app is not crawling these urls!
Any way to find out why this is happening?
phranque
11:14 am on Jun 2, 2015 (gmt 0)
are you internally linking to these articles?
have you tried other site crawlers?
born2run
2:08 pm on Jun 2, 2015 (gmt 0)
Hi can you explain the words internally linking please ?
born2run
5:33 pm on Jun 3, 2015 (gmt 0)
Hi please let me know what you meant by "internally linking"? thanks!
phranque
6:44 pm on Jun 3, 2015 (gmt 0)
can you navigate to these articles from your homepage?
lucy24
7:24 pm on Jun 3, 2015 (gmt 0)
the app is not crawling these urls
Why do you expect it to? That's not meant as a hostile rejoinder, sorry, but a question of fact. What action have you taken that would lead the app to crawl these specific URLs? (It isn't enough for them to exist, hence the questions about internal linking.)
Also, to double-check: Do you mean "not crawling" or do you mean "not requesting"? I'd like to think the app would tell you if it tried to crawl an URL and wasn't allowed in-- but you never know.
born2run
5:35 pm on Jun 4, 2015 (gmt 0)
You guys are right I'm not internally linking these articles. Any suggestions how I can link these old articles? Thanks again!
lucy24
8:26 pm on Jun 4, 2015 (gmt 0)
Think of your humans. Under what circumstances would you like them to be able to find the old articles?
<tangent> On rare occasions, I find an interesting article via a search engine, and I then proceed to wander around the rest of the site. Later, when I want to go back to the original article, I can't find it for love or money. Is this intentional? Does the site owner know or care that some old URLs-- forever in a search engine's memory-- are no longer reachable from within the site itself?
Unless you particularly enjoy infuriating your readers, try very hard to avoid this. Either noindex old articles, or keep them reachable. At a minimum, create an Archives section where people can eventually find everything. </tangent>
born2run
9:04 pm on Jun 4, 2015 (gmt 0)
Thanks Lucy for your advice. I shall have to open an Archives section then!