Forum Moderators: phranque

Message Too Old, No Replies

Screaming Frog not crawling all my urls

         

born2run

2:16 am on Jun 2, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hi I have a license for screaming frog seo app

I have older articles on my site about 3000 of them

For some reason the app is not crawling these urls!

Any way to find out why this is happening?

phranque

11:14 am on Jun 2, 2015 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



are you internally linking to these articles?

have you tried other site crawlers?

born2run

2:08 pm on Jun 2, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hi can you explain the words internally linking please ?

born2run

5:33 pm on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hi please let me know what you meant by "internally linking"? thanks!

phranque

6:44 pm on Jun 3, 2015 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



can you navigate to these articles from your homepage?

lucy24

7:24 pm on Jun 3, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



the app is not crawling these urls

Why do you expect it to? That's not meant as a hostile rejoinder, sorry, but a question of fact. What action have you taken that would lead the app to crawl these specific URLs? (It isn't enough for them to exist, hence the questions about internal linking.)

Also, to double-check: Do you mean "not crawling" or do you mean "not requesting"? I'd like to think the app would tell you if it tried to crawl an URL and wasn't allowed in-- but you never know.

born2run

5:35 pm on Jun 4, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



You guys are right I'm not internally linking these articles. Any suggestions how I can link these old articles? Thanks again!

lucy24

8:26 pm on Jun 4, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Think of your humans. Under what circumstances would you like them to be able to find the old articles?

<tangent>
On rare occasions, I find an interesting article via a search engine, and I then proceed to wander around the rest of the site. Later, when I want to go back to the original article, I can't find it for love or money. Is this intentional? Does the site owner know or care that some old URLs-- forever in a search engine's memory-- are no longer reachable from within the site itself?

Unless you particularly enjoy infuriating your readers, try very hard to avoid this. Either noindex old articles, or keep them reachable. At a minimum, create an Archives section where people can eventually find everything.
</tangent>

born2run

9:04 pm on Jun 4, 2015 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks Lucy for your advice. I shall have to open an Archives section then!