URL analysis

Forum Moderators: phranque

Message Too Old, No Replies

URL analysis

need a tool

MadeWillis

6:38 pm on Dec 2, 2009 (gmt 0)

Does anyone know of a tool that can scan a list of URLs and then display what is returned in the header? I want to scan a large list of URLs used in my Adwords campaign to see if any are going to 404's or other dead pages. For some reason Google only scans URLs at the ad level for errors, not the keyword level.

Thanks!

phranque

11:31 am on Dec 3, 2009 (gmt 0)

i would try xenu link sleuth.

MadeWillis

5:33 pm on Dec 4, 2009 (gmt 0)

link sleuth is a very useful tool, but can you upload a list of URLs to scan? I don't want it to scan my site. I want it to scan a list of URLs that I provide to see if any are now 404's.

Jonesy

7:16 pm on Dec 4, 2009 (gmt 0)

Here's something for the start of a linux script that might be useful:


$ wget -S -i url-list -O /dev/null 2>&1 Ś egrep '^ 1 HTTP'
 1 HTTP/1.1 200 OK
 1 HTTP/1.1 200 OK
 1 HTTP/1.1 404 Not Found

Load the input file (url-list) with the urls of interest (there were 3 in the example above.)
It directs the html body(s) to /dev/null.
It displays just (via `egrep`) the response code.

Further embellishment of the 'report' is left as an exercise for the student. :-)

Jonesy

phranque

11:52 pm on Dec 4, 2009 (gmt 0)

provide a list to xenu; set level to 1

MadeWillis

3:06 pm on Dec 7, 2009 (gmt 0)

I found a tool that does exactly what I need. Thanks for you responses.

[edited by: phranque at 2:48 am (utc) on Dec. 8, 2009]
[edit reason] specifics [/edit]

URL analysis

need a tool

MadeWillis

phranque

MadeWillis

Jonesy

phranque

MadeWillis

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week