Crawl a mobile m.site.com website with Xenu Link Sleuth

Forum Moderators: not2easy

Message Too Old, No Replies

Crawl a mobile m.site.com website with Xenu Link Sleuth

How to avoid that the Xenu crawler gets redirected to desktop version?

seidelbast

4:48 pm on Jan 10, 2017 (gmt 0)

Hello,

I would like to use Xenu Link Sleuth to crawl the mobile version m.site.com of a site which is served to mobile users, and not the desktop version www.site.com which is served to desktop users.

But when I enter m.site.com into Xenu, the homepage gets redirected to the desktop version and that’s it. I suppose because the server considers that the connection is coming from a desktop computer and therefore redirects to the desktop version.

How can I configure Xenu so that it crawls the mobile website?

Thanks a lot in advance for your help!

lucy24

6:52 pm on Jan 10, 2017 (gmt 0)

:: detour to assorted docs ::

Oh, I see. Xenu's developer has chosen not to let users change the UA string, because of the possibility of abuse (since it crawls from your own IP rather than a knowable address). Even the w3 link checker--which does crawl from a fixed IP--doesn't seem to have a UA editor.

That leaves us with Option B: add or change a line in the site's configuration file so the Xenu user-agent is exempt from redirection. Er ... this is your own site, right? Exact wording will, of course, depend on your server type.

Come to think of it, you might find it useful to exempt your own IP from mandatory redirection anyway. So that's another way to get the same result (either exempt the Xenu UA, or exempt an IP). In fact it's safer, since anyone could claim to be Xenu Link Sleuth.

keyplyr

9:17 pm on Jan 10, 2017 (gmt 0)

anyone could claim to be Xenu Link Sleuth

...and often do.

seidelbast

9:56 am on Jan 11, 2017 (gmt 0)

Thank you Lucy24 for the insightful answer. Option B with excluding IP address from redirection sound like sound advice.