Site maps / Crawling

Forum Moderators: open

Message Too Old, No Replies

Site maps / Crawling

dave1236

3:40 pm on Aug 28, 2006 (gmt 0)

I think this is the proper forum for these questions -
I run a site that authenticates users using PHP. When I look at how my site is indexed via search engines, I obviously have a problem getting the robots to crawl pages requiring a password.

I thought I might have overcame this when I added a button that allowed access to the site 'anonymously'. it submitted a form to my DB with a username/ pw combination automatically. So, I thought a spider would try the button and that would allow indexing of my site. It does not appear to work.

I have read with a site map i can direct a crawl to access pw protected pages - But I am having a problem finding examples to show how to do this.

My question is: how do I get a robot to crawl the proper way, and to utilize the 'auto login' button to access and index the protected pages. The pages are protected simply to get users to login for purposes of a rewards system.

Thanks!

FiRe

9:30 am on Sep 4, 2006 (gmt 0)

this should be in the PHP section but anyways...


<?php$useragent = getenv("HTTP_USER_AGENT");
if (!eregi("GoogleBot", $useragent)) {
// If not googlebot redirect to login page
header ("Location: http://www.site.com/login.php");
exit();}
?>

[edited by: FiRe at 9:30 am (utc) on Sep. 4, 2006]

caveman

3:39 pm on Sep 5, 2006 (gmt 0)

Perhaps this thread on password protected pages and robots [webmasterworld.com] will help?: