Forum Moderators: phranque
Google says it cannot index my site due to the robots.txt file. But I don't even have a robots.txt file at the moment. Godaddy, the hosting provider, has said to try to take out the .htaccess file to see if it works. But all it has is the following, which was the default:
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME}!-f
RewriteCond %{REQUEST_FILENAME}!-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress
Even after taking it out completely, Google still thinks there is a robot.txt file with the following:
User-agent: *
Disallow: /
But there is NOT ROBOTS.TXT FILE.
Does anyone know what the problem could be?
Thanks for the help.
[edited by: encyclo at 4:46 pm (utc) on Aug. 22, 2007]
[edit reason] no links to personal sites please [/edit]
You should remove the reference to your specific site before a moderator does. Incidentally, your site does appear to contain a (rather complex) robots.txt file which may be accidentally excluding content you want indexed. Your current robots file is as below:
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: */trackback/
Disallow: */feed/
Disallow: /category/*
Disallow: /trackback
Disallow: /category/*/*
Disallow: */trackback
Disallow: */*/trackback
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?*
Disallow: /*?
Disallow: /docs
Disallow: /docs/
Disallow: /docs/*
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Allow: /wp-content/uploadsUser-agent: Googlebot-Image
Disallow:
Allow: /*User-agent: Mediapartners-Google*
Allow: /
Have you set it for search engine friendly URLs?
Make sure your ftp/shell is using ls -latr to see hidden files.
