Is a huge increase in the number of blocked pages a bad sign?

Forum Moderators: phranque

Message Too Old, No Replies

Is a huge increase in the number of blocked pages a bad sign?

JS_Harris

7:42 pm on Jul 13, 2009 (gmt 0)

My site uses robots.txt to block any page generated via the site search function. I did this primarily because I disabled the site search function and I use a G/Y/MSN combo much like WW and I didn't want the pages showing up.

In looking through my G WMT's account a moment ago I found over 6,000 new entries under "blocked by robots'txt", all of them having the search parameter. Now, if you try to use the search parameter a 404 is returned so I know no pages were generated and the feature has been disabled since day one so this isn't just an update.

I'm not sure how Google would have found pages that never existed to begin with but I'm more concerned with why someone might have done a search for almost every word on my site.

Has anyone else experienced this recently?

tsalmark

3:27 am on Jul 14, 2009 (gmt 0)

Those are pages it knows about or knew about, and now can not reach. So long as there are no real links to these, no longer existing pages, they will just fade away as they expire from Googles cache. Of course there could be links to these pages on the internet that will keep them listed as 404.

phranque

5:42 am on Jul 14, 2009 (gmt 0)

the search engines will never know that a 404 is generated if they are excluded by robots.txt.
if you want G or the others to stop crawling those pages you will have to remove the exclusion and serve them the 404 Not Found responses.
you should be able to determine through GWT if those requests are caused by inbound links found by G and if so the source of those links.