Search for domain.com in google, and for [a unique string of text from an inside page]

What does that tell you?

As this is a question that apparently concerns all the SE's, you should check Quadrille's question against G, Y and MSN, and report back.

When I entered domain.com [string of text],

Google:
It showed me a list of urls where that string of text is present

Yahoo:
It showed me a list of urls where that string of text is present

Msn:
It showed me a list of urls where that string of text is present

the results in the Se's where somewhere around 1000 +.

Let me know what do you infere from it.

Thanks.

Then your site *is* indexed.

Maybe you now need to look again and change your question ...

Quadrille,

Thanks for ur help.

The thing here is my sites home page is not cached and its not showing in the results which i saw in the domain search.

There is a javascript at the top of the home page. Do u think this might can create the problem as I think it might.

Let me know ur thoughts.

Best Regards.

Domain searches don't matter; if someone searches for a domain, then they don't really need to search, because they know, right?

What matters is searches made by potential visitors.

Your site is indexed, it appears in searches.

The question you are NOT asking (I wonder why?) is "Why isn't my site appearing BETTER in the results".

That's the question that matters, and you need to read around these froums for the answers you need, and ask specific questions if we can help.

There's no magic formula; most of the thousands of members and readers here want EXACTLY what you want; read, learn, ask and learn some more ... but don't expect miracles, and remember that not every site can be at the top.

Question one is "Does your site DESERVE to be at the top?" - if not, then your priority is to work on the site. No point worrying about search results until there's something good to be found! :)

It showed me a list of urls where that string of text is present

I think he meant the following by that: he could see a list of sites where his domain name is linked from, but he couldn't see his domain in the search (even for domain.com).

Is that what is happening?

Habtom

[edited by: Habtom at 9:14 am (utc) on Sep. 25, 2007]

Dear Quadrille,

To make this more clear let me tell you a few more things.

I am not concerned abt the rankings in SE at present.

My site has x pages and y of them appear when I do site:domain.com
These y pages are pdf pages, doc pages.

But I cannot see my sites home page and rest pages present in that site:domain.com. I hope you are getting my question. When ever we search for site:domain.com, most of the time it will show home page and other pages. But here its not showing.

Besides when I use google toolbar to see the cached snapshot of my home page it says " your search cache:domain.com did not match any documents.

Again, my question goes why is my home page not getting cached.

My question should have been "my sites home page not getting indexed since past 3 yrs" instead of the present question.

Thanks for ur inputs.

hi ashish ...
same is problem with me.
i m having a website (more than 5 years old) each and every page is crawled except home page.

I think there must be some technical problem with page.

also luking for perfect answer ;)

please tell me if you find anything relevane

Yuvi

Yes Yuvi,

I will definitely try to tell you the answer if I get.

Start looking at the code and try to figure out some coding.

Best Regards

Are there enough links to the home page?

Also check your robots.txt file to see if there is a mistake there.

Finally, just rewrite the home page from scratch. Maybe there is a problem with the code somewhere that spiders can't read it.

If my site page has viewstate enabled with large amount of viewstate already in it. Can anybody suggest me whether it could make a deference to the page crawl by a robot.

Thanks for any views.

Best Regards.

It sounds like your site has a technical problem, but let's first see if we have this right so far:

When you do a site search, you only see pdf pages on your site? No regular Web pages? Is that right?

So for example, if you take the exact URL of your homepage (usually but not always in the form of "www.example.com/"), or any other important page on your site, are you getting messages like: "Your search - www.example.com/wiggly-widgets/ - did not match any documents."?

A few other questions:

How many backlinks to your site in total? (Use Yahoo Site Explorer to find out.)

Of the backlinks to your site, how many are recips?

Are the SE bots crawling all of your pages? Only a few pages? No pages?

Also try this: Take a string of text from your homepage that is unique and only exists on your homepage, and search on the string in the SE's. Make sure you surround the phrase with quoation marks when you search on it, which tells the SE's to look only for that exact string of text. Find anything?

Also check your robots.txt file to see if there is a mistake there.

Finally, just rewrite the home page from scratch. Maybe there is a problem with the code somewhere that spiders can't read it.

It is a very good idea to check your robots.txt file if you have one, as that could be causing the problem you describe.

OTOH, it is a very bad idea to completely rewrite a page, especially the homepage, when you have no idea what the problem is. The idea is to find the problem or likely problem, and methodically work to address it.

Firstly, Caveman, Hats of 2 u for asking me such cool questions and analzing the things before getting into some major change.

I have read many forums and many other things but this one was really awesome as this kind of troubling shooting is hardly seen but the most importnat 1.

So once again thanks for taking time.

Now answers to your questions.

It sounds like your site has a technical problem, but let's first see if we have this right so far:
When you do a site search, you only see pdf pages on your site? No regular Web pages? Is that right?
>> Yes its right

So for example, if you take the exact URL of your homepage (usually but not always in the form of "www.example.com/"), or any other important page on your site, are you getting messages like: "Your search - www.example.com/wiggly-widgets/ - did not match any documents."?
>> Yes when I do cached view of my site it says:
Your search - cache:http://www.dmain.com - did not match any documents

A few other questions:

How many backlinks to your site in total? (Use Yahoo Site Explorer to find out.)
>> can you tell me how do i do it? should I do it as in Google.
link:domain.com or is there any specific way.

Of the backlinks to your site, how many are recips?
>> will give u in 1-2 days

Are the SE bots crawling all of your pages? Only a few pages? No pages?
>> give me 1-2 days for this also.

Also try this: Take a string of text from your homepage that is unique and only exists on your homepage, and search on the string in the SE's. Make sure you surround the phrase with quoation marks when you search on it, which tells the SE's to look only for that exact string of text. Find anything?
>> i took a string and searched it in google like this "string" and it showed me following result:
Your search - "string" - did not match any documents

Also check your robots.txt file to see if there is a mistake there.
>> give me 1-2 days for this also.

Finally, just rewrite the home page from scratch. Maybe there is a problem with the code somewhere that spiders can't read it.
>> I will wait for this till we conclude from the above research.

It is a very good idea to check your robots.txt file if you have one, as that could be causing the problem you describe.

OTOH, it is a very bad idea to completely rewrite a page, especially the homepage, when you have no idea what the problem is. The idea is to find the problem or likely problem, and methodically work to address it.

Please do reply me on the above answers.

I look forward to hearing from you.

Best Regards.
Thanks for the time taken.

Hi ashish
In my case ... google is showing alomost every page cached but NOT HOME PAGE and this site also have text only version ... and homepage under text only version is cached like http://www.example.com/textonly/default.html ... but not the .aspx page.

any comments from anyone?

Thanks
yuvi

[edited by: pageoneresults at 3:25 pm (utc) on Oct. 10, 2007]
[edit reason] Examplified URI Reference [/edit]

What is strange is that your pdf files are being indexed but nothing else. That suggests that the SE's don't have an issue with the site per se, but are not indexing most of the normal pages on the site. Certainly sounds like a technical problem.

Are the pdf files in a directory apart from everything else?

Also, do we have it right that the same issue exists across all three major search engines?

It would help to get answers to the other questions, especially the robots.txt question.

Re Site Explorer, just search on "Yahoo Site Explorer" to find it. Then drop your URL(s) in the box and search on them to see what Yahoo is indexing (and what backlinks exist).

As for your Viewstate question, I'm no expert on ASP issues, but if the Viewstate feature is dropping large amounts of data into a form on the page, I guess that might be an issue for the bots WRT crawling the pages. Doesn't strike me as something that would likely prevent crawling or indexing of every page on a site though. Maybe we can get POR or one of our other resident experts to comment on that one. :p

I have been having a similar problem with Google on one of my sites for about 9 months.

Early this year all HTML/SHTML pages were dropped from the Google index, leaving only the Home Page, PDF files, and DOC files in the index - about 90 URL's.

Yahoo! Site Explorer shows 518 pages.

MSN shows 2,720 URL's

I have not been able to work on this issue for a while, but I am back investigating. I will start with the robots.txt file, but any other advice would be welcome.

Thanks.

MY website got cached ... :)