I add it to robots.txt on my sites. I don't see any reason it should be spidered anyway.

A few years back I had a terms and conditions page outrank the home page due to the repetition of the company name in the text. Since that time, I always add a meta noindex (and/or exclude in robots.txt) on pages of this sort - privacy policies, contact forms, popups, or anything which isn't a landing page and you don't want ranked.

whether or not we should use a robots.txt file to exclude the privacy policy.

Might be a BAD idea.

Google Adwords now checks for the existence of a privacy policy, and penalizes sites that don't have one. "Contact Us" page, as well. The rationale apparently is that "quality" sites have a privacy policy and a "contact us" page.

This may or may not be a factor right now for organic search, but I'd bet it will become one.

It would make sense to make sure these pages are clearly marked for what they are. If the search engine is doing a good job (sigh... don't get me started... I know - they DON'T) it should then be able to figure out that this is not your main page.

I think in any case you SHOULD allow it to be indexed. What if somebody is searching for your privacy policy? It's a prudent thing to do before doing business with somebody, and a reasonable thing for people to search for.

And CERTAINLY you should allow "contact us" to be indexed. What if you just want to contact a company, and have no interest in wading through pages of widget promotion to find out how? Unfortunately, companies often bury their phone numbers on their web sites. Sometimes, a search is the only practical way to find their phone number.

Hmm, should one try and make each privacy policy or contact us page different in some way?

Why do you care if your privacy policy is top ten or not?

Why do you care if your privacy policy is top ten or not?

You probably don't.

But neither do you want to have it excluded from indexing, for the reasons I previously stated.

And CERTAINLY you should allow "contact us" to be indexed. (...) Unfortunately, companies often bury their phone numbers on their web sites. Sometimes, a search is the only practical way to find their phone number.

As your example shows, the problem is that the sites have poor usability, not that the contact form should be indexed.

Any well-designed commercial site should be making it extremely clear how to contact the company on every page of their site, either by a obvious link or by having an address or telephone number visible.

There should be no reason to have a contact form indexed, especially as they tend to have email addresses and form scripts which can attract spammers.

As for privacy policies, a better long-term solution is to look towards P3P:

[w3.org...]

What we're worried about is getting penalised by the search engines for having many privacy policies that (apart from a word or two) are identical to each other.

Would this happen?

If so would the whole site get penalised?

Or would we get penalised a little bit for NOT having a privacy policy (as we've banned search engines from examing that page)?

It's actually a very good question. I have the same concerns

I think this topic is being a little over analyzed.

It's a bit negative to think that a whole site would get penalized for duplicate content on a single page.

The better search engines are pretty clever these days - not only will they probably be able to tell that it's really no big deal that these pages are similar (I imagine there are many hundreds of thousands of privacy pages that use the same stock phrases), but if the sites are on the same server, and they share either a single IP (or are in the same "block"), the the SE will also know that the sites are probably "related" that way, too.