We all know “A description of this page is not available because of this site’s robots.txt”. Has anyone else met the new version?
In place of the snippet we get “No information is available for this page.” On the following line is a “Learn why” link, leading in turn to two categories of action: one for “I don’t own this page” and a second for “I own this page”. Did market research finally disclose that the average human searcher hasn’t the remotest idea what robots.txt is, so the former wording wasn’t especially useful?
I most recently met it while searching for {specific name} {type-of-business} -- a type of search where there’s a clear #1 result, whether or not the search engine has seen the page, because there are bound to be links to it. Further snooping reveals that
(a) the site’s robots.txt* is, in fact, a comprehensive
User-Agent: *
Disallow: /
(Why, for god's sake? Doesn’t that defeat the point of even
having a website for your business? But I digress...)
(b) That Other Search Engine mysteriously manages to say {brief and extremely generic description of business}, which in turn is identical to the wording offered by two minor search engines I tried at random.
Once again, I feel as if I have missed a chapter.
* I'm going to assume they are not clever enough to serve up a different robots.txt to every visitor. It’s not that big a business.