RSS feeds appearing in search... desired behaviour or not?

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

RSS feeds appearing in search... desired behaviour or not?

james007

4:03 am on Feb 23, 2018 (gmt 0)

Doing a few idle searches for content that appears in my website on Google today, and I notice that the (full-content) RSS feed is very visible in Google Search results. It's normally #2 or #3.

Obviously, if you click on the RSS feed then it rewards you with a page full of gibberish.

I've checked the RSS code is returning the correct Content-Type - it is (content-type:text/xml; charset=utf-8). Obviously, I don't want to block my RSS feed from Google; I want it to follow all the links in there, and I want it to index the RSS feed itself for the various things it does with it. (Specifically, it's a podcast feed as well, so I definitely want Google indexing it). I just don't want it to appear in the main Google Search results.

Is this a bug - is there any time where Google would want to return an XML document for a search? What would be advice to remove it from SERPS but not to block Google from looking at it and using it?

keyplyr

9:39 am on Feb 23, 2018 (gmt 0)

There's no effective method to just lower the ranking in Google's index.

But you can remove it from the index while still having crawlers follow the links contained in the feed.

If you organize the RSS file is in a specific directory, you can try this in that htaccess:

X-Robots-Tag: noindex,follow

[google.com...]

Alternatively, if there is a HEAD section in the file, you could use the equivalent meta tag:

<META NAME="robots" CONTENT=noindex, follow">

keyplyr

8:24 pm on Feb 23, 2018 (gmt 0)

correction (sorry, lost a quote)

<META NAME="robots" CONTENT="noindex, follow">

lucy24

9:49 pm on Feb 23, 2018 (gmt 0)

If you organize the RSS file is in a specific directory

. . . and if you don't, you can put the directive inside a <Files> envelope, or equivalent in non-Apache servers.

I don't know about the X-Robots tag, but in a robots meta, "follow" is the default, so you don't formally need to spell it out.

james007

1:16 am on Feb 26, 2018 (gmt 0)

I still want Google indexing it, so that it'll correctly list my podcast. So if I add a "no-index" directive, surely that'll break the podcast index?

This is the podcast data documentation, incidentally: [developers.google.com...]

I just don't want it appearing in SERPS. I'm having difficulty understanding any use-case where someone searching on Google would want to see a jumble of XML in the search results.

keyplyr

1:38 am on Feb 26, 2018 (gmt 0)

SERP (search engine result pages) are populated by the index, but asking Google not to index a file does not mean Google doesn't' know about the file and follow the links.

james007

1:56 am on Feb 26, 2018 (gmt 0)

I - and anyone - want to continue allowing Google and other robots to index the RSS feed, since it has a podcast in it, and I want to get Google to index the podcast. It would be really bad to tell robots to "no-index" my RSS feed, since that would mean my podcast disappears.

I'm having trouble understanding why Google wants to place an RSS feed inside its main SERPS. What possible benefit is it for any normal user to click a page and get a set of gibberish as a result?

not2easy

2:31 am on Feb 26, 2018 (gmt 0)

Surely the RSS feed is not the ideal place for people to find this podcast. Removing the RSS feed from the index would only noindex the podcast if that is the only place it is found.

keyplyr

2:35 am on Feb 26, 2018 (gmt 0)

I'm having trouble understanding why Google wants to place an RSS feed inside its main SERPS.

Sorry, there is only one search index (SERP) as I said above.

You don't have to "noindex' this file, that was just one suggestion.

Point is, there's no way to control where Google places this link. That is up to Google and the way it fulfills search term requests.

james007

2:46 am on Feb 26, 2018 (gmt 0)

Understood. My original question was:

Is this a bug - is there any time where Google would want to return an XML document for a search?

My conclusion is that it must be.

james007

2:50 am on Feb 26, 2018 (gmt 0)

@not2easy: "Surely the RSS feed is not the ideal place for people to find this podcast."

An RSS feed is the *only* way. A podcast is essentially an RSS feed with enclosures. That's how you publish a podcast. If I were to block the RSS feed from being visible by robots, all the robots that ping my RSS feed - iTunes, Stitcher, etc - wouldn't be able to any more, and my podcast would fall off the entire internet. This would be a bad thing.

My question here is really why Google would ever want to return a content-type:text/html document in the main SERPS. Not just that, but Google is now highlighting the RSS feed as one of my sitelinks, which is clearly not great user experience.

keyplyr

2:52 am on Feb 26, 2018 (gmt 0)

Is this a bug

No, various file types sometimes show in the SERP.

james007

3:14 am on Feb 26, 2018 (gmt 0)

They do, but they're mostly PDF files or similar which are humanly readable. Again: I cannot conceive of a single time where someone who does a web search for "cheese" wants to see, in the search results, an XML document. This simply cannot be good user experience for anyone. (A search for "cheese filetype:xml"? Absolutely.)

OK, well, I've done some more research. While many RSS feeds use Content-type: text/xml, the "correct" is Content-type: application/rss+xml so I've changed it to that, and we'll see what effect it has. Here's hoping it stops being, at least, highlighted as a site-link.

Samizdata

3:51 am on Feb 26, 2018 (gmt 0)

Obviously, if you click on the RSS feed then it rewards you with a page full of gibberish.

Not if you use an XSL stylesheet linked to a CSS file it doesn't.

The feed will display nicely in a browser, formatted however you wish.

"XSL (eXtensible Stylesheet Language) is a styling language for XML."

Basic webmaster stuff, I would have thought.

...

james007

6:49 am on Feb 26, 2018 (gmt 0)

But I'm not; and nor do I want that.

Basic search engine stuff is not to return gobbledegook, I would have thought; but it's clear that I've asked a more complicated question than this forum can deal with. Thanks for your help

<snip>

[edited by: goodroi at 2:59 pm (utc) on Feb 26, 2018]
[edit reason] TOS #19 [/edit]

Samizdata

12:05 pm on Feb 26, 2018 (gmt 0)

Basic search engine stuff is not to return gobbledegook

The search engine will only return gobbledegook if you omit the formatting.

it's clear that I've asked a more complicated question than this forum can deal with

I have given you the perfect solution for your problem.

Having your XML podcast feed ranking well in the SERPs is highly desirable behaviour.

You just need to format it so it displays nicely in a browser.

This is not difficult, and it is not new.

[w3schools.com ]

<snip>

I wish you luck.

...

[edited by: goodroi at 2:59 pm (utc) on Feb 26, 2018]
[edit reason] TOS #19 [/edit]