Forum Moderators: open

Message Too Old, No Replies

sitemap.xml link in robots.txt ?

Is this configuration advisable?

         

Suki_Cotton

9:29 pm on Dec 8, 2020 (gmt 0)

5+ Year Member



Is it advisable to configure a robots.txt file like this:

Sitemap: https://www.example.com/sitemap.xml
User-agent: *
Allow: /

not2easy

10:47 pm on Dec 8, 2020 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Hi Suki_Cotton and welcome to WebmasterWorld [webmasterworld.com]

The line for your sitemap's URL should be the last line after all the robots directives. Google offers instructions: [developers.google.com...]

phranque

11:24 pm on Dec 8, 2020 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



welcome to WebmasterWorld, Suki_Cotton!

The line for your sitemap's URL should be the last line after all the robots directives.

i would also include a blank line after each User-agent group of directives.

Suki_Cotton

11:53 pm on Dec 8, 2020 (gmt 0)

5+ Year Member



Thank you. In Google I read: "Each group consists of multiple rules or directives (instructions), one directive per line." I interpret this to say that blank lines are not to be inserted between directives, and the following would be OK…

User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml

Also, can you please confirm that within the site, this is the correct way to reference the sitemap.xml

phranque

12:23 am on Dec 9, 2020 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



I interpret this to say that blank lines are not to be inserted between directives

it is not a requirement, but see Robots.txt Specifications [developers.google.com]:
Note the optional use of white-space and empty lines to improve readability.

like this:
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml