SEO Suite Ultimate M2

sitemap.xml and robots.txt Issues When Using DocumentRoot /pub

When using /pub as DocumentRoot, you can face the following issues:

  • sitemap.xml:

    The file sitemap.xml is solely created in the Magento root directory. In the column ‘Link for Google’ it incorrectly says: http://example.com/sitemap.xml. Further on, after clicking on the sitemap entry in the table, changing the path to /pub, and ‘saving and generating’, sitemap.xml is created in the pub directory and you once again see incorrect http://example.com/pub/sitemap.xml in the same column.

  • robots.txt:

    The file robots.txt is solely created in the Magento root directory, but not in the /pub directory, which makes it not accessible for web requests.

MageWorx developers offer the following issue solution options

First Option: Problem Solution Without any Additional Settings (Not Advisable)

  • for XML Sitemap

    First, you must specify the path /pub/. Then you will see ‘Link for Google’ in the admin panel as http://example.com/pub/sitemap.xml . However, xml sitemap will be available via URL http://example.com/sitemap.xml - this is exactly what should be entered in the Google Search Console.

  • for robots.txt

    First, store 2 robots.txt files on the server, one file must be stored in <your Magento install dir>, another one - in <your Magento install dir>/pub.

    Each and every time you make changes in <your Magento install dir>/robots.txt, make sure to duplicate the file in pub/.

Second Option: Use nginx Configuration

Add the following code lines to nginx config for your website:

location ~ ^/(robots\.txt|sitemap.*\.xml)$ {
    alias <your Magento install dir>/$1;
}

Third Option: Use Symlinks

This will fail if a symlink already exists.

1) To create a new symlink, complete the following actions:

ln -s <your Magento install dir>/robots.txt <your Magento install dir>/pub/robots.txt
ln -s <your Magento install dir>/sitemap.xml <your Magento install dir>/pub/sitemap.xml

2) To create or update a symlink, complete the following actions:

ln -sf <your Magento install dir>/robots.txt <your Magento install dir>/pub/robots.txt
ln -sf <your Magento install dir>/sitemap.xml <your Magento install dir>/pub/sitemap.xml

When using clauses number 1 and 2, the symlink ownership will remain the same as in the source file.

You can use chown to change the symlink’s ownership manually. To change the ownership of a symlink, complete the following actions:

cd <your Magento install dir>/pub
chown -h <Magento file system owner>:<owner group> robots.txt
chown -h <Magento file system owner>:<owner group> sitemap.xml

MageWorx hopes that the offered solutions will be of help when dealing with the above stated issues.