Improve search with Robots.txt and the sitemap

Creating a robots.txt file lets search crawlers accurately find and identify the website's sitemap while reducing the potential workload to the site. If a robots.txt file is not included, the sitemap must be submitted manually to the search engine directly.

Multi-language sites include the hreflang attribute in the URL tag for all storefront webpages, sitemaps, and pages. This attribute identifies the specific language on the page for search engines and enables them to serve relevant language results to users searching in that language.

Submitting your sitemap directly to a search engine enables you to receive status information and any processing errors. See each search engine's documentation.

Add the following line, including the full URL, to the sitemap:

Sitemap: http://www.example.com/sitemap.xml

Because the directive is independent of the user agent-line, you can place it anywhere in the file. If a sitemap index file exists, you can include its location in the file. You do not need to include each sitemap in the file. However, if you want to specify more than one sitemap per robots.txt file, add the following URL to the file:

Sitemap: http://www.example.com/sitemap-host1.xml

Sitemap: http://www.example.com/sitemap-host2.xml

HTTP requests perform an action on a specific server resource. Submitting a sitemap request with a HTTP request ensures that the search engine indexes the site.

To submit a sitemap using an HTTP request replace <searchengine_URL> with the URL provided by the specific search engine.

You can issue the HTTP request using wget, curl, or another content retriever. A successful request returns an HTTP 200 response code; if you receive a different response, resubmit your request. The HTTP 200 response code indicates that the search engine has received the sitemap. However, the returned HTTP 200 code does not validate the sitemap or the URL. To ensure validation, set up an automated job to generate and submit sitemaps on a regular basis.