---
title: Edit the Robots.txt file
slug: configured-commerce/docs/edit-the-robots-txt-file
docTags: 
createdAt: 2026-08-13T10:46:30.000Z
---

The robots.txt file prevents web crawlers, such as Googlebot, from crawling certain pages of your site. The file contains a list commands and URLs that dictate the behavior of web crawlers on a website. Keep in mind that not all web crawlers adhere to the directives in the robots.txt. There are several ways to make edits to the Robots.txt file. This article provides the steps for editing the robots.txt file through the Content Management System (CMS).

:::hint{type="info"}
**Note**

The robots.txt file may already contain some sitemap URLs which are generated by the system and cannot be edited.
:::

1. Go to **Admin Console**.
2. Click **View Website**.
3. Toggle **Content Editor** to display **On**.
4. Select the desired website from the list of websites.
5. Click **Content Tree**.
6. Click **Edit**.
7. Click **Home** and expand the content tree.
8. Click the **Robots.txt** file.
9. Click **Edit** this page to make edits to the page.
10. (Optional) Add the **Disallow: /** command in the **Text** field, and click enter or return to move to the next line.

    ![](https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/mDFRo0sUghTvaPDtOYCMK-20260720-144617.png)

:::Paragraph{indent="1"}
You can increase the size of the text box by dragging the resize area in the lower right corner.
:::

::Image[]{src="https://api.archbee.com/api/optimize/BrbvVORKJ5iIeSMZye5kI/0c_YBDrEcBgN9KtGAFxJ--20260720-144617.png" size="100" isUploading="false" showCaption="false" indent="2"}

11. Add the desired URLs that should be ignored by web crawlers. Enter each URL on its own line and spaces between lines are not necessary. Precede the URLs with **sitemap:** `sitemap: https://website.com/pagename`
12. Click **Save** after adding the URLs. They now display on the page.
13. Click **Publish**.
14. Decide whether to publish immediately or sometime in the future. Click **Publish**.