Shopify stores can now edit their robots.txt file, giving owners more control over how search engines crawl their website.
Shopify CEO Tobi Lutke broke the news on Twitter tonight, adding that this feature may not be available to everyone until Monday.
All Shopify stores begin with the same robots.txt file, which the company says is suitable for most websites, but it can now be edited through the robots.txt.liquid theme template.
Also read: seo strategy 2021
Site owners can make the following edits to the robots.txt file:
- Allow or disallow certain URLs from being crawled
- Add crawl-delay rules for certain crawlers
- Add extra sitemap URLs
- Block certain crawlers
Shopify recommends using Liquid to add or remove instructions from the robots.txt.liquid template because it retains the ability to keep the files updated automatically.
How To Edit Robots.txt in Shopify
Here are the steps to customizing a Shopify store’s robots.txt file:
- From your Shopify admin, go to Online Store > Themes.
- Click Actions, and then click Edit Code.
- Click Add a new template, and then select robots.
- Click Create template.
- Make the changes that you want to make to the default template.
- Save changes to the robots.txt.liquid file in your published theme.
Also read: Twitter Unmentions
Alternatively, the site owner can remove content from the template and replace it with plain text rules, but Shopify does not recommend this because it will not update automatically.
Changes to ThemeKit or the command line will preserve the robots.txt.liquid file. Uploading a theme from the theme section of the Shopify backend will not import robots.txt.liquid.
Shopify recommends editing the robots.txt.liquid file as an unsupported customization, which means that their support team cannot help with editing.
“Using this feature incorrectly may result in the loss of all traffic,” Shopify warned. However, this is true for any site’s robots.txt file.
Source: robots.txt liquid template