How to Write a Robots.txt File
Robots.txt is a plain text file at the root of your site, such as https://example.com/robots.txt, that tells search engine crawlers which paths they may crawl. It uses simple lines: User-agent names the crawler, Disallow lists paths to skip and Sitemap points to your sitemap. Robots.txt controls crawling, not indexing.
A simple example
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
- User-agent: * applies the rules to all crawlers.
- Disallow: /admin/ asks crawlers not to crawl anything under /admin/.
- Sitemap: gives the full URL of your XML sitemap.
The robots.txt generator builds this file from your sitemap URL and a list of paths to block.
Rules worth knowing
- The file must be at the root of the host. A robots.txt in a subfolder is ignored.
- Paths are case sensitive.
- An empty Disallow line, or no robots.txt at all, means everything may be crawled.
- Disallow: / blocks the whole site, a common and costly mistake on a launched site.
- Each subdomain needs its own robots.txt.
Robots.txt does not hide pages
A blocked page can still appear in search results if other sites link to it, usually without a description. To keep a page out of search results, let it be crawled and add a noindex robots meta tag, or protect it with a password. Never rely on robots.txt for private content, since anyone can read the file.
Adding your sitemap
A sitemap lists the URLs you want crawled. If you do not have one, the XML sitemap generator creates a simple sitemap from a list of URLs. Add its address to robots.txt and submit it in Google Search Console.
Common things to block
- Admin and login areas.
- Internal search results pages.
- Cart and checkout steps.
- Endless filtered or sorted versions of the same listing.
For duplicate versions of a page, such as those with tracking parameters, a canonical tag is usually better than blocking. The canonical URL builder gives the clean URL to use, and the meta tag generator writes the canonical and robots tags.
Test before you publish
Check the file in Google Search Console's robots.txt report after changes, and confirm important pages are not blocked.
Questions and answers
Does robots.txt stop a page from being indexed?
No. It stops crawling. A blocked page can still be indexed from links. Use a noindex tag to keep a page out of search results.
Where do I put robots.txt?
At the root of the domain, for example https://example.com/robots.txt.