To create a robots.txt file, set your allow and disallow rules and sitemap URL in the Robots.txt Generator and copy the result to a file named robots.txt in your site root. It tells crawlers which paths they may fetch.
How to use the Robots.txt Generator
- Choose the user agent, or use all crawlers.
- Add allow and disallow rules.
- Add your sitemap URL.
- Save the output as robots.txt at the root of your domain.
Worked example
User-agent: * with Disallow: /admin/ and Sitemap: https://example.com/sitemap.xml blocks the admin area and points crawlers to your sitemap.
Tips and common mistakes
- robots.txt is a request, not security; do not use it to hide private pages.
- Disallowing a page does not remove it from search if it is linked elsewhere.
- Never block CSS or JavaScript that pages need to render.
Frequently asked questions
Where does robots.txt go?
At the root of the domain, such as example.com/robots.txt.
Does it stop pages being indexed?
Not reliably. Use noindex for that.
Is it private?
Yes.