Generator
Generate a robots.txt file with custom rules.
Validator
Validate and analyze any robots.txt file.
Robots.txt Generator
Generate a robots.txt file with custom rules.
Build your robots.txt rules below, then copy the output.
Allow
Disallow
User-agent: * Allow: / Disallow:
How to create a robots.txt file
Most sites need one group for all crawlers, a few Disallow paths and a sitemap line. Start there.
- 01
Start with the * group for every crawler
The first group targets User-agent: * and starts with Allow: / and an empty Disallow, which together allow everything. Add Disallow paths here for folders no crawler should visit.
- 02
Add groups only for crawlers that need different rules
Click to add a group and pick from Googlebot, Bingbot, Slurp, DuckDuckBot or Baiduspider, or type any other name, such as GPTBot. A crawler with its own group ignores the * group completely.
- 03
Add your sitemap URL
Enter the full address, including https://. It's added as a Sitemap line at the end of the file, where every crawler can read it.
- 04
Copy the output and upload it to your site's root
Save it as robots.txt in the top folder, so it loads at https://yoursite.com/robots.txt. A file in a subfolder is ignored. Test it in the validator before you upload.
Ready-made rules for common situations
These are the patterns most sites need. Paths are matched from the start of the URL, and * stands for any run of characters.
| You want to | Group | Rule |
|---|---|---|
| Block a whole staging site | User-agent: * | Disallow: / |
| Keep crawlers out of admin and checkout | User-agent: * | Disallow: /admin/ and Disallow: /checkout/ |
| Stop internal search pages being crawled | User-agent: * | Disallow: /search |
| Skip sorted and filtered duplicates | User-agent: * | Disallow: /*?sort= |
| Keep a folder blocked but one page open | User-agent: * | Disallow: /private/ and Allow: /private/press-kit |
| Opt out of OpenAI's training crawler | User-agent: GPTBot | Disallow: / |
| Opt out of Google's AI training | User-agent: Google-Extended | Disallow: / |
Google-Extended isn't a separate crawler. Googlebot does the crawling, and this token tells Google whether the pages may be used to train its Gemini models. Blocking it doesn't affect Google Search. Other AI crawlers you may want named groups for include ClaudeBot, CCBot and PerplexityBot.
Two rules people get backwards
First, robots.txt isn't a privacy tool. The file is public, so listing /secret-reports/ in it tells everyone where to look. Polite crawlers respect it and others don't. Anything private needs a login, not a Disallow line.
Second, Disallow doesn't remove a page from Google. If the page is already indexed or linked from elsewhere, blocking it stops Google from reading the noindex tag that would have removed it. To drop a page from results, leave it crawlable and add noindex. Once it's gone, you can block it if you want.
When you'd build a new file
Launching a new site
Start from the default allow-all file, add the sitemap, and you have a valid robots.txt in under a minute.
Taking a staging copy out of search
A single Disallow: / for * keeps polite crawlers off a test server. Just remember to swap it out before launch.
Deciding on AI crawlers
Add separate groups for the AI crawlers you want to block, while keeping search engines on the default rules.