Robots.txt Tools
Generate and validate robots.txt files to control how search engines crawl your website.
Generator
Create a robots.txt file with custom allow and disallow rules for any user-agent.
Validator
Validate and analyze any robots.txt file. Check syntax and test URL rules.
How to use the robots.txt tools
Two tools, one file. Build it with the generator, then test it with the validator before it goes live.
- 01
Use the Generator when you don't have a file yet
Build groups for all crawlers or for specific ones, add Allow and Disallow paths and a sitemap, then copy the text.
- 02
Use the Validator when you already have one
Paste the live file or a draft, read the warnings, and test real URLs against Googlebot, * or any other group in the file.
- 03
Upload to the root of each host
The file must load at /robots.txt on every host it should cover. shop.example.com and example.com each need their own, and so do the http and https versions if both are served.
Which directives crawlers actually obey
RFC 9309, published in 2022, standardised the core of robots.txt: User-agent, Allow and Disallow. Other lines are extensions, and support varies. Google publishes the list it follows; anything outside it is silently ignored.
| Directive | What it does | Bing | |
|---|---|---|---|
| User-agent | Starts a group for one crawler, or * for all | Yes | Yes |
| Disallow | Path prefix the group shouldn't crawl | Yes | Yes |
| Allow | Exception inside a disallowed path | Yes | Yes |
| Sitemap | Full URL of an XML sitemap | Yes | Yes |
| Crawl-delay | Seconds to wait between requests | Ignored | Yes |
| Noindex | Was an unofficial way to drop pages | Ignored since Sept 2019 | No |
| Host | Preferred domain, a Yandex extension | Ignored | No |
The validator flags Noindex, Host and other unknown lines as warnings, so you can spot rules that look active but do nothing.
What happens when the file is missing or broken
Crawlers request robots.txt before anything else, and the HTTP status changes how they behave. A 404 or other 4xx response means there are no rules, so everything may be crawled. That's fine for a small site that doesn't need a file.
A 5xx server error is treated far more cautiously. Google reads it as "don't crawl yet" and holds back on the whole site until the file can be fetched, so a broken robots.txt route can quietly stop crawling for days. If your host or CDN serves the file dynamically, check that it returns 200 after every deploy. Google also reads only the first 500 KiB, which you'll only hit with a very long, generated file.
Reading a typical robots.txt line by line
Most real files are short. Here's one for a small online shop, with what each line tells crawlers. Rows in the same group sit together in the file, and groups are separated by a blank line.
| Line | What it means |
|---|---|
| User-agent: * | The rules below apply to every crawler without its own group |
| Disallow: /cart | Don't crawl the cart, or anything starting with /cart |
| Disallow: /*?sort= | Skip any URL that sorts a listing, wherever it appears |
| Allow: /cart/help | Except this help page, which is a longer and so stronger match |
| User-agent: GPTBot | A separate group for OpenAI's training crawler |
| Disallow: / | That crawler may not fetch anything |
| Sitemap: https://shop.example/sitemap.xml | Where to find every page that should be crawled |
One detail surprises people: GPTBot follows only its own group and ignores the * rules completely. If you want it to keep the cart rules as well, repeat them in its group. The validator shows each group's rules separately, which makes this easy to check.
What robots.txt is good for
Keeping crawlers off low-value URLs
Internal search results, cart pages and endless filter combinations waste crawling on large sites. Disallow them so crawlers spend their time on real pages.
Hiding a staging or test site
One Disallow: / keeps polite crawlers off a test server. Add a password too, since robots.txt is only a request.
Choosing which AI crawlers may visit
Named groups for GPTBot, ClaudeBot or Google-Extended let you opt out of AI training without touching search engines.
Pointing crawlers at your sitemap
A Sitemap line helps engines you haven't registered with, such as smaller search engines, find every page.