Generator
Generate a robots.txt file with custom rules.
Validator
Validate and analyze any robots.txt file.
Robots.txt Validator
Validate and analyze any robots.txt file.
Paste your robots.txt content below to validate and analyze it.
How to test a robots.txt file
Paste the file, check the warnings, then test the URLs you care about against the crawlers you care about.
- 01
Paste the file's contents
Open yoursite.com/robots.txt in a browser, copy everything and paste it in. The validator doesn't fetch files itself, which also means you can test a draft before it goes live.
- 02
Read the warnings and the parsed groups
Each User-agent group is listed with its Allow and Disallow rules, along with any Sitemap lines. Lines the parser doesn't understand, such as Noindex or Host, show up as warnings with their line number.
- 03
Test a path against a crawler
Pick a user-agent from the groups in your file and type a path such as /search?q=shoes, or paste a full URL. The result says Allowed or Blocked for that crawler.
How a crawler decides between Allow and Disallow
The validator follows the rules Google documents and RFC 9309 sets out. The crawler first picks the group whose User-agent matches its name; if none does, it uses the * group. Then it compares every rule in that group with the path. The longest matching rule wins, and if an Allow and a Disallow are the same length, Allow wins.
Two characters are special. An asterisk matches any run of characters, and a $ at the end means the URL has to end there. Everything else matches from the start of the path, so Disallow: /search also blocks /search-results. We ran this test file through the validator to show it:
| Path tested | Result | Why |
|---|---|---|
| /search-results | Blocked | Disallow: /search matches the start of the path |
| /search/help/faq | Allowed | Allow: /search/help is longer than Disallow: /search |
| /shoes?sort=price | Blocked | Disallow: /*?sort= matches anywhere after the wildcard |
| /shoes?color=red | Allowed | No rule matches, so the default is allowed |
| /guide.pdf | Blocked | Disallow: /*.pdf$ matches a URL ending in .pdf |
| /guide.pdf?v=2 | Allowed | The $ means it must end in .pdf, and this one doesn't |
| /blog/post | Allowed | Allow: /blog/ and Disallow: /blog/ tie, so Allow wins |
| /admin/ | Allowed | Disallow: /Admin/ is a different path, since paths are case-sensitive |
Mistakes the validator helps you catch
The costliest one is a leftover Disallow: / from a staging site. It blocks everything, and it's often copied to production when a site launches. Test your home page and a key product page against Googlebot every time robots.txt changes.
Next is blocking CSS and JavaScript. Google renders pages much like a browser does, so if /assets/ or /_next/ is disallowed, it may see a broken page. Directives Google doesn't support are another trap. Noindex in robots.txt stopped working in September 2019, and Crawl-delay is ignored by Google, though Bing still reads it.
When to run the validator
Finding the rule behind a Search Console warning
When Search Console reports "Blocked by robots.txt", test that exact URL here to see whether it's blocked for Googlebot or only for one group.
Checking a change before you deploy it
Paste the edited file and test the five or six URLs that matter most. It's quicker than waiting for Google to recrawl and report back.
Making sure AI crawler rules work
If you've added a GPTBot or ClaudeBot group, test a page against that agent and against * to confirm each one sees the rules you meant.
What the validator doesn't check
- It doesn't fetch your live file or check that it returns HTTP 200. A 5xx error on robots.txt can make Google pause crawling the site, so check the status code separately.
- It shows Allowed or Blocked, not the specific rule that decided it. Use the table above to work out which rule won.
- Agent matching is by exact name, falling back to *. Real crawlers sometimes match on part of the name, so test with the name as it appears in your file.
- It doesn't check file size. Google reads only the first 500 KiB of a robots.txt file and ignores the rest.
- A blank line or comment between a User-agent line and its rules ends the group in this parser, so keep each group's lines together.