Robots.txt tester
TechnicalPaste a robots.txt file, pick a crawler and a URL, and see whether it is allowed or blocked and exactly which rule decides it. Free, no signup, and it runs entirely in your browser.
How to use it
- Paste the full contents of your robots.txt file into the text box.
- Enter or pick a user-agent, such as Googlebot or GPTBot.
- Type the URL or path to test and read the allowed or blocked verdict with its deciding rule.
Why test before you publish
A single misplaced slash in robots.txt can block an entire section of a site, and the mistake often goes unnoticed until rankings or traffic drop. Testing the file against your most important URLs takes a minute and catches problems before crawlers ever see them. This tool runs entirely in your browser, so you can test a draft that is not live yet.
How crawlers pick a group
Robots.txt rules are organized into groups, each starting with one or more User-agent lines. A crawler looks for the group that names it most specifically and follows only that group. If none names it, it falls back to the User-agent: * group. If there is no matching group at all, everything is allowed.
This catches people out. If you add a group for GPTBot, GPTBot ignores every rule in the * group, including any general blocks you meant to apply to all crawlers. The tester shows which group it used so you can see this happening.
How the deciding rule is chosen
Within the chosen group, the rule with the longest matching path wins. If an Allow rule and a Disallow rule match with the same length, Allow wins. This is the behavior documented by Google and followed by the major search engines. So with Disallow: /checkout/ and Allow: /checkout/help, the help page stays crawlable because its rule is longer and more specific.
Two wildcards are supported. An asterisk matches any sequence of characters, so Disallow: /*?sort= blocks any URL containing that sort parameter. A dollar sign anchors the end of the URL, so Disallow: /*.pdf$ blocks URLs that end in .pdf. Paths are case sensitive, which means /Blog/ and /blog/ are different rules.
What to test
Start with your money pages: home, key service or product pages, and top blog posts. Check them for Googlebot and Bingbot. Then test the URLs you intended to block, such as cart, internal search and filtered URLs, to confirm the rules actually catch them.
If you manage AI crawler access, test with OAI-SearchBot, ClaudeBot, PerplexityBot or Google-Extended too. Blocking a training crawler is a business choice, but blocking a search or retrieval crawler can keep your pages out of that assistant's answers. The AI crawler robots.txt generator explains the difference. For a full review of crawl and index issues across your site, see our SEO audit.
Questions, answered
Does this tester behave the same way as Google?
It follows the published robots.txt standard and Google's documented rules: most specific user-agent group, longest matching path, Allow wins ties, and support for the * and $ wildcards. Edge cases with malformed files can differ, so for a live site you can also confirm results with the robots.txt report in Google Search Console.
Why is my URL allowed when I blocked it for all crawlers?
Most likely the crawler you are testing has its own user-agent group. A crawler follows only the most specific group that names it and ignores the general * rules. Copy the blocks you need into that crawler's group, or remove the dedicated group if you meant it to follow the general rules.
Are robots.txt paths case sensitive?
Yes. Paths in rules are matched case sensitively, so Disallow: /Private/ will not block /private/. User-agent names, on the other hand, are matched case insensitively. If your site serves the same page at different capitalizations, it is better to fix that with redirects than to write rules for every variation.
Can I test a full URL instead of a path?
Yes. You can paste a full URL and the tester will check its path and query string against the rules. Remember that robots.txt applies per host, so the rules you paste only govern URLs on the domain or subdomain where that file is published, not other subdomains.
Find out what is holding your rankings back
Send us your site. Within 2 business days you get a written audit, an AI visibility check and a short fix list ranked by impact. Free, with no contract attached.