A robots.txt file looks simple until a single rule blocks a page it was never meant to block. The difference between a correct file and a costly one often comes down to one path, one wildcard, or one rule that a longer, more specific rule should have overridden. The Robots.txt Tester on FreeToolZones lets you check your file before you trust it, by testing real URL paths against the rules and showing, for each path, whether it is allowed or blocked and which rule decided the outcome.
This page explains how the tester parses and matches rules, how to use it step by step, and how to interpret its verdicts. It also explains the matching standard the tool follows, so you can reason about edge cases with confidence instead of guessing.
What Is the Robots.txt Tester?
The Robots.txt Tester is a free browser-based tool that evaluates URL paths against robots.txt content you paste. You provide the full contents of a robots.txt file, choose the crawler identity to test, and list the paths you care about. The tool reports a verdict for each path, Allowed or Blocked, together with the matching rule that produced the verdict, or a note that no rule matched.
The tester follows the robots.txt matching standard. An asterisk in a pattern matches any sequence of characters, a dollar sign at the end of a pattern matches the end of the URL, the longest matching rule wins when several rules match, and an Allow rule wins when an Allow and a Disallow rule of equal length both match. A path with no matching rule is allowed. These rules are stated in the tool’s own output, so the verdicts are transparent rather than mysterious.
The tool arrives with example content already filled in, so you can click the test button immediately and see how it works. The example file allows all crawlers, disallows a private directory, allows one public page inside it, disallows PDF files, and declares a sitemap. The example paths test the homepage, a private page, the allowed public page, and a PDF file. Replace the examples with your own file and paths when you are ready to test for real.
All testing happens in your browser. The tool does not fetch a live robots.txt file from a URL, does not visit the paths you test, and does not send your file anywhere. It parses the text you paste and applies the matching rules to the paths you list, which makes it safe to use with draft files and staging configurations.
How to Use the Robots.txt Tester
The tester has three inputs and one button. The value comes from choosing test paths that cover the important cases, including the edge cases where rules interact.
- Paste your robots.txt content. Copy the full contents of your robots.txt file into the robots.txt Content box. Comments, lines beginning with a hash character, are ignored during parsing, so you can paste the file exactly as it exists, comments included. If the box is empty, the tool asks you to paste content before testing.
- Choose the User-agent to test. The User-agent field starts with an asterisk, meaning all crawlers. Leave it for a general test, or type a specific name such as Googlebot to test how that crawler is treated. When the file contains both general and specific groups, the tool prefers the specific groups that match the agent you entered, falling back to the general group when no specific group matches.
- Enter the URL paths to test. Type one path per line in the URL Paths box, such as “/”, “/private/page”, or “/files/report.pdf”. Full URLs are accepted; the tool extracts the path and query parts and tests those. A path typed without a leading slash is treated as if the slash were present. If no paths are entered, the tool asks for at least one.
- Click Test Paths. The tool parses the file and evaluates each path. If no User-agent groups or rules are found in the pasted content, the tool shows an error explaining the problem, so you know the file needs attention before any verdict can be trusted.
- Read the results table. The table has three columns: the path tested, the verdict (Allowed or Blocked), and the matching rule, labeled as the longest match that won. When no rule matches a path, the table says so explicitly, and the verdict is Allowed, because unmentioned paths are open by default.
- Review the matching explanation. Below the table, the tool restates the matching rules: wildcards, the end anchor, longest match wins, Allow wins ties, and no match means allowed. Use this explanation to understand any surprising verdict, then adjust the file and test again if needed.
Testing is iterative. When a verdict surprises you, find the matching rule shown in the table, reconsider whether that rule’s pattern is too broad or too narrow, edit the pasted content, and test again. Repeat until every important path produces the verdict you intend.
Key Features of the Robots.txt Tester
The tester combines faithful parsing with clear reporting. The features below describe exactly what it does.
Full file parsing with comment handling
The parser reads the pasted text line by line, strips comments, and groups rules under their User-agent lines. Lines without a colon are skipped, and empty Disallow values, which mean nothing is disallowed, are not treated as blocking rules. Sitemap lines and other non-rule lines do not affect verdicts. The result is a faithful model of how the file’s groups and rules relate to each other.
Specific agent groups preferred over general groups
When a file contains a group for a specific crawler and a general group for all crawlers, the tester applies the specific group’s rules to that crawler and ignores the general group for it. This matches the standard behavior and matters greatly in practice: a site that writes special rules for one crawler needs to know exactly what that crawler sees, not a blend of both groups.
Longest match wins, with Allow winning ties
When several rules match a path, the tester selects the longest matching pattern, measured by pattern length. When an Allow and a Disallow pattern of the same length both match, the Allow rule wins. This is the standard resolution, and it is why a specific Allow rule inside a broadly disallowed directory works: the longer, more specific pattern takes precedence over the shorter, broader one.
Wildcard and end-anchor support
Patterns support the asterisk wildcard, which matches any sequence of characters, and a trailing dollar sign, which anchors the match to the end of the URL. The tester converts each pattern into a matching expression that respects these characters while treating other special characters literally, so patterns behave as the standard intends.
Full URLs accepted, paths tested
You can paste complete addresses such as “https://example.com/private/page?ref=news” and the tool tests the path plus query portion. This is convenient when copying URLs from a browser or a report, because you do not need to strip the domain by hand. The tested path shown in the table reflects what was actually evaluated.
Clear verdicts with the deciding rule shown
Every row shows not only Allowed or Blocked but also the rule that decided it. That transparency turns the tester from a black box into a debugging aid: when a path is blocked unexpectedly, the table points directly at the rule responsible, and you can fix the file with confidence.
Practical Use Cases With Concrete Examples
Testing pays for itself the first time it catches a mistake before launch. The examples below show the tester at work in realistic situations.
Verifying a WordPress file before launch
A developer prepares a robots.txt file that disallows “/wp-admin/” but allows “/wp-admin/admin-ajax.php”. She pastes the file and tests “/”, “/wp-admin/”, “/wp-admin/admin-ajax.php”, and a blog post path. The table confirms the homepage and post are allowed, the administration directory is blocked, and the Ajax endpoint is allowed by the longer matching rule. The launch proceeds with confidence.
Catching a wildcard that blocks too much
A site owner adds a rule disallowing “/*.pdf$” to keep PDF files out of the crawl, then tests “/guides/handbook.pdf” and “/guides/handbook.pdf?download=1”. The first is blocked as intended, but the second is allowed, because the query string means the URL does not end with “.pdf”. The owner learns that the end anchor interacts with query strings and revises the rule to match the intended set of URLs.
Checking a staging file that blocks everything
Before launch, a developer tests the production file that should replace a restrictive staging file. She tests the homepage, key category pages, and the sitemap path, confirming all are allowed. She also tests a private path that should remain blocked. The check takes two minutes and prevents the classic launch failure of carrying staging restrictions into production.
Comparing crawler-specific groups
A large site maintains a general group and a stricter group for one aggressive crawler. The owner tests the same paths twice, once with the general agent and once with the specific crawler name, and compares the verdicts. The comparison confirms that the specific group applies to its crawler while other crawlers see the general rules, exactly as intended.
Auditing a file inherited from a previous developer
A new maintainer inherits a long robots.txt file with overlapping rules and comments in several styles. She pastes the file and tests a list of twenty important paths drawn from the site’s analytics. Three paths produce surprising verdicts, and the deciding rules shown in the table identify two outdated patterns. She removes them, retests, and documents the cleaned file for the team.
Tips and Best Practices
- Test every important path type. Include the homepage, a typical content page, a category page, a search or filtered URL, a file download, and every path you intend to block. Edge cases hide in the paths you forget to test.
- Test the allowed exception inside a blocked area. When you allow one page inside a disallowed directory, test both the directory and the exception. The longest-match rule should favor the exception, but only a test confirms it.
- Test with each relevant User-agent. If your file has crawler-specific groups, test the same paths under each agent name. A file can be correct for one crawler and wrong for another.
- Include query strings in test paths. Real URLs often carry parameters for tracking, filtering, or pagination. Test representative URLs with their query strings, because patterns and end anchors behave differently when a query is present.
- Keep a record of your test set. Save the list of paths and expected verdicts alongside the file. When the file changes, rerun the same set to catch regressions. A saved test set turns a one-time check into a repeatable safeguard.
- Remember that testing is not the same as live verification. The tester evaluates the text you paste. After uploading, confirm the live file at your domain plus “/robots.txt” contains the same text, because an upload error or caching can leave a different file in place.
Limitations of the Robots.txt Tester
The tester evaluates pasted text against the matching standard. It does not fetch the live file from your site, so it cannot detect an upload failure, a caching problem, or a difference between your draft and the published file. It does not visit the tested paths, so it cannot confirm that a path exists or returns the expected content. It applies the standard matching rules, but individual crawlers may have minor differences in behavior, and the tester cannot reproduce every crawler’s exact implementation.
The tool also focuses on Allow and Disallow verdicts. It does not evaluate sitemap lines, crawl delay, or other directives for correctness, and it does not judge whether your rules serve your goals. A file can test exactly as written and still be the wrong file for the site, if the rules themselves reflect a misunderstanding of what should be crawled. Pair testing with a review of intent: for each rule, ask what it protects and what it might accidentally exclude.
Frequently Asked Questions
Does the tester fetch my live robots.txt file?
No. You paste the file contents, and the tool tests that text. To check the live file, open it in a browser at your domain plus “/robots.txt”, copy its contents, and paste them into the tester. That way you test exactly what crawlers see.
What does it mean when no rule matches a path?
It means the file says nothing about that path, so the path is allowed. Robots.txt is permissive by default: only paths matched by a winning Disallow rule are blocked. The table states explicitly when no rule matched, so there is no ambiguity.
Why did an Allow rule win over a Disallow rule?
Either the Allow pattern was longer than the Disallow pattern, making it the more specific match, or both patterns were the same length and the tie was resolved in favor of Allow, as the standard requires. The table shows the winning rule, so you can see which case applies.
How do wildcards and dollar signs work in patterns?
An asterisk matches any sequence of characters, including none. A dollar sign at the end of a pattern matches only the end of the URL, so the pattern must match through the final character. Other characters in a pattern match literally. The tool’s output restates these rules below the results table.
Can I test full URLs, or only paths?
You can enter full URLs. The tool extracts the path and query parts and tests those, and the table shows the tested path. Entering paths directly works equally well, and paths typed without a leading slash are handled correctly.
Does a Blocked verdict mean the page cannot appear in search results?
No. Blocking crawling prevents the crawler from reading the page, but the URL may still appear in results if other pages link to it. The tester reports crawling verdicts only. Keeping a page out of an index requires a different mechanism, such as a noindex directive on the page itself.
Related Tools
These tools work with the tester to build and verify your crawling setup.
- Robots.txt Generator – builds a robots.txt file from rules you define.
- XML Sitemap Generator – creates the sitemap your robots.txt file can reference.
- Sitemap Validator – checks a sitemap for errors, duplicates, and missing entries.
- Canonical Tag Generator – declares the preferred URL for pages with similar versions.
- Htaccess Redirect Generator – builds redirect rules for moved pages.