Robots.txt Tester
Test pasted robots.txt rules locally against crawler tokens and paths, with winning lines and CSV export.
This tool runs entirely in your browser. Your data is never uploaded, never stored, and never leaves your device.
Models pasted robots.txt Allow/Disallow rules for exact crawler tokens and paths, showing applicable groups and the winning source line.
How to use it
- 1Paste or load UTF-8 robots.txt and enter exact crawler product tokens.
- 2Enter paths or HTTP(S) URLs and test the bounded rule matrix.
- 3Inspect parser notices, winning lines and the downloadable CSV.
Example
- Input
- User-agent: * Disallow: /private/ Allow: /private/public/ Test Googlebot: /private/public/help
- Output
- Allowed by line 3: Allow: /private/public/
What happens to your data
Pasted files and paths stay local. No website is fetched or changed. CSV reports include supplied paths/rules; review before sharing.
Last updated October 2026
Test pasted robots.txt rules before publishing a change. Enter crawler product tokens and several URL paths, then inspect which group and Allow or Disallow line controls each decision. A matrix can reveal an overly broad wildcard, a missing specific group or an Allow exception that is shorter than the blocking rule. Nothing is fetched or changed on your website.
The result models a bounded set of RFC 9309 rules with explicitly named product-token matching. It does not prove what Google has currently cached, whether a URL is indexed, or whether a crawler will visit or cite it. A robots.txt file is a public request to cooperative crawlers, not authentication or a privacy boundary. Protect private content with real access controls.
How it works
Paste UTF-8 robots.txt content, including an empty file when that is what you intend to test. Field names and crawler tokens are matched case-insensitively. Each requested crawler token matches exact named user-agent entries. Matching repeated groups are combined; a specific token uses its named groups instead of the wildcard groups. If no named group exists, wildcard groups apply. If neither exists, there is no applicable rule. The tester does not infer crawler aliases or inheritance, so enter the precise product token you want to model.
Consecutive user-agent entries before any Allow/Disallow record belong to the same group. A new user-agent after rules starts another group. Unknown records such as Sitemap and Crawl-delay do not end a group and do not affect this access test. Empty Allow or Disallow patterns do not restrict access. Rules before the first user-agent and malformed lines are ignored with notices. This strict parser does not guess at missing colons or common directive misspellings.
Paths are compared from their beginning and remain case-sensitive. An unanchored /fish pattern also matches /fishheads; it is not automatically a directory boundary. The wildcard * matches zero or more characters, and a dollar sign only at the end anchors the rule to the end of the path plus query. For example, /*.pdf$ matches /guide.pdf but not /guide.pdf?download=1. To match a literal asterisk or dollar sign, use its percent-encoded form in the rule.
Among matching rules, the longest normalized pattern is selected. Allow wins an equal-length conflict. The report gives the winning rule's original line number and the start lines of the applicable groups. If no path rule matches, access is allowed under these rules. The exact /robots.txt URI is implicitly allowed by RFC 9309. A URL fragment is ignored because it is not sent in an HTTP request, while the query remains relevant to matching.
Full HTTP(S) URLs are parsed to their normalized path and query. They all use the same pasted rule file, regardless of hostname; the tester does not verify that this file actually belongs to those origins. Dot segments and other path syntax may be normalized by the browser URL parser. Raw Unicode is converted to percent-encoded UTF-8 for comparison. Percent-encoded unreserved ASCII is decoded and other percent escapes are normalized to uppercase. Reserved encoded separators remain encoded, so an encoded slash is not silently treated as a directory separator. Invalid percent escapes are rejected or reported on rules.
This is not a complete emulation of Google's legacy C++ parser or every crawler. Google-specific typo tolerance, index.html aliases and some legacy percent-encoding behavior can differ. Fetch status codes, redirects, caching, non-UTF-8 recovery, automatic product aliases and nonstandard directives are outside scope. The primary specification and Google's documentation are linked so the profile is reviewable rather than hidden behind an SEO score.
The tester accepts up to 500 KiB, 10,000 lines, 100 groups and 300 Allow/Disallow rules. A rule has at most 1,024 characters, each test path at most 2,048, and the matrix at most 200 decisions from up to ten tokens and fifty paths. A shared matching-work budget stops pathological wildcard combinations without presenting an incomplete matrix as complete. These are local testing limits, not recommendations for a site's robots.txt size or complexity.
The CSV contains your test paths, winning rules and group locations. Formula-like fields receive an apostrophe prefix before spreadsheet export. Downloads may reveal private-looking paths: robots.txt itself is public, and this tool does not anonymize a report. Inputs are processed locally, never fetched from an entered origin and not intentionally saved in browser storage. Editing any field removes stale decisions.
Worked example: A longer Allow exception wins
- Input
- User-agent: * Disallow: /private/ Allow: /private/public/ Crawler: Googlebot Paths: /private/report and /private/public/help
- Result
- /private/report: blocked by line 2 /private/public/help: allowed by line 3
The more specific Allow prefix controls the second path. This is a rule-model result, not an indexing result.
Common use cases
- Review wildcard and end-anchor changes before publishing robots.txt
- Compare exact crawler tokens against the same planned rule file
- Find the line responsible for blocking a particular path
- Save a local rule-decision matrix for an SEO change review
Frequently asked questions
Does Allowed mean Google will index the page?＋
No. It means the pasted rules permit the tested path under this model. Fetchability, noindex, canonical choices, content assessment and actual crawl behavior are separate matters. A blocked URL may still appear in an index.
Can robots.txt protect private content?＋
No. It is public and cooperative crawlers may follow it, but it is not authorization. Use authentication and appropriate server-side access controls for private resources.
Why does a specific crawler ignore my wildcard group?＋
A matching exact product-token group takes precedence here. Its repeated groups are combined; global wildcard rules are used only when no named match exists. No automatic crawler aliases are inferred.
Why does a PDF rule stop matching when a query is added?＋
A trailing $ anchors the end of the path plus query. /*.pdf$ matches /guide.pdf but not /guide.pdf?download=1. Remove or change the anchor only if that broader behavior is intended.
Is this identical to Googlebot?＋
No. It is an explicitly bounded RFC rule model. Google's complete implementation includes legacy tolerance and crawler behavior outside this tool, including fetch, cache and some parser quirks. Review notices and the linked primary documentation.
Does the tester change or resubmit my robots.txt?＋
No. It tests pasted content locally. There is no write, fetch, deployment, sitemap submission or indexing-request action.
Related tools
Sitemap Validator
Inspect sitemap XML fields, dates, URLs and duplicates locally, without a live crawl.
Website Meta Tag Checker
Inspect a public URL or pasted HTML for title, description, canonical, robots and social sharing tags.
HTTP Header Inspector & Compare
Inspect and compare pasted HTTP response headers locally, keeping repeated fields and response blocks.
HTTP Status Codes & Log Analyzer
Look up HTTP status codes and analyze response lines or access logs with counts and explanations.