Test whether pasted robots.txt rules allow a crawler to fetch a URL path.
Use the Robots.txt Tester
What this tool helps you check
A robots.txt file communicates crawler access rules, but its user-agent groups and wildcard matching can be easy to misread. Paste rules, enter a full URL and choose a crawler. The tester selects the most specific applicable group and reports the rule that decides the path.
How to use it
- Paste the robots.txt rules exactly as served
- Enter the target page URL and crawler name
- Check the matching allow or disallow rule in the result
Interpreting the result
This is a rules simulator. It does not confirm that the live robots.txt file is reachable, that the page is indexed or that every crawler implements extensions identically. Test live behavior in the relevant search engine tools.
Limitations and next steps
Check important decisions against your original business data and the relevant provider documentation. Results describe the inputs supplied, rather than a guarantee of future outcomes.
Related free tools
Continue with Redirect Checker, Canonical Tag Checker, XML Sitemap Validator or browse all free tools.
Frequently asked questions
Does Allow override Disallow?
When matching patterns have equal length, the allow rule wins in this tester. A longer matching pattern takes precedence.
Is this Robots.txt Tester free?
Yes. This tool is available without an account. URL inspection requests are rate limited to keep the service responsive.