Test whether pasted robots.txt rules allow a crawler to fetch a URL path.
Use the Robots.txt Tester
What this tool helps you check
A robots.txt file communicates crawler access rules, but its user-agent groups and wildcard matching can be easy to misread. Paste rules, enter a full URL and choose a crawler. The tester selects the most specific applicable group and reports the rule that decides the path.
How to use it
- Paste the robots.txt rules exactly as served
- Enter the target page URL and crawler name
- Check the matching allow or disallow rule in the result
Interpreting the result
This is a rules simulator. It does not confirm that the live robots.txt file is reachable, that the page is indexed or that every crawler implements extensions identically. Test live behavior in the relevant search engine tools.
Limitations and next steps
Does Allow override Disallow?|When matching patterns have equal length, the allow rule wins in this tester. A longer matching pattern takes precedence.
Related free tools
Continue with Redirect Checker, Canonical Tag Checker, XML Sitemap Validator or browse all free tools.