robots.txt Tester
Test a URL against the live robots.txt or your own draft. We show you the rule that applies and check all important bots at once.
Free
No sign-up
No tracking.
How is it tested?
- According to the official standard RFC 9309 and Google’s implementation: most specific user-agent group, longest matching rule, Allow wins ties.
- Supports the wildcards
*and$as well as percent-encoding. - Detects syntax errors, unknown directives, relative sitemap URLs and directives Google ignores such as crawl-delay or noindex.
Frequently asked questions
Does robots.txt prevent indexing?
No. robots.txt only controls crawling. Blocked URLs can still be indexed if they are linked. To exclude a page you need noindex, and for that the page must not be blocked.
What if robots.txt is missing?
On 404 everything may be crawled. On 5xx errors Google assumes a full block.