Start SEO check
  1. oen.de
  2. All tools
  3. robots.txt Tester

robots.txt Tester

Test a URL against the live robots.txt or your own draft. We show you the rule that applies and check all important bots at once.

Free No sign-up No tracking.
Test your own robots.txt (optional)Leave empty to use the live robots.txt of the domain.

How is it tested?

  • According to the official standard RFC 9309 and Google’s implementation: most specific user-agent group, longest matching rule, Allow wins ties.
  • Supports the wildcards * and $ as well as percent-encoding.
  • Detects syntax errors, unknown directives, relative sitemap URLs and directives Google ignores such as crawl-delay or noindex.

Frequently asked questions

Does robots.txt prevent indexing?
No. robots.txt only controls crawling. Blocked URLs can still be indexed if they are linked. To exclude a page you need noindex, and for that the page must not be blocked.
What if robots.txt is missing?
On 404 everything may be crawled. On 5xx errors Google assumes a full block.

Related tools