Start SEO check
  1. oen.de
  2. All tools
  3. robots.txt Generator

robots.txt Generator

Decide what search engines, AI crawlers and SEO bots may crawl. The robots.txt is built live and can be downloaded right away.

Free No sign-up No tracking.
Default for all crawlers
One path per line. * and $ work as wildcards.
One path per line. * and $ work as wildcards.
AI crawlers
Training bots collect data for language models. Search bots fetch pages live for answers with citations. Blocking search bots means ChatGPT, Perplexity & co. will not cite you.

robots.txt

Done? Test single URLs against your draft in the robots.txt Tester

What can the generator do?

  • Your own rules: Disallow and Allow one per line, wildcards * and $ are supported.
  • AI bots: allow all, block training only or block all AI crawlers, with an up-to-date bot list.
  • SEO crawlers like AhrefsBot or SemrushBot can be blocked.
  • Sitemap reference at the end of the file.

Frequently asked questions

Does robots.txt prevent indexing?
No. It only controls crawling. Blocked pages can still be indexed if they are linked. Use noindex for that.
Should I block AI bots?
Blocking training bots does not hurt visibility. Blocking search bots like OAI-SearchBot or PerplexityBot means you will no longer be cited in AI answers.
Where does the file go?
Always in the root: https://www.example.com/robots.txt. Subdirectories are not read.

Related tools