robots.txt Generator
Decide what search engines, AI crawlers and SEO bots may crawl. The robots.txt is built live and can be downloaded right away.
Free
No sign-up
No tracking.
robots.txt
Done? Test single URLs against your draft in the robots.txt Tester
What can the generator do?
- Your own rules: Disallow and Allow one per line, wildcards * and $ are supported.
- AI bots: allow all, block training only or block all AI crawlers, with an up-to-date bot list.
- SEO crawlers like AhrefsBot or SemrushBot can be blocked.
- Sitemap reference at the end of the file.
Frequently asked questions
Does robots.txt prevent indexing?
No. It only controls crawling. Blocked pages can still be indexed if they are linked. Use noindex for that.
Should I block AI bots?
Blocking training bots does not hurt visibility. Blocking search bots like OAI-SearchBot or PerplexityBot means you will no longer be cited in AI answers.
Where does the file go?
Always in the root: https://www.example.com/robots.txt. Subdirectories are not read.