Build a correct robots.txt file with allow and disallow rules, crawler-specific directives, crawl delay and sitemap references — with the common mistakes designed out.
Getting started
Choose whether to allow or disallow crawling by default.
Add paths to disallow, and any crawler-specific rules you need.
Add your sitemap URL, then copy the file and place it at your site's root.
Why it helps
Generates syntactically valid output, avoiding the errors that silently deindex a site.
Supports per-crawler rules for search engines and AI crawlers separately.
Includes the sitemap directive, which is easy to forget and genuinely useful.
In practice
Setting up robots.txt for a new website.
Blocking crawlers from admin, search or duplicate-parameter URLs.
Adding rules for AI training crawlers specifically.
Everything people ask most about the robots.txt generator, answered plainly.
Keep exploring
More from SEO Tools and beyond — most people who use this one reach for these next.
