Robots.txt Generator
Build a valid robots.txt — rules, crawl-delay, sitemap. Copy and go.
Stays on your device. This tool runs in your browser — nothing you paste or open ever leaves it. Nothing uploaded, nothing to leak.
↳ received from
Preset
robots.txt
Help us improve
Was this tool useful? Tap a star.
Thanks — your rating helps others find it.
A robots.txt generator builds the file that tells search-engine crawlers which parts of your site they may crawl. Pick a preset or add your own allow/disallow rules, set a sitemap URL, and copy the file to your site root. It generates in your browser.
What robots.txt does (and doesn’t)
It guides crawling, not indexing — a blocked page can still be indexed if linked elsewhere. To keep a page out of search, use a noindex meta tag instead. Robots.txt is best for steering crawl budget away from low-value paths.
FAQ
What does a robots.txt file actually do?It tells search engine crawlers which parts of your site they may or may not request, using allow and disallow rules. This generator builds those rules plus your sitemap line to copy into a robots.txt at your site root.
Does blocking a page in robots.txt keep it out of Google?Not reliably. Disallow stops crawling, but a blocked URL can still be indexed if other pages link to it, just without a description. To truly keep a page out of results, use a noindex tag instead of, or in addition to, robots.txt.
Where does the robots.txt file have to go?It must sit at the root of your domain, reachable at yourdomain.com/robots.txt, because crawlers only look there. Placing it in a subfolder means it will be ignored.
Should I include my sitemap in robots.txt?Yes. Adding a Sitemap line pointing to your sitemap URL helps crawlers discover all your pages efficiently. The generator includes a field for it so the reference is formatted correctly.