Free Robots.txt Generator
Build a robots.txt file for your website in seconds with this free robots.txt generator. Set user agents, allow and disallow rules, a sitemap address and crawl delay, then copy or download the finished file.
Note: major search engines treat crawl delay as a hint, not a strict rule, and some ignore it completely.
Add your rules above and click Generate robots.txt.
How to use this robots.txt generator
- Add a row for each user agent, starting with * for all crawlers.
- Type an Allow path and a Disallow path for that agent.
- Add your sitemap URL and an optional crawl delay.
- Click Generate, check the file, then Copy or Download it.
How do I create a robots.txt file with this tool?
Add one row for each user agent, type its allowed and disallowed paths, then add your sitemap address and an optional crawl delay. Click Generate to see the finished file, check every line carefully, then use Copy or Download to save it as robots.txt for your website root.
Where should I put the robots.txt file?
Place robots.txt in the root folder of your website, so it is available at your domain followed by robots.txt. Search engines look there first before crawling. A file in a subfolder will not work for the whole site, so upload it to the root and test that it loads in a browser.
Does crawl delay work for all search engines?
No. Major search engines treat crawl delay as a hint, not a strict rule, and some ignore it completely. Use it as a polite request to slow crawling on a small server, but do not rely on it to block traffic. For sensitive pages, use disallow rules, passwords or noindex tags instead of crawl delay alone.