Robots.txt and Sitemap Checker
Check a site's robots.txt and XML sitemap free: see which crawlers may fetch key paths, count sitemap URLs,...
Open toolFree technical SEO utility
Robots.txt Generator
Choose crawler rules and generate a robots.txt file you can review and download. Rules only affect compliant crawlers; they do not secure private content.
One user-agent per line. Use * for all crawlers.
All rules are generated in your browser. Check the result before replacing your live robots.txt file.
How to use the Robots.txt Generator
- List your crawler groups in the User-agent groups field — one per line, or * for all crawlers.
- Add the paths you want blocked in Disallow paths (one per line) and any exceptions in Allow paths (one per line); optionally paste your sitemap URL in the Sitemap URL field.
- Click Generate robots.txt and read the warnings — the tool flags dangerous rules like Disallow: / that would block the whole site.
- Use Copy file contents or Download robots.txt, then upload the file to your domain root and test it before it goes live.
Why it matters
Robots.txt tells search and AI crawlers which parts of your site they may fetch, so a careful file protects staging areas, admin paths, and thin pages while keeping important content open. Mistakes are expensive: one wrong line can deindex an entire site overnight, and overly permissive rules waste crawl budget on junk URLs. A generator with built-in warnings helps you write clean, predictable rules and declare your sitemap in the right format — small work that prevents large, silent traffic losses.
Robots.txt Generator FAQs
Does robots.txt keep private pages safe?
No. Robots.txt is a polite request to crawlers, not access control. Anyone can read the file and visit the paths it lists, and non-compliant bots ignore it entirely. Never list genuinely private directories there. Use passwords, authentication, or server permissions for anything sensitive, and treat robots.txt as a crawl-guidance tool only.
What does Disallow: / actually do?
It asks every matching crawler not to crawl any path on the site. The generator warns you loudly if this rule appears, because people usually add it by accident while staging and forget to remove it at launch. Review this line carefully before publishing, and use the checker tool to confirm the live file afterward.
Should I add a Crawl-delay directive?
This generator leaves Crawl-delay out on purpose. Googlebot does not support it, and support among other crawlers is inconsistent. If crawl load is a problem, adjust crawl rate settings in Search Console or fix server performance instead of relying on a directive most major crawlers ignore.
Do I need to list my sitemap in robots.txt?
It is optional but useful. A Sitemap directive points crawlers to your XML sitemap without waiting for a Search Console submission. The generator adds the line when you paste a valid sitemap URL. Keep the URL absolute and reachable, since a broken sitemap reference wastes the discovery benefit.