Free technical SEO utility
Robots.txt and Sitemap Checker
Check whether a public website serves a robots.txt file and a readable XML sitemap. Review sitemap entries and a basic path access check for common search and AI crawlers.
Check a website
Discovery report
Sitemap XML
This is a technical fetch and basic parser. robots.txt is a crawler directive, not access control, and crawler implementations can differ.
What it checks
It reads robots.txt, looks for Sitemap directives, checks the first five declared sitemap files, and tests whether common crawlers are allowed to fetch the homepage path.
What it does not mean
A successful sitemap fetch does not confirm that search engines indexed its URLs. A robots.txt allow result also does not guarantee crawling or ranking.
How to use the Robots.txt and Sitemap Checker
- Enter a domain or homepage URL in the Website domain or homepage URL field and click Check robots.txt & sitemap.
- Review the robots.txt card: its HTTP status, the rules for Googlebot, Bingbot, GPTBot, ClaudeBot and other crawlers, and the full file under View robots.txt contents.
- Check the Sitemap XML card for each declared sitemap — URL count, valid and unique URLs, and any invalid XML.
- Use the findings to fix blocked pages, broken sitemaps, or missing Sitemap directives before the next crawl.
Why it matters
Robots.txt and XML sitemaps are the first things crawlers read when they visit your site. A single stray Disallow can hide your most important pages, while a broken sitemap leaves new content waiting to be discovered. Google's and AI search crawlers both respect these files, so errors here quietly cost you visibility in both classic and AI-generated results. Checking them regularly catches misconfigurations from site updates, migrations, and plugin changes before they turn into lost traffic.
Robots.txt and Sitemap Checker FAQs
Which sitemap does the checker request?
The checker reads up to five Sitemap directives listed in the site's robots.txt and fetches each one. If no Sitemap directive exists, it falls back to the conventional /sitemap.xml path. Each sitemap is parsed as XML, so you see its URL count, valid entries, and parse problems — which helps you confirm crawlers are being pointed at a working file.
Does an allowed result mean Google will index the page?
No. robots.txt only gives crawlers access instructions; indexing depends on content quality, links, canonicals, and whether Google chooses to include the page. The checker's crawler list shows a basic path allow-or-block result for names like Googlebot and GPTBot, which is a starting point for debugging — not a guarantee of crawling or ranking.
Does the checker save my scan results?
No. The scan runs when you submit a public URL and the results are returned to your browser only. Nothing is stored in an account or database. Only publicly accessible robots.txt and sitemap files are requested, so nothing private about the site is exposed or retained.
What should I fix if a sitemap shows invalid entries?
Open the sitemap details and look for invalid XML or URLs that fail validation. Common causes include HTML pages served at a sitemap URL, unescaped characters, redirects, or entries with non-HTTP schemes. Fix the generating plugin or file, make sure it returns XML with a proper urlset or sitemapindex root, and re-run the check.