Robots.txt Explained: Generate Yours in 60 Seconds (2026)
Short answer: robots.txt is a small text file at the root of your website that tells search engine crawlers which parts of your site they may visit and which they should skip. It takes about a minute to create, it cannot hide pages from search results on its own, and one wrong line — Disallow: / — can de-index your entire site. This guide explains how the file works, gives you a safe starter version, and shows you how to generate and check yours in under a minute.
1. What robots.txt actually does
Every time Googlebot visits your site, the first thing it does is request yoursite.com/robots.txt. That file is a set of instructions — not orders, but a protocol that well-behaved crawlers respect. Think of it as a sign on the door: it tells crawlers where they are welcome and where they should not go.
It is useful for three things:
- Saving crawl budget: keeping Googlebot out of admin panels, internal search pages, filter combinations and other low-value URLs so it spends its time on pages that matter.
- Protecting staging and duplicate copies: blocking a development or staging subdomain's copy of your site from being crawled.
- Pointing crawlers to your sitemap: the file can declare your sitemap location, which helps crawlers discover new URLs faster.
What it cannot do is just as important. Robots.txt does not make a page private, does not remove a page from search results, and does not stop people or bad bots from visiting a URL. Anyone can open your robots.txt file in a browser and read it — so never list sensitive paths there expecting them to stay secret.
2. How the file works: the only rules you need
A robots.txt file is made of groups. Each group starts with a User-agent line naming the crawler it applies to, followed by Allow and Disallow rules. The asterisk * means "all crawlers". Here is the anatomy:
- User-agent: * — the rules below apply to every crawler, including Googlebot, Bingbot and others.
- Disallow: /admin/ — crawlers should not fetch any URL starting with
/admin/. - Allow: / — crawlers may fetch everything else (this is the default when no rule matches).
- Sitemap: https://yoursite.com/sitemap.xml — tells crawlers where your sitemap lives.
Rules match by prefix: Disallow: /blog blocks /blog, /blog/post-one and /blogging-tips alike. When an Allow and a Disallow both match a URL, the more specific (longer) rule wins. And if a crawler cannot fetch your robots.txt at all — say your server returns a 500 error — most crawlers will assume they may crawl everything, while a persistent 5xx can make Googlebot stop crawling entirely. Keep the file small, valid and served with a 200 status.
3. A safe starter robots.txt
For most small business and ecommerce sites, a good starting point looks like this:
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /search?
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlThat blocks the admin area, login page, cart and checkout flows and internal search results — pages with no search value — while leaving everything else open. Swap the WordPress paths for your platform's equivalents: Shopify stores, for example, block /admin/ and /cart/ by default in the file Shopify generates. If your site is a simple brochure site with no admin area or checkout, even this is enough:
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xmlWhen in doubt, allow more rather than less. An over-permissive robots.txt costs you a little crawl budget; an over-restrictive one can cost you your rankings.
4. Generate yours in 60 seconds
Writing the file by hand is fine, but a generator removes the syntax guesswork — one misplaced slash or a rule in the wrong group is all it takes to block the wrong pages. Here is the 60-second workflow I use:
- Step 1: open my free robots.txt generator, enter your domain and tick the sections you want blocked — admin, cart, checkout, search pages and any custom paths.
- Step 2: copy the generated file and upload it to the root of your site, so it is reachable at
yoursite.com/robots.txt. On WordPress that means the public_html folder; on Shopify, edit therobots.txt.liquidtemplate; on a static host like Vercel or Netlify, drop the file in the project root. - Step 3: verify it with the robots.txt and sitemap checker — it confirms the file is reachable, parses your rules, and flags syntax errors before Googlebot ever sees them.
After uploading, ask Google Search Console to re-crawl the file: in the robots.txt report you can see the version Google has fetched and when it last checked. Google re-fetches robots.txt periodically on its own, but a manual check after a change is worth the thirty seconds.
5. Five mistakes that break sites
Most robots.txt disasters I see in audits come from the same handful of errors. Check yours against this list:
| Mistake | What happens | Fix |
|---|---|---|
Disallow: / on the live site | Every page is blocked from crawling | Usually a staging file pushed live — remove the rule immediately |
| Blocking CSS and JS files | Google cannot render pages properly | Never disallow /*.css, /*.js or asset folders |
| Using robots.txt to "remove" a page | URL can still appear in results, without a description | Use a noindex tag instead — and do not block the page in robots.txt, or Google cannot see the tag |
| File in a subfolder | Crawlers ignore it completely | It must live at the domain root: yoursite.com/robots.txt |
| Listing sensitive paths | Attackers read the file to find admin URLs | Protect sensitive areas with passwords, not robots.txt |
The single most dangerous line in SEO is Disallow: / — it is one character away from a perfectly normal file, and it has taken down more sites than any algorithm update. Always verify the live file after every deploy.
6. Robots.txt vs noindex vs passwords: pick the right tool
People reach for robots.txt when they actually need a different mechanism. Here is how to choose:
- Robots.txt — use it to manage crawl budget: keep crawlers out of areas they should not waste time on. It does not hide content.
- Noindex tag — use it when a page should not appear in search results at all, such as thank-you pages, internal search results or thin tag archives. Google must be able to crawl the page to see the tag, so never disallow it in robots.txt at the same time.
- Password protection — use it when content must genuinely stay private, such as a client portal or staging site. This is the only option of the three that actually keeps people out.
They work together, not against each other: block the crawl of your admin area with robots.txt, noindex your thank-you pages, and password-protect staging. Each tool does one job well.
7. How to check your file is working
Once the file is live, confirm it behaves the way you intended:
- Open it in a browser at
yoursite.com/robots.txt— you should see the current file, not a 404 or an old cached copy. - Test individual URLs in Google Search Console's robots.txt tester to see which rule applies to a given path.
- Watch the Crawl Stats report for a week after a change: blocked URLs should stop appearing in crawl requests, and important pages should keep being fetched.
- Re-check after every redesign or migration — platform changes and staging pushes are when bad robots.txt files slip through. I include a robots.txt check in every technical SEO audit for exactly this reason.
Robots.txt is a small file with outsized consequences. Get it right once, verify it after every change, and it quietly does its job for years. If you want a second pair of eyes on yours, my SEO services include a full technical audit — crawlability, indexing, structured data and speed, prioritized by impact.
Sources and further reading
Frequently asked questions
Does robots.txt hide a page from Google search results?
No — robots.txt only asks crawlers not to fetch a URL. If other pages link to it, Google may still index the URL without its content. To remove a page from search results, use a noindex tag or delete the page.
Is a robots.txt file required for every website?
No, it is optional. Without one, search engines simply crawl everything they can reach. A robots.txt file is worth having when you want to keep crawlers out of admin areas, search-result pages or staging copies, or point them to your sitemap.
Where do I put the robots.txt file?
At the root of your domain — for example https://example.com/robots.txt. A file in any subfolder is ignored by crawlers, and only one robots.txt per host is used.
Want your site's crawlability checked properly?
I do hands-on technical SEO audits for businesses in Pakistan — robots.txt, indexing, structured data and speed, prioritized by impact.
