Does Disallow in robots.txt remove a page from Google?

No. Disallow only stops crawling. A blocked URL that is linked from elsewhere can still be indexed without a description. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header.

Where does robots.txt live?

At the root of the host, e.g. https://example.com/robots.txt. Each subdomain and protocol needs its own file.

Is robots.txt case-sensitive?

Yes - paths in Disallow and Allow rules are case-sensitive, and the filename itself must be lowercase robots.txt.

Do I even need a robots.txt file?

Small sites can live without one - no file means crawl everything. But a minimal file with a Sitemap line is worth having, and any site with admin areas, filters or duplicate paths benefits from sensible rules.

Robots.txt Explained: Syntax, Examples & the Mistakes That Kill SEO

Key takeaways

robots.txt controls crawling, not indexing — Disallow does not remove pages from Google.
It lives at your domain root: yoursite.com/robots.txt.
One typo — Disallow: / — can hide your entire site from search.
Always include a Sitemap: line, and test the file before you trust it.

robots.txt is the smallest file on your site with the biggest blast radius. Written correctly, it quietly steers crawlers away from junk pages and toward your content. Written wrong — sometimes by a single character — it can wipe your site from search results for weeks. Here's how it actually works, with copy-paste examples and the mistakes to avoid.

What robots.txt does (and doesn't do)

robots.txt is a plain-text file at the root of your host — https://yoursite.com/robots.txt — that tells crawlers which paths they may request. Googlebot, Bingbot and other well-behaved bots check it before crawling anything else.

The critical nuance: it controls crawling, not indexing. A URL you block with Disallow can still appear in Google — with no description — if other pages link to it. Google can't read the page (you blocked it), but it knows the page exists. To truly keep a page out of results, do the opposite: allow crawling and add a noindex meta tag or X-Robots-Tag header so Google can see the instruction.

The syntax: four directives cover 99% of cases

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

User-agent — which crawler the rules below apply to. * means everyone; name a specific bot (e.g. Googlebot) to give it its own group.
Disallow — a path prefix the bot must not crawl. Disallow: /admin/ blocks everything under /admin/. An empty Disallow means "allow everything".
Allow — an exception inside a blocked area, e.g. allow one file inside a disallowed folder.
Sitemap — the absolute URL of your XML sitemap. Costs one line, helps every crawler find your content faster.

Rules are grouped: each User-agent line starts a group, and the directives under it apply to that bot. Paths are case-sensitive. Comments start with #.

Test your robots.txt in seconds

Paste your file to validate the syntax, see each crawler's rules grouped, and catch site-killing mistakes — free, in your browser.

Open the Robots.txt Analyzer →

The five mistakes that quietly kill rankings

Disallow: / left in production. The classic. A developer blocks the staging site, the file ships to the live site, and the whole domain drops out of search. Check your robots.txt after every migration or relaunch.
Using Disallow to "deindex" a page. As above — blocking crawling doesn't remove the URL. Worse: if the page already carried a noindex tag, blocking it means Google can never see that tag again.
Blocking CSS and JavaScript. Google renders pages. If it can't fetch your CSS/JS, it may judge the page broken or not mobile-friendly, and rankings suffer.
Directives before any User-agent line. Rules that don't belong to a group are ignored — your "protection" silently does nothing.
No Sitemap line. Not fatal, but you're skipping a free discovery boost for every search engine that reads the file.

Sensible starter files

For most small sites:

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

For WordPress:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap.xml

After editing, validate with our robots.txt analyzer, then confirm in Google Search Console's robots.txt report. If you're also optimizing for AI search, pair the file with an llms.txt at the same root.

Frequently asked questions

Does Disallow remove a page from Google?

No — it only stops crawling. Use a noindex tag (with crawling allowed) to remove a page from results.

Where exactly does the file go?

The root of each host: https://example.com/robots.txt. Subdomains need their own file.

Do all bots obey robots.txt?

Reputable crawlers do. Scrapers and bad bots often ignore it — robots.txt is a signpost, not a lock. Use server rules for real blocking.

How do I check what Google sees?

Search Console → Settings → robots.txt report shows the fetched file, when it was read, and any parse errors.

Related tools

See our Disclaimer for how to use WorkIQ content and tools.