Create a robots.txt file
Disallow can still appear in search results if other sites link to it — Google just shows it without a description. To keep a page out of the index you need a noindex meta tag or X-Robots-Tag header, and the page must stay crawlable for the crawler to see it.| Directive | What it does | Honoured by Google? |
|---|---|---|
User-agent: | Opens a block that applies to a named crawler, or * for all | Yes |
Disallow: | Path prefix the crawler should not fetch | Yes |
Allow: | Carves an exception out of a broader Disallow | Yes |
Sitemap: | Absolute URL of a sitemap; applies site-wide, not per block | Yes |
Crawl-delay: | Seconds to wait between requests | No — ignored. Bing and Yandex honour it |
Noindex: | Was an unofficial Google extension | No — dropped 1 September 2019 |
Host: | Preferred mirror | No — Yandex only, now deprecated |
| Pattern | Meaning | Example |
|---|---|---|
* | Any sequence of characters | Disallow: /*?sort= blocks every sorted URL |
$ | End of URL | Disallow: /*.pdf$ blocks PDFs but not /file.pdf?x=1 |
/ alone | The entire site | Disallow: / blocks everything |
empty Disallow: | Nothing is blocked | Equivalent to allowing all |
Allow: /blog/public/ beats Disallow: /blog/ for that subdirectory.User-agent: Googlebot exists, Googlebot ignores the * block entirely — you must repeat any shared rules inside its block.example.com/blog/robots.txt is not read.| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection |
| OAI-SearchBot | OpenAI | ChatGPT search index |
| ChatGPT-User | OpenAI | Live fetch when a user asks about a URL |
| ClaudeBot | Anthropic | Training data collection |
| Claude-SearchBot | Anthropic | Search indexing |
| PerplexityBot | Perplexity | Search index |
| Google-Extended | Gemini training opt-out — does not affect Search ranking | |
| Applebot-Extended | Apple | Apple Intelligence training opt-out |
| CCBot | Common Crawl | Open crawl corpus many models train on |
| meta-externalagent | Meta | AI training |
| Bytespider | ByteDance | AI training |
A typical content site that wants search traffic and AI citations
User-agent: * then Disallow: /admin/, Disallow: /*?sessionid=, Allow: /
A separate block naming each AI crawler you want to permit, repeating the same Disallow lines
A final Sitemap: https://example.com/sitemap.xml line outside any block
Blocking AI crawlers keeps you out of AI answers. If citations in ChatGPT or Perplexity are a goal, those user agents have to be allowed.
Test with a parser that implements the real specification. Python’s built-in urllib.robotparser does not support the * and $ wildcards and will report false failures on any modern file — use a Google-compatible parser such as protego, or the robots.txt report in Search Console. Then confirm live: fetch a blocked URL with the crawler’s user agent and check what actually comes back, because a CDN or edge firewall can block a bot long before your robots.txt is consulted.
No. It stops crawling. A disallowed URL can still be listed in search results, without a snippet, if other pages link to it. To exclude a page from the index use a noindex meta tag or X-Robots-Tag header, and leave the page crawlable so the directive is actually seen.
No. Google ignores Crawl-delay entirely and manages its own crawl rate. Bing, Yandex and Seznam do honour it. To slow Googlebot, use the crawl rate settings in Search Console or return 503 responses under load.
Add a User-agent block for each one — GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Applebot-Extended and others — with Disallow: /. Be aware that this also removes you from AI answer engines, so it is a trade-off between training-data control and citation visibility.
At the root of the host, as https://example.com/robots.txt. It applies to that exact host and port only, so a subdomain needs its own file, and one placed in a subdirectory is never read.
Google treats a persistent 5xx as a signal that the whole site may be disallowed and can stop crawling. A 404 is safe and means everything is allowed. Never let this file depend on a database or an application server.
Everything on this page runs in your browser — nothing you type is uploaded or stored.
Short, exact answers to the questions people ask alongside this tool — the number first, then the working.
For teams, not just visitors
We build the calculators on SnoopTool as embeddable widgets in your branding, wired to your own lead form — the same maths, your colours, your domain, your leads. Tell us which one and we will send back a working demo you can put in front of your team.
Thanks — we have it, and you will get a reply from a person within one business day. Meanwhile every tool on SnoopTool stays free, with no signup.