Create a robots.txt file for your website in seconds, then download it as a ROBOTS file—no installation required.
User-agent: *
A robots.txt file lives at https://yourdomain.com/robots.txt and tells well-behaved crawlers which paths they may fetch. XConvert's free generator builds a valid, RFC 9309-compliant file with per-user-agent Allow/Disallow rules, Sitemap declarations, optional Crawl-delay, and ready-made blocks for the AI training crawlers (GPTBot, Google-Extended, CCBot, anthropic-ai, Bytespider) that publishers increasingly want to opt out of. Everything runs in your browser — no account, no upload — and you copy or download the finished file in seconds.
* (applies to every crawler). Add a separate group for Googlebot, Bingbot, GPTBot, Google-Extended, CCBot, anthropic-ai, Bytespider, or any token from the robotstxt.org database. Per-bot groups override the wildcard.Disallow: /admin/, Disallow: /search, Allow: /public/. Paths are case-sensitive, start with /, and support Google's two wildcards: * (zero or more characters) and $ (end of URL).https://example.com/sitemap.xml). You may declare multiple Sitemap lines — Google, Bing, and Yandex all read them regardless of which user-agent group they appear under.Crawl-delay in seconds if Bingbot or Yandexbot is hammering your origin (Googlebot ignores this directive). Click Generate, then Copy or Download the file and upload it to your web root so it resolves at /robots.txt.robots.txt is the Robots Exclusion Protocol, formalized as IETF RFC 9309 in September 2022 by Martijn Koster, Gary Illyes, Henner Zeller, and Lizzi Sassman. Before that it was a 1994 de facto convention. The file is the first thing a compliant crawler requests when it visits your site, and getting it wrong can de-index pages you wanted ranked or, conversely, leak staging URLs into search results. A generator catches the small syntax bugs (missing leading slash, wrong directive name, Disallow: with nothing after it) that silently break enforcement.
User-agent: GPTBot / Disallow: / to opt out of OpenAI training data, User-agent: Google-Extended for Gemini and Vertex AI (does NOT affect Google Search ranking), User-agent: CCBot for Common Crawl (the dataset most LLMs are trained on), User-agent: anthropic-ai and User-agent: ClaudeBot for Anthropic, and User-agent: Bytespider for TikTok/ByteDance. Each needs its own group.Disallow: /*?sort=), internal search results (Disallow: /search), session-ID parameters, and printer-friendly variants so Googlebot spends its crawl budget on real product or article pages.Disallow: /wp-admin/, Disallow: /api/, Disallow: /staging/. Note this only stops crawling, not access — sensitive data still needs authentication; robots.txt itself is publicly readable.Sitemap: line per file (news sitemap, image sitemap, video sitemap, language-split sitemaps). Search Console picks them up faster than waiting for crawler discovery.Crawl-delay: 10 slows Bingbot, Yandexbot, and many smaller bots to one request per 10 seconds. Googlebot ignores it (use Search Console's crawl-rate setting instead).Disallow: / line can drop your entire site from the SERPs.| Directive | Purpose | Googlebot | Bingbot | RFC 9309 |
|---|---|---|---|---|
User-agent |
Names the crawler(s) the group applies to (or * for all) |
Yes | Yes | Yes (required) |
Disallow |
Path the crawler must not fetch | Yes | Yes | Yes |
Allow |
Path that overrides a parent Disallow |
Yes | Yes | Yes |
Sitemap |
Absolute URL of an XML sitemap | Yes | Yes | Yes (non-group) |
Crawl-delay |
Seconds the crawler should wait between requests | Ignored | Honored | Not in RFC |
Host |
Preferred hostname (formerly used by Yandex) | Ignored | Ignored | Deprecated |
Noindex |
Block indexing | Ignored since 1 Sep 2019 | Ignored | Not in RFC |
Google's documentation is explicit: "Google supports the following fields: user-agent, allow, disallow, sitemap. Other fields such as crawl-delay aren't supported." The noindex in robots.txt directive was removed on September 1, 2019 — use a <meta name="robots" content="noindex"> tag or X-Robots-Tag HTTP header on the page itself instead.
| Goal | Lines |
|---|---|
| Allow everything | User-agent: * then Disallow: (empty) |
| Block entire site (staging) | User-agent: * then Disallow: / |
| Block admin and search results | Disallow: /admin/ and Disallow: /search |
| Block all query strings | Disallow: /*?* |
| Block PDFs only | Disallow: /*.pdf$ |
| Allow CSS/JS Google needs to render | Allow: /*.css$ and Allow: /*.js$ |
| Opt out of OpenAI training | User-agent: GPTBot / Disallow: / |
| Opt out of Common Crawl | User-agent: CCBot / Disallow: / |
| Opt out of Gemini / Vertex training | User-agent: Google-Extended / Disallow: / |
| Opt out of Anthropic Claude | User-agent: anthropic-ai and User-agent: ClaudeBot / Disallow: / |
| Declare a sitemap | Sitemap: https://example.com/sitemap.xml |
It must live at the root of the host: https://example.com/robots.txt — not in a subdirectory and not at a different filename. Crawlers will only check that one location, and the file applies to that host and port. Subdomains need their own file (https://blog.example.com/robots.txt is separate from https://example.com/robots.txt). Per RFC 9309 the response must return HTTP 200; a 404 is treated as "no restrictions" and a 5xx as "temporarily disallow everything."
Each AI vendor publishes its own user-agent token. The ones currently in active use: GPTBot (OpenAI training), OAI-SearchBot (OpenAI's ChatGPT search), ChatGPT-User (on-demand fetches from ChatGPT), Google-Extended (Gemini Apps and Vertex AI training — separate from Googlebot), CCBot (Common Crawl, used as training data by most LLMs), anthropic-ai and ClaudeBot (Anthropic), PerplexityBot (Perplexity), Bytespider (TikTok/ByteDance), Applebot-Extended (Apple Intelligence training, distinct from Applebot which feeds Siri/Spotlight). Add one User-agent: / Disallow: / group per token. Note: blocking only stops future crawls — content already ingested cannot be un-trained.
The original 1994 spec did not, but Google, Bing, and most major crawlers support two wildcards as a de facto extension: * matches zero or more of any character, and $ anchors the pattern to the end of the URL. For example, Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /private/* blocks everything under /private/. RFC 9309 itself does not require wildcard support, so smaller or older crawlers may treat * as a literal asterisk.
noindex in robots.txt removed?Google announced in July 2019 that it would stop honoring Noindex: lines in robots.txt as of September 1, 2019, because the directive was never part of the official protocol and behaved inconsistently across implementations. Use a <meta name="robots" content="noindex"> tag in the page's <head>, or send an X-Robots-Tag: noindex HTTP header — both are official, supported across every major crawler, and apply per-page rather than per-pattern.
robots.txt controls crawling (whether the bot fetches the URL at all). The meta robots tag and X-Robots-Tag header control indexing (whether a fetched URL appears in search results). Counterintuitively, blocking a page in robots.txt can prevent Google from seeing a noindex tag on that page — so if a URL is already indexed and you want it removed, allow crawling and add noindex to the page itself. Use robots.txt for crawl-budget management (huge sites, infinite-scroll URLs, faceted navigation) and meta robots for "this specific page shouldn't show in search."
No. Google can still index a disallowed URL based on links pointing to it, displaying a result with "No information is available for this page" and no snippet. To fully suppress a URL from search results you need either authentication, a noindex directive (which requires allowing crawling), or a removal request via Search Console. The Google documentation states explicitly that robots.txt "is not a mechanism for keeping a web page out of Google."
RFC 9309 is the IETF Internet Standards Track document that formalized the Robots Exclusion Protocol in September 2022, ~28 years after Martijn Koster's original 1994 convention. It defines the file format, parsing rules, error handling (e.g., 4xx = unrestricted, 5xx = fully blocked), caching behavior (max 24 hours), and the merge logic for multiple matching user-agent groups (longest match wins). Major crawlers including Googlebot, Bingbot, and DuckDuckBot conform to RFC 9309; AI training crawlers generally do too, though enforcement is voluntary.
Within a single User-agent group, Google and Bing use longest-match-wins rules — the most specific path pattern takes precedence, regardless of whether it's Allow or Disallow. So Disallow: /folder/ plus Allow: /folder/public/ correctly unblocks the public subfolder because /folder/public/ is the longer match. For multiple User-agent blocks, the crawler picks the most specific token that names it (e.g., Googlebot-Image prefers its own block over the generic Googlebot block, falling back to * only if neither is present).
Both. Disallow: /admin/ blocks everything under that directory. Disallow: /private/secret.html blocks one file. Disallow: /*.pdf$ blocks every PDF anywhere on the site. The path is matched as a prefix unless wildcards are used, so Disallow: /admin (no trailing slash) blocks both /admin/dashboard and /admin.html — usually you want the trailing slash.
A robots.txt usually ships with a sitemap and supporting headers. XConvert has companion tools for related infrastructure tasks: Cron Expression Generator for scheduling robots.txt regeneration jobs, XML Validator to check the sitemap files you reference, and JSON Formatter for the structured-data files that complement crawl-control headers.