Robots.txt Code Generator & Validator
Generate RFC 9309 compliant robots.txt files with visual user-agent controls, AI scraper blocking presets, XML sitemap directives, and real-time crawl path validation.
1-Click Industry Standard Presets
Apply instant best-practice rules tailored for SEO and crawler security
Robots Rule Builder
Production Output & Validator
Status: BLOCKED for Googlebot
Matched directive: "disallow: /admin/" (Line 7)
# -------------------------------------------------- # Robots.txt generated via TwisterTools.com # Standard Robots Exclusion Protocol (REP) Compliance # -------------------------------------------------- User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /dashboard/ Disallow: /checkout/ Allow: / # Sitemaps Sitemap: https://example.com/sitemap.xml Host: example.com
Robots Exclusion Protocol (RFC 9309) & Search Engine Crawl Mechanics
The Robots Exclusion Protocol (REP), formalized as an official Internet standard in IETF RFC 9309, establishes how automated web agents (such as Googlebot, Bingbot, and AI LLM crawlers) interact with web properties. A website's robots.txt file acts as the primary gateway, instructing crawlers on which resource paths may be fetched and indexed into search engine databases.
To function properly, the file must always reside at the absolute root of the domain origin (e.g., https://example.com/robots.txt) and be served with a text/plain Content-Type encoded in standard UTF-8. Subdirectories (such as example.com/blog/robots.txt) are ignored by search engines.
RFC 9309 Longest-Match Resolution
When a URL path matches both an Allow: and a Disallow: directive, crawlers determine the winner by the longest character length of the matched path. If pattern lengths are identical, the Allow directive wins.
Crawl Access vs. Indexation Shield
Blocking a page in robots.txt prevents bots from crawling its HTML, but Google may still index the bare URL if external backlinks point to it. To guarantee total de-indexing, remove the robots.txt disallow and use a noindex robots meta tag.
AI Scraping Governance & Major LLM Crawler User-Agents
With the rise of Generative AI, modern websites require differentiated bot management. Blocking AI scraping bots allows publishers to safeguard copyright and intellectual property without sacrificing organic Google and Bing search rankings.
| Organization / Platform | User-Agent Token | Crawler Purpose | Disallow Impact |
|---|---|---|---|
| OpenAI (ChatGPT) | GPTBot | Foundation model training datasets | Blocks AI Training Only |
| OpenAI (Browsing) | ChatGPT-User | Real-time user queries inside ChatGPT | Blocks Live Chat Retrieval |
| Anthropic (Claude) | ClaudeBot | Claude model training & web scraping | Blocks AI Training Only |
| Perplexity AI | PerplexityBot | Real-time AI search indexing & scraping | Excludes from Perplexity |
| Google Gemini & Vertex | Google-Extended | Gemini model training datasets | Does Not Hurt Google SEO |
| Common Crawl | CCBot | Open-source web archive used by AI labs | Blocks Global AI Datasets |
Directives, Pattern Wildcards & Syntax Reference
Modern search engine crawlers support pattern matching with asterisks (*) and end-of-URL anchors ($). Leverage these standard syntax rules to manage complex URL architectures:
Disallow URL Query Parameters
Disallow: /*?*
Disallow: /*?sort=*
Prevents crawling of duplicate faceted navigation and sorting parameters while keeping canonical URLs indexable.
Block Specific File Extensions
Disallow: /*.pdf$
Disallow: /*.zip$
The $ anchor signifies the exact end of the path string, protecting PDFs and archive downloads from indexing.
Allow Single Subfolder Within Disallowed Tree
Disallow: /private/
Allow: /private/public-press-kit/
Longest-match RFC 9309 resolution ensures the specific allow path takes precedence over the parent directory block.
Multiple Sitemap Indexing
Sitemap: https://example.com/blog-sitemap.xml
You can declare unlimited XML Sitemap endpoints anywhere in your robots.txt to accelerate discovery across large web applications.
Step-by-Step Production Implementation & Verification Guide
Generate & Download
Configure your bot directives, sitemaps, and test paths above. Click Download robots.txt to obtain the UTF-8 text file.
Deploy to Public Root
Upload to your server root directory (e.g. Next.js public/robots.txt, Apache public_html/, or NGINX root).
Verify in Search Console
Open Google Search Console > Settings > Robots.txt to verify Googlebot reads the fresh version with HTTP 200 status code.
Frequently Asked Questions (FAQ)
What is robots.txt and where must it be hosted?
A robots.txt file is a plain text file placed strictly at the root directory of your website (e.g., https://example.com/robots.txt). It instructs compliant search engine crawlers, bots, and AI scrapers which URL paths they are allowed or disallowed from accessing according to the Robots Exclusion Protocol.
Does robots.txt guarantee that private web pages will not be indexed or seen?
No. Robots.txt is an advisory access control for compliant crawlers, not a security firewall. If other public websites link to a disallowed URL, search engines like Google may still index the URL without reading its content. To keep pages strictly confidential or unindexed, use HTTP Authentication, password walls, or the <meta name="robots" content="noindex"> tag.
How do I block modern AI bots like ChatGPT, Claude, and Perplexity?
You can declare explicit User-agent rule blocks targeting AI scrapers. For example: User-agent: GPTBot, User-agent: ClaudeBot, User-agent: PerplexityBot, User-agent: CCBot, and User-agent: Google-Extended followed by Disallow: /. Our generator includes a preconfigured 1-click "Search Allowed + AI Blocked" preset.
How does RFC 9309 handle conflicting Allow and Disallow rules?
Under the official RFC 9309 Robots Exclusion Protocol specification, crawlers evaluate path rules using longest-match precedence. If both an Allow and Disallow rule match a URL path, the rule with the longest character pattern wins. If both patterns have identical lengths, the Allow rule takes priority.
What is the Crawl-delay directive and is it supported by Google?
The Crawl-delay directive instructs bots to wait a specified number of seconds between successive requests to prevent server overloading. It is supported by Bingbot, Yandex, and Baidu, but Googlebot does not recognize Crawl-delay in robots.txt (Google manages crawl rates dynamically through Google Search Console).
Can I use wildcard asterisks (*) and dollar signs ($) in robots.txt?
Yes. The asterisk (*) matches any sequence of zero or more characters (e.g., Disallow: /*?sort=*), while the dollar sign ($) anchors the end of a URL pattern (e.g., Disallow: /*.pdf$ to block all PDF files).
Related & Complementary Utilities
Explore more privacy-first client-side web tools.
URL Encoder / Decoder & URI Sanitizer
Encode special characters into percent-encoded URI strings or decode encoded URLs back to human-readable paths in real time. 100% client-side web utility.
Regex Tester, Explainer & Cheat Sheet
Test, debug, and explain regular expressions in real-time with native JavaScript RegExp engine, flag toggles, match highlighting, group captures, and a comprehensive syntax cheat sheet — 100% client-side.
Diff Checker & Text Comparison Tool
Compare text differences with precision — line-by-line or character-by-character. Split and unified views with real-time performance metrics.