Robots.txt Generator
Choose what search engines and AI crawlers may visit, then download a ready-to-upload robots.txt.
- Free
- No sign-up
- Runs in your browser
Start from a preset
Rules for all crawlers
Default access (User-agent: *)
Paths are case-sensitive. * matches any characters and $ marks the end of a URL, e.g. /*.pdf$.
Re-opens a path inside a disallowed folder. The most specific (longest) matching rule wins.
Your robots.txt
User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /search
Sitemap: https://www.example.com/sitemap.xml
Upload it as robots.txt to the root of your domain, e.g. https://www.example.com/robots.txt. Each subdomain (blog.example.com, shop.example.com) needs its own file.
Rules for specific bots
“Default” follows the rules above. A bot with its own rule ignores the * section completely.
Googlebot
Google Search
Crawls pages for Google Search. Blocking it removes you from Google.
Googlebot-Image
Google Images
Crawls images for Google Images.
Bingbot
Bing (also powers DuckDuckGo, Yahoo, Ecosia results)
Crawls pages for Bing and the search engines that use its index.
GPTBotAI
OpenAI — AI model training
Collects content that may be used to train OpenAI’s models.
OAI-SearchBotAI
OpenAI — ChatGPT search
Lets your pages appear as sources in ChatGPT search answers.
ChatGPT-UserAI
OpenAI — user-requested visits
Fetches a page when a ChatGPT user asks it to.
ClaudeBotAI
Anthropic — AI model training
Collects content that may be used to train Anthropic’s Claude models.
Claude-SearchBotAI
Anthropic — Claude search
Indexes pages so they can be cited in Claude’s search answers.
Google-ExtendedAI
Google — Gemini training
A control token, not a crawler: blocks use of your content for Gemini models. Does not affect Google Search.
Applebot-ExtendedAI
Apple — AI training
A control token: opts out of Apple using your content to train its AI models. Applebot itself still crawls for Siri/Spotlight.
PerplexityBotAI
Perplexity — AI search
Indexes pages for Perplexity’s answer engine.
CCBotAI
Common Crawl
Open web archive widely used to train AI models.
BytespiderAI
ByteDance (TikTok)
Crawler associated with ByteDance’s AI products.
meta-externalagentAI
Meta — AI training
Collects content for Meta’s AI models.
AmazonbotAI
Amazon
Crawls for Alexa answers and other Amazon services.
How robots.txt works
robots.txt is a plain-text file at the root of your site that tells crawlers which URLs they may request. It is made of groups: each group starts with one or more User-agent lines naming the bots it applies to, followed by Disallow and Allow rules. User-agent: * covers every bot that doesn’t have its own group. A crawler only obeys the single most specific group that names it — so if you add a group for Googlebot, it ignores everything in the * group.
When rules conflict, Google uses the most specific one: the rule with the longest matching path. For example, with Disallow: /admin/ (7 characters) and Allow: /admin/help/ (12 characters), the URL /admin/help/faq is allowed because the Allow rule is longer, while /admin/users stays blocked. An empty Disallow: means “nothing is blocked”.
Should I block AI crawlers like GPTBot?
It depends on what you want. Training bots (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) collect content for AI models; blocking them doesn’t affect your Google or Bing rankings. Search and user bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch pages to cite them in AI answers, which can send you visitors — many publishers block training but allow these. robots.txt is a request, not a lock: reputable companies honour it, but it can’t stop bots that choose to ignore it.
Does Disallow remove a page from Google?
No. Disallow stops crawling, not indexing — a blocked URL can still appear in results (without a description) if other sites link to it. To keep a page out of Google, allow crawling and add a noindex robots meta tag, or protect it with a password.
Frequently asked questions
- Does Google respect Crawl-delay?
- No. Googlebot ignores Crawl-delay and sets its own crawl rate based on how quickly your server responds. Bing and Yandex do honour it.
- Are robots.txt paths case-sensitive?
- Yes. Disallow: /Admin/ does not block /admin/. The User-agent name, on the other hand, is matched case-insensitively.
- Do I need to list my sitemap in robots.txt?
- It is optional but helpful: any crawler that reads robots.txt can discover the sitemap. Also submit it in Google Search Console and Bing Webmaster Tools.
- How do I test my robots.txt?
- After uploading, open the robots.txt report in Google Search Console to see the version Google fetched and any parsing errors, and use the URL Inspection tool to check whether a specific URL is blocked.