How to Allow or Block AI Crawlers in Robots.txt (2026 Guide)
Jump To
Table of Contents
- What Are AI Crawlers?
- Complete List of AI Crawlers (2026)
- How to Block All AI Crawlers
- How to Block Specific AI Crawlers
- How to Allow All AI Crawlers (Recommended)
- Partial Blocking: Allow Some Paths, Block Others
- The AI Opt-Out Meta Tag
- How to Test Your Robots.txt Configuration
- Step 1: Check File Accessibility
- Step 2: Validate Syntax
- Step 3: Test Crawler Access
- Step 4: Monitor Your Server Logs
- Step 5: Verify Sitemap Accessibility
- Why You Should Allow AI Crawlers
- AI Citations Drive Real Traffic
- The AI Search Audience Is Growing Rapidly
- Competition for AI Citations Is Still Moderate
- AI Visibility and Traditional SEO Are Complementary
- Common Robots.txt Mistakes
- Mistake 1: Blocking All Bots with a Blanket Disallow
- Mistake 2: Missing Sitemap Directive
- Mistake 3: Overly Broad Disallow Rules
- Mistake 4: Conflicting Rules
- Mistake 5: Not Updating After Platform Changes
- Mistake 6: Case Sensitivity Assumptions
- Mistake 7: Blocking CCBot Unintentionally
- Action Plan: Optimize Your Robots.txt for AI Search
- Related Guides
Your robots.txt file is the single most important control point for deciding whether AI systems like ChatGPT, Perplexity, DeepSeek, and Google AI Overview can access and cite your content. Get it wrong and you vanish from AI search entirely. Get it right and you unlock a growing source of highly targeted traffic.
In 2026, AI-powered search handles billions of queries every month. When ChatGPT or Perplexity generates a response and cites your page as a source, that citation drives visitors who already trust the AI recommendation. But none of that happens if your robots.txt file blocks the crawlers that feed these systems.
This guide covers every major AI crawler active in 2026, shows you exactly how to allow or block each one, and explains why most websites should be letting AI bots in rather than keeping them out.
What Are AI Crawlers?
AI crawlers, also called AI bots or AI user-agents, are automated programs that browse the web to collect content for AI model training and real-time AI search responses. They work similarly to traditional search engine crawlers like Googlebot, but instead of building a search index for blue-link results, they gather content that AI systems use to generate synthesized answers.
There are two main categories of AI crawlers:
- Training crawlers collect large-scale web data to train or improve AI models. These bots gather content for model training datasets and may not revisit frequently.
- Live search crawlers fetch pages in real time when a user asks an AI assistant a question. These bots access your content on-demand to generate cited responses.
Most major AI platforms operate both types. Understanding which crawlers serve which purpose helps you make informed decisions about access control.
Complete List of AI Crawlers (2026)
This table lists every significant AI crawler that you may encounter in your server logs or need to configure in your robots.txt file.
| User-agent | Company | Platform | Purpose |
|---|---|---|---|
| GPTBot | OpenAI | ChatGPT | Training data collection |
| ChatGPT-User | OpenAI | ChatGPT | Live search for user queries |
| PerplexityBot | Perplexity | Perplexity AI | Live search and indexing |
| CCBot | Common Crawl | Multiple (open source) | Open web crawl dataset |
| DeepSeekBot | DeepSeek | DeepSeek AI | Training and live search |
| Bytespider | ByteDance | Doubao / TikTok AI | Training data collection |
| Baiduspider | Baidu | Baidu AI Search | Indexing for Baidu search and AI |
| ClaudeBot | Anthropic | Claude | Training data collection |
| Google-Extended | Google AI Overview / Gemini | AI training opt-out control | |
| Applebot-Extended | Apple | Apple Intelligence | AI training data collection |
| FacebookBot | Meta | Meta AI | Training data collection |
| YouBot | You.com | You.com AI | Live search and indexing |
Important note on Google-Extended: This user-agent specifically controls whether your content is used to train Google's AI models. It does not affect whether your pages appear in Google search results or Google AI Overview. Google search indexing is controlled by Googlebot, which is a separate user-agent.
Important note on Baiduspider: Baiduspider handles both traditional Baidu search indexing and Baidu AI Search. Blocking Baiduspider removes your site from all Baidu services, including AI-powered results. If you want to maintain Chinese search visibility, keep Baiduspider allowed.
How to Block All AI Crawlers
If you have decided that you do not want any AI platform to access your content, you can block all known AI crawlers in your robots.txt file. This is the most restrictive approach and will prevent your content from appearing in AI-generated responses across all platforms.
Here is a robots.txt configuration that blocks every major AI crawler:
User-agent: GPTBot Disallow: /
User-agent: ChatGPT-User Disallow: /
User-agent: PerplexityBot Disallow: /
User-agent: CCBot Disallow: /
User-agent: DeepSeekBot Disallow: /
User-agent: Bytespider Disallow: /
User-agent: ClaudeBot Disallow: /
User-agent: Google-Extended Disallow: /
User-agent: Applebot-Extended Disallow: /
User-agent: FacebookBot Disallow: /
User-agent: YouBot Disallow: /
Each block is separate because robots.txt requires individual User-agent directives for each bot you want to control. There is no wildcard syntax that reliably blocks all AI crawlers across all platforms.
Do NOT use this approach as a blanket default. As we explain later in this guide, blocking AI crawlers removes your content from AI search responses, which is where a growing share of user attention and traffic is heading. Only use full blocking if you have a specific reason, such as protecting paywalled content or proprietary research.
How to Block Specific AI Crawlers
In many cases, you may want to allow most AI crawlers while blocking specific ones. For example, you might want to appear in ChatGPT and Perplexity responses but prevent your content from being used for AI model training.
Block OpenAI crawlers only (GPTBot + ChatGPT-User):
User-agent: GPTBot Disallow: /
User-agent: ChatGPT-User Disallow: /
Block training crawlers but allow live search crawlers:
User-agent: GPTBot Disallow: /
User-agent: CCBot Disallow: /
User-agent: ClaudeBot Disallow: /
User-agent: Google-Extended Disallow: /
User-agent: Applebot-Extended Disallow: /
User-agent: FacebookBot Disallow: /
This configuration blocks crawlers that collect training data while allowing live search crawlers like ChatGPT-User and PerplexityBot to access your content for real-time citation. This is a balanced approach for publishers who want AI-driven traffic without contributing their content to model training.
Block ByteDance (Doubao) crawler:
User-agent: Bytespider Disallow: /
Block Anthropic crawler:
User-agent: ClaudeBot Disallow: /
How to Allow All AI Crawlers (Recommended)
For most websites, the best robots.txt configuration is to allow all AI crawlers. By default, if no Disallow directive targets a specific User-agent, that bot is allowed to crawl your entire site. This means a minimal robots.txt file naturally permits AI crawler access.
Here is the simplest robots.txt that allows all crawlers, including AI bots:
User-agent: * Allow: /
Sitemap: https://example.com/sitemap.xml
This configuration allows every crawler to access every page on your site and points them to your sitemap for efficient discovery. The Sitemap directive is optional but recommended because it helps all crawlers, including AI bots, find your content faster.
If you already have a robots.txt file that restricts certain paths (like /admin/ or /private/), make sure your general Allow directive is in place and that you are not accidentally blocking AI crawlers with overly broad Disallow rules.
A more explicit version that specifically names AI crawlers:
User-agent: * Allow: /
User-agent: GPTBot Allow: /
User-agent: ChatGPT-User Allow: /
User-agent: PerplexityBot Allow: /
User-agent: CCBot Allow: /
User-agent: DeepSeekBot Allow: /
User-agent: ClaudeBot Allow: /
Sitemap: https://example.com/sitemap.xml
This explicit approach is useful if you want to document your AI crawler policy clearly or if you have previously blocked these bots and need to override those rules.
Partial Blocking: Allow Some Paths, Block Others
Many websites need a nuanced approach that allows AI crawlers to access public content while keeping them out of private or duplicate areas. Robots.txt supports path-specific rules for this purpose.
Allow your blog and public pages, block admin and private areas:
User-agent: * Allow: /blog/ Allow: /guides/ Allow: /tools/ Allow: /about/ Disallow: /admin/ Disallow: /private/ Disallow: /staging/ Disallow: /api/
Sitemap: https://example.com/sitemap.xml
This configuration lets AI crawlers access your valuable public content (blog posts, guides, tools) while keeping them out of administrative, private, staging, and API areas.
Allow AI crawlers but block them from parameter-based duplicate pages:
User-agent: * Allow: / Disallow: /*?session_id= Disallow: /*?ref= Disallow: /*?utm_ Disallow: /search?
This prevents AI crawlers from indexing URL variations that create duplicate content, such as session parameters, referral codes, and internal search result pages.
Block specific paths for specific AI crawlers only:
User-agent: GPTBot Allow: / Disallow: /internal-reports/
User-agent: PerplexityBot Allow: /
User-agent: * Allow: / Disallow: /admin/
This shows how you can give different AI crawlers different levels of access. In this example, GPTBot is allowed everywhere except internal reports, PerplexityBot gets full access, and all other bots follow the general rules.
The AI Opt-Out Meta Tag
In addition to robots.txt, there is an emerging HTML meta tag standard for opting out of AI training. The tag uses the robots meta tag with AI-specific directives:
The noai directive signals that the page content should not be used for AI model training. The noimageai directive specifically prevents images on the page from being used for AI image generation training.
This meta tag approach has some important differences from robots.txt:
- Page-level control — The meta tag applies to individual pages, not your entire site
- Training only — It targets AI training, not live AI search crawling
- Opt-in compliance — Each AI company must choose to honor this tag
- Complementary to robots.txt — Use both for the strongest protection
As of 2026, adoption of the noai meta tag is growing but not yet universal. Some AI platforms respect it while others do not. For maximum protection against AI training use, combine the meta tag with robots.txt User-agent blocks for training crawlers.
To add the meta tag to a specific page, place it in the HTML head section:
How to Test Your Robots.txt Configuration
After updating your robots.txt file, you must verify that it works as intended. A misconfigured robots.txt can silently block AI crawlers, and you may not realize your content is missing from AI search results for weeks or months.
Step 1: Check File Accessibility
Your robots.txt file must be accessible at the root of your domain:
https://example.com/robots.txt
Open this URL in your browser and confirm the file loads correctly. If you get a 404 error, your file is missing. If you get a 500 error, there may be a server configuration issue.
Step 2: Validate Syntax
Robots.txt has specific syntax rules. Even a small typo can change the meaning of your directives. Common syntax errors include:
- Missing colon after User-agent or Disallow
- Extra spaces in directives
- Incorrect path formatting (paths must start with /)
- Missing newline between directive groups
Step 3: Test Crawler Access
Use the Robots.txt Tester to verify whether specific AI crawlers can access your pages. Enter your URL and the tool will fetch your robots.txt, parse the rules, and tell you exactly which bots are allowed or blocked for each path.
This is the most reliable way to confirm that GPTBot, PerplexityBot, and other AI crawlers can reach your content.
Step 4: Monitor Your Server Logs
After deploying your robots.txt changes, check your server access logs for crawler activity. Look for requests from AI user-agents like GPTBot and PerplexityBot. If you see 200 status codes, the crawlers are successfully accessing your pages. If you see 403 (Forbidden) or 429 (Too Many Requests) responses, there may be an issue beyond robots.txt, such as server-level blocking.
Step 5: Verify Sitemap Accessibility
Your sitemap.xml should also be accessible and properly formatted. Use the Sitemap Validator to confirm that your sitemap is valid and contains all your important pages. AI crawlers use sitemaps to discover content efficiently.
Why You Should Allow AI Crawlers
The default instinct for many website owners is to block AI crawlers. This is understandable given concerns about content scraping and unauthorized use. However, blocking AI crawlers in 2026 is often a strategic mistake. Here is why.
AI Citations Drive Real Traffic
When ChatGPT cites your page as a source, users click through. The same is true for Perplexity, Google AI Overview, and other AI platforms. AI-driven traffic tends to be highly engaged because the user has already received a recommendation from a system they trust.
Early data from 2025-2026 shows that AI citation traffic has conversion rates comparable to or higher than traditional organic search traffic for many content types.
The AI Search Audience Is Growing Rapidly
ChatGPT processes billions of queries monthly. Perplexity serves over 100 million queries per month. Google AI Overview appears on a significant and growing percentage of Google searches. Users are shifting from clicking blue links to asking AI assistants for answers.
If your content is not accessible to AI crawlers, you are invisible to this growing audience.
Competition for AI Citations Is Still Moderate
Most websites have not yet optimized for AI search visibility. This creates a window of opportunity. Sites that allow AI crawler access and optimize their content structure (headings, FAQs, schema markup) are more likely to be cited because there are fewer competing sources.
As more websites optimize for AI search, competition will increase. Establishing yourself as a trusted source now builds an advantage that compounds over time.
AI Visibility and Traditional SEO Are Complementary
Many GEO best practices also improve traditional SEO. Adding FAQ content helps with featured snippets. Structured data helps with rich results. Clear heading structure improves crawlability. Allowing AI crawlers does not hurt your traditional search performance.
Check your overall AI readiness with the AI SEO Analyzer to see how your site performs across both traditional and AI search factors.
Common Robots.txt Mistakes
Even experienced webmasters make robots.txt errors that silently hurt their AI visibility. These are the most common mistakes and how to fix them.
Mistake 1: Blocking All Bots with a Blanket Disallow
User-agent: * Disallow: /
This blocks every crawler, including Googlebot, Bingbot, and all AI crawlers. Your site will disappear from all search results, both traditional and AI-powered. Only use this for staging sites or private development environments.
Mistake 2: Missing Sitemap Directive
Without a Sitemap directive, crawlers must discover your pages through links alone. Adding a sitemap reference helps all crawlers, including AI bots, find your content faster and more completely.
# Always include your sitemap User-agent: * Allow: /
Sitemap: https://example.com/sitemap.xml
Validate your sitemap regularly with the Sitemap Validator.
Mistake 3: Overly Broad Disallow Rules
User-agent: * Disallow: /a
This single rule blocks access to every path starting with /a, including /about, /articles, /api-docs, and any other path beginning with the letter "a". Always test your Disallow paths carefully to avoid unintended blocking.
Mistake 4: Conflicting Rules
User-agent: GPTBot Disallow: /
User-agent: * Allow: /
In this case, GPTBot sees the specific Disallow rule and will be blocked, even though the general rule allows access. Specific User-agent rules always take precedence over the wildcard rule. This is correct behavior, but it can be confusing if you expected the general Allow to override.
Mistake 5: Not Updating After Platform Changes
AI companies regularly add new crawlers and change user-agent strings. A robots.txt file written in 2024 may not account for DeepSeekBot, ClaudeBot, or Applebot-Extended. Review your robots.txt quarterly and update it with any new AI crawler user-agents.
Mistake 6: Case Sensitivity Assumptions
The User-agent field in robots.txt is case-sensitive in practice, even though the specification says matching should be case-insensitive. Always use the exact user-agent string published by each AI company. For example, use "GPTBot" not "gptbot" or "GPTBOT".
Mistake 7: Blocking CCBot Unintentionally
CCBot (Common Crawl) provides open web data that many smaller AI platforms and research projects depend on. Blocking CCBot removes your content not just from Common Crawl datasets, but from any AI system that builds on that data. Be intentional about whether you block CCBot, and understand the downstream effects.
Action Plan: Optimize Your Robots.txt for AI Search
Follow these steps to ensure your robots.txt configuration supports maximum AI visibility:
- Audit your current robots.txt — Review your existing file for any rules that block AI crawlers
- Allow all major AI crawlers — Add explicit Allow directives for GPTBot, ChatGPT-User, PerplexityBot, CCBot, and DeepSeekBot at minimum
- Block only necessary paths — Restrict access to /admin/, /private/, and other non-public areas
- Add your sitemap — Include the Sitemap directive pointing to your current XML sitemap
- Test with the Robots.txt Tester — Verify that each AI crawler can access your important pages
- Check your GEO score — Use the GEO Score Checker to see how your site's AI readiness scores across platforms
- Review quarterly — Check for new AI crawlers and update your configuration as the landscape evolves
Your robots.txt file is the gateway between your content and AI search engines. Get it right, and your content becomes discoverable by the fastest-growing segment of search traffic. Combined with strong content optimization — clear headings, FAQ sections, and structured data — proper robots.txt configuration is the foundation of effective AI search visibility.
Start by testing your current configuration with the Robots.txt Tester, then check your overall AI readiness with the AI SEO Analyzer. For a complete picture of your citation potential across 10 AI platforms, run the GEO Score Checker.
Related Guides
- AI SEO in 2026 Guide — Comprehensive AI search optimization
- Schema Markup for AI Search — Structured data for AI citation
- ChatGPT Optimization — Get cited by ChatGPT
- Perplexity Optimization — Increase Perplexity citations