Is Your Website Blocking AI From Ever Recommending You? A 5-Minute robots.txt Check

Short answer: If your robots.txt disallows GPTBot, ClaudeBot, PerplexityBot or Google-Extended, those engines will never quote your website. In 2026, that's the single fastest way to make yourself invisible to AI search — and it's the first check we run on every audit.

Why This One File Decides Whether AI Cites You

robots.txt is a plain-text file at the root of your site (yourdomain.com/robots.txt). It tells crawlers what they can and can't fetch. For years the only crawlers most SMBs cared about were Googlebot and Bingbot. In 2026 there are a dozen more that matter, and most sites don't know they exist — much less that a default WordPress or Shopify template quietly blocked several of them a year ago.

If Claude can't fetch your services page, Claude will never recommend you when a prospect asks "who's the best digital marketing agency in Vancouver." The site will not appear in the answer. Not lower — not at all.

The 8 AI Crawlers Your robots.txt Should Allow in 2026

• GPTBot — OpenAI's training crawler (ChatGPT model training).
• OAI-SearchBot — OpenAI's live search crawler (ChatGPT Search citations).
• ChatGPT-User — Fetches pages when a ChatGPT user asks a question that requires a live lookup.
• ClaudeBot / anthropic-ai — Anthropic's Claude crawlers.
• PerplexityBot / Perplexity-User — Perplexity's index and on-demand fetch bots.
• Google-Extended — Controls whether Google can use your content for Gemini and AI Overviews training (separate from Googlebot, which handles Search).
• CCBot — Common Crawl. Used by dozens of open LLMs including Mistral, Llama fine-tunes, and many enterprise tools.
• Applebot-Extended — Apple Intelligence and Siri answer generation.

The 5-Minute Check

1. Open yourdomain.com/robots.txt in a browser.
2. Search the file for each bot name above (case-insensitive).
3. Under each match, look for Disallow: /. If you find it, that bot is blocked from your entire site.
4. Also check for a blanket User-agent: * block that disallows everything — many WordPress "SEO privacy" plugins add this and forget to remove it after launch.
5. Fix anything blocked (see the copy-paste template below), redeploy, and re-verify.

Copy-Paste: A Sane 2026 robots.txt for AI Visibility

# Allow all major search + AI crawlers
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Applebot-Extended
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

When You Might Want to Block One

Only two cases where selective blocking makes sense for most SMBs:

• Paid content or gated research. Block training bots (GPTBot, ClaudeBot, CCBot), keep live-search bots (OAI-SearchBot, Perplexity-User) — your content still gets cited but isn't used for model training.
• News/publisher with a licensing deal. Block the specific bot your competitor is licensing your content to.

For a typical service business, agency, ecommerce store, or local trade — allow everything. The visibility upside is enormous and the "training data" downside is theoretical.

Common Mistakes We See Weekly

• A blanket Disallow: / left over from staging. Blocks every bot from the entire site. Devastating and shockingly common.
• WordPress "Discourage search engines" checkbox still ticked in Settings → Reading, months after launch.
• Blocking Google-Extended "for safety." This removes you from Gemini and Google AI Overviews entirely while providing zero SEO benefit.
• Blocking crawlers at the CDN/WAF layer (Cloudflare "Block AI Bots" toggle). The robots.txt looks fine, but the request never reaches your origin.

What to Do Next

1. Run the free audit above — it checks all 8 crawlers in one pass.
2. Fix any blocks in the robots.txt template.
3. If you use Cloudflare, disable "Block AI Bots" or add explicit allow rules for the bots above.
4. Once unblocked, layer on the citation signals: llms.txt, schema, and answer-first content structure covered in our AI citation checklist.