Cloudflare's AI Bot Policy Changed: Check Googlebot Access Today

Crawler access used to be a file in your web root. It is now a setting in front of your web root, and the two do not have to agree. That is the whole of today's issue: a control chosen for one purpose can affect a crawler you depend on for another, and nothing in your own site will tell you it happened.

What Cloudflare confirmed

Cloudflare confirmed a change to its AI bot defaults effective September 15, 2026. The detail that matters most for search visibility is Cloudflare's own caveat: Training blocks can also affect mixed-purpose crawlers such as Googlebot, Applebot and Bingbot. Some crawlers fetch for more than one reason, so a rule expressed in terms of purpose cannot always be enforced with the precision the label implies. That is not a criticism of the feature; it is the reason to verify.

Three decisions, not one

Write your stance down for each row before changing a setting. A decision you cannot explain in a sentence is a decision that will be reversed by the next person who reads the dashboard.

robots.txt requests, the edge enforces

robots.txt is a published preference. A compliant crawler reads it and honours it; a non-compliant one ignores it; either way the request reaches your server. An edge control sits earlier in the path and can challenge or refuse a request outright, before your application, your logs or your analytics ever see it. So an allow rule in robots.txt tells you nothing about whether a crawler is actually getting through. Check the edge.

The six-step audit

1. Read the current bot settings on every zone that serves the site, including any staging or country-specific hostname, and record which purpose-based controls are enabled.
2. Run a live test in Search Console URL Inspection on the home page and two important service pages. A live fetch is evidence; the last indexed state is history.
3. Check edge analytics and firewall events filtered by user agent for the last seven days, looking for challenges or blocks on crawlers you intend to allow.
4. Fetch key URLs directly and record status codes, including robots.txt and your sitemap. A challenge page returning 200 with no content is a silent failure.
5. Confirm robots.txt agrees with the stance you just wrote down, and that nothing stale is still disallowed from an old project.
6. Document the result with a date, so the next change has a before state to compare against.

Implementation safeguards

• Change one control at a time, and note the time. Two simultaneous changes cannot be attributed later.
• Never rely on a rule you have not observed working. Verify each change with a live fetch.
• Keep an allowlist for crawlers you depend on, and review it whenever a platform default changes.
• Watch the Search Console Pages report for new crawl-related exclusions in the days after any edge change.
• Give someone ownership of the setting. Crawler access fails quietly when it belongs to nobody.

What we are not claiming

This article covers what Cloudflare has confirmed and what its documentation states about mixed-purpose crawlers. It does not claim a measured traffic impact, a ranking effect, a crawl-volume figure or a recovery timeline. Fresh Search Console metrics were unavailable at the time of writing, and search volume, difficulty, CPC, traffic, ranking, backlink and crawl-total data are not included because they were unavailable and were not invented.

Sources

• Cloudflare's confirmed AI bot policy change effective September 15, 2026, including its statement that Training blocks can also affect mixed-purpose crawlers such as Googlebot, Applebot and Bingbot.
• Google Search Central documentation on Google crawlers, robots.txt, and debugging crawl access.
• Google Search Console Help on URL Inspection and the Pages report.
• Search Engine Roundtable and Search Engine Journal coverage, reviewed for context.

Our SEO audit page covers crawler-access checks in detail, SEO services covers the ongoing monitoring, and AEO services explains the answer-engine side of the same decision. To have your crawler access checked today, contact us.