AI article
Your CDN can overrule robots.txt: finding the layer that refuses AI crawlers
Community description: In the first post I found Japanese tourism sites whose robots.txt welcomes GPTBot while the server...
Dev.to | Oct 4, 2026 | Mahiro Hirakawa
Automated excerpt
If ExampleFetchBot gets 200 and GPTBot gets 403, the rule names AI crawlers. AI bot policies (Security settings, "Configure AI bot policies", which replaced the older "Block AI bots" toggle). At server level it refuses every crawler, /robots. txt included.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.