AI article

robots.txt is a sign, not a fence: 8 technical vectors through which AI still reads your website

You configure robots.txt like this: User-agent: GPTBot Disallow: / User-agent: CCBot Disallow:...

Dev.to | Mar 23, 2026 | carlosortet

Read the original article