diff --git a/web/public/robots.txt b/web/public/robots.txt index 8958c9c..6cf3dfc 100644 --- a/web/public/robots.txt +++ b/web/public/robots.txt @@ -1,13 +1,16 @@ # Everything here is public documentation for a free, open-source app. +# AI search and assistant bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, +# Claude-User, PerplexityBot, Perplexity-User) fall into this group on purpose: +# they fetch pages to cite them in answers, which is how the site gets found. User-agent: * Allow: / -# AI crawlers are blocked at the edge, with a WAF skip for the two surfaces +# Training crawlers are blocked at the edge, with a WAF skip for the two surfaces # built for them: the llms.txt index and the markdown behind every docs page. # Saying so here spends a crawler's budget on the two paths that answer instead # of on HTML that will 403. This list is advisory and safe to let drift — the -# WAF is the enforcement, and a crawler missing from it falls back to the group -# above and is stopped at the edge anyway, which is the same answer, just later. +# WAF is the enforcement. Keep it in step with the WAF rule: a search bot listed +# here or matched there disappears from AI answers. # # The Allow lines come first on purpose. An RFC 9309 parser takes the longest # match and doesn't care about order, but a first-match parser reads `Disallow: /` @@ -15,13 +18,7 @@ Allow: / # open. The .md line needs `*` and `$`, which are Google extensions: the crawlers # named here support them, and one that doesn't just won't open the .md files. User-agent: GPTBot -User-agent: OAI-SearchBot -User-agent: ChatGPT-User User-agent: ClaudeBot -User-agent: Claude-User -User-agent: Claude-SearchBot -User-agent: PerplexityBot -User-agent: Perplexity-User User-agent: meta-externalagent User-agent: Bytespider User-agent: CCBot