From d2df7b87a6cf1627dc09d8324c24303b262cbc6e Mon Sep 17 00:00:00 2001 From: Julkar Naen Nahian Date: Sat, 19 Sep 2026 00:56:00 +0600 Subject: [PATCH] fix(web): let AI search bots crawl the site OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User fetch pages to cite them in answers. Blocking them alongside the training crawlers kept the site out of ChatGPT, Claude and Perplexity results. They now fall into the open * group; the training crawlers stay blocked except for llms.txt and the docs markdown. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01CTg9MF9vuf9d8kpg4bUoAq --- web/public/robots.txt | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/web/public/robots.txt b/web/public/robots.txt index 8958c9c..6cf3dfc 100644 --- a/web/public/robots.txt +++ b/web/public/robots.txt @@ -1,13 +1,16 @@ # Everything here is public documentation for a free, open-source app. +# AI search and assistant bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, +# Claude-User, PerplexityBot, Perplexity-User) fall into this group on purpose: +# they fetch pages to cite them in answers, which is how the site gets found. User-agent: * Allow: / -# AI crawlers are blocked at the edge, with a WAF skip for the two surfaces +# Training crawlers are blocked at the edge, with a WAF skip for the two surfaces # built for them: the llms.txt index and the markdown behind every docs page. # Saying so here spends a crawler's budget on the two paths that answer instead # of on HTML that will 403. This list is advisory and safe to let drift — the -# WAF is the enforcement, and a crawler missing from it falls back to the group -# above and is stopped at the edge anyway, which is the same answer, just later. +# WAF is the enforcement. Keep it in step with the WAF rule: a search bot listed +# here or matched there disappears from AI answers. # # The Allow lines come first on purpose. An RFC 9309 parser takes the longest # match and doesn't care about order, but a first-match parser reads `Disallow: /` @@ -15,13 +18,7 @@ Allow: / # open. The .md line needs `*` and `$`, which are Google extensions: the crawlers # named here support them, and one that doesn't just won't open the .md files. User-agent: GPTBot -User-agent: OAI-SearchBot -User-agent: ChatGPT-User User-agent: ClaudeBot -User-agent: Claude-User -User-agent: Claude-SearchBot -User-agent: PerplexityBot -User-agent: Perplexity-User User-agent: meta-externalagent User-agent: Bytespider User-agent: CCBot