From 5cf71940f7d08f4cdd23b7ad24ea1b04dd65190f Mon Sep 17 00:00:00 2001 From: Julkar Naen Nahian Date: Mon, 7 Sep 2026 01:07:45 +0600 Subject: [PATCH] chore(web): publish the agent surface in robots.txt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AI crawlers stay blocked at the edge, with a WAF skip for /llms.txt and /docs/*.md — the two surfaces that exist for them. robots.txt still said `Allow: /` to everyone, which invites a compliant crawler to spend its budget earning 403s on HTML it can't have. The named group points them at the two paths that answer instead. The list is advisory and drifts safely: the WAF is the enforcement, and a crawler missing from the list falls back to the wildcard group and is stopped at the edge, which is where it would have ended up anyway. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JGMQ49hvA936om3GQfNTAn --- web/public/robots.txt | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/web/public/robots.txt b/web/public/robots.txt index e74db4e..d1465b2 100644 --- a/web/public/robots.txt +++ b/web/public/robots.txt @@ -2,4 +2,25 @@ User-agent: * Allow: / +# AI crawlers are blocked at the edge, with a WAF skip for the two surfaces +# built for them: the llms.txt index and the markdown behind every docs page. +# Saying so here spends a crawler's budget on the two paths that answer instead +# of on HTML that will 403. This list is advisory and safe to let drift — the +# WAF is the enforcement, and a crawler missing from it falls back to the group +# above and is stopped at the edge anyway, which is the same answer, just later. +User-agent: GPTBot +User-agent: OAI-SearchBot +User-agent: ChatGPT-User +User-agent: ClaudeBot +User-agent: Claude-User +User-agent: Claude-SearchBot +User-agent: PerplexityBot +User-agent: Perplexity-User +User-agent: meta-externalagent +User-agent: Bytespider +User-agent: CCBot +Disallow: / +Allow: /llms.txt +Allow: /docs/*.md$ + Sitemap: https://reader-md.jnahian.me/sitemap.xml