From d77d72622c27a8dc332639f1c351b7fa13fb33c9 Mon Sep 17 00:00:00 2001 From: Julkar Naen Nahian Date: Mon, 7 Sep 2026 01:49:11 +0600 Subject: [PATCH] fix(web): put the Allow lines before the Disallow in robots.txt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The AI-crawler group led with `Disallow: /`, which an RFC 9309 parser handles fine — it takes the longest match, so `Allow: /llms.txt` still wins — but a first-match parser reads the Disallow and stops. Checked against Python's urllib.robotparser: every path denied, including the two the group exists to open. With the Allow lines first, /llms.txt is allowed and /docs/reading/ is still denied, under both readings. The .md line still depends on `*` and `$` being understood. Those are Google extensions, supported by the crawlers named here; a parser without them opens llms.txt and skips the .md files, which is a degradation rather than a break. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01JGMQ49hvA936om3GQfNTAn --- web/public/robots.txt | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/web/public/robots.txt b/web/public/robots.txt index d1465b2..8958c9c 100644 --- a/web/public/robots.txt +++ b/web/public/robots.txt @@ -8,6 +8,12 @@ Allow: / # of on HTML that will 403. This list is advisory and safe to let drift — the # WAF is the enforcement, and a crawler missing from it falls back to the group # above and is stopped at the edge anyway, which is the same answer, just later. +# +# The Allow lines come first on purpose. An RFC 9309 parser takes the longest +# match and doesn't care about order, but a first-match parser reads `Disallow: /` +# and stops there — leading with it denies the two paths this group exists to +# open. The .md line needs `*` and `$`, which are Google extensions: the crawlers +# named here support them, and one that doesn't just won't open the .md files. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User @@ -19,8 +25,8 @@ User-agent: Perplexity-User User-agent: meta-externalagent User-agent: Bytespider User-agent: CCBot -Disallow: / Allow: /llms.txt Allow: /docs/*.md$ +Disallow: / Sitemap: https://reader-md.jnahian.me/sitemap.xml