Put the Allow lines before the Disallow in robots.txt - #75
Merged
Merged
Conversation
The AI-crawler group led with `Disallow: /`, which an RFC 9309 parser handles fine — it takes the longest match, so `Allow: /llms.txt` still wins — but a first-match parser reads the Disallow and stops. Checked against Python's urllib.robotparser: every path denied, including the two the group exists to open. With the Allow lines first, /llms.txt is allowed and /docs/reading/ is still denied, under both readings. The .md line still depends on `*` and `$` being understood. Those are Google extensions, supported by the crawlers named here; a parser without them opens llms.txt and skips the .md files, which is a degradation rather than a break. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JGMQ49hvA936om3GQfNTAn
jnahian
added
the
web
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #73, fixing a bug in it.
The AI-crawler group led with
Disallow: /. An RFC 9309 parser takes the longest match, soAllow: /llms.txtstill wins there — but a first-match parser reads the Disallow and stops, denying the two paths the group exists to open.Checked against Python's
urllib.robotparser:/llms.txt/docs/reading//docs/reading/staying denied is the point — the block is still doing its job.The
.mdline depends on*and$being understood. Those are Google extensions, supported by the crawlers named here; a parser without them opensllms.txtand skips the.mdfiles. That's a degradation, not a break, and there's no wildcard-free prefix that covers/docs/*.mdwithout also opening the HTML.Still outstanding, and not fixable in this repo
Cloudflare's Managed robots.txt is on, and prepends its own block to this file. The live file therefore has two groups for
GPTBot,ClaudeBot,CCBotandBytespider(Cloudflare'sDisallow: /first, then ours) and twoUser-agent: *groups.Under longest-match semantics the merged groups resolve our way. Under first-match semantics Cloudflare's blanket
Disallow: /wins for those four, regardless of this fix —PerplexityBot, which Cloudflare doesn't name, is allowed either way. Verified against the live file.The clean resolution is to turn off Cloudflare's Managed robots.txt so this file is the only one, moving the
Content-Signal: search=yes,ai-train=no,use=referencedeclaration into it to preserve that policy. That's a dashboard change.🤖 Generated with Claude Code
https://claude.ai/code/session_01JGMQ49hvA936om3GQfNTAn