Skip to content

Put the Allow lines before the Disallow in robots.txt #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jnahian merged 1 commit into from
Sep 6, 2026
Merged

Put the Allow lines before the Disallow in robots.txt #75

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion web/public/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@ Allow: /
# of on HTML that will 403. This list is advisory and safe to let drift — the
# WAF is the enforcement, and a crawler missing from it falls back to the group
# above and is stopped at the edge anyway, which is the same answer, just later.
#
# The Allow lines come first on purpose. An RFC 9309 parser takes the longest
# match and doesn't care about order, but a first-match parser reads `Disallow: /`
# and stops there — leading with it denies the two paths this group exists to
# open. The .md line needs `*` and `$`, which are Google extensions: the crawlers
# named here support them, and one that doesn't just won't open the .md files.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Expand All @@ -19,8 +25,8 @@ User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: CCBot
Disallow: /
Allow: /llms.txt
Allow: /docs/*.md$
Disallow: /

Sitemap: https://reader-md.jnahian.me/sitemap.xml
Loading