Skip to content

Publish the agent surface in robots.txt #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jnahian merged 1 commit into from
Sep 6, 2026
Merged

Publish the agent surface in robots.txt #73

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions web/public/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,25 @@
User-agent: *
Allow: /

# AI crawlers are blocked at the edge, with a WAF skip for the two surfaces
# built for them: the llms.txt index and the markdown behind every docs page.
# Saying so here spends a crawler's budget on the two paths that answer instead
# of on HTML that will 403. This list is advisory and safe to let drift — the
# WAF is the enforcement, and a crawler missing from it falls back to the group
# above and is stopped at the edge anyway, which is the same answer, just later.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: CCBot
Disallow: /
Allow: /llms.txt
Allow: /docs/*.md$

Sitemap: https://reader-md.jnahian.me/sitemap.xml
Loading