GEO Improvement: Add robots.txt and llms.txt to the docs site
Source audit: GEO Audit Report — 2026-07-20
Audit date: 2026-07-20
Run: §29761233522
Finding
robots.txt score: 0/18 — file not found at https://github.github.com/gh-aw/robots.txt. All AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended) have no explicit permission signal.
llms.txt score: 0/18 — file not found at https://github.github.com/gh-aw/llms.txt. No LLM-friendly structured index exists for the docs site.
Why this matters
These two files are the largest scoring categories (18 pts each, 36 pts combined out of 100). Their absence is the single biggest reason the docs site scores only 44/100 (Foundation band). Fixing both would push the site into the Good band (80+) and ensure every one of the 20+ sitemap pages benefits simultaneously, since they share the same domain.
Suggested fix
1. Add robots.txt to the docs site static assets (e.g. docs/public/robots.txt):
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://github.github.com/gh-aw/sitemap.xml
2. Generate and publish llms.txt at https://github.github.com/gh-aw/llms.txt:
geo llms --base-url https://github.github.com/gh-aw --output docs/public/llms.txt
Or author it manually following the [llms.txt spec]((llmstxt.org/redacted) At minimum include: project name, description, and links to key documentation pages.
3. Verify by re-running the GEO audit after deployment.
Generated by 🌍 GEO Optimizer Daily Audit · 31.7 AIC · ⌖ 9.91 AIC · ⊞ 4.9K · ◷
GEO Improvement: Add robots.txt and llms.txt to the docs site
Source audit: GEO Audit Report — 2026-07-20
Audit date: 2026-07-20
Run: §29761233522
Finding
Why this matters
These two files are the largest scoring categories (18 pts each, 36 pts combined out of 100). Their absence is the single biggest reason the docs site scores only 44/100 (Foundation band). Fixing both would push the site into the Good band (80+) and ensure every one of the 20+ sitemap pages benefits simultaneously, since they share the same domain.
Suggested fix
1. Add
robots.txtto the docs site static assets (e.g.docs/public/robots.txt):2. Generate and publish
llms.txtathttps://github.github.com/gh-aw/llms.txt:Or author it manually following the [llms.txt spec]((llmstxt.org/redacted) At minimum include: project name, description, and links to key documentation pages.
3. Verify by re-running the GEO audit after deployment.