Make the site discoverable to agents, and serve the docs as markdown

[jnahian]

The site was crawlable — static HTML, titles, descriptions, canonicals, OG tags — but nothing told a machine what was here. Nothing enumerated the docs pages, nothing typed the claims the prose makes, and an agent that wanted a page had to parse the prose back out of the HTML.

Two commits.

Discoverability

sitemap.xml	@astrojs/sitemap, since site was already configured. 18 URLs, 404 excluded.
robots.txt	Points at the sitemap.
llms.txt	llmstxt.org index, generated from the docs collection.
JSON-LD	SoftwareApplication on /, FAQPage on /docs/faq.

Nothing here is a list to maintain by hand. llms.txt is built off the same collection the /docs hub is, so a new page under docs/ lists itself. SoftwareApplication reads softwareVersion from latestVersion, so it tracks releases. A static llms.txt in public/ would have drifted on the next page added, which is what plugins/docs-pages.mjs exists to prevent.

The integration hard-codes <filenameBase>-index.xml, so a build hook renames the index to the /sitemap.xml a crawler actually guesses. Renaming the index rather than the chunk keeps the split intact.

Markdown twins + copy button

/docs/reading is the page, /docs/reading.md is the markdown it was rendered from — all 15 pages, including /docs/faq.md, where the raw ### question headings read better for a machine than the accordion does.

Served byte for byte except the links. Those are on-disk relative paths (../cli.md) so they resolve in Reader.md and on GitHub, and flattening docs/features/ onto /docs/ breaks them — so they're resolved to absolute URLs through the same rewriteLink the rendered page uses, now exported, so a link can't mean one thing in HTML and another in markdown. No markdown parser: stringifying would reflow the prose, and the corpus has no link titles or reference definitions (66 destinations, all inline).

The Copy markdown button on each page fetches its own twin, so the HTML doesn't carry a second copy of the prose. A failed fetch leaves the button alone — claiming "Copied" over an empty clipboard is worse than appearing to do nothing.

The FAQ's answers now reach a reader three ways — accordion, JSON-LD, /docs/faq.md — all derived from docs/faq.md alone.

Verification

Clean rebuild: 15 .md twins, JSON-LD in dist/index.html and dist/docs/faq/index.html, copy button on all 15 docs pages, no unrewritten relative links in the markdown, and the existing href invariant (only absolute GitHub URLs) still holds.

Driven in a browser: 5,469 characters of reading.md on the clipboard, the label and tick through the full cycle, no console errors, FAQ accordion and search unaffected.

Two things worth a look

• Cloudflare's "Block AI Scrapers and Crawlers" / Bot Fight Mode is a zone setting, not a file in this repo. If it's on, none of this is reachable by the audience it's for.
• The .md files will serve as text/markdown, which most browsers download rather than display. That's right for agents, so it's left alone. Forcing text/plain needs a _headers rule, and I didn't want to guess whether Cloudflare's wildcard supports a /docs/*.md suffix — if it degraded to /docs/* it would serve all 15 docs pages as plain text. Easy to add and verify once deployed.

web/CLAUDE.md gains a Machine readers section, including the silent failure mode this introduces: the FAQPage JSON-LD comes from rehype-faq-accordion via remarkPluginFrontmatter, and a stale Astro content cache drops it with the build still green. That bit me once while building this.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JGMQ49hvA936om3GQfNTAn