seo/geo: AgentXRay crawlability + LLM retrieval

[alloevil]

Makes the GitHub Pages demo at https://alloevil.github.io/AgentXRay/ crawlable by search engines and retrievable/citable by LLM answer engines. No behaviour change to the app.

Publish-root category: GENERATED (vite build) — artifacts added to the SOURCE, not the output

frontend/dist is a build output and is gitignored (.gitignore line frontend/dist/), so nothing may be hand-committed there. Evidence used to decide where the files belong:

• .github/workflows/pages.yml runs cd frontend && npm ci && VITE_DEMO=1 npm run build, then actions/upload-pages-artifact with path: frontend/dist.
• frontend/vite.config.ts sets build.outDir: 'dist' and does not set publicDir, so vite's default publicDir of frontend/public applies and its contents are copied verbatim into dist. It also sets base: '/AgentXRay/' when VITE_DEMO === '1', which the workflow does set.
• frontend/public/ did not exist before this PR; it is created here.
• No .nojekyll file exists anywhere in the repo, and it is moot here: Jekyll never runs on a workflow-built Pages site, and this sitemap lists no converted-markdown URL to begin with.

I did not guess — I ran the real build (VITE_DEMO=1 npm run build) and listed the output. See Verification.

Files

• frontend/public/robots.txt (new) — the site had no robots.txt at all. Byte-format copied verbatim from agent-harness-evals/docs/robots.txt, with no deviation, because nothing published here needs excluding. Scope caveat, stated plainly: this file is advisory, not authoritative. The Robots Exclusion Protocol is origin-scoped at the root path, and this site is a subpath of the single origin alloevil.github.io, so compliant crawlers read https://alloevil.github.io/robots.txt and never this copy — its Sitemap: directive included. Effective sitemap discovery for this origin lives in the root robots.txt, handled separately. The file is still worth shipping: it is the fleet convention, some tools and AI crawlers do probe subpaths, and it becomes authoritative if this site ever moves to its own domain.
• frontend/public/sitemap.xml (new) — one <loc>, the trailing-slash home URL. AgentXRay is a single-page app: frontend/index.html is the only HTML entry point in the whole frontend and vite.config.ts declares no rollupOptions.input, so dist contains exactly one HTML file. Listing anything else would be a 404.
• frontend/public/llms.txt (new) — llms.txt convention: definition, prose, then Docs / Evidence / Source link sections. Every link points at a file that exists on master or at a URL I fetched.
• frontend/public/llms-full.txt (new) — self-contained answer-engine context: what it is, install, quickstart, verifiable claims, when to use / when NOT to use, comparison, FAQ. Written so an LLM can answer "what is AgentXRay / how do I install it / does it send my data anywhere" without fetching anything else.
• frontend/public/claims.json (new) — machine-readable claims. See "Notes for review" for exactly what is and is not claimed.
• frontend/index.html — added one SoftwareApplication JSON-LD block before </head>. The site had no structured data. The head already had title/description/canonical/og/twitter, so no meta tag was added, changed or duplicated; in particular the existing og:image was left exactly as it was (it points at screenshots/tool-calls.png, which I confirmed exists on disk, 352 KB PNG — so it is a valid raster and there was no reason to touch it).
• README.md, README.zh-CN.md — GEO restructure: added an H1 plus a front-loaded one-sentence definition as the first body text, relocated (never deleted) the hero/badge/nav blocks below it, added ## What it is, ## When to use it, ## When NOT to use it, ## FAQ, and a comparison table against LangSmith / Langfuse — the only alternatives the repo itself discusses. ## Quick Start was renamed to ## Install and the nav anchor updated to match so the link still resolves. Untouched sections were not reflowed.

Two factual corrections in the READMEs while restructuring: npm i -g agent-xray -> npm i -g @alloevil/agent-xray (that is the published name), and the stale "(once the package is published to npm)" caveat was dropped, because @alloevil/agent-xray is live on the registry at 1.17.1.

Notes for review

• JSON-LD facts are all repo-derived: license from the MIT LICENSE file, programmingLanguage: "JavaScript, TypeScript" from the Express/Node backend plus the React + TypeScript frontend, codeRepository and url from the actual origins. No version number is embedded, so it cannot go stale.
• claims.json contains exactly one claim: 7 registered log-format adapters. That number is the key count of the PLATFORMS registry in lib/platforms/index.js, and the file ships the exact one-line command that prints it. No performance, accuracy or benchmark numbers are claimed, because this repository measures none — claims.json says so explicitly in a notClaimed field, and llms-full.txt repeats it, so an answer engine cannot infer a benchmark that does not exist.
• No markdown URL is listed, so the .md vs .html question does not arise. The sitemap holds exactly one entry, the trailing-slash origin, served by the built dist/index.html. Nothing was inferred from the Pages build type.
• No generator owns any file I touched. frontend/index.html is the hand-maintained vite entry template, not a file rendered from data; vite only copies frontend/public verbatim. There is therefore no generator to move sitemap/robots generation into, and no --check-style artifact test to extend. The sitemap is drift-proof by construction: it has one URL because the app has one HTML entry point, and gaining a second would require an explicit rollupOptions.input change to vite.config.ts.
• No existing test was blocked, narrowed or skipped. Searched test/ for frontend/index.html, canonical, ld+json, <head and og: — no matches, so nothing in the suite asserts against the head I edited.

Published-content audit

Nothing was withheld from the sitemap for content reasons — I checked rather than skipped. The published surface here is narrow by construction, because only frontend/dist is uploaded, and I listed it after the real build: 8 files total — index.html, assets/index-*.js, assets/index-*.css, and the five artifacts this PR adds. No markdown is published at all, so there is no .md versus .html question and no repository doc, draft, planning note or internal file reaches the site. Nothing needed a Disallow.

One thing here does deserve a second look, since it ships inside the published JS bundle rather than as a file: the demo session logs under frontend/demo/sample-logs, which the live site renders. I verified they are genuinely synthetic rather than trusting the README's word for it — across all five log files the only cwd values are /demo/acme-cache, /demo/api-service, /demo/orders-service and /demo/webapp, and a scan found zero /Users/… or /home/… home paths, zero API-key-shaped strings, zero GitHub tokens and zero email addresses. No real session data or personal path is published.

The sitemap lists pages only. Assets resolve 200 but are not pages and are deliberately excluded, so it is not padded with the JS, CSS or screenshots.

Verification

Everything below was actually run, not assumed.

• Ran the real Pages build: cd frontend && npm ci && VITE_DEMO=1 npm run build. It succeeded, and dist/ then contained robots.txt, sitemap.xml, llms.txt, llms-full.txt and claims.json alongside index.html and assets/. cmp reported all five copies byte-identical to their frontend/public/ sources, which is the proof that publicDir reaches the published artifact.
• Confirmed the demo base path is applied in the build output: dist/index.html references /AgentXRay/assets/index-*.js and /AgentXRay/assets/index-*.css.
• XML parses: python3 -c "import xml.dom.minidom;xml.dom.minidom.parse('frontend/public/sitemap.xml')" — OK.
• JSON parses: python3 -m json.tool frontend/public/claims.json — OK.
• JSON-LD parses: extracted the application/ld+json payload from both frontend/index.html and the built dist/index.html and loaded it with json.loads — OK in both, so the block survives the vite HTML transform.
• Claim reproduced: node -e 'console.log(Object.keys(require("./lib/platforms/index.js").PLATFORMS).join(", "))' prints openclaw, codex, claude-code, omp, dsh, gemini, hermes — 7, matching claims.json exactly.
• og:image target verified present on disk: screenshots/tool-calls.png (352 KB PNG). The other screenshots referenced by the READMEs (main-view.png, spawn-tracking.png, codex-view.png, settings-panel.png) also all exist.
• Every path referenced by llms.txt verified to exist on master: README.md, README.zh-CN.md, SPEC.md, SPAWN_TREE_SPEC.md, GRAPH_SPEC.md, docs/ROADMAP.md, lib/platforms/index.js, test/, .github/workflows/test.yml. The npm link was verified live: the registry returns 200 with dist-tags.latest = 1.17.1.
• Sitemap enumeration verified against reality, not guessed: frontend/index.html is the only HTML file under frontend/ (excluding node_modules), hence a single-URL sitemap.
• No formatter, linter or test suite was run, and no CI config was modified. The build was run only as evidence; frontend/dist and frontend/node_modules are gitignored and are not part of this PR.