docs: crawlability runbook and CI crawl artifact checks

.github/workflows/ci.yml

@@ -124,6 +124,12 @@ jobs:
           test -f .output/public/help/index.html
           test -f .output/public/support/index.html
 
+      - name: Verify crawlability artifacts
+        run: |
+          test -f .output/public/sitemap.xml
+          test -f .output/public/robots.txt
+          grep -q 'https://devcentr.org/sitemap.xml' .output/public/robots.txt
+
       - name: Add .nojekyll for GitHub Pages
         run: touch .output/public/.nojekyll
 

CHANGELOG.adoc

@@ -1,5 +1,10 @@
 = Changelog
 
+== 2026-09-14 -- Crawlability maintainer docs
+
+* Antora site architecture and local development pages document `site-routes.mjs`, postbuild crawl files, and progressive diagram sync scripts.
+* Detailed Antora changelog for 2026-09-10 public crawlability; CI checks `sitemap.xml` and `robots.txt` after build.
+
 == 2026-09-10 -- Public crawlability
 
 * Publish `/robots.txt` and a build-generated `/sitemap.xml` from the shared prerender route list.

docs/modules/ROOT/pages/changelog-details/2026-09-10 - public-crawlability.adoc

@@ -0,0 +1,57 @@
+= 2026-09-10 — Public crawlability
+:navtitle: 2026-09-10 Public crawlability
+:page-audience: DevCentr site maintainers
+:page-usage-context: SEO, GitHub Pages deploy, and route additions
+:page-orig-author: Auto on behalf of Ryan Johnson
+:page-last-author: Auto on behalf of Ryan Johnson
+
+== Summary
+
+Search engines and link previews now get a consistent route list, `robots.txt`, `sitemap.xml`, and a no-JavaScript HTML fallback on the static app shell.
+
+== Single route list
+
+`scripts/site-routes.mjs` exports `getPrerenderRoutes()` — the only list of indexable paths.
+
+It is consumed by:
+
+* `app.config.ts` — Vinxi `router.prerender.routes` (static HTML per path)
+* `scripts/spa-fallback.mjs` — copies the built SPA shell into each prerender directory on GitHub Pages
+* `scripts/write-crawlability.mjs` — writes crawl files after postbuild
+
+`STATIC_ROUTES` covers marketing pages and redirects. News slugs come from `src/lib/news-posts.generated.json` (or `content/news/` before build). Idea slugs are parsed from `src/lib/apps-catalog.ts`.
+
+When you add a public route, update `STATIC_ROUTES` or the catalog/news sources above — do not maintain a second sitemap list.
+
+== Crawl files (postbuild)
+
+`scripts/spa-fallback.mjs` (npm `postbuild`) calls `writeCrawlability()` on `.output/public`:
+
+* `robots.txt` — `Allow: /` and a `Sitemap:` line pointing at `https://devcentr.org/sitemap.xml`
+* `sitemap.xml` — one `<loc>` per prerender route at `SITE_ORIGIN` (`https://devcentr.org`)
+
+Excluded from the sitemap (still prerendered for humans and probes):
+
+* `/health`, `/status` — operational probes
+* `/blog` and `/blog/:slug` — SEO aliases of `/news` (canonical URLs use `/news`)
+
+A committed `public/robots.txt` documents the policy for local preview; production values are regenerated on every build.
+
+== No-JS fallback
+
+`injectHtmlFallback()` ensures prerendered `index.html` has:
+
+* `<title>` and `<meta name="description">` when the Vinxi shell omits them
+* A `<noscript id="crawl-fallback">` block with heading and purpose copy from `HTML_FALLBACK` in `site-routes.mjs`
+
+After injection, postbuild re-copies the enhanced shell into every SPA fallback directory and `404.html`.
+
+== Verification
+
+CI builds the site and checks that `.output/public/sitemap.xml` and `.output/public/robots.txt` exist after postbuild.
+
+== Knowledge gaps addressed
+
+* Where to edit routes when shipping a new marketing page or idea slug
+* Why `/blog` URLs are prerendered but omitted from the sitemap
+* How crawl metadata relates to the SPA fallback pass

docs/modules/ROOT/pages/changelog.adoc

@@ -2,10 +2,17 @@
 
 Timeline of notable documentation and site changes in the devcentr.org repository.
 
+== 2026-09-14 — Crawlability maintainer docs
+
+* Antora: xref:explanation/site-architecture.adoc#public-crawlability[Public crawlability] on the site architecture page; diagram sync notes on the same page.
+* Detailed changelog for the 2026-09-10 ship: xref:changelog-details/2026-09-10 - public-crawlability.adoc[Public crawlability].
+* CI verifies `sitemap.xml` and `robots.txt` after postbuild.
+
 == 2026-09-10 — Public crawlability
 
 * Publish `/robots.txt` and generate `/sitemap.xml` from the shared prerender route list in postbuild (`spa-fallback`).
 * Sitemap omits `/health`, `/status`, and `/blog` aliases of `/news`.
+* See xref:changelog-details/2026-09-10 - public-crawlability.adoc[Detailed changelog].
 
 == 2026-09-08 — Themed PlayTime news diagrams
 

docs/modules/ROOT/pages/explanation/site-architecture.adoc

@@ -57,7 +57,46 @@ Catalogue and nested article pages (`PageTrail`) show the access path as `Sectio
 
 . `prebuild` — `scripts/sync-advisor-catalog.mjs`, `scripts/sync-bootstrap-profiles.mjs`, then `scripts/build-news.mjs`
 . `vinxi build` — static route artifacts
-. `postbuild` — `scripts/spa-fallback.mjs` for client-side routing on Pages
+. `postbuild` — `scripts/spa-fallback.mjs` for client-side routing on Pages, crawl files, and no-JS shell metadata
+
+[[public-crawlability]]
+== Public crawlability
+
+Indexable URLs share one list: `getPrerenderRoutes()` in `scripts/site-routes.mjs`.
+
+[cols="1,2",options="header"]
+|===
+| Consumer | Role
+
+| `app.config.ts` `router.prerender.routes`
+| Static HTML per path from Vinxi
+
+| `scripts/spa-fallback.mjs`
+| Copy SPA shell into each route directory on GitHub Pages
+
+| `scripts/write-crawlability.mjs`
+| Emit `robots.txt` and `sitemap.xml` under `.output/public`
+|===
+
+The sitemap uses `https://devcentr.org` and lists prerender paths except `/health`, `/status`, and `/blog` (and `/blog/:slug`) aliases of `/news`.
+
+`injectHtmlFallback()` adds title, description, and a `<noscript>` crawl block to `index.html` when the build shell is sparse.
+
+Add a new marketing or idea route by extending `STATIC_ROUTES` or the news/apps catalog sources in `site-routes.mjs` — do not edit sitemap XML by hand.
+
+See xref:changelog-details/2026-09-10 - public-crawlability.adoc[2026-09-10 crawlability changelog] for the full pipeline.
+
+[[progressive-diagrams]]
+== Progressive themed SVGs
+
+News diagrams and the home toolchain diagrams ship as pinned SVG artifacts, not inline drawings in TSX.
+
+* **PlayTime (news)** — `public/news/media/`; sync via `pnpm news:diagrams:sync`, verify with `pnpm news:diagrams:check` and `pnpm news:diagrams:test`. AsciiDoc uses `[.themed-svg]` blocks; runtime upgrade via `@dev-centr/themed-svg` in `src/components/themed-news-content.tsx`.
+* **Semantic architecture (home)** — `public/media/diagrams/`; sync via `pnpm semantic:diagrams:sync`, verify with `pnpm semantic:diagrams:check`. Components: `src/components/progressive-diagram.tsx`.
+
+Provenance JSON beside each artifact set records upstream repo, commit, and SHA-256. CI runs check scripts without sibling clones.
+
+xref:changelog-details/2026-09-08 - themed-playtime-news-diagrams.adoc[2026-09-08 progressive diagrams changelog] covers the migration from hand-authored SVGs.
 
 CI (`.github/workflows/ci.yml`) additionally checks out sibling repos into the workspace root before `build-news.mjs`:
 
@@ -80,4 +119,4 @@ Custom domain `devcentr.org` is configured in the repository's Pages settings.
 * `src/lib/` — catalog data, changelog loader, theme reveal, site links
 * `content/news/` — authored news (see xref:explanation/news-and-changelog.adoc[News & changelog pipeline])
 * `public/brand/` — SVG logo marks (canonical org assets live in `.github` profile repos)
-* `scripts/` — news/changelog build, advisor and bootstrap-profile sync, SPA fallback, logo rasterize
+* `scripts/` — news/changelog build, advisor and bootstrap-profile sync, SPA fallback and crawlability (`site-routes.mjs`, `write-crawlability.mjs`), diagram sync, logo rasterize

docs/modules/ROOT/pages/how-to/local-development.adoc

@@ -65,8 +65,16 @@ Without siblings, news still builds; changelog ingest logs `changelog skip (miss
 
 | `pnpm brand:raster`
 | Rasterize SVG logos to PNG sizes under `public/brand/`
+
+| `pnpm news:diagrams:sync` / `pnpm news:diagrams:check`
+| Refresh or verify pinned PlayTime news SVGs (`scripts/sync-playtime-diagrams.mjs`)
+
+| `pnpm semantic:diagrams:sync` / `pnpm semantic:diagrams:check`
+| Refresh or verify home semantic architecture SVGs (`scripts/sync-semantic-diagrams.mjs`)
 |===
 
+After `pnpm build`, inspect `.output/public/sitemap.xml` and `.output/public/robots.txt` (written in postbuild). New public routes must be added through `scripts/site-routes.mjs` — see xref:explanation/site-architecture.adoc#public-crawlability[Public crawlability].
+
 == Editor setup
 
 Committed `.vscode/settings.json` and `.vscode/extensions.json` enable dprint format-on-save and recommend the SolidJS extension.