diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d31f12b..366a167 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -124,6 +124,12 @@ jobs: test -f .output/public/help/index.html test -f .output/public/support/index.html + - name: Verify crawlability artifacts + run: | + test -f .output/public/sitemap.xml + test -f .output/public/robots.txt + grep -q 'https://devcentr.org/sitemap.xml' .output/public/robots.txt + - name: Add .nojekyll for GitHub Pages run: touch .output/public/.nojekyll diff --git a/CHANGELOG.adoc b/CHANGELOG.adoc index aa9c26d..5275d08 100644 --- a/CHANGELOG.adoc +++ b/CHANGELOG.adoc @@ -1,5 +1,10 @@ = Changelog +== 2026-09-14 -- Crawlability maintainer docs + +* Antora site architecture and local development pages document `site-routes.mjs`, postbuild crawl files, and progressive diagram sync scripts. +* Detailed Antora changelog for 2026-09-10 public crawlability; CI checks `sitemap.xml` and `robots.txt` after build. + == 2026-09-10 -- Public crawlability * Publish `/robots.txt` and a build-generated `/sitemap.xml` from the shared prerender route list. diff --git a/docs/modules/ROOT/pages/changelog-details/2026-09-10 - public-crawlability.adoc b/docs/modules/ROOT/pages/changelog-details/2026-09-10 - public-crawlability.adoc new file mode 100644 index 0000000..0c587c1 --- /dev/null +++ b/docs/modules/ROOT/pages/changelog-details/2026-09-10 - public-crawlability.adoc @@ -0,0 +1,57 @@ += 2026-09-10 — Public crawlability +:navtitle: 2026-09-10 Public crawlability +:page-audience: DevCentr site maintainers +:page-usage-context: SEO, GitHub Pages deploy, and route additions +:page-orig-author: Auto on behalf of Ryan Johnson +:page-last-author: Auto on behalf of Ryan Johnson + +== Summary + +Search engines and link previews now get a consistent route list, `robots.txt`, `sitemap.xml`, and a no-JavaScript HTML fallback on the static app shell. + +== Single route list + +`scripts/site-routes.mjs` exports `getPrerenderRoutes()` — the only list of indexable paths. + +It is consumed by: + +* `app.config.ts` — Vinxi `router.prerender.routes` (static HTML per path) +* `scripts/spa-fallback.mjs` — copies the built SPA shell into each prerender directory on GitHub Pages +* `scripts/write-crawlability.mjs` — writes crawl files after postbuild + +`STATIC_ROUTES` covers marketing pages and redirects. News slugs come from `src/lib/news-posts.generated.json` (or `content/news/` before build). Idea slugs are parsed from `src/lib/apps-catalog.ts`. + +When you add a public route, update `STATIC_ROUTES` or the catalog/news sources above — do not maintain a second sitemap list. + +== Crawl files (postbuild) + +`scripts/spa-fallback.mjs` (npm `postbuild`) calls `writeCrawlability()` on `.output/public`: + +* `robots.txt` — `Allow: /` and a `Sitemap:` line pointing at `https://devcentr.org/sitemap.xml` +* `sitemap.xml` — one `` per prerender route at `SITE_ORIGIN` (`https://devcentr.org`) + +Excluded from the sitemap (still prerendered for humans and probes): + +* `/health`, `/status` — operational probes +* `/blog` and `/blog/:slug` — SEO aliases of `/news` (canonical URLs use `/news`) + +A committed `public/robots.txt` documents the policy for local preview; production values are regenerated on every build. + +== No-JS fallback + +`injectHtmlFallback()` ensures prerendered `index.html` has: + +* `` and `<meta name="description">` when the Vinxi shell omits them +* A `<noscript id="crawl-fallback">` block with heading and purpose copy from `HTML_FALLBACK` in `site-routes.mjs` + +After injection, postbuild re-copies the enhanced shell into every SPA fallback directory and `404.html`. + +== Verification + +CI builds the site and checks that `.output/public/sitemap.xml` and `.output/public/robots.txt` exist after postbuild. + +== Knowledge gaps addressed + +* Where to edit routes when shipping a new marketing page or idea slug +* Why `/blog` URLs are prerendered but omitted from the sitemap +* How crawl metadata relates to the SPA fallback pass diff --git a/docs/modules/ROOT/pages/changelog.adoc b/docs/modules/ROOT/pages/changelog.adoc index df54721..aa26cd8 100644 --- a/docs/modules/ROOT/pages/changelog.adoc +++ b/docs/modules/ROOT/pages/changelog.adoc @@ -2,10 +2,17 @@ Timeline of notable documentation and site changes in the devcentr.org repository. +== 2026-09-14 — Crawlability maintainer docs + +* Antora: xref:explanation/site-architecture.adoc#public-crawlability[Public crawlability] on the site architecture page; diagram sync notes on the same page. +* Detailed changelog for the 2026-09-10 ship: xref:changelog-details/2026-09-10 - public-crawlability.adoc[Public crawlability]. +* CI verifies `sitemap.xml` and `robots.txt` after postbuild. + == 2026-09-10 — Public crawlability * Publish `/robots.txt` and generate `/sitemap.xml` from the shared prerender route list in postbuild (`spa-fallback`). * Sitemap omits `/health`, `/status`, and `/blog` aliases of `/news`. +* See xref:changelog-details/2026-09-10 - public-crawlability.adoc[Detailed changelog]. == 2026-09-08 — Themed PlayTime news diagrams diff --git a/docs/modules/ROOT/pages/explanation/site-architecture.adoc b/docs/modules/ROOT/pages/explanation/site-architecture.adoc index 81ae7d8..60355a3 100644 --- a/docs/modules/ROOT/pages/explanation/site-architecture.adoc +++ b/docs/modules/ROOT/pages/explanation/site-architecture.adoc @@ -57,7 +57,46 @@ Catalogue and nested article pages (`PageTrail`) show the access path as `Sectio . `prebuild` — `scripts/sync-advisor-catalog.mjs`, `scripts/sync-bootstrap-profiles.mjs`, then `scripts/build-news.mjs` . `vinxi build` — static route artifacts -. `postbuild` — `scripts/spa-fallback.mjs` for client-side routing on Pages +. `postbuild` — `scripts/spa-fallback.mjs` for client-side routing on Pages, crawl files, and no-JS shell metadata + +[[public-crawlability]] +== Public crawlability + +Indexable URLs share one list: `getPrerenderRoutes()` in `scripts/site-routes.mjs`. + +[cols="1,2",options="header"] +|=== +| Consumer | Role + +| `app.config.ts` `router.prerender.routes` +| Static HTML per path from Vinxi + +| `scripts/spa-fallback.mjs` +| Copy SPA shell into each route directory on GitHub Pages + +| `scripts/write-crawlability.mjs` +| Emit `robots.txt` and `sitemap.xml` under `.output/public` +|=== + +The sitemap uses `https://devcentr.org` and lists prerender paths except `/health`, `/status`, and `/blog` (and `/blog/:slug`) aliases of `/news`. + +`injectHtmlFallback()` adds title, description, and a `<noscript>` crawl block to `index.html` when the build shell is sparse. + +Add a new marketing or idea route by extending `STATIC_ROUTES` or the news/apps catalog sources in `site-routes.mjs` — do not edit sitemap XML by hand. + +See xref:changelog-details/2026-09-10 - public-crawlability.adoc[2026-09-10 crawlability changelog] for the full pipeline. + +[[progressive-diagrams]] +== Progressive themed SVGs + +News diagrams and the home toolchain diagrams ship as pinned SVG artifacts, not inline drawings in TSX. + +* **PlayTime (news)** — `public/news/media/`; sync via `pnpm news:diagrams:sync`, verify with `pnpm news:diagrams:check` and `pnpm news:diagrams:test`. AsciiDoc uses `[.themed-svg]` blocks; runtime upgrade via `@dev-centr/themed-svg` in `src/components/themed-news-content.tsx`. +* **Semantic architecture (home)** — `public/media/diagrams/`; sync via `pnpm semantic:diagrams:sync`, verify with `pnpm semantic:diagrams:check`. Components: `src/components/progressive-diagram.tsx`. + +Provenance JSON beside each artifact set records upstream repo, commit, and SHA-256. CI runs check scripts without sibling clones. + +xref:changelog-details/2026-09-08 - themed-playtime-news-diagrams.adoc[2026-09-08 progressive diagrams changelog] covers the migration from hand-authored SVGs. CI (`.github/workflows/ci.yml`) additionally checks out sibling repos into the workspace root before `build-news.mjs`: @@ -80,4 +119,4 @@ Custom domain `devcentr.org` is configured in the repository's Pages settings. * `src/lib/` — catalog data, changelog loader, theme reveal, site links * `content/news/` — authored news (see xref:explanation/news-and-changelog.adoc[News & changelog pipeline]) * `public/brand/` — SVG logo marks (canonical org assets live in `.github` profile repos) -* `scripts/` — news/changelog build, advisor and bootstrap-profile sync, SPA fallback, logo rasterize +* `scripts/` — news/changelog build, advisor and bootstrap-profile sync, SPA fallback and crawlability (`site-routes.mjs`, `write-crawlability.mjs`), diagram sync, logo rasterize diff --git a/docs/modules/ROOT/pages/how-to/local-development.adoc b/docs/modules/ROOT/pages/how-to/local-development.adoc index c3d4944..65ae1ce 100644 --- a/docs/modules/ROOT/pages/how-to/local-development.adoc +++ b/docs/modules/ROOT/pages/how-to/local-development.adoc @@ -65,8 +65,16 @@ Without siblings, news still builds; changelog ingest logs `changelog skip (miss | `pnpm brand:raster` | Rasterize SVG logos to PNG sizes under `public/brand/` + +| `pnpm news:diagrams:sync` / `pnpm news:diagrams:check` +| Refresh or verify pinned PlayTime news SVGs (`scripts/sync-playtime-diagrams.mjs`) + +| `pnpm semantic:diagrams:sync` / `pnpm semantic:diagrams:check` +| Refresh or verify home semantic architecture SVGs (`scripts/sync-semantic-diagrams.mjs`) |=== +After `pnpm build`, inspect `.output/public/sitemap.xml` and `.output/public/robots.txt` (written in postbuild). New public routes must be added through `scripts/site-routes.mjs` — see xref:explanation/site-architecture.adoc#public-crawlability[Public crawlability]. + == Editor setup Committed `.vscode/settings.json` and `.vscode/extensions.json` enable dprint format-on-save and recommend the SolidJS extension.