Skip to content

feat(fixtures): consolidate corpus around a flagship demo fork + mutation tests - #32

Merged
arielshulman29 merged 1 commit into
mainfrom
feat/flagship-fixture-corpus
Aug 28, 2026
Merged

arielshulman29 merged 1 commit into
mainfrom
feat/flagship-fixture-corpus

Conversation

@arielshulman29

Copy link
Copy Markdown
Collaborator

What

Consolidates the e2e fixture corpus from 21 miniature apps to 11 meaningful ones, centered on a flagship fixture forked from the ora-air demo, and adds a multi-build mutation-test harness for the build-N+1 bugs the one-shot golden layer can't see. Three product bugs the new corpus immediately surfaced are fixed in the same PR.

Fixture corpus (21 → 11)

Fixture Role
flagship (new) Fork of the ora-air demo, enriched: dynamic segment (destinations/[slug] + generateStaticParams), hand-owned llms.txt route, /guide via @next/mdx, sitemap.ts, hand-authored /destinations twin, gated /account page, two mcp-handler servers (public + withMcpAuth-gated), withAx(manifest, botGate) middleware. Absorbs middleware, mcp-adapter, mcp-adapter-gated, mcp-multi-server, llms-txt, openapi, discovery, config-overrides.
flagship-pages (new) Full Pages Router port of the flagship IA — the one deep axis that forks the router model and everything downstream. Absorbs pages-bare, pages-mcp, pages-webmcp-declarative.
webmcp (new) Declarative <form toolname> + imperative registerTool() on one page (merges webmcp-declarative + webmcp-imperative, keeps the false-positive-prose regression).
next-auth (new) Thin single-axis fixture: real next-auth dependency + [...nextauth] mount — net-new real-build coverage for provider detection.
bare, bare-js, deploy-variants, edge-cases, monorepo, markdown-twins, hybrid Kept unchanged (absence baseline / JS axis / basePath / WebMCP decoys / nested app / twin-ladder rungs / app+pages union).

Design notes:

  • Zero secrets: the demo's Clerk wiring is replaced with a stub withMcpAuth verifier + hand-written static RFC 9728/8414 .well-known documents — same detection surface, deterministic CI builds. Provider detection stays covered by unit tests + the next-auth fixture.
  • The golden/conformance pipeline needed no changes — snapshot discovery is dynamic. Only fixtures-integration.test.ts (every deleted fixture's assertion mapped to an explicit new home) and dogfood-middleware.mjs (retargeted to flagship, plus a new composed-middleware probe: scripted UAs still get bot-gated through withAx's second argument) reference fixtures by name.
  • The real win is CI wall-clock (17 → 8 report builds) and maintenance surface; the flagship additionally tests detector composition, which no isolated fixture could.

Mutation-test harness (pnpm mutation:test, wired into CI)

Copies the flagship's tracked sources to a tmp dir (one absolute symlink reuses its pnpm node_modules), then runs build → mutate → rebuild sequences with targeted delta assertions:

  1. Idempotence — steady-state rebuilds are byte-identical (report + every twin), robots.txt append never doubles, nothing is deleted on a no-change rebuild.
  2. Auth-method change — flipping the declared entry auth api_key ↔ oauth2 propagates to the report, catalog entry, and auth.md with no remnant of the old method, and never re-opens the MCP review gate (a declared auth marks its mount reviewed).
  3. User-edit preservation — a filled-in scaffold TODO survives the next build.
  4. Staleness — deleting a page removes its orphaned twin and manifest entry.
  5. Additive — a new page and a new MCP tool land in twins, cards, and the report.
  6. Manifest drift — removing the gated page refreshes gatedPaths via the prebuild.

Bugs the new corpus caught (fixed here)

  • MDX pages were invisible to the router model: /guide built and got a twin, but was missing from the serving manifest — agents requesting it got the 404 wayfinding page for a route that exists. buildRouterModel now takes pageExtensions (from next.config) and counts page.mdx/.mdx pages as routes exactly when Next serves them, staying conservative otherwise.
  • 404.md was swept as a stale twin on every build after the first: it carries the generated-by marker but is lifecycle-owned by agent-404, so the sweep deleted it and its owner rewrote it — a phantom markdownTwins.deleted entry CI never saw (it always builds once from clean). The sweep now skips auth.md/404.md; stale-twin deletions are also logged per-path like server cards.
  • The init multiselect garbled on arrow keys: the raw-mode selector paused the shared readline interface but resumed stdin, leaving readline's keypress listener live — arrow-up recalled the previous question's answer from history and painted it over the option rows; printable keys were echoed at the cursor. Reproduced under a real PTY, fixed by suspending readline's keypress listeners for the duration of the selector (Node's keypress machinery re-attaches on restore — verified with follow-up questions). Also guards a PTY reporting columns: 0, which truncated every row to the empty string.

Verification

  • 736 unit tests, 8/8 report/twin/card snapshots (hermetic reports:regen), 8/8 ARD conformance, 18 dogfood probes, 29 mutation probes, typecheck, prettier — all green.
  • Only golden diff from the fixes: flagship's 404.md grew one line (/guide now listed as a real route).

🤖 Generated with Claude Code

…tion tests

Replace 14 single-scenario fixtures with a flagship fixture (a fork of the
ora-air demo enriched with a dynamic segment, hand-owned llms.txt, MDX page,
sitemap, gated page, and two MCP servers behind a secret-free auth stub), a
full Pages Router twin, and thin single-axis fixtures (webmcp, next-auth).
21 fixtures become 11; the golden/conformance pipeline needed no changes.

Add scripts/mutation-tests.mjs: multi-build sequences over the flagship
(idempotence, user-edit preservation, orphaned-twin cleanup, additive change,
auth-method flip, manifest drift) — the build-N+1 bugs the one-shot golden
layer can't see. Wired into CI after the dogfood step.

Fix three product bugs the new corpus surfaced:
- the router model now counts MDX pages as routes exactly when next.config
  pageExtensions serves them, so /guide is in the serving manifest instead of
  404-wayfinding for agents
- the stale-twin sweep no longer deletes ax's own 404.md every second build
  (phantom markdownTwins.deleted entry); twin deletions now log per-path
- the raw-mode multiselect no longer garbles on arrow keys: the shared
  readline interface's keypress listeners are suspended during the selector
  (history recall was rewriting earlier answers over the rows), and a
  zero-column PTY no longer truncates every row to nothing

Retarget dogfood-middleware.mjs to the flagship (adds a composed-middleware
probe: withAx's second argument still bot-gates scripted UAs).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arielshulman29
arielshulman29 merged commit 7a273cb into main Aug 28, 2026
2 checks passed
arielshulman29 added a commit that referenced this pull request Aug 28, 2026
Brings in the flagship fixture-corpus consolidation (PR #32). Old
fixtures deleted on main resolved as deletions; the surviving goldens
were regenerated from the merged source (8 snapshots now, down from 17).
The new corpus and mutation harness were authored against the
pre-rename names, so the sweep re-applied @ora-ai/ax-nextjs across the
new fixtures and fixed scripts/mutation-tests.mjs reading
.ora/report.json (4 spots) — all mutation probes pass after the fix.
dogfood-middleware.mjs keeps main's flagship retarget plus the renamed
import path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant