feat(fixtures): consolidate corpus around a flagship demo fork + mutation tests - #32
Merged
Merged
Conversation
…tion tests Replace 14 single-scenario fixtures with a flagship fixture (a fork of the ora-air demo enriched with a dynamic segment, hand-owned llms.txt, MDX page, sitemap, gated page, and two MCP servers behind a secret-free auth stub), a full Pages Router twin, and thin single-axis fixtures (webmcp, next-auth). 21 fixtures become 11; the golden/conformance pipeline needed no changes. Add scripts/mutation-tests.mjs: multi-build sequences over the flagship (idempotence, user-edit preservation, orphaned-twin cleanup, additive change, auth-method flip, manifest drift) — the build-N+1 bugs the one-shot golden layer can't see. Wired into CI after the dogfood step. Fix three product bugs the new corpus surfaced: - the router model now counts MDX pages as routes exactly when next.config pageExtensions serves them, so /guide is in the serving manifest instead of 404-wayfinding for agents - the stale-twin sweep no longer deletes ax's own 404.md every second build (phantom markdownTwins.deleted entry); twin deletions now log per-path - the raw-mode multiselect no longer garbles on arrow keys: the shared readline interface's keypress listeners are suspended during the selector (history recall was rewriting earlier answers over the rows), and a zero-column PTY no longer truncates every row to nothing Retarget dogfood-middleware.mjs to the flagship (adds a composed-middleware probe: withAx's second argument still bot-gates scripted UAs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arielshulman29
added a commit
that referenced
this pull request
Aug 28, 2026
Brings in the flagship fixture-corpus consolidation (PR #32). Old fixtures deleted on main resolved as deletions; the surviving goldens were regenerated from the merged source (8 snapshots now, down from 17). The new corpus and mutation harness were authored against the pre-rename names, so the sweep re-applied @ora-ai/ax-nextjs across the new fixtures and fixed scripts/mutation-tests.mjs reading .ora/report.json (4 spots) — all mutation probes pass after the fix. dogfood-middleware.mjs keeps main's flagship retarget plus the renamed import path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Consolidates the e2e fixture corpus from 21 miniature apps to 11 meaningful ones, centered on a flagship fixture forked from the ora-air demo, and adds a multi-build mutation-test harness for the build-N+1 bugs the one-shot golden layer can't see. Three product bugs the new corpus immediately surfaced are fixed in the same PR.
Fixture corpus (21 → 11)
flagship(new)destinations/[slug]+generateStaticParams), hand-ownedllms.txtroute,/guidevia@next/mdx,sitemap.ts, hand-authored/destinationstwin, gated/accountpage, two mcp-handler servers (public +withMcpAuth-gated),withAx(manifest, botGate)middleware. Absorbsmiddleware,mcp-adapter,mcp-adapter-gated,mcp-multi-server,llms-txt,openapi,discovery,config-overrides.flagship-pages(new)pages-bare,pages-mcp,pages-webmcp-declarative.webmcp(new)<form toolname>+ imperativeregisterTool()on one page (mergeswebmcp-declarative+webmcp-imperative, keeps the false-positive-prose regression).next-auth(new)next-authdependency +[...nextauth]mount — net-new real-build coverage for provider detection.bare,bare-js,deploy-variants,edge-cases,monorepo,markdown-twins,hybridDesign notes:
withMcpAuthverifier + hand-written static RFC 9728/8414.well-knowndocuments — same detection surface, deterministic CI builds. Provider detection stays covered by unit tests + thenext-authfixture.fixtures-integration.test.ts(every deleted fixture's assertion mapped to an explicit new home) anddogfood-middleware.mjs(retargeted to flagship, plus a new composed-middleware probe: scripted UAs still get bot-gated throughwithAx's second argument) reference fixtures by name.Mutation-test harness (
pnpm mutation:test, wired into CI)Copies the flagship's tracked sources to a tmp dir (one absolute symlink reuses its pnpm
node_modules), then runs build → mutate → rebuild sequences with targeted delta assertions:api_key↔oauth2propagates to the report, catalog entry, andauth.mdwith no remnant of the old method, and never re-opens the MCP review gate (a declared auth marks its mount reviewed).gatedPathsvia the prebuild.Bugs the new corpus caught (fixed here)
/guidebuilt and got a twin, but was missing from the serving manifest — agents requesting it got the 404 wayfinding page for a route that exists.buildRouterModelnow takespageExtensions(from next.config) and countspage.mdx/.mdxpages as routes exactly when Next serves them, staying conservative otherwise.404.mdwas swept as a stale twin on every build after the first: it carries the generated-by marker but is lifecycle-owned by agent-404, so the sweep deleted it and its owner rewrote it — a phantommarkdownTwins.deletedentry CI never saw (it always builds once from clean). The sweep now skipsauth.md/404.md; stale-twin deletions are also logged per-path like server cards.keypresslistener live — arrow-up recalled the previous question's answer from history and painted it over the option rows; printable keys were echoed at the cursor. Reproduced under a real PTY, fixed by suspending readline's keypress listeners for the duration of the selector (Node's keypress machinery re-attaches on restore — verified with follow-up questions). Also guards a PTY reportingcolumns: 0, which truncated every row to the empty string.Verification
reports:regen), 8/8 ARD conformance, 18 dogfood probes, 29 mutation probes, typecheck, prettier — all green.404.mdgrew one line (/guidenow listed as a real route).🤖 Generated with Claude Code