feat(404): generate a /404.md wayfinding guide, drop the not-found scaffold

[arielshulman29]

Why

The agent-aware 404 previously altered HTML that humans also see: an opt-in scaffolded not-found.tsx with an "If you are an AI agent" section. Two problems:

1. Intrusive. A 404 page is the site owner's design domain. The unstyled scaffold never matched anyone's design system and got rewritten anyway — and since ax never overwrites it, it was really a one-shot template.
2. Root-only detection was false comfort. App Router not-found.* is per-segment: a notFound() thrown inside app/docs/[slug]/page.tsx renders the nearest not-found file, bypassing the root one. Those dynamic-route misses are exactly the ones the wayfinding middleware structurally can't answer (the URL matches a route pattern, so existence is only knowable at request time).

New division of labor: the human-visible 404 belongs entirely to the user; ax only owns machine artifacts.

What

• Generate public/404.md every build — the "404 wayfinding guide": discovery links + real routes, rendered by the same renderWayfinding the middleware serves (new URL-neutral title variant), under the standard generated-markdown frontmatter. Gated on markdownTwins like auth.md, shown at the review gate, size-measured, generated-marker overwrite/delete guard. Rendered from serving-manifest data with this run's own outputs (planned twins, auth guide, catalog, cards) overlaid, so the guide describes the build it ships with — the CLI test pins that the twin written in the same run is linked.
• agent-404.ts is detect-and-recommend only. Scans every app/**/not-found.* (segment-level included) plus pages/404.*, across both routers. Tiered recommendations: no 404 page → create a standard one in your own design system plus one <link rel="alternate" type="text/markdown" href="/404.md" /> tag; pages without the tag → one message naming exactly the unlinked files. The tag is invisible to humans; an agent that fetched the HTML 404 follows it to the guide in one hop.
• Deleted: the scaffold, the not-found-agent-data module, and the scaffoldAgent404 config option (schema, defaults, init wizard — now six setup items; package is unpublished, so no deprecation path needed).
• Report: agent404 now carries per-page granularity (pages: [{source, agentAware}]) plus the written markdownGuide path.
• Manifest contract: artifacts.notFoundMd on both shapes, and /404.md joins the middleware's protectDiscovery set so a wrapped bot-gate can't silently un-publish it.
• Gating fix surfaced by the goldens: renderWayfinding listed gated routes under "Pages that do exist" (the markdown-twins fixture's /private landed in the public file). Gated paths are now filtered — a login-walled page is no place to send a lost agent, and the gating policy never advertises a gated surface as open. Applies to the middleware body too, with a test.

Verification

• 734 unit/integration tests pass; typecheck and prettier clean.
• All 17 fixture golden reports regenerated from real next build runs (pnpm reports:regen); every fixture now snapshots its 404.md in twins.golden/, and reports:verify passes.
• pnpm dogfood:middleware (real next start + HTTP probes) passes — negotiation, wayfinding, gated-path fall-through all intact.

🤖 Generated with Claude Code