feat(404): generate a /404.md wayfinding guide, drop the not-found scaffold

.gitignore

@@ -37,6 +37,7 @@ fixtures/mcp-multi-server/public/index.md
 fixtures/discovery/public/index.md
 fixtures/middleware/public/index.md
 fixtures/*/public/auth.md
+fixtures/*/public/404.md
 
 # The serving-manifest data module a fixture's `prebuild: ax manifest` regenerates every build —
 # build output there (real projects may commit theirs; a fixture must rebuild it from scratch).

README.md

@@ -37,9 +37,11 @@ ax detects gated MCP and OpenAPI surfaces, lets you declare the real auth flow w
 see one, and publishes both in a generated `auth.md`. Your 401/403 responses stay honest — ax never
 touches your route handlers.
 
-**Steer lost agents back — agent-aware 404 + wayfinding middleware**
-An opt-in `not-found.tsx` lists your real routes and discovery artifacts for agents that hit a dead
-end; the runtime middleware gives the same wayfinding response to any URL that matches nothing.
+**Steer lost agents back — `/404.md` wayfinding guide + middleware**
+Every build generates `/404.md` — your real routes and discovery artifacts, for agents that hit a
+dead end. Your 404 page stays yours: ax only asks it to carry one invisible
+`<link rel="alternate" type="text/markdown" href="/404.md">` tag. The runtime middleware serves the
+same wayfinding response directly for any URL that matches nothing.
 
 **Tell agents who you are and gain their trust — JSON-LD scaffolding**
 An opt-in `Organization` JSON-LD component is scaffolded from your `package.json`. ax prints the
@@ -136,13 +138,12 @@ const config: AxConfig = {
   emit: 'static',
   // Scaffold a starter llms.txt filled with your real routes and artifacts. Opt-in.
   scaffoldLlmsTxt: true,
-  // Scaffold an agent-aware app/not-found.tsx plus a regenerated route-manifest module. Opt-in.
-  scaffoldAgent404: true,
   // Add Sitemap:/Agentmap: discovery lines to robots.txt (or write one if you have none). Opt-in.
   scaffoldRobots: true,
   // Scaffold an Organization JSON-LD component. Opt-in; ax never wires it into your layout.
   scaffoldJsonLd: true,
-  // Generate markdown twins of your pages, plus /auth.md when surfaces are gated. Default on.
+  // Generate markdown twins of your pages, /auth.md when surfaces are gated, and the /404.md
+  // wayfinding guide lost agents continue from. Default on.
   markdownTwins: true,
   // Write .ora/report.json, the machine-readable build report. Opt-in.
   report: true,

docs-internal/PLAN.md

@@ -859,6 +859,16 @@ Two additions shipped alongside Phase 4, both following existing conventions:
       sync, the agreed pattern is a composing higher-order middleware that wraps the user's
       existing middleware instead of owning the file, which removes the invasiveness objection.
       See the Vercel strategy sync section.
+      **Update (2026-08-28): redesigned — the scaffold, data module, and `scaffoldAgent404` option
+      are gone.** A scaffolded human-visible page was intrusive (users design their own 404s; the
+      unstyled scaffold got rewritten anyway), and root-only detection was false comfort: App
+      Router `not-found.*` is per-segment, and dynamic-route misses render the *nearest* one,
+      bypassing both the root page and the middleware's wayfinding branch (the URL matches a route
+      pattern). Now: ax generates only `public/404.md` (the wayfinding-guide artifact, same
+      `renderWayfinding` body as the middleware, gated on `markdownTwins` like `auth.md`);
+      detection scans every `app/**/not-found.*` plus `pages/404.*` and recommends one
+      `<link rel="alternate" type="text/markdown" href="/404.md">` tag per unlinked page — the
+      human 404 surface belongs to the user, ax only owns machine artifacts.
 - [x] **Machine-readable build report** (`src/report.ts`; `--report[=path]` / `ax.config`
       `report`, default off; default path `.ora/report.json`). The structured twin of the CLI
       output: entries + written paths, MCP mounts + server card, WebMCP sites, per-artifact
@@ -922,8 +932,10 @@ consumer's source tree unasked), **write-once or append-only** (never overwrite,
     deliberately does **not** wire the component into `app/layout.tsx` (editing the file every page
     renders through, behind the owner's back, is not a postbuild step's call) — it prints the exact
     import + element instead.
-  - **`scaffoldAgent404`** (Phase 4.5) — the same pattern: a user-owned `not-found.tsx` written once,
-    backed by a data module regenerated every build from the real route tree.
+  - **`scaffoldAgent404`** (Phase 4.5) — was the same pattern (a user-owned `not-found.tsx` written
+    once, backed by a regenerated data module) until the 2026-08-28 redesign removed it; see the
+    Phase 4.5 update above. The 404 story is now the generated `public/404.md` + a link-tag
+    recommendation, with no scaffold and no config option.
 
 **3. The report became a handoff, not a log (`src/report.ts` v2 + `src/ora-checks.ts`).** The plugin's
 half of the loop is deterministic detection/emission/scaffolding; the other half — authoring content,
@@ -1147,8 +1159,9 @@ ceremony.
      entries, with the default floor (`/api/auth/**`, `/api/webhooks/**`) pre-noted. Answers
      become an `isGated` matcher (compose `defaultIsGated` unless the user deselects the floor).
      A reviewed detection beats free-text glob authoring.
-   - **Scaffold opt-ins** (`scaffoldLlmsTxt`, `scaffoldJsonLd`, `scaffoldRobots`,
-     `scaffoldAgent404`) — **default yes in the wizard.** This is not a contradiction of the
+   - **Scaffold opt-ins** (`scaffoldLlmsTxt`, `scaffoldJsonLd`, `scaffoldRobots`; historically
+     also `scaffoldAgent404`, removed 2026-08-28) — **default yes in the wizard.** This is not a
+     contradiction of the
      config's `false` defaults: config defaults are `false` because *silent* writes into a source
      tree are invasive; in a wizard the user is present and the ask itself is the opt-in.
      Default-yes-when-asked / default-no-when-silent is one coherent policy — state it in the docs.

fixtures/bare/report.golden.json

@@ -16,7 +16,9 @@
   },
   "agent404": {
     "notFoundPresent": false,
-    "agentAware": false
+    "agentAware": false,
+    "pages": [],
+    "markdownGuide": "public/404.md"
   },
   "middleware": {
     "present": false,
@@ -67,6 +69,13 @@
       "bytes": 87,
       "chars": 87,
       "tokens": 22
+    },
+    {
+      "artifact": "404.md",
+      "path": "public/404.md",
+      "bytes": 643,
+      "chars": 637,
+      "tokens": 159
     }
   ],
   "ora": {
@@ -160,7 +169,7 @@
     "No robots.txt found — add one (app/robots.ts or public/robots.txt) that explicitly Allows the reputable AI agents you want to reach, scoped to specific User-agent groups (never \"User-agent: *\"), plus a \"Sitemap:\" line. Note: Next's MetadataRoute.Robots (app/robots.ts) has no field for the \"Agentmap:\" catalog pointer — put that line in a static public/robots.txt. Set scaffoldRobots: true in ax.config to have one written for you.",
     "No agents.md found — consider adding one (public/agents.md) with when-to-use / when-NOT-to-use guidance for agents. ax won't write its content for you; the companion skill can help author it from your repo.",
     "No JSON-LD structured data found in your layouts/pages — structured data is how registries and agents disambiguate and rank your site. Add an Organization block in your root layout <head> (<script type=\"application/ld+json\">) with a \"sameAs\" array linking your LinkedIn/GitHub/npm/social profiles (the entity-disambiguation signal registries value), and at least one more schema.org @type beyond Organization — SoftwareApplication or Product for an app/API, or a FAQPage — since covering more types helps registries understand your site more fully. JSON-LD pairs with an llms.txt: llms.txt tells agents what your site is for, JSON-LD identifies it as an entity registries can rank — add both, not one alone. ax won’t author the block (the fields are judgment content); the companion skill can help draft it from your repo.",
-    "No app/not-found.tsx found — agents that hit a missing URL get Next.js’s bare default 404, a dead end that makes them give up or guess. Add one that tells agents why the 404 happened and how to continue (links to llms.txt, the ai-catalog, and your real routes), or set scaffoldAgent404: true in ax.config to have an agent-aware page (plus a build-time route manifest) written for you.",
+    "No app/not-found.tsx found — agents that hit a missing URL get Next.js’s bare default 404, a dead end that makes them give up or guess. Add a standard 404 page in your own design system (a clear “page not found” and a way back home), and include `<link rel=\"alternate\" type=\"text/markdown\" href=\"/404.md\" />` so agents can hop from it to /404.md — the generated wayfinding guide (real routes + discovery links) this build writes. Rendered in the page’s JSX, React hoists the tag into <head> (App Router); use next/head in the Pages Router.",
     "Serve agents the generated markdown automatically: First run `npx ax manifest` (and wire it as the \"prebuild\" script so it stays fresh — `middleware.ts` is compiled during `next build`, so the manifest must exist before the build starts). Then: add `import { withAx } from '@ora-ai/ax/middleware';` and `import { axManifest } from './ax-manifest';`, create middleware.ts with `export default withAx({ manifest: axManifest });`, and export the matcher literal `export const config = { matcher: ['/((?!_next|api|.*\\\\..*|favicon|robots|health|status).*)'] };`. The middleware rewrites a request to its markdown twin only when the manifest lists one, never touches gated paths, and answers unknown URLs from detected agents with a 200 markdown wayfinding body — it composes with (never replaces) your existing middleware, and keeps your own gating (UA filters, bot walls) from ever blocking the published discovery artifacts."
   ]
 }

fixtures/bare/twins.golden/404.md

@@ -0,0 +1,21 @@
+---
+title: "Page not found — bare"
+description: "Where to continue when a URL on this site does not exist: discovery links and real routes."
+canonical_url: /404.md
+last_updated: <last_updated>
+generated-by: "@ora-ai/ax"
+---
+
+# Page not found
+
+The URL you requested does not exist on this site. It may have moved, or it may never have existed —
+requesting it again will keep returning this response. Do not retry; start from the links
+below instead.
+
+## Start here
+
+- [/.well-known/ai-catalog.json](/.well-known/ai-catalog.json) — machine-readable catalog of everything this site offers agents
+
+## Pages that do exist
+
+- [/](/)

fixtures/config-overrides/report.golden.json

@@ -35,7 +35,9 @@
   },
   "agent404": {
     "notFoundPresent": false,
-    "agentAware": false
+    "agentAware": false,
+    "pages": [],
+    "markdownGuide": "public/404.md"
   },
   "middleware": {
     "present": false,
@@ -86,6 +88,13 @@
       "bytes": 658,
       "chars": 658,
       "tokens": 165
+    },
+    {
+      "artifact": "404.md",
+      "path": "public/404.md",
+      "bytes": 655,
+      "chars": 649,
+      "tokens": 162
     }
   ],
   "ora": {
@@ -181,7 +190,7 @@
     "No robots.txt found — add one (app/robots.ts or public/robots.txt) that explicitly Allows the reputable AI agents you want to reach, scoped to specific User-agent groups (never \"User-agent: *\"), plus a \"Sitemap:\" line. Note: Next's MetadataRoute.Robots (app/robots.ts) has no field for the \"Agentmap:\" catalog pointer — put that line in a static public/robots.txt. Set scaffoldRobots: true in ax.config to have one written for you.",
     "No agents.md found — consider adding one (public/agents.md) with when-to-use / when-NOT-to-use guidance for agents. ax won't write its content for you; the companion skill can help author it from your repo.",
     "No JSON-LD structured data found in your layouts/pages — structured data is how registries and agents disambiguate and rank your site. Add an Organization block in your root layout <head> (<script type=\"application/ld+json\">) with a \"sameAs\" array linking your LinkedIn/GitHub/npm/social profiles (the entity-disambiguation signal registries value), and at least one more schema.org @type beyond Organization — SoftwareApplication or Product for an app/API, or a FAQPage — since covering more types helps registries understand your site more fully. JSON-LD pairs with an llms.txt: llms.txt tells agents what your site is for, JSON-LD identifies it as an entity registries can rank — add both, not one alone. ax won’t author the block (the fields are judgment content); the companion skill can help draft it from your repo.",
-    "No app/not-found.tsx found — agents that hit a missing URL get Next.js’s bare default 404, a dead end that makes them give up or guess. Add one that tells agents why the 404 happened and how to continue (links to llms.txt, the ai-catalog, and your real routes), or set scaffoldAgent404: true in ax.config to have an agent-aware page (plus a build-time route manifest) written for you.",
+    "No app/not-found.tsx found — agents that hit a missing URL get Next.js’s bare default 404, a dead end that makes them give up or guess. Add a standard 404 page in your own design system (a clear “page not found” and a way back home), and include `<link rel=\"alternate\" type=\"text/markdown\" href=\"/404.md\" />` so agents can hop from it to /404.md — the generated wayfinding guide (real routes + discovery links) this build writes. Rendered in the page’s JSX, React hoists the tag into <head> (App Router); use next/head in the Pages Router.",
     "Serve agents the generated markdown automatically: First run `npx ax manifest` (and wire it as the \"prebuild\" script so it stays fresh — `middleware.ts` is compiled during `next build`, so the manifest must exist before the build starts). Then: add `import { withAx } from '@ora-ai/ax/middleware';` and `import { axManifest } from './ax-manifest';`, create middleware.ts with `export default withAx({ manifest: axManifest });`, and export the matcher literal `export const config = { matcher: ['/((?!_next|api|.*\\\\..*|favicon|robots|health|status).*)'] };`. The middleware rewrites a request to its markdown twin only when the manifest lists one, never touches gated paths, and answers unknown URLs from detected agents with a 200 markdown wayfinding body — it composes with (never replaces) your existing middleware, and keeps your own gating (UA filters, bot walls) from ever blocking the published discovery artifacts."
   ]
 }

fixtures/config-overrides/twins.golden/404.md

@@ -0,0 +1,21 @@
+---
+title: "Page not found — config-overrides"
+description: "Where to continue when a URL on this site does not exist: discovery links and real routes."
+canonical_url: /404.md
+last_updated: <last_updated>
+generated-by: "@ora-ai/ax"
+---
+
+# Page not found
+
+The URL you requested does not exist on this site. It may have moved, or it may never have existed —
+requesting it again will keep returning this response. Do not retry; start from the links
+below instead.
+
+## Start here
+
+- [/.well-known/ai-catalog.json](/.well-known/ai-catalog.json) — machine-readable catalog of everything this site offers agents
+
+## Pages that do exist
+
+- [/](/)

fixtures/deploy-variants/report.golden.json

@@ -16,7 +16,9 @@
   },
   "agent404": {
     "notFoundPresent": false,
-    "agentAware": false
+    "agentAware": false,
+    "pages": [],
+    "markdownGuide": "public/404.md"
   },
   "middleware": {
     "present": false,
@@ -67,6 +69,13 @@
       "bytes": 98,
       "chars": 98,
       "tokens": 25
+    },
+    {
+      "artifact": "404.md",
+      "path": "public/404.md",
+      "bytes": 674,
+      "chars": 668,
+      "tokens": 167
     }
   ],
   "ora": {
@@ -165,7 +174,7 @@
     "basePath \"/app\" means the catalog is served at /app/.well-known/ai-catalog.json, not at the conventional /.well-known/ai-catalog.json a crawler probes. Point agents at it (ARD §6.1):",
     "  • Add to your root layout <head>: <link rel=\"ai-catalog\" href=\"/app/.well-known/ai-catalog.json\" />",
     "  • Add to robots.txt (app/robots.ts or public/robots.txt): Agentmap: /app/.well-known/ai-catalog.json",
-    "No app/not-found.tsx found — agents that hit a missing URL get Next.js’s bare default 404, a dead end that makes them give up or guess. Add one that tells agents why the 404 happened and how to continue (links to llms.txt, the ai-catalog, and your real routes), or set scaffoldAgent404: true in ax.config to have an agent-aware page (plus a build-time route manifest) written for you.",
+    "No app/not-found.tsx found — agents that hit a missing URL get Next.js’s bare default 404, a dead end that makes them give up or guess. Add a standard 404 page in your own design system (a clear “page not found” and a way back home), and include `<link rel=\"alternate\" type=\"text/markdown\" href=\"/app/404.md\" />` so agents can hop from it to /app/404.md — the generated wayfinding guide (real routes + discovery links) this build writes. Rendered in the page’s JSX, React hoists the tag into <head> (App Router); use next/head in the Pages Router.",
     "Serve agents the generated markdown automatically: First run `npx ax manifest` (and wire it as the \"prebuild\" script so it stays fresh — `middleware.ts` is compiled during `next build`, so the manifest must exist before the build starts). Then: add `import { withAx } from '@ora-ai/ax/middleware';` and `import { axManifest } from './ax-manifest';`, create middleware.ts with `export default withAx({ manifest: axManifest });`, and export the matcher literal `export const config = { matcher: ['/((?!_next|api|.*\\\\..*|favicon|robots|health|status).*)'] };`. The middleware rewrites a request to its markdown twin only when the manifest lists one, never touches gated paths, and answers unknown URLs from detected agents with a 200 markdown wayfinding body — it composes with (never replaces) your existing middleware, and keeps your own gating (UA filters, bot walls) from ever blocking the published discovery artifacts."
   ]
 }

fixtures/deploy-variants/twins.golden/404.md

@@ -0,0 +1,21 @@
+---
+title: "Page not found — deploy-variants"
+description: "Where to continue when a URL on this site does not exist: discovery links and real routes."
+canonical_url: /app/404.md
+last_updated: <last_updated>
+generated-by: "@ora-ai/ax"
+---
+
+# Page not found
+
+The URL you requested does not exist on this site. It may have moved, or it may never have existed —
+requesting it again will keep returning this response. Do not retry; start from the links
+below instead.
+
+## Start here
+
+- [/app/.well-known/ai-catalog.json](/app/.well-known/ai-catalog.json) — machine-readable catalog of everything this site offers agents
+
+## Pages that do exist
+
+- [/app/](/app/)