From cafab7c8dc925b965698c7cd802b7c7ae4d660d8 Mon Sep 17 00:00:00 2001 From: Ariel Shulman Date: Thu, 27 Aug 2026 17:20:47 +0300 Subject: [PATCH 1/3] =?UTF-8?q?docs:=20rewrite=20README=20=E2=80=94=20prob?= =?UTF-8?q?lem-first,=20value-first,=20half=20the=20words?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opens with the problem (the web was built for humans), one-time setup, the before+after-build map mechanism, six benefit-first feature entries, and the coding-agent report loop. Contract sections (status, design posture, supported matrix, init, config, layout) kept in trimmed form; deep-dive sections folded into the feature entries or cut. 596 → 186 lines. Co-Authored-By: Claude Fable 5 --- README.md | 624 ++++++++++-------------------------------------------- 1 file changed, 107 insertions(+), 517 deletions(-) diff --git a/README.md b/README.md index d40b0b9..9ccb131 100644 --- a/README.md +++ b/README.md @@ -1,119 +1,83 @@ # `@ora-ai/ax` — Agent Experience for Next.js -AI agents are becoming every site's newest user segment, and most sites are invisible to them. -`ax` is one `postbuild` line that makes a Next.js app **discoverable, legible, and usable by -agents**: +The web was built for humans. Now it has a new kind of visitor — AI agents that browse, read, and +act on a site's behalf — and most sites are invisible to them: no map, no context, no way in. + +`@ora-ai/ax` is a build plugin and CLI that makes a Next.js app discoverable, legible, and usable by +agents. + +## One-time setup ```sh npm install --save-dev @ora-ai/ax ``` -```json -{ - "scripts": { - "build": "next build", - "postbuild": "ax" - } -} +```sh +npx ax init ``` -**Try it locally first.** Run your build once on your machine and `ax` shows you what agents can -already do with your site, what it generated for you, and a short, ranked list of quick wins to make -your site more agent-ready — each with the exact next step to take. Like what you see? Commit the -generated catalog (it lives in `public/`, right where Next.js serves it) and every build after that -just works. - -> On the first publish, `ax` shows the surface it's about to expose and asks a quick y/N — so nothing -> goes public by surprise. Automating in CI? Add `--yes` to run unattended, or `--dry-run` to preview -> anytime without writing. - -One run — offline, deterministic, about a second — then: - -- **Generates** a spec-valid [AI Catalog](https://github.com/Agent-Card/ai-catalog) (Agentic - Resource Discovery) at `/.well-known/ai-catalog.json`, validated against the official ARD schema - before a byte is written, so agents and registries can discover the site's capabilities. -- **Detects** the agent surfaces already in your code and references what's unambiguous: MCP - servers, W3C WebMCP in-page tools (declarative and imperative), OpenAPI docs, `llms.txt`, - `robots.txt` / sitemap / `agents.md` / JSON-LD. -- **Scaffolds (opt-in)** the mechanical parts a build tool is uniquely placed to write, from data - it already has: an `llms.txt` filled with your real routes and artifacts, an agent-aware 404 - page carrying your real route table, `robots.txt` discovery pointers, an `Organization` JSON-LD - component. -- **Hands off the judgment work**: `.ora/report.json` maps every finding to [Ora](https://ora.ai)'s - agent-readiness checks (`addressed` / `actionable`) and points your coding agent at Ora's live - skill server — fix, scan, rescan until the site is agent-ready. +That's it. From now on, every build generates your agent-friendly artifacts automatically. -``` -[ax] ✓ wrote public/.well-known/ai-catalog.json -[ax] ⚠ Scaffolded a starter llms.txt at app/llms.txt/route.ts — ax filled in what it can derive… -[ax] ✓ wrote .ora/report.json (machine-readable build report) -[ax] Find your report at: .ora/report.json -[ax] Prompt for your coding agent (copy-paste): -[ax] Read .ora/report.json and work through every check marked "actionable": create or improve -[ax] those artifacts to make this site more agent-ready… -``` +## How it works -No network calls and no AI at build time — every byte is derived from your source tree. Three -runtime dependencies. Atomic writes, and the CLI exits non-zero rather than emit an invalid -catalog. 348 tests, including the spec's official conformance tool run over a corpus of real -fixture apps in CI. +`ax init` wires two build hooks. `prebuild: ax manifest` builds a lightweight map of your site — +routes, gated paths, discovery artifacts — before Next.js compiles your middleware. +`postbuild: ax` uses the finished build to generate and publish the real artifacts against that map: +the catalog, markdown twins, scaffolds, and the report. At runtime, `@ora-ai/ax/middleware` reads +the same map to serve agents directly. Wiring the middleware (and any JSON-LD component) into your +app is left to you — ax prints the exact lines to add, it never edits `middleware.ts` or your layout +itself. -> **Status:** pre-release, under active development. The detect-and-reference core, WebMCP -> detection, the agent-aware 404, the opt-in scaffolds, the Ora-mapped build report, -> review-before-publish, gated-surface detection (never advertise an auth-walled endpoint as -> open), and the `ax init` onboarding wizard are implemented and tested. Before the first public npm -> release: an end-to-end run against -> Ora's production crawler. Full phased roadmap and every design decision: -> [`docs-internal/PLAN.md`](./docs-internal/PLAN.md). +## What ax does -## `ax init` — one-command setup +**Teach agents how to use your website — `agents.md`** +ax detects an existing `agents.md` and recommends adding one when it's missing. Writing the content +is a job for a docs-authoring skill, not the build — ax never writes it for you. -Rather than hand-write `ax.config` and wire the build yourself, run the onboarding wizard: +**Make your site readable to agents — markdown twins** +Every static page gets a `.md` twin (`/docs` → `/docs.md`), generated from your real build output or +a markdown source, and kept in sync on every build. -```sh -npx ax init -``` +**Help agents navigate your website — AI Catalog + `llms.txt`** +A spec-valid AI Catalog at `/.well-known/ai-catalog.json` lists what your site offers; an opt-in +scaffolded `llms.txt` points agents at your key pages and machine-readable resources. -It runs the same source-tree detection a build does (no `next build` needed), prints what it found, -then asks **only what the code can't answer** — your production `siteUrl`, which detected surfaces -agents can use without signing in (the rest are gated and never advertised as open), how agents -authenticate to the gated ones (API key/bearer with a docs URL — the realistic common case — or -OAuth 2.0; written as a declared `auth` entry in `ax.config`. Endpoint questions only appear when -the source tree has no OAuth to read: committed authorization-server metadata is adopted verbatim, -and a wired RFC 9728 protected-resource route means agents discover the endpoints at runtime, so -nothing is asked), and one -pre-selected checklist of every opt-in scaffold, each line stating why agents need it — deselect -anything you don't want, then press Enter. It writes an `ax.config.ts` (or `.js`, -matching your project) with a one-line comment on every field, so the config it commits doubles as -its own documentation, and adds `"postbuild": "ax"` to `package.json` — but only when no `postbuild` -already exists; if one does, it prints the exact edit to make instead of chaining into a script it -doesn't own. It never overwrites an existing `ax.config.*`, and it generates no public artifact -itself — the first real build is still the moment the [review-before-publish](#the-catalog) gate -runs, now pre-answered by your choices. - -It is a plain command, never a `postinstall` hook — installing the package stays inert. For CI or -scripting, run it unattended: +**Help agents authenticate to your website — auth detection, declared auth, `auth.md`** +ax detects gated MCP and OpenAPI surfaces, lets you declare the real auth flow when detection can't +see one, and publishes both in a generated `auth.md`. Your 401/403 responses stay honest — ax never +touches your route handlers. -```sh -npx ax init --yes --site-url https://yourdomain.com -``` +**Steer lost agents back — agent-aware 404 + wayfinding middleware** +An opt-in `not-found.tsx` lists your real routes and discovery artifacts for agents that hit a dead +end; the runtime middleware gives the same wayfinding response to any URL that matches nothing. + +**Tell agents who you are and gain their trust — JSON-LD scaffolding** +An opt-in `Organization` JSON-LD component is scaffolded from your `package.json`. ax prints the +exact import and element to add — it never edits your layout itself. + +Detection and the catalog run automatically on every build. Everything that writes into your source +tree is opt-in, listed in `ax.config`, and never overwrites a file you already have. -`--yes` accepts every default; `siteUrl` has no default (it's written verbatim into your public -catalog, so it must be given via `--site-url` or a `SITE_URL` / `NEXT_PUBLIC_SITE_URL` env var), and -the wizard exits non-zero with a clear message if it's missing or a localhost/preview URL. +## The report -> **Yes-when-asked, no-when-silent.** The scaffolds default to **off** in `ax.config` but the -> wizard's checklist lists every one of them **pre-selected**. That's one coherent policy, not a -> contradiction: a _silent_ write into your source tree on an unattended build is invasive, so -> config defaults are `false`; but in the wizard you're present and the list itself is the opt-in, -> so everything starts checked. +With `report: true` (the wizard's default), every build writes `.ora/report.json` — a +machine-readable summary of what was generated, detected, and skipped (and why), with every finding +mapped to [Ora](https://ora.ai)'s agent-readiness checks as `addressed` or `actionable`. Point your +coding agent at it after a build and let it work through what's left. + +Welcome to the agentic web. + +--- + +> **Status:** Pre-release, under active development. Detection, the opt-in scaffolds, the +> Ora-mapped build report, review-before-publish, and the `ax init` wizard are implemented and +> tested. Full roadmap and design rationale: [`docs-internal/PLAN.md`](./docs-internal/PLAN.md). ## Design posture -**Spec follower, never spec inventor.** The plugin translates code developers already wrote into -whatever shape the spec defines. **Precision over recall** — a wrong or dangerous catalog entry is -worse than a missing one, so route-level tool entries are explicit opt-in and zero-config publishes -only what is unambiguous. +**Spec follower, never spec inventor** — ax translates code you already wrote into spec shape; it +never invents an entry. **Precision over recall** — a wrong or dangerous catalog entry is worse than +a missing one, so anything ambiguous is opt-in rather than automatic. ## Supported matrix (v1) @@ -129,127 +93,70 @@ This matrix is a public contract from day one. Anything outside it is out of sco | Bundler | Webpack **and** Turbopack (CLI is bundler-agnostic) | — | | Monorepo | Turborepo: **detect-and-warn** planned for v1 | Full nested-workspace resolution | -> The monorepo support level is still an open decision — see the open-questions table in -> `docs-internal/PLAN.md`. +## `ax init` -## Repository layout +Rather than hand-write `ax.config` and wire the build yourself, run the onboarding wizard: +```sh +npx ax init ``` -packages/ax the plugin / CLI (`@ora-ai/ax`) — the npm package (3 runtime deps: ajv, ajv-formats, jiti) -spec/ vendored AI Catalog spec + hand-written JSON Schema + validator oracle -fixtures/* minimal-but-real Next.js apps — the test suite, docs examples, and eval corpus + +It detects your surfaces from source (no build needed), asks only what code can't answer — your +production `siteUrl`, which surfaces agents can use without signing in, how agents authenticate to +the gated ones — and writes a documented `ax.config.ts`, adding `"postbuild": "ax"` and +`"prebuild": "ax manifest"` to `package.json`. It never overwrites an existing script or config; +where one already exists, it prints the exact edit to make instead. + +For CI, run it unattended: + +```sh +npx ax init --yes --site-url https://yourdomain.com ``` -## The catalog - -The `postbuild` run writes `public/.well-known/ai-catalog.json` with: - -- **Site-level metadata** — `displayName` / `description` from `package.json`. -- **Zero-config artifact detection** (Phase 2.2) — detects and references what's already there: - - An **MCP server** mounted via [`mcp-handler`](https://www.npmjs.com/package/mcp-handler) (or - its legacy alias `@vercel/mcp-adapter`) → `application/mcp-server-card+json`. - - A static **`public/openapi.json`** → `application/vnd.oai.openapi+json`. - - An **`llms.txt`** served either as `app/llms.txt/route.ts` or `public/llms.txt` → - `text/markdown`. -- **Config-declared entries** — anything you list in `ax.config`'s `entries`, e.g. docs/skills - pointers (`text/html` / `application/ai-skill+md`). - -Every entry above is validated against the AI Catalog spec before writing; the CLI refuses to -write (and exits non-zero) if generation ever produces an invalid catalog. The plugin only ever -**detects and references** — it never invents a per-route entry or synthesizes a doc from route -handlers (see `docs-internal/PLAN.md`'s _Scope_ and _Core design decisions_). - -**Absolute URLs need a known site origin.** The spec requires every entry's `url` to be an -absolute URI, so a detector skips emitting its entry (with a warning) unless it can resolve the -site's origin. This must be your **public production URL** (e.g. `https://yourdomain.com`) — it's -written verbatim into the catalog's entry URLs, so a `localhost` or preview URL would publish broken -links. It's resolved in this order, first match wins: - -1. `ax.config`'s `siteUrl` — an explicit declaration, always wins. -2. `SITE_URL`, then `NEXT_PUBLIC_SITE_URL` — the two env-var names Next.js apps most commonly use - for a stable production URL. Expected to be an absolute `https://…` origin. -3. Vercel's build-time `VERCEL_PROJECT_PRODUCTION_URL` — injected automatically **on Vercel only**. - -**Iterating locally.** `VERCEL_PROJECT_PRODUCTION_URL` exists only during a build _on Vercel_, so a -plain local `next build` can't resolve an origin from it — the catalog would come out without its -URL-bearing entries (MCP, OpenAPI, llms.txt) and you couldn't check your work before deploying. -Your production URL is the same string locally and in prod, so declare that (still the public -domain, **not** `localhost`): set `siteUrl` in `ax.config`, or run the build with -`SITE_URL=https://yourdomain.com next build` (or export `NEXT_PUBLIC_SITE_URL`). Any of these lets -you generate and preview the real catalog locally before deploying. - -`ax.config.*` is evaluated as real code at build time (via [`jiti`](https://github.com/unjs/jiti)), -not parsed as static JSON — so if your host/CI uses a different variable name, `siteUrl: -process.env.DEPLOY_URL` (or whatever it is) works with no special support needed. - -### Config (`ax.config.{ts,js,mjs,cjs}`) - -Optional. Loaded from your project root; `.ts`/`.mjs`/`.cjs`/`.js` all work — named after the `ax` -tool that reads it, so the file you commit says plainly which tool it configures. - -> If you still have an `ard.config.*` from before the 2026-07-27 rename, it's no longer read at -> all — rename it to `ax.config.*`. A build with only an `ard.config.*` fails loudly with that -> instruction rather than silently building with defaults, so this is safe to miss and easy to fix. +`--yes` accepts every default; `siteUrl` has no default, since it's written verbatim into your +public catalog — give it via `--site-url` or a `SITE_URL` / `NEXT_PUBLIC_SITE_URL` env var. + +> Scaffolds default to **off** in `ax.config`, but the wizard's checklist shows them all +> **pre-selected** — you're present and choosing, so nothing is written silently on an unattended +> build. + +## Configuration (`ax.config.{ts,js,mjs,cjs}`) + +Optional, loaded from your project root. Evaluated as real code (via +[`jiti`](https://github.com/unjs/jiti)), not parsed as static JSON. ```ts import { defaultIsGated, type AxConfig } from '@ora-ai/ax'; const config: AxConfig = { - // Your production origin — every detected entry's URL is resolved against this. Optional: falls - // back to a SITE_URL / NEXT_PUBLIC_SITE_URL env var, then (on Vercel) to - // VERCEL_PROJECT_PRODUCTION_URL. Set it here to iterate locally, or to pin the URL on any host. - // This file is real code, so reading it from a different env var works too: - // siteUrl: process.env.DEPLOY_URL, + // Your production origin. Falls back to SITE_URL / NEXT_PUBLIC_SITE_URL, then (on Vercel) + // VERCEL_PROJECT_PRODUCTION_URL. Real code, so any env var name works: + // siteUrl: process.env.DEPLOY_URL siteUrl: 'https://example.com', - // Where to write the catalog. 'static' (default) writes public/.well-known/ai-catalog.json; - // 'route' writes an App Router handler at app/.well-known/ai-catalog.json/route.ts instead - // (for proxy setups and future dynamic catalogs). See the basePath note below. + // Where to write the catalog: 'static' (default) writes public/.well-known/ai-catalog.json; + // 'route' writes an App Router handler instead. emit: 'static', - // Scaffold a starter app/llms.txt/route.ts when neither it nor public/llms.txt exists, filled in - // with your real routes and artifacts. Opt-in (defaults to false) — it writes into your source - // tree, not just the catalog file. + // Scaffold a starter llms.txt filled with your real routes and artifacts. Opt-in. scaffoldLlmsTxt: true, - // Scaffold an agent-aware app/not-found.tsx (written once, yours to edit) plus a route-manifest - // data module regenerated on every build. Opt-in (defaults to false) — see "Agent-aware 404". + // Scaffold an agent-aware app/not-found.tsx plus a regenerated route-manifest module. Opt-in. scaffoldAgent404: true, - // Append the Sitemap:/Agentmap: discovery pointers to your public/robots.txt, or write one when - // you have no robots source at all. Opt-in (defaults to false) — see "Generated artifacts". + // Add Sitemap:/Agentmap: discovery lines to robots.txt (or write one if you have none). Opt-in. scaffoldRobots: true, - // Scaffold an app/organization-json-ld.tsx server component when no JSON-LD is rendered - // anywhere. Opt-in (defaults to false); ax never edits your layout to wire it up. + // Scaffold an Organization JSON-LD component. Opt-in; ax never wires it into your layout. scaffoldJsonLd: true, - // Generate markdown twins of your pages (route /docs → public/docs.md) plus /auth.md when - // surfaces are gated. Default ON — twins are regenerated build artifacts, not scaffolds, and - // their first write is confirmed at the review gate. See "Markdown twins". + // Generate markdown twins of your pages, plus /auth.md when surfaces are gated. Default on. markdownTwins: true, - // Write .ora/report.json, the machine-readable twin of the CLI output (true, or a custom path). - // Opt-in (defaults to false); the CLI flag --report[=path] does the same per run. + // Write .ora/report.json, the machine-readable build report. Opt-in. report: true, - // Mark an artifact (MCP server, OpenAPI/REST surface, or a declared entry) as gated behind auth, - // so it is never advertised as an open surface. Replaces the old denylist/allowlist pair — a - // single matcher subsumes both: return `false` to re-include a path the floor would gate. A - // gated artifact ax can describe (a detected withMcpAuth / OpenAPI securitySchemes) is emitted - // with a secret-free `auth` descriptor; one it can't describe is dropped, not published — except - // an MCP server, whose gated status *is* its description: it is always published with the auth - // marker, never dropped. Usually you don't need this field for MCP at all: the gating answer - // from `ax init` (or a build's review gate) is recorded in the committed server card, and only a - // server with no recorded decision is asked about. With no isGated, a built-in floor gates - // `/api/auth/**` and `/api/webhooks/**`; supplying isGated replaces that floor wholesale, so - // compose `defaultIsGated` to keep it: + // Mark an artifact as gated behind auth so it's never advertised as open. Compose + // defaultIsGated to extend the built-in floor (which gates /api/auth/** and /api/webhooks/**) + // rather than replace it: isGated: (target) => defaultIsGated(target) && target.path !== '/api/auth/status', - // Hand-declared entries — e.g. docs/skills pointers zero-config detection can't guess at. An - // `identifier` matching a detected entry overrides/extends it field-by-field (never replaces it - // outright); anything else is appended as a new entry. + // Hand-declared entries zero-config detection can't guess at, e.g. docs/skills pointers. entries: [ { identifier: 'urn:example:docs', type: 'text/html', url: 'https://example.com/docs' }, - // An entry's `auth` declares how agents authenticate when ax can't derive it — the endpoints - // detection can never see (a withMcpAuth-wrapped MCP mount is only ever detectable as - // "requires auth, scheme unknown"). Exactly the secret-free EntryAuth shape registries read: - // status, OAuth endpoint URLs, scope keys, and a human docs URL — never credentials. Declared - // once, it flows to the catalog entry, the MCP server card, and the generated /auth.md, and — - // like a detected scheme — marks the surface gated. URL fields must be absolute http(s); the - // config gate rejects anything else loudly, and a declared status that contradicts a detected - // one wins with a warning. + // Declare the real auth flow when detection can only see "requires auth, scheme unknown" + // (e.g. a withMcpAuth-wrapped MCP mount). Flows to the catalog, the server card, and /auth.md. { identifier: 'urn:air:example.com:mcp-server', auth: { @@ -267,330 +174,13 @@ const config: AxConfig = { export default config; ``` -An invalid config (unknown top-level key, wrong field type, ...) fails the build loudly with a -specific, actionable message — it is never silently ignored or partially applied. - -### `next.config` reading - -The CLI reads your `next.config.{ts,js,mjs,cjs}` (object or function form) to extract `basePath`, -`distDir`, and `output`, so you never repeat them in `ax.config`. Unlike the plugin's own -config above, a `next.config` that fails to load only warns and falls back to defaults — it's not -this plugin's place to fail your build over your Next.js config. - -**`basePath` and where the catalog is served.** If your app sets `basePath`, the catalog is served -under that prefix (e.g. `/app/.well-known/ai-catalog.json`), not at the domain root crawlers probe. -Switching `emit` to `'route'` does **not** change this — an App Router route handler is subject to -`basePath` too. The in-spec fix (ARD §6.1) is to point crawlers at wherever the catalog actually -lives: on a `basePath` build the CLI prints a recommendation to add an HTML -`` tag to your root layout and an `Agentmap:` line to your -`robots.txt`. - -## WebMCP detection (Phase 4) - -The CLI detects **in-page WebMCP tools** (the W3C Web Machine Learning CG draft that lets a page -register tools a browser-resident agent can call) in both styles: - -- **Declarative** — `
`. Markup survives into - server-rendered HTML, so these are also visible to HTML-reading scanners. Tools on - statically-addressable App Router pages become `text/html` catalog entries whose `capabilities` - carry the tool names. -- **Imperative** — `document.modelContext.registerTool()` / `provideContext()` in `'use client'` - components, and the `useWebMCP()` hook (`@mcp-b/react-webmcp` / `usewebmcp`). These are - runtime-only with no spec-defined manifest, so they are surfaced in the build summary and - recommendations — never invented as catalog entries. - -The detector also warns on the two mistakes that silently produce zero working tools: registration -via the **deprecated `navigator.modelContext`** alias (the entry point moved to -`document.modelContext` in the May 2026 draft; Chrome 150+ deprecates the alias), and registration -in a **server component** (no `'use client'`), where the API doesn't exist at render time. A -user-defined function that merely happens to be called `registerTool` is not detected. When an app -has `` elements but no WebMCP at all, the CLI points out that every form is a latent agent -tool that two attributes make callable. - -## Agent-aware 404 (`scaffoldAgent404`) - -An agent that fetches a URL that doesn't exist gets a dead-end 404 and either gives up or -guesses. Setting `scaffoldAgent404: true` in `ax.config` scaffolds an **agent-aware -`app/not-found.tsx`** — written once, yours to edit, never overwritten — that tells agents why the -404 happened (the URL doesn't exist; don't retry) and how to continue: links to the site's -discovery artifacts (`ai-catalog.json`, `llms.txt`, sitemap — only the ones that actually exist) -and a list of the app's real routes, as visible HTML plus a schema.org `ItemList` in JSON-LD. - -The route list lives in a companion data module (`app/not-found-agent-data.ts`) that **is -regenerated on every build** from the App Router source tree — the piece only a build-time tool -can supply, since nothing at runtime knows the route table. Dynamic (`[slug]`) routes are never -guessed. Without the opt-in, the CLI detects your existing `not-found.*` and recommends adding -agent signposts if it has none. - -## Generated artifacts (opt-in scaffolds) - -Most of what makes a site agent-ready is judgment work — what your site is _for_, which crawlers you -want, who you are as an entity. But the skeleton around that judgment is mechanical, and a build -step is better placed to write it than a person is: it already knows your route table, your -package.json, and which artifacts this build produced. So where the plugin can derive real content -it generates instead of advising, and stops at exactly the line where a guess would start. - -Every scaffold follows the same three rules: **opt-in** via a config flag (it writes into your -source tree, not just the one file the plugin exists to produce), **write-once** (the file is yours -the moment it exists — ax never overwrites it), and **honest** (nothing invented; anything ax can't -derive ships as a marked TODO rather than plausible-looking filler). - -### `scaffoldLlmsTxt` — a starter `llms.txt` with your actual content - -`app/llms.txt/route.ts` (or `.js`), written once, containing: your `package.json` name and -description, a **Key pages** section listing your app's real statically-addressable routes (dynamic -segments are never guessed), and a **Machine-readable resources** section linking the artifacts this -build actually produced or detected — the catalog, `openapi.json`, an MCP endpoint — as absolute -URLs when the site origin resolved and served paths otherwise. - -The **When to use** section is deliberately a TODO, and the comment says why: agent-readiness checks -look for real guidance about which tasks belong on your site, and an unedited placeholder scores the -same as no section at all. That paragraph is the one part of an llms.txt no build tool can derive, -so it's the one part left for you (or your coding agent) to write. - -### `scaffoldRobots` — discovery pointers in `robots.txt` - -What ax knows that your `robots.txt` doesn't is where the catalog it just generated lives, and -whether you actually have a sitemap. So: - -- **You have a `public/robots.txt`** → ax _appends_ a `Sitemap:` line (only when a sitemap really - exists) and an `Agentmap:` line pointing at the generated catalog, in a block marked - `# Added by @ora-ai/ax`, and only when they're missing. Existing lines are never modified or - reordered, a directive you already wrote counts as written (in any casing), and running twice - appends nothing the second time. -- **You have an `app/robots.ts` route** → ax doesn't touch it. That file is code, and it owns your - policy. (Next's `MetadataRoute.Robots` has no field for `Agentmap:` at all, so ax says so and - leaves the choice to move to you.) -- **You have neither** → ax writes `public/robots.txt` with `User-agent: *` / `Allow: /`, explicit - `Allow` blocks for reputable AI agent crawlers — the retrieval and search families across OpenAI - (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), - Google (Google-Extended), Perplexity, Meta, Amazon, and others, kept in one shared corpus so the - list never drifts — and the pointer lines above. - -The generated file also carries a **commented-out** example of restricting training-only crawlers -(CCBot, Bytespider). It stays commented out on purpose: whether to block a crawler is a decision -about your content and your business, and the plugin says as much in the file rather than making it -for you. - -### `scaffoldJsonLd` — an `Organization` block you can fill in - -When nothing in your app renders JSON-LD, ax writes `app/organization-json-ld.tsx` — a small server -component rendering one `