diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e40484..351cecf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,42 @@ loosely follows [Keep a Changelog](https://keepachangelog.com/); dates are ## [Unreleased] +### Added: `?view=index` — a whole-cell projection cheap enough for an agent to read (ADR 0012 §4, ADR 0017 §2, ADR 0018 §7 amendments) + +On a real 42-export/47-dataset cell, `GET /context` measured 429 KB +compact — `exports[]` alone was 383 KB (`semantic` 158 KB, `schema` 122 KB, +`check` 87 KB) — and agents refused to read it; `?model=`/`?terms=` fared no +better (467 KB and 431 KB respectively, of which the actually-requested +content was 38 KB and 1.4 KB). Additive; `datamk_context` stays 4. + +- **`?view=index` / `datamk context --view index`.** Projects every + `exports[]` entry to identity, claims (`description`, `grain`, + `freshness`, `from`), affordances (`query`/`binding`, `deployed`), + declared column *names* (`columns[]`, in place of `schema`'s full column + bodies), and bound Ossie dataset names (`semantic_datasets[]`, in place + of `semantic[]`'s full bodies) — everything cell-level stays. Composes + with `include`/`terms`/`model`; refused (400) on `/context/`, a + single-export door already being the index's whole point. New + `index_request` affordance, always present. New `~index` `ETag` suffix. + MCP resource `datamk:///context/index`. +- **`?model=` and `?terms=` (without a route) now apply the same + projection automatically** — this door never narrows by route, so asking + for one model or a term subset has no reason to drag every export's full + body along. `/context/?terms=` is exempt: that door's whole point + is the route's full export. `datamk context --model`/`--terms` (without + `--export`) match. +- **The per-column census (`exports[].check.columns`) moves behind + `?include=check`** — omitted by default (including under `?view=index`), + present under `?include=check`, and always inlined in the portable + `datamk context` emission (no `--no-check` flag; a file cannot be + re-requested). The rollup (`check`, `grain`, `rows`, `distinct_grain`, + `null_rows`, `at`) stays on every record. `include` becomes `docs` | + `check`; `included` lists `check` when inlined. New `~check` `ETag` + suffix. +- Measured on a synthetic 40-export/30-column/one-dataset-per-export cell: + `?view=index` and `?model=` both land under 10% of the full document's + bytes. + ### Added: `semantic_model:` — Apache Ossie ingest (ADR 0018) datamake ingests business meaning it does not author: `semantic_model:` diff --git a/docs/adr/0012-cell-context-document.md b/docs/adr/0012-cell-context-document.md index 749115f..28c5f56 100644 --- a/docs/adr/0012-cell-context-document.md +++ b/docs/adr/0012-cell-context-document.md @@ -841,3 +841,54 @@ reused, not reimplemented. What would reverse it: a design partner needing MCP against a hosted cell they cannot run locally (then `POST /mcp` on `serve`, same module, inheriting bearer auth); agents demonstrably unable to compose `describe_export` → `query_export` in two hops. + +## Amendment (2026-09-11): `?view=index` — the index projection (context bloat) + +Measured on a real 42-export/47-dataset cell: default `/context` is 429 KB +compact, of which `exports[]` is 383 KB (`semantic` 158 KB, `schema` 122 KB, +`check` 87 KB, `description` 25 KB, `deployed` 13 KB) — agents refuse it. +§2's "the document is flat, one schema" stands; what was missing was a +*view* of that one schema cheap enough to pick a route from before paying +for its body. + +- **`?view=index` / `datamk context --view index`** — closed vocabulary + `full` (default) | `index`. `index` returns the full document with every + `exports[]` entry projected through `ExportDoc::to_index`: identity + (`name`, `version`, `route`, `contract`), the claims that fit in a + sentence (`description`, `grain`, `freshness`, `from`), the affordances + (`query`/`binding`, `deployed`), declared column *names* (`columns[]`, + §3's `schema` minus type/unit/description/`from`), and bound Ossie + dataset names (`semantic_datasets[]`, `model/dataset`, §7's `semantic[]` + minus every field/relationship/metric body). Every cell-level field + (`semantic_models[]`, `definitions[]`, `upstreams[]`, `docs[]` identity, + `build`, `source_check` rollup, `data`, `notes`) stays — this is an + `exports[]`-only projection, not a second document. +- **Composes**, never collapses: `include`, `terms`, `model` all still work + under `view=index` — the projection runs on whatever `exports[]` those + narrowed it to. Refused on `/context/` (400): a single-export door + is already the index's whole point, so the combination names something + incoherent, the same refusal `?model=` already draws there. +- **`index_request`** joins `include_request`/`definitions_request` as a + constant, always-present affordance, relative per §2's RFC 3986 rule. +- **ETag** gains a `~index` component, same style as `~docs`/`~model`; the + plain default response is unaffected byte for byte. +- **One schema, still.** `ExportDoc` gained two fields (`columns`, + `semantic_datasets`), both empty and omitted on the full document — no + second struct, no second document to keep in sync. A consumer that has + only ever parsed `view=index` reads `view=full` unmodified; it just sees + more fields populated. +- **MCP:** `datamk:///context/index`, alongside `context` and + `context/` — the same projection over stdio. + +Measured on a synthetic 40-export/30-column/one-dataset-per-export cell: +`?view=index` and `?model=` both land under 10% of the full document's +bytes (`serve::smoke:: +view_index_and_model_are_under_a_tenth_of_the_full_document_on_a_synthetic_cell`). +See the ADR 0017 §2 and ADR 0018 §7 amendments (same date) for `?model=` +and `?terms=` (without a route) applying this same projection +automatically — the other half of the same measurement. + +Premises: an agent's first move is almost always "what exists," not "give +me everything" — the index answers the first cheaply. What would reverse +it: a client that always wants the full body anyway, for which `view=index` +is simply unused, at zero cost to it. diff --git a/docs/adr/0017-definitions.md b/docs/adr/0017-definitions.md index ae8e209..e13ae64 100644 --- a/docs/adr/0017-definitions.md +++ b/docs/adr/0017-definitions.md @@ -342,3 +342,33 @@ exits 0: the emission is truthful about its own state. Falsified if operators keep misreading a stale emission as validated — that argues for a non-zero exit, which is a behavior change this amendment deliberately does not take. + +## Amendment (2026-09-11): `?model=`/`?terms=` (no route) carry the index projection + +§2's "the document keeps its shape" still holds — one schema, additive +fields, `datamk_context` stays 4 — but on the un-routed `/context` door, +asking for `terms=` alone was paying for every export's full `schema`, +`check`, `probe`, and `semantic[]` body to answer a lookup that only ever +touches `definitions[]`/`docs[]`. Measured on the same 42-export/47-dataset +cell ADR 0012's `view=index` amendment measures: `?terms=` is 431 KB, of +which the matches (`definitions[]` + `semantic_matches[]`) are 1.4 KB. + +`GET /context?terms=...` and `GET /context?model=...`, absent a route, now +also project `exports[]` through `ExportDoc::to_index` (ADR 0012 §4 +amendment 2026-09-11) — automatically, without `view=index` — because this +door never narrows by route, so the narrower ask (a term subset, one model) +has no reason to drag the wider one along. `datamk context --model`/ +`--terms` (without `--export`) match. **`/context/?terms=` +is exempt**: §3's composition rule already holds one route's document to +its full export, and that full export is the whole point of asking a route +door — projecting it there would take away the one thing a route-scoped +request is for. + +The document's shape is unchanged by this — same `ExportDoc` schema, same +fields, just fewer of them populated for this request. There is no escape +hatch back to full `exports[]` under `terms=`/`model=`: the narrowed +request is the signal that `exports[]` is not what's being asked for. +Falsified if a caller demonstrably wants `terms=`/`model=` composed with +full export bodies — the fix then is a route-scoped request +(`/context/?terms=`, exempt above) or the plain default document, +not a third parameter on this door. diff --git a/docs/adr/0018-ossie-ingest.md b/docs/adr/0018-ossie-ingest.md index cefb180..c19499c 100644 --- a/docs/adr/0018-ossie-ingest.md +++ b/docs/adr/0018-ossie-ingest.md @@ -262,3 +262,29 @@ not-found message. answer is `unverified`, never a join planner. 3. Ossie 0.2 moves metrics to dataset-level measures ⇒ the version pin fails loudly on upgrade and the lookup keys change shape under authors. + +## Amendment (2026-09-11): `?model=`/`?terms=` (no route) project `exports[]` too + +§7's table says `/context?model=` returns "one semantic model in +full, composable with `terms`" — true, but on the real 42-export/47-dataset +design-partner cell it arrived beside all 42 exports' full `schema`/ +`check`/`semantic[]` bodies anyway: 467 KB, of which the requested model +was 38 KB. The door answered "the one thing you asked for" and "everything +else" in the same breath. + +`GET /context?model=` and `GET /context?terms=...`, absent a route, +now project `exports[]` through `ExportDoc::to_index` (ADR 0012 §4 +amendment 2026-09-11, same date) — every export reduced to identity, its +claims, `columns[]` (names), and `semantic_datasets[]` (`model/dataset` +names) in place of `semantic[]`'s full dataset/field/relationship/metric +bodies. `semantic_model`/`semantic_matches` themselves are unaffected — +`?model=` still returns that one model in full; the projection only trims +the tag-along `exports[]`. `datamk context --model`/`--terms` (without +`--export`) match. `/context/?terms=` keeps the route's full export +— see the ADR 0017 §2 amendment (same date) for the full reasoning, which +applies identically here since both doors share `?terms=`'s grammar. + +MCP is unaffected structurally: `describe_export` already narrows to one +route (never carries the whole-cell `exports[]` to begin with), and the new +`datamk:///context/index` resource (ADR 0012 §4 amendment) is the +whole-cell projection's own door. diff --git a/docs/guides/context.md b/docs/guides/context.md index aecb016..1ad5957 100644 --- a/docs/guides/context.md +++ b/docs/guides/context.md @@ -11,20 +11,37 @@ to maintain. # Served (same auth as the data routes) curl -H "Authorization: Bearer $TOKEN" https://orders.data.internal/context curl -H "Authorization: Bearer $TOKEN" "https://orders.data.internal/context?include=docs" +curl -H "Authorization: Bearer $TOKEN" "https://orders.data.internal/context?include=check" +curl -H "Authorization: Bearer $TOKEN" "https://orders.data.internal/context?view=index" curl -H "Authorization: Bearer $TOKEN" "https://orders.data.internal/context/orders_daily@2" curl -H "Authorization: Bearer $TOKEN" "https://orders.data.internal/context?terms=net_revenue,nr" curl -H "Authorization: Bearer $TOKEN" "https://orders.data.internal/context?model=invoice" -# Portable (no server, no token). Inlines docs by default. +# Portable (no server, no token). Inlines docs and the census by default. datamk context -f cell.yaml [-p prod] [--out context.json] [--no-docs] \ - [--export orders_daily@2] [--terms net_revenue,nr] [--model invoice] + [--export orders_daily@2] [--terms net_revenue,nr] [--model invoice] \ + [--view index] ``` -Query params on `/context` are a closed set: `include` (`docs` only), -`terms`, and `model`. `/context/` accepts `include` and `terms` only -— `model` there is 400 (`model` is a whole-cell view). Anything else is 400. -An unknown `` is 404 naming the routes that exist; an unknown `model` -is 404 naming the models that exist. +Query params on `/context` are a closed set: `include` (`docs`, `check`), +`terms`, `model`, and `view` (`full` default, `index`). `/context/` +accepts `include` and `terms` only — `model` and `view=index` are both 400 +there (`model` is a whole-cell view; a single-export door is already the +index's whole point). Anything else is 400. An unknown `` is 404 +naming the routes that exist; an unknown `model` is 404 naming the models +that exist. + +`?view=index` projects every `exports[]` entry to identity, claims, and +affordances — no `schema`, `check`, `probe`, or `semantic[]` bodies; +declared column names ride `columns[]` and bound Ossie datasets ride +`semantic_datasets[]` instead. `?model=` and `?terms=` (without a route) +apply the same projection automatically, since this door never narrows by +route — asking for one model or a term subset has no reason to also pay for +every export's full body. `/context/?terms=` is exempt: that door's +whole point is the route's full export. Measured on a real 42-export/ +47-dataset cell: default `/context` is 429 KB, of which `exports[]` is +383 KB; `?view=index` and `?model=` both land under 10% of the full +document's bytes on an equivalent synthetic cell. Every data-route response (200 and 404) carries: @@ -46,12 +63,13 @@ Flat document. A record with `from` is a **claim** (origin per field: | `status` | `draft` \| `verified_at_source` (live check passed) \| `verified` (published, verify-gated build). | | `grain_verified` | Whether grain uniqueness was checked. | | `description`, `from` | Cell prose and its origin. | -| `exports[]` | Per export: `name`, `version`, `route`, `contract`, `description`, `grain`, `freshness`, `schema`, and exactly one of `query` / `binding`. | -| `exports[].schema.` | `type`, optional `unit`, `description`, `from`. | +| `exports[]` | Per export: `name`, `version`, `route`, `contract`, `description`, `grain`, `freshness`, `schema`, and exactly one of `query` / `binding`. Under `?view=index`/`?model=`/`?terms=` (no route): projected — see below. | +| `exports[].schema.` | `type`, optional `unit`, `description`, `from`. Empty (`{}`) under the index projection. | +| `exports[].columns[]` | Index projection only: declared column names, in order — no type/unit/description/`from`. Empty (omitted) on the full document. | | `exports[].query` | `filters`, `filter_semantics`, `limit_default` 100, `limit_max` 1000, `offset_max` 1000000, `sample_request`. | -| `exports[].probe` | At swap: `at`, `rows`, `coverage` (min/max per grain col), `values` (per col, with `complete`), `null_rows` (per grain col), `example_request` (from one real row). | -| `exports[].check` | Live-check measurement (bound exports): `at`, `check` (`grain_unique`, or `schema` when no grain is declared), `grain`, `rows`, `distinct_grain`, `null_rows`, `columns`. | -| `exports[].check.columns` | The column census, bound exports only: per declared column, `null_rows`; for a non-grain column of at most 50 distinct values, `distinct` and up to five `top_values` (`{value, rows}`); otherwise `distinct_over_50: true`. `top_values` withheld under `--no-data`. | +| `exports[].probe` | At swap: `at`, `rows`, `coverage` (min/max per grain col), `values` (per col, with `complete`), `null_rows` (per grain col), `example_request` (from one real row). Absent under the index projection. | +| `exports[].check` | Live-check measurement (bound exports): `at`, `check` (`grain_unique`, or `schema` when no grain is declared), `grain`, `rows`, `distinct_grain`, `null_rows`, and `columns` under `?include=check`. Absent under the index projection. | +| `exports[].check.columns` | The column census, bound exports only, `?include=check` (served) or the portable `datamk context` emission (always) — see "The census on the wire" under `bind:` below. | | `exports[].freshness` | Author's `freshness:` verbatim. Advisory only, never compared against anything. | | `exports[].freshness_observed` | Discovered exports with a `freshness` claim and a live check: `at`, `intervals_end` (`deployed.intervals.end` as last synced), `age_seconds`. A number beside the claim, not a verdict on it. | | `upstreams[]` | `ref`, pinned `version` (usually `null`), attached `execution`, `data_as_of`. Last two absent for direct-attach upstreams. | @@ -59,19 +77,22 @@ Flat document. A record with `from` is a **claim** (origin per field: | `source_check` | `outcome`, `checked_at`, `datamk_version`, optional `data_as_of`. | | `freshness` (top level) | Server poll telemetry, not data age. | | `docs[]` | Per page: `target`, `source_path`, `media_type`; `sha256`, `bytes` after a release; `content` only under `?include=docs`. | -| `included` | Section names inlined (`[]` or `["docs"]`). Absent means the server predates docs pages. | +| `included` | Section names inlined — any subset of `["docs", "check"]`, `[]` on the default served variant, both on the default portable emission. Absent means the server predates this field. | | `include_request` | Relative URL to fetch the docs variant. | +| `index_request` | Relative URL to fetch the index projection (`context?view=index`). | | `definitions[]`, `missing_terms` | Glossary (always present) and unmatched `terms=` tokens. | | `semantic_models[]` | Apache Ossie index (always present, `[]` without a bound `semantic_model:`): `name`, `description`, `file`, `datasets`, `bound`, `metrics` counts. | | `semantic` | Ossie provenance, a measurement outside every digest: `synced_at`, `content_sha256`, `resolved`, `files`; `checked_at` once `datamk verify` has run. Present iff `semantic_model:` is declared and fresh. | | `semantic_model` | One semantic model in full, `?model=`/`--model` only: `name`, `description`, `ai_context`, `file`, `datasets[]` (unbound ones carry `routes: []`), `relationships[]`, `metrics[]`. | | `semantic_matches[]` | `?terms=`/`--terms` hits against an Ossie dataset, field, metric, or synonym: `token`, `kind`, `model`, `dataset`, `field`, `description`. Every hit returned — a token colliding across two models lists both. | -| `exports[].semantic[]` | Every Ossie dataset bound to this route: fields (with `verified`/`reason` once checked), `primary_key`/`primary_key_check`, relationships touching it, metrics fully checkable from this route alone plus `metric_refs` pointers to the rest. | +| `exports[].semantic[]` | Every Ossie dataset bound to this route: fields (with `verified`/`reason` once checked), `primary_key`/`primary_key_check`, relationships touching it, metrics fully checkable from this route alone plus `metric_refs` pointers to the rest. Absent under the index projection. | +| `exports[].semantic_datasets[]` | Index projection only: `model/dataset` names bound to this route, in place of `semantic[]`'s full bodies. Empty (omitted) on the full document. | | `data` | `served_here`, `channels`. | | `notes` | Engine notes, e.g. why status is `draft`, or that a bound semantic model is stale. | -`sample_request`, `example_request`, and `include_request` are relative to -the document's own URL (RFC 3986), so mounts in a multi-cell server work. +`sample_request`, `example_request`, `include_request`, `index_request`, +and `definitions_request` are relative to the document's own URL (RFC +3986), so mounts in a multi-cell server work. ## Author the meaning @@ -122,11 +143,13 @@ see [semantic-model.md](semantic-model.md) for authoring, sync, and verify. |---|---| | `/context` | `semantic_models[]` — name, description, file, dataset/bound/metric counts. | | `/context/` | `exports[].semantic[]` — every dataset bound to the route, in full. | -| `/context?model=` | `semantic_model` — one model in full, composable with `terms`. | -| `/context?terms=` | `semantic_matches[]` — Ossie dataset/field/metric names and synonyms join the `definitions:` lookup. | +| `/context?model=` | `semantic_model` — one model in full, composable with `terms`; `exports[]` projected to the index shape. | +| `/context?terms=` | `semantic_matches[]` — Ossie dataset/field/metric names and synonyms join the `definitions:` lookup; `exports[]` projected to the index shape. | +| `/context?view=index` | Every `exports[]` entry projected: identity, claims, `columns[]`, `semantic_datasets[]` — no `semantic[]` bodies. | `datamk mcp` adds a resource per model, `datamk:///semantic/` -(the same document `?model=` returns). +(the same document `?model=` returns), plus `datamk:///context/index` +(the same document `?view=index` returns). ## Meaning without rows: `--no-data` @@ -158,6 +181,7 @@ interface: | Document | `query: null` plus `binding: {source, object, connection}`, verbatim from `cell.yaml`, never profile-resolved. | | `datamk run` | Never computes a bound export. An all-bound cell has no snapshot; `run` refuses, use `verify` and `context`. | | `datamk verify -p prod` | Live-checks schema and grain against the warehouse (native types where available, BigQuery today) and takes the column census (`check.columns`). Writes `.cell/source_check.json` with a `cell.yaml` digest. | +| The census on the wire | `check.columns` (per declared column: `null_rows`; for a non-grain column of at most 50 distinct values, `distinct` and up to five `top_values`; otherwise `distinct_over_50: true`, `top_values` withheld under `--no-data`) is served only under `?include=check` — omitted by default, including under `?view=index`. The rollup (`check`, `grain`, `rows`, `distinct_grain`, `null_rows`, `at`) is on every record regardless. The portable `datamk context` artifact always inlines it when present — no `--no-check` flag; a file cannot be re-requested. | | Prose vs data | A column `description` or a definition with `applies_to: [route.column]` that says the column is empty (`currently null`, `backfill pending`, `not yet populated`, `not populated`, `always null`, `empty`) while the census measured rows populated draws a warning from `verify` and `release` and a note in `notes[]`. Never a failure. | | `datamk context` | Embeds `source_check` only while that digest matches the current `cell.yaml`. | | Status | Passing live check gives `verified_at_source`, never `verified`. | @@ -181,7 +205,7 @@ description, exports, and `context_digest`. Static file; never served by - `verify` fails on a `description` for a column the source no longer has, and on `contract: supported` without a non-empty export `description` (a `docs:` page does not count). - On a bound export, `verify` scans the object once per low-cardinality column for the census, on top of the grain check's full scan. Id-like columns cost one shared aggregate. Run it on a schedule, not per request. - `datamk release` digests `cell.yaml` prose and docs content; a change without a version bump warns. Warehouse prose is never in that digest. -- Interface digest = `/context` `ETag` = `/openapi.json` `info.version` = manifest `context_digest`. Prose and docs-content edits do not move it. Adding, removing, or renaming a `docs:` page does. `?include=docs` has its own `ETag` (`"~docs."`); `If-None-Match` gives 304. +- Interface digest = `/context` `ETag` = `/openapi.json` `info.version` = manifest `context_digest`. Prose and docs-content edits do not move it. Adding, removing, or renaming a `docs:` page does. `?include=docs` has its own `ETag` (`"~docs."`); `?include=check` adds `~check`; `?view=index` adds `~index`; `If-None-Match` gives 304 for the exact variant it names. - Docs edits move `content_hash`, so a deploy rolls the workload. - The Kubernetes ConfigMap carrying cell content is capped at 1 MiB, shared with `cell.yaml` and transform SQL. - No caller-supplied SQL, filters, or projections on any route. Use `datamk attach` for SQL. diff --git a/docs/guides/mcp.md b/docs/guides/mcp.md index 3196196..45364a2 100644 --- a/docs/guides/mcp.md +++ b/docs/guides/mcp.md @@ -43,6 +43,7 @@ count. Follow `next.offset`. | URI | Content | | --- | --- | | `datamk:///context` | The context document. | +| `datamk:///context/index` | The whole-cell document with `exports[]` projected to identity, column names, and affordances — the same document `?view=index` returns. | | `datamk:///context/` | One export, docs inlined. | | `datamk:///docs/` | One docs page. Targets: `cell`, ``, `definition:`. | | `datamk:///semantic/` | One Apache Ossie semantic model in full (ADR 0018) — the same document `?model=` returns. One resource per bound model. | diff --git a/src/catalog/mod.rs b/src/catalog/mod.rs index d4ab27b..81d66a2 100644 --- a/src/catalog/mod.rs +++ b/src/catalog/mod.rs @@ -786,6 +786,7 @@ mod tests { None, None, None, + false, ) .expect("the portable door still emits on a stale record"); assert!(doc.exports.is_empty()); diff --git a/src/cli.rs b/src/cli.rs index c6c5d1d..4266482 100644 --- a/src/cli.rs +++ b/src/cli.rs @@ -293,6 +293,24 @@ pub struct ContextArgs { /// whole-cell view: mutually exclusive with `--export`. #[arg(long, value_name = "NAME")] pub model: Option, + /// Project `exports[]` to identity, column names, and affordances + /// (item 2) — the portable twin of `GET /context?view=index`. A + /// whole-cell view: mutually exclusive with `--export`, same reason as + /// `--model`. + #[arg(long, value_enum, default_value = "full")] + pub view: ContextView, +} + +/// `--view`'s closed vocabulary (item 2) — mirrors `serve::VIEW_SECTIONS`; +/// kept a plain two-variant enum rather than a boolean because the wire +/// vocabulary (`full`/`index`) is the contract, and a third view is additive +/// here without touching every call site's signature. +#[derive(Debug, Clone, Copy, PartialEq, Eq, clap::ValueEnum, Default)] +#[clap(rename_all = "lowercase")] +pub enum ContextView { + #[default] + Full, + Index, } #[derive(Args)] diff --git a/src/context.rs b/src/context.rs index 6aa8a67..b07e7d3 100644 --- a/src/context.rs +++ b/src/context.rs @@ -110,6 +110,11 @@ pub struct ContextDocument { pub docs: Vec, /// The affordance to fetch docs content — a constant, always present. pub include_request: String, + /// The affordance to fetch the index projection (item 2, ADR 0012 §4 + /// amendment 2026-09-11) — a constant, always present, beside + /// `include_request`/`definitions_request`. Relative to the document's + /// own URL, same rule as those two (RFC 3986, see `INCLUDE_DOCS_REQUEST`). + pub index_request: String, /// The Ossie semantic-model index (ADR 0018 §7) — always present, /// `[]` without a bound `semantic_model:`. Full detail lives behind /// `?model=`/`--model` (`semantic_model`) or per-route in @@ -155,11 +160,14 @@ pub struct ContextDocument { pub data: DataBlock, /// Engine-emitted only — no author-supplied string ever lands here. pub notes: Vec, - /// Which optional sections this response inlines (ADR 0013) — - /// engine-emitted, always present: `[]` on the default variant, `["docs"]` - /// under `?include=docs` (served) or the default portable emission. Lets - /// an agent distinguish "server predates this field" (absent — an old - /// binary) from "this cell has no docs" (present, and `docs` is `[]`). + /// Which optional sections this response inlines (ADR 0013, extended by + /// item 4) — engine-emitted, always present: `[]` on the default + /// variant, any subset of `["docs", "check"]` under `?include=` (served) + /// — `docs` under `?include=docs`, `check` under `?include=check` — + /// or the default portable emission, which carries both (no + /// `--no-check` flag; a file cannot be re-requested). Lets an agent + /// distinguish "server predates this field" (absent — an old binary) + /// from "this cell has no docs" (present, and `docs` is `[]`). pub included: Vec, /// Portable artifact only (`datamk context`): when the document was /// emitted. A hosted `/context` omits it — the response is always now. @@ -202,6 +210,11 @@ const INCLUDE_DOCS_REQUEST: &str = "context?include=docs"; /// the reader, not resolved server-side. const DEFINITIONS_REQUEST: &str = "context?terms=[,]&include=docs"; +/// `context?view=index` — the affordance naming how to ask for the index +/// projection (item 2, ADR 0012 §4 amendment 2026-09-11). A constant, like +/// `INCLUDE_DOCS_REQUEST`. +const INDEX_REQUEST: &str = "context?view=index"; + /// One glossary term (ADR 0017 §2) — the index an agent reads before it /// asks. Always present in full on the default document (never gated /// behind `include=`): an agent recovering from a `missing_terms` miss @@ -344,6 +357,55 @@ pub struct ExportDoc { /// dataset/field claims themselves are structural, from `interface`. #[serde(skip_serializing_if = "Vec::is_empty")] pub semantic: Vec, + /// `view=index`/`--view index` only (`to_index`): declared column names, + /// no type/unit/description/from — enough for an agent to pick a route + /// before paying for `schema`'s full cost. Empty (omitted) on the full + /// document, where `schema` already carries this and more. + #[serde(skip_serializing_if = "Vec::is_empty")] + pub columns: Vec, + /// `view=index`/`--view index` only (`to_index`): `model/dataset` names + /// bound to this route — the index's stand-in for `semantic[]`'s full + /// bodies. Empty (omitted) on the full document. + #[serde(skip_serializing_if = "Vec::is_empty")] + pub semantic_datasets: Vec, +} + +impl ExportDoc { + /// The index projection (`?view=index`/`--view index`, added alongside + /// per-route docs and `?model=`/`?terms=` without a route): identity, + /// the claims that fit in a sentence, and the affordances — every field + /// that costs bytes without answering "what is this and how do I call + /// it" is dropped. Same struct as the full document (one schema, ADR + /// 0012 §2), so a consumer that has only ever seen `view=index` parses + /// `view=full` with no new code — it just sees more fields populated. + pub fn to_index(&self) -> ExportDoc { + ExportDoc { + name: self.name.clone(), + version: self.version.clone(), + route: self.route.clone(), + contract: self.contract, + description: self.description.clone(), + freshness: self.freshness.clone(), + freshness_observed: self.freshness_observed.clone(), + grain: self.grain.clone(), + from: self.from.clone(), + schema: IndexMap::new(), + query: self.query.clone(), + binding: self.binding.clone(), + depends_on: self.depends_on.clone(), + depends_on_unselected: self.depends_on_unselected, + deployed: self.deployed.clone(), + probe: None, + check: None, + semantic: Vec::new(), + columns: self.schema.keys().cloned().collect(), + semantic_datasets: self + .semantic + .iter() + .map(|d| format!("{}/{}", d.model, d.dataset)) + .collect(), + } + } } fn is_zero(n: &usize) -> bool { @@ -1114,6 +1176,8 @@ pub fn interface( .as_ref() .map(|index| route_semantic_docs(index, route)) .unwrap_or_default(), + columns: Vec::new(), + semantic_datasets: Vec::new(), } }) .collect(); @@ -2096,6 +2160,7 @@ pub fn assemble(facts: Facts) -> ContextDocument { definitions_request: interface.definitions_request, docs, include_request: interface.include_request, + index_request: INDEX_REQUEST.to_string(), semantic_models: interface.semantic_models, semantic, semantic_model: None, @@ -2269,15 +2334,18 @@ impl ContextDocument { true } - /// Inline docs content (ADR 0013): marks `included: ["docs"]` and sets + /// Inline docs content (ADR 0013): adds `"docs"` to `included` and sets /// `content` on each `docs[]` entry whose `target` a loaded page names. - /// `included` is set even when there are no pages — a truthful + /// `included` gains the entry even when there are no pages — a truthful /// (possibly empty) answer, distinct from "server predates this field". + /// Idempotent and order-independent with `inline_check` — both push + /// their own name rather than overwrite, so `?include=docs,check` + /// carries both regardless of call order. pub fn inline_docs<'a>( &mut self, pages: impl IntoIterator, ) { - self.included = vec!["docs".to_string()]; + mark_included(&mut self.included, "docs"); for p in pages { if let Some(d) = self.docs.iter_mut().find(|d| d.target == p.target) { d.media_type = p.media_type.clone(); @@ -2285,6 +2353,53 @@ impl ContextDocument { } } } + + /// `?include=check`/portable emission (item 4): marks `included` as + /// carrying the census — the data itself is already on `exports[]. + /// check.columns` from `assemble`; the served default's mirror call is + /// `omit_check_columns`, which clears it. A no-op on the columns + /// themselves, so calling this after `omit_check_columns` does not + /// resurrect them — callers choose exactly one per response. Marks + /// `included` unconditionally, exactly as `inline_docs` does: `included` + /// exists so an agent can tell "the census was not requested" from + /// "this cell has no census" (ADR 0013), and a cell with no bound + /// export answers the second question with empty `check`s, not with a + /// missing marker. + pub fn inline_check(&mut self) { + mark_included(&mut self.included, "check"); + } + + /// The served default (item 4): the per-column census is row-derived + /// detail, the same class `--no-data` already withholds from the probe + /// — cleared unless `?include=check` asked for it. The rollup + /// (`check`, `grain`, `rows`, `distinct_grain`, `null_rows`, `at`) is + /// never touched; it stays on every record regardless. + pub fn omit_check_columns(&mut self) { + for e in &mut self.exports { + if let Some(check) = &mut e.check { + check.columns = BTreeMap::new(); + } + } + } + + /// `?view=index`/`--view index`: every export projected through + /// `ExportDoc::to_index` — same cell-level fields, `exports[]` reduced + /// to identity, claims, and affordances. Refused on `/context/` + /// by the caller (a single-export door is already the index's whole + /// point); composes with `include`/`terms`/`model` because it only + /// touches `exports[]`. + pub fn apply_index_view(&mut self) { + self.exports = self.exports.iter().map(ExportDoc::to_index).collect(); + } +} + +/// Push `section` onto `included` iff it isn't already there — the shared +/// primitive `inline_docs`/`inline_check` use so `?include=docs,check` (in +/// either order) yields the same `included` array. +fn mark_included(included: &mut Vec, section: &str) { + if !included.iter().any(|s| s == section) { + included.push(section.to_string()); + } } /// Whether an `applies_to` entry (`name@major` or `name@major.column`) @@ -2616,18 +2731,21 @@ pub fn build_document( profile: &str, no_docs: bool, ) -> Result { - build_document_for(file, profile, no_docs, None, None, None) + build_document_for(file, profile, no_docs, None, None, None, false) } /// `build_document`, optionally narrowed to one export's route (`datamk /// context --export `, the portable twin of `GET /context/`), /// to a `--terms` list (ADR 0017 §6, composing with `--export` exactly as -/// `terms=` composes with `/context/`), and/or to one semantic model -/// in full (`--model`, ADR 0018 §7) — mutually exclusive with `--export` -/// (`model` is a whole-cell view). An unknown route or model is an error -/// naming the ones that exist; an unknown term is an error naming the known -/// ones — the deliberate CLI asymmetry with the served door (ADR 0013 §7): -/// a file written by `--out` cannot be re-requested. +/// `terms=` composes with `/context/`), to one semantic model in +/// full (`--model`, ADR 0018 §7) — mutually exclusive with `--export` +/// (`model` is a whole-cell view) — and/or to the index projection +/// (`--view index`, item 2), also mutually exclusive with `--export` for the +/// same reason `/context/` refuses `?view=index`. An unknown route or +/// model is an error naming the ones that exist; an unknown term is an error +/// naming the known ones — the deliberate CLI asymmetry with the served door +/// (ADR 0013 §7): a file written by `--out` cannot be re-requested. +#[allow(clippy::too_many_arguments)] pub fn build_document_for( file: &std::path::Path, profile: &str, @@ -2635,10 +2753,16 @@ pub fn build_document_for( export: Option<&str>, terms: Option<&[String]>, model: Option<&str>, + index_view: bool, ) -> Result { if export.is_some() && model.is_some() { anyhow::bail!("--model is a whole-cell view; drop --export"); } + if export.is_some() && index_view { + anyhow::bail!( + "--view index is a whole-cell projection; drop --export (it is already one export)" + ); + } let loaded = crate::config::load(file, profile)?; let routes = discoverable_routes(&loaded.def)?; let direct_attach = crate::config::direct_attach(&loaded.bindings); @@ -2894,6 +3018,14 @@ pub fn build_document_for( } } + // item 2/item 3: whole-cell only (refused on --export above), same + // trigger the served door uses — explicit `--view index`, or `--model`, + // or `--terms` without `--export` (which would otherwise narrow to one + // route's full export, ADR 0017 §6's composition). + if index_view || model.is_some() || (terms.is_some() && export.is_none()) { + doc.apply_index_view(); + } + // ADR 0013 §7 / ADR 0017 §6: inline by default — a portable artifact // with null content pointing at a path the reader doesn't have is a // dangling pointer. `--no-docs` withholds page content only; a term's @@ -2904,6 +3036,9 @@ pub fn build_document_for( let pages = crate::config::docs::load_declared(&loaded.dir, &loaded.def, &routes)?; doc.inline_docs(&pages); } + // item 4: the portable artifact always carries the census when one + // exists — no `--no-check` flag (a file cannot be re-requested). + doc.inline_check(); doc.emitted_at = Some(crate::timeutil::rfc3339_utc(crate::timeutil::unix_now())); doc.cell_yaml_digest = Some(cell_yaml_digest); @@ -2925,10 +3060,11 @@ pub fn emit( export: Option<&str>, terms: Option<&[String]>, model: Option<&str>, + index_view: bool, ) -> Result<()> { use anyhow::Context as _; - let doc = build_document_for(file, profile, no_docs, export, terms, model)?; + let doc = build_document_for(file, profile, no_docs, export, terms, model, index_view)?; let json = serde_json::to_string_pretty(&doc)?; match out { Some(path) => { @@ -3962,11 +4098,18 @@ interface: None, None, None, + false, ) .unwrap(); let default_doc: serde_json::Value = serde_json::from_str(&std::fs::read_to_string(&default_out).unwrap()).unwrap(); - assert_eq!(default_doc["included"], serde_json::json!(["docs"])); + // The portable emission inlines both optional sections (a file + // cannot be re-requested), and says so even when this cell has no + // census to inline — the marker answers "was it asked for". + assert_eq!( + default_doc["included"], + serde_json::json!(["docs", "check"]) + ); assert_eq!(default_doc["docs"][0]["target"], "cell"); assert!( default_doc["docs"][0]["content"] @@ -3985,11 +4128,13 @@ interface: None, None, None, + false, ) .unwrap(); let no_docs_doc: serde_json::Value = serde_json::from_str(&std::fs::read_to_string(&no_docs_out).unwrap()).unwrap(); - assert_eq!(no_docs_doc["included"], serde_json::json!([])); + // `--no-docs` withholds docs only; the census marker is unaffected. + assert_eq!(no_docs_doc["included"], serde_json::json!(["check"])); assert!( no_docs_doc["docs"][0].get("content").is_none(), "{no_docs_doc}" @@ -4053,7 +4198,7 @@ interface: ); let out = dir.join("context.json"); - emit(&file, "local", Some(&out), false, None, None, None) + emit(&file, "local", Some(&out), false, None, None, None, false) .expect("emit the context document"); let v: serde_json::Value = serde_json::from_str(&std::fs::read_to_string(&out).unwrap()).unwrap(); @@ -4147,7 +4292,7 @@ interface: crate::verify::run(&file, "local", false).expect("live-verify the all-bound cell"); let out = dir.join("context.json"); - emit(&file, "local", Some(&out), false, None, None, None) + emit(&file, "local", Some(&out), false, None, None, None, false) .expect("emit the context document"); let v: serde_json::Value = serde_json::from_str(&std::fs::read_to_string(&out).unwrap()).unwrap(); @@ -4185,7 +4330,7 @@ interface: std::fs::write(&file, yaml).unwrap(); let out = dir.join("context.json"); - emit(&file, "local", Some(&out), false, None, None, None) + emit(&file, "local", Some(&out), false, None, None, None, false) .expect("emit must still succeed, just without the stale record"); let v: serde_json::Value = serde_json::from_str(&std::fs::read_to_string(&out).unwrap()).unwrap(); @@ -4656,7 +4801,7 @@ interface: crate::verify::run(&file, "local", false).expect("live-verify the all-bound cell"); let out = dir.join("context.json"); - emit(&file, "local", Some(&out), false, None, None, None) + emit(&file, "local", Some(&out), false, None, None, None, false) .expect("emit the context document"); let v: serde_json::Value = serde_json::from_str(&std::fs::read_to_string(&out).unwrap()).unwrap(); @@ -5508,7 +5653,7 @@ interface: .unwrap(); let file = dir.join("cell.yaml"); - let doc = build_document_for(&file, "local", true, None, None, None) + let doc = build_document_for(&file, "local", true, None, None, None, false) .expect("datamk context must still emit, not refuse"); assert!( doc.notes @@ -5561,7 +5706,7 @@ interface: ) .unwrap(); - let doc = build_document_for(&file, "local", true, None, None, None) + let doc = build_document_for(&file, "local", true, None, None, None, false) .expect("datamk context must still emit over a drifted source"); assert!( doc.notes @@ -5603,7 +5748,7 @@ interface: let file = dir.join("cell.yaml"); crate::catalog::sync(&file, "local", false, false).expect("datamk sync (Ossie half)"); - let err = build_document_for(&file, "local", true, None, None, Some("nope")) + let err = build_document_for(&file, "local", true, None, None, Some("nope"), false) .expect_err("an unknown model must fail"); let msg = err.to_string(); assert!(msg.contains("no semantic model 'nope'"), "{msg}"); @@ -5611,7 +5756,7 @@ interface: // A near-miss (a prefix of the real name) is offered as the nearest // match, not the whole vocabulary. - let err = build_document_for(&file, "local", true, None, None, Some("invoi")) + let err = build_document_for(&file, "local", true, None, None, Some("invoi"), false) .expect_err("an unknown model must fail"); let msg = err.to_string(); assert!(msg.contains("nearest: invoice"), "{msg}"); @@ -5651,6 +5796,7 @@ interface: None, Some(&["flight_spen".to_string()]), None, + false, ) .expect_err("an unknown term must fail"); let msg = err.to_string(); @@ -5676,9 +5822,123 @@ interface: Some("orders_daily@2"), None, Some("x"), + false, ) .expect_err("--export and --model together must fail"); assert!(err.to_string().contains("--model is a whole-cell view")); let _ = std::fs::remove_dir_all(&dir); } + + /// item 2: `/context/`'s refusal of `?view=index` (ADR 0012 §4 + /// amendment) mirrors on the CLI — `--view index` composed with + /// `--export` is a whole-cell projection asked of a single-export door. + #[test] + fn view_index_and_export_together_is_an_error() { + let def = sample_def(); + let dir = semantic_context_dir("cli-view-export"); + std::fs::write(dir.join("cell.yaml"), serde_yaml::to_string(&def).unwrap()).unwrap(); + let file = dir.join("cell.yaml"); + let err = build_document_for( + &file, + "local", + true, + Some("orders_daily@2"), + None, + None, + true, + ) + .expect_err("--export and --view index together must fail"); + assert!(err.to_string().contains("whole-cell projection"), "{err}"); + let _ = std::fs::remove_dir_all(&dir); + } + + /// item 2: the index projection drops the bulky per-export fields and + /// carries column names plus bound Ossie dataset names instead. + #[test] + fn view_index_projects_exports_to_identity_and_affordances() { + let dir = semantic_context_dir("cli-view-index"); + std::fs::write( + dir.join("cell.yaml"), + "cell: orders\n\ + interface:\n\ + \x20 - name: orders_daily\n\ + \x20 version: 1.0.0\n\ + \x20 description: One row per order.\n\ + \x20 grain: [order_date]\n\ + \x20 schema:\n\ + \x20 order_date: date\n\ + \x20 revenue: decimal\n", + ) + .unwrap(); + let file = dir.join("cell.yaml"); + let doc = build_document_for(&file, "local", true, None, None, None, true) + .expect("--view index must still emit"); + assert_eq!(doc.exports.len(), 1); + let e = &doc.exports[0]; + assert_eq!(e.name, "orders_daily"); + assert_eq!(e.route, "orders_daily@1"); + assert_eq!( + e.columns, + vec!["order_date".to_string(), "revenue".to_string()] + ); + assert!(e.schema.is_empty(), "{:?}", e.schema); + assert!(e.semantic_datasets.is_empty(), "{:?}", e.semantic_datasets); + // `schema` (the full column bodies) is emitted empty, never + // populated — `columns` (names only) is the index's replacement. + let v = serde_json::to_value(&doc).unwrap(); + assert_eq!(v["exports"][0]["schema"], serde_json::json!({}), "{v}"); + let _ = std::fs::remove_dir_all(&dir); + } + + /// item 3: `datamk context --model` (no `--export`) projects `exports[]` + /// automatically — the portable twin of `?model=` doing the same. + #[test] + fn model_without_export_projects_exports_and_lists_the_bound_dataset() { + let dir = semantic_context_dir("cli-model-projects"); + std::fs::create_dir_all(dir.join("osi")).unwrap(); + std::fs::write( + dir.join("osi/invoice.yaml"), + "version: 0.1.1\n\ + semantic_model:\n\ + \x20 - name: invoice\n\ + \x20 datasets:\n\ + \x20 - name: flight_spend\n\ + \x20 source: flight_spend\n", + ) + .unwrap(); + std::fs::write(dir.join("data.csv"), "id,amount\n1,10.0\n").unwrap(); + std::fs::write( + dir.join("cell.yaml"), + "cell: t\n\ + sources:\n\ + \x20 raw: ./data.csv\n\ + interface:\n\ + \x20 - name: flight_spend\n\ + \x20 version: 1.0.0\n\ + \x20 grain: [id]\n\ + \x20 bind: raw\n\ + \x20 schema:\n\ + \x20 id: bigint\n\ + \x20 amount: decimal\n\ + semantic_model:\n\ + \x20 dir: osi\n", + ) + .unwrap(); + let file = dir.join("cell.yaml"); + crate::catalog::sync(&file, "local", false, false).expect("datamk sync (Ossie half)"); + + let doc = build_document_for(&file, "local", true, None, None, Some("invoice"), false) + .expect("--model must still emit"); + assert_eq!(doc.semantic_model.as_ref().unwrap().name, "invoice"); + assert_eq!(doc.exports.len(), 1); + let e = &doc.exports[0]; + assert!(e.schema.is_empty(), "{:?}", e.schema); + assert_eq!(e.columns, vec!["id".to_string(), "amount".to_string()]); + assert_eq!( + e.semantic_datasets, + vec!["invoice/flight_spend".to_string()] + ); + + let _ = std::fs::remove_dir_all(&dir); + } } diff --git a/src/main.rs b/src/main.rs index df8e8ef..55a38f2 100644 --- a/src/main.rs +++ b/src/main.rs @@ -150,6 +150,7 @@ async fn dispatch(command: Command) -> Result<()> { a.export.as_deref(), a.terms.as_deref(), a.model.as_deref(), + a.view == cli::ContextView::Index, ), Command::Mesh(a) => match a.command { cli::MeshCommand::Emit(e) => mesh::emit( diff --git a/src/serve/mcp.rs b/src/serve/mcp.rs index efe6709..3124761 100644 --- a/src/serve/mcp.rs +++ b/src/serve/mcp.rs @@ -461,6 +461,14 @@ impl McpServer { schema, meaning, query grammar, and provenance (GET /context).", "mimeType": "application/json", })); + out.push(json!({ + "uri": format!("datamk://{}/context/index", m.name), + "name": format!("{} context (index)", doc.cell), + "description": "The whole-cell context document with exports[] projected to \ + identity, column names, and affordances (GET /context?view=index) — for \ + picking a route before paying for its full document.", + "mimeType": "application/json", + })); for e in &doc.exports { out.push(json!({ "uri": self.context_uri(m, Some(&e.route)), @@ -516,7 +524,16 @@ impl McpServer { not_found(format!("mounted cells: {}", known.join(", "))) })?; let text = if path == "context" { - serde_json::to_string(&build_context_document(&m.state)) + let mut doc = build_context_document(&m.state); + doc.omit_check_columns(); + serde_json::to_string(&doc) + } else if path == "context/index" { + // item 2: the whole-cell projection — checked before the + // generic `context/` arm below, or `index` would read as + // an unknown route name. + let mut doc = build_context_document(&m.state); + doc.apply_index_view(); + serde_json::to_string(&doc) } else if let Some(route) = path.strip_prefix("context/") { if !m.state.routes.contains_key(route) { return Err(not_found(unknown_route_message(&m.state, route))); @@ -580,7 +597,8 @@ impl McpServer { serde_json::to_string(&doc) } else { return Err(not_found( - "paths are `context`, `context/`, `docs/`, `semantic/`" + "paths are `context`, `context/index`, `context/`, `docs/`, \ + `semantic/`" .to_string(), )); } @@ -647,6 +665,9 @@ fn narrowed_with_docs(s: &AppState, route: &str) -> crate::context::ContextDocum .filter(|p| targets.contains(p.target.as_str())) .collect(); doc.inline_docs(pages); + // item 4: MCP has no `?include=check` equivalent — mirrors the REST + // served default, which withholds the census unless asked. + doc.omit_check_columns(); doc } @@ -1097,6 +1118,59 @@ mod tests { .contains("mounted cells: smoke")); } + /// item 2: `datamk:///context/index` — listed in `resources()` + /// and readable, and the whole-cell projection it returns matches + /// `?view=index`'s server-side twin exactly. + #[tokio::test] + async fn index_resource_is_listed_readable_and_projected() { + let s = single(false); + let v = call(&s, 1, "resources/list", Value::Null).await; + let uris: Vec<&str> = v["result"]["resources"] + .as_array() + .unwrap() + .iter() + .map(|r| r["uri"].as_str().unwrap()) + .collect(); + assert!(uris.contains(&"datamk://smoke/context/index"), "{uris:?}"); + + let v = call( + &s, + 1, + "resources/read", + json!({ "uri": "datamk://smoke/context/index" }), + ) + .await; + let c = &v["result"]["contents"][0]; + assert_eq!(c["mimeType"], "application/json"); + let doc: Value = serde_json::from_str(c["text"].as_str().unwrap()).unwrap(); + let mut expected = build_context_document(&s.mounts[0].state); + expected.apply_index_view(); + assert_eq!( + doc, + serde_json::to_value(expected).unwrap(), + "the resource matches ?view=index's projection" + ); + assert_eq!(doc["exports"][0]["schema"], json!({}), "{doc}"); + assert!( + !doc["exports"][0]["columns"].as_array().unwrap().is_empty(), + "{doc}" + ); + + // `index` is checked before the generic `context/` arm, so + // it never reads as an unknown route. + let v = call( + &s, + 1, + "resources/read", + json!({ "uri": "datamk://smoke/bogus" }), + ) + .await; + assert!(v["error"]["message"] + .as_str() + .unwrap() + .contains("context/index")); + } + /// `single()`, plus a bound semantic model — one model (`invoice`), one /// dataset bound by name to `orders_daily@2`. The same construction /// `serve::smoke::router_with_semantic` uses for the REST door's diff --git a/src/serve/mod.rs b/src/serve/mod.rs index 2492582..d8a6d9d 100644 --- a/src/serve/mod.rs +++ b/src/serve/mod.rs @@ -1349,10 +1349,15 @@ async fn health(State(s): State>) -> Json { Json(body) } -/// The closed `include=` vocabulary (ADR 0013 §4) — the single source both -/// `validate_context_query` and `openapi::generate`'s documented `enum` -/// read, so the two can never drift. -pub(crate) const INCLUDE_SECTIONS: &[&str] = &["docs"]; +/// The closed `include=` vocabulary (ADR 0013 §4, extended by item 4) — the +/// single source both `validate_context_query` and `openapi::generate`'s +/// documented `enum` read, so the two can never drift. +pub(crate) const INCLUDE_SECTIONS: &[&str] = &["docs", "check"]; + +/// The closed `view=` vocabulary (item 2) — same discipline as +/// `INCLUDE_SECTIONS`. `full` is the default and never needs to be named; +/// listed here so both the validator and `openapi::generate` read one list. +pub(crate) const VIEW_SECTIONS: &[&str] = &["full", "index"]; /// `?terms=` cap (ADR 0017 §3) — the largest known estate at request time. const MAX_TERMS: usize = 64; @@ -1367,24 +1372,28 @@ struct ContextQuery { include: Vec, terms: Vec, model: Option, + /// `true` under `?view=index`; `false` (the default) is `view=full`. + index_view: bool, } /// Validate `/context`'s query string against the closed `{include, terms, -/// model}` grammar (ADR 0013 §4, extended by ADR 0017 §3 and ADR 0018 §7): -/// `include`'s value is a comma-separated list drawn from +/// model, view}` grammar (ADR 0013 §4, extended by ADR 0017 §3, ADR 0018 +/// §7, and item 2): `include`'s value is a comma-separated list drawn from /// `INCLUDE_SECTIONS`; `terms`'s value is a comma-separated list of tokens /// matching `[A-Za-z0-9_.-]`, capped at `MAX_TERMS`; `model`'s value is one /// token matching `[A-Za-z0-9_.-]{1,128}` — given more than once, a 400 (no -/// comma-list semantics: a document names one model, not several). Any -/// other parameter, an empty value, an empty segment, or an out-of-grammar -/// token is a 400. Takes the raw pairs (not a `HashMap`) so repeated keys -/// are seen rather than silently collapsed — the discipline -/// `serve_export`'s `validate_params` already established. +/// comma-list semantics: a document names one model, not several); `view`'s +/// value is one of `VIEW_SECTIONS`, also given at most once. Any other +/// parameter, an empty value, an empty segment, or an out-of-grammar token +/// is a 400. Takes the raw pairs (not a `HashMap`) so repeated keys are +/// seen rather than silently collapsed — the discipline `serve_export`'s +/// `validate_params` already established. fn validate_context_query(pairs: &[(String, String)]) -> std::result::Result { let mut include: Vec = Vec::new(); let mut terms: Vec = Vec::new(); let mut terms_seen = 0usize; let mut model: Option = None; + let mut view: Option = None; for (k, v) in pairs { match k.as_str() { "include" => { @@ -1394,14 +1403,15 @@ fn validate_context_query(pairs: &[(String, String)]) -> std::result::Result std::result::Result { + if view.is_some() { + return Err("`view` can only be given once".to_string()); + } + if !VIEW_SECTIONS.contains(&v.as_str()) { + return Err(format!( + "unknown `view` '{v}' — `/context` accepts: {}", + VIEW_SECTIONS.join(", ") + )); + } + view = Some(v.clone()); + } other => { return Err(format!( "unknown query parameter '{other}' — `/context` accepts `include` \ - (sections: docs), `terms`, and `model`" + (sections: {}), `terms`, `model`, and `view` (sections: {})", + INCLUDE_SECTIONS.join(", "), + VIEW_SECTIONS.join(", ") )); } } @@ -1469,6 +1493,7 @@ fn validate_context_query(pairs: &[(String, String)]) -> std::result::Result?terms=` (the +/// other door) is exempt — a route's own export stays full. async fn context_doc( State(s): State>, Query(pairs): Query>, @@ -1580,6 +1613,7 @@ async fn context_doc( Err(msg) => return (StatusCode::BAD_REQUEST, msg).into_response(), }; let want_docs = query.include.iter().any(|s| s == "docs"); + let want_check = query.include.iter().any(|s| s == "check"); // ADR 0017 §3: an unresolved token still selects the (empty) `terms=` // variant — `narrow_terms` reports it in `missing_terms`, never a 400. let terms = (!query.terms.is_empty()).then(|| resolve_terms(&s, &query.terms)); @@ -1595,17 +1629,19 @@ async fn context_doc( }; // The interface digest names the interface; the ETag names a - // representation of it (ADR 0013 §6, ADR 0017 §4, ADR 0018 §7): every - // suffix below is either precomputed at startup or, for + // representation of it (ADR 0013 §6, ADR 0017 §4, ADR 0018 §7, item 2, + // item 4): every suffix below is either precomputed at startup or, for // `~terms`/`~docs`/`~model` under a filter, a bounded in-memory hash (or // a plain name) over data already resident in `AppState` — never a // filesystem, store, or DuckDB read. let etag = context_etag( &s, want_docs, + want_check, None, terms.as_deref(), query.model.as_deref(), + query.index_view, ); if matches_etag(&headers, &etag) { return not_modified(etag); @@ -1618,6 +1654,16 @@ async fn context_doc( doc.narrow_terms(&query.terms, &s.interface.definitions, &all_docs); } doc.semantic_model = semantic_model; + // item 2/item 3: a whole-cell projection, so only the un-routed door + // offers it — `context_export` 400s `?view=index` before reaching here, + // and never auto-projects under `terms=` (a route's export stays full, + // ADR 0017 §2's composition). Also triggers without `?view=index` + // whenever `model=` is set, or `terms=` is set here (this door never + // narrows by route) — the whole point of asking for one key is not + // paying for every export's `schema`/`check`/`semantic[]` bodies too. + if query.index_view || query.model.is_some() || terms.is_some() { + doc.apply_index_view(); + } if want_docs { // Whatever `doc.docs` holds at this point (the full list, or the // `terms=`-selected subset) — inlining only ever adds `content` to @@ -1631,6 +1677,14 @@ async fn context_doc( .collect(); doc.inline_docs(pages); } + // item 4: the served default withholds the per-column census — the same + // row-derived class `--no-data` withholds from the probe — unless + // `?include=check` asked for it. + if want_check { + doc.inline_check(); + } else { + doc.omit_check_columns(); + } context_response(etag, doc) } @@ -1661,12 +1715,30 @@ async fn context_export( ) .into_response(); } + // item 2: same reasoning — `/context/` is already one export, so + // a whole-cell projection over `exports[]` has nothing to narrow. + if query.index_view { + return ( + StatusCode::BAD_REQUEST, + "view=index is a whole-cell projection; /context/ is already one export", + ) + .into_response(); + } let want_docs = query.include.iter().any(|s| s == "docs"); + let want_check = query.include.iter().any(|s| s == "check"); let terms = (!query.terms.is_empty()).then(|| resolve_terms(&s, &query.terms)); if !s.routes.contains_key(&route) { return (StatusCode::NOT_FOUND, unknown_route_message(&s, &route)).into_response(); } - let etag = context_etag(&s, want_docs, Some(&route), terms.as_deref(), None); + let etag = context_etag( + &s, + want_docs, + want_check, + Some(&route), + terms.as_deref(), + None, + false, + ); if matches_etag(&headers, &etag) { return not_modified(etag); } @@ -1687,6 +1759,11 @@ async fn context_export( .collect(); doc.inline_docs(pages); } + if want_check { + doc.inline_check(); + } else { + doc.omit_check_columns(); + } context_response(etag, doc) } @@ -1717,22 +1794,32 @@ fn unknown_route_message(s: &AppState, route: &str) -> String { /// distinct tag by construction); `~docs.` adds the docs-content /// bundle for `?include=docs`, computed over the *selected* pages' sha256s /// in declared order when `terms` narrows it, or `s.docs_bundle_sha12` -/// (precomputed at startup) otherwise; `~model.` (ADR 0018 §7) names a +/// (precomputed at startup) otherwise; `~check` (item 4) marks `?include= +/// check` — no hash needed, the census travels with the document itself, +/// the same "name it" style as `~model`; `~index` (item 2) marks +/// `?view=index`, same style. `~model.` (ADR 0018 §7) names a /// `?model=` narrowing — the plain name, not a hash: bounded, closed /// grammar, no benefit to hashing it. A cell with neither observed input -/// present and no `terms=`/`model=` keeps the exact byte-identical default -/// ETag it always had (mesh.rs copies this verbatim into `context_digest`). +/// present and none of `terms=`/`model=`/`include=`/`view=` keeps the exact +/// byte-identical default ETag it always had (mesh.rs copies this verbatim +/// into `context_digest`). +#[allow(clippy::too_many_arguments)] fn context_etag( s: &AppState, want_docs: bool, + want_check: bool, export: Option<&str>, terms: Option<&[String]>, model: Option<&str>, + index_view: bool, ) -> String { let mut etag = format!("\"{}", s.digest); if let Some(obs) = &s.observed_bundle_sha12 { etag.push_str(&format!("~observed.{obs}")); } + if index_view { + etag.push_str("~index"); + } if let Some(route) = export { etag.push_str(&format!("~export.{route}")); } @@ -1769,6 +1856,9 @@ fn context_etag( } None => {} } + if want_check { + etag.push_str("~check"); + } etag.push('"'); etag } @@ -4299,6 +4389,249 @@ pub(in crate::serve) mod smoke { }) } + /// A synthetic cell with `n_exports` bound exports of `n_columns` + /// columns each, plus one Apache Ossie dataset bound to every export by + /// name — the shape the motivating measurement (a real 42-export, + /// 47-dataset cell) approximates, built from scratch so the test is + /// hermetic. No transform runs (every export is `bind:`, like + /// `built_all_never_cell`): `engine::open` tolerates the never-published + /// catalog, so there is no DuckDB build cost even at this export count. + fn synthetic_router(n_exports: usize, n_columns: usize) -> Router { + let dir = std::env::temp_dir().join(format!( + "datamk-serve-smoke-synthetic-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + std::fs::create_dir_all(dir.join("profiles")).unwrap(); + std::fs::write(dir.join("data.csv"), "col_00\n1\n").unwrap(); + + let mut yaml = String::from( + "cell: synthetic\ndescription: A synthetic cell for the index-projection size test.\n\ + sources:\n\x20 raw: ./data.csv\naccess:\n\x20 shareable: true\ninterface:\n", + ); + for e in 0..n_exports { + yaml.push_str(&format!( + "\x20 - name: export_{e}\n\x20 version: 1.0.0\n\ + \x20 description: Synthetic export {e}, one row per col_00.\n\ + \x20 grain: [col_00]\n\x20 bind: raw\n\x20 schema:\n" + )); + for c in 0..n_columns { + yaml.push_str(&format!( + "\x20 col_{c:02}: {{ type: bigint, description: \"Column {c} of export {e}: a synthetic bigint measure used only to size-test the index projection against the full per-export schema, check census, and Ossie field bodies.\" }}\n" + )); + } + } + std::fs::write(dir.join("cell.yaml"), &yaml).unwrap(); + std::fs::write( + dir.join("profiles/local.yaml"), + "catalog: ./.cell/catalog.ducklake\nstorage: ./.cell/data\n", + ) + .unwrap(); + + let cell_yaml = dir.join("cell.yaml"); + + // A synthetic `.cell/source_check.json` (bypassing a real live + // verify pass — DuckDB never touched, per-column census fabricated + // directly) so the full document actually carries `check.columns` + // at the same scale the motivating measurement did: the census is + // 87 KB of a real 429 KB document, and a full document that omits + // it would understate what `?view=index`/`?include=check` are + // actually saving. + let cell_yaml_digest = crate::context::cell_yaml_digest_of(&cell_yaml).unwrap(); + let exports: BTreeMap = (0..n_exports) + .map(|e| { + let columns: BTreeMap = (0..n_columns) + .map(|c| { + let m = if c == 0 { + // col_00 is the grain column: census reports + // only its NULL count (the probe/grain check + // already own its value set). + crate::manifest::ColumnMeasurement { + null_rows: 0, + distinct: None, + distinct_over_50: false, + top_values: Vec::new(), + } + } else { + crate::manifest::ColumnMeasurement { + null_rows: 0, + distinct: Some(1), + distinct_over_50: false, + top_values: vec![crate::manifest::ValueRows { + value: "1".to_string(), + rows: 1, + }], + } + }; + (format!("col_{c:02}"), m) + }) + .collect(); + ( + format!("export_{e}@1"), + crate::manifest::ExportMeasurement { + check: "grain_unique".to_string(), + grain: vec!["col_00".to_string()], + rows: 1, + distinct_grain: Some(1), + null_rows: BTreeMap::from([("col_00".to_string(), 0)]), + columns, + }, + ) + }) + .collect(); + std::fs::create_dir_all(dir.join(".cell")).unwrap(); + std::fs::write( + dir.join(".cell/source_check.json"), + serde_json::to_string(&crate::manifest::SourceCheckRecord { + outcome: "passed".to_string(), + checked_at: "2026-09-11T00:00:00Z".to_string(), + data_as_of: None, + datamk_version: env!("CARGO_PKG_VERSION").to_string(), + cell_yaml_digest, + profile: "local".to_string(), + exports, + }) + .unwrap(), + ) + .unwrap(); + + let mut cell = + engine::open(&cell_yaml, "local", true).expect("open the synthetic cell read-only"); + let routes = crate::context::discoverable_routes(&cell.def).unwrap(); + let fields: Vec = (0..n_columns) + .map(|c| crate::ossie::Field { + name: format!("col_{c:02}"), + expression: crate::ossie::Expression { + dialects: vec![crate::ossie::DialectExpression { + dialect: crate::ossie::Dialect::AnsiSql, + expression: format!("col_{c:02}"), + }], + }, + dimension: None, + label: None, + description: Some(format!( + "Column {c}, synthetic Ossie field: a bound measure whose prose exists only to give \ + the full document realistic per-field mass for the size-projection test." + )), + datatype: Some(crate::ossie::DataType::Integer), + ai_context: None, + custom_extensions: vec![], + }) + .collect(); + // Datasets round-robin across several models (`n_models`) — the + // motivating cell's 47 datasets are not one model's worth (a real + // Ossie estate splits by domain), and `?model=` only ever + // returns one. Bundling every dataset into a single model would + // make `?model=` re-request the whole estate by definition, which + // is a real (unhelped) case, just not this cell's shape — one + // model per export/dataset here, the extreme of that split. + let n_models = n_exports; + let models: Vec = (0..n_models) + .map(|m| { + let datasets: Vec = (0..n_exports) + .filter(|e| e % n_models == m) + .map(|e| crate::ossie::Dataset { + name: format!("export_{e}"), + source: format!("export_{e}"), + primary_key: vec!["col_00".to_string()], + unique_keys: vec![], + description: Some(format!("Synthetic dataset {e}.")), + ai_context: None, + fields: fields.clone(), + custom_extensions: vec![], + }) + .collect(); + crate::ossie::SemanticModel { + name: format!("synthetic_model_{m}"), + description: Some(format!("The synthetic size-test model {m}.")), + ai_context: None, + datasets, + relationships: vec![], + metrics: vec![], + custom_extensions: vec![], + } + }) + .collect(); + let model_files: indexmap::IndexMap = (0..n_models) + .map(|m| { + ( + format!("synthetic_model_{m}"), + format!("synthetic_{m}.yaml"), + ) + }) + .collect(); + let record = crate::ossie::record::SemanticModelRecord { + datamk_version: "0.0.0".to_string(), + cell_yaml_digest: "d".to_string(), + synced_at: "2026-09-11T00:00:00Z".to_string(), + source: crate::ossie::source::SemanticModelSource::Dir { + dir: "osi".to_string(), + }, + resolved: crate::ossie::record::Resolved::Dir { + dir: "/abs/osi".to_string(), + }, + content_sha256: "abc123".to_string(), + files: (0..n_models) + .map(|m| format!("synthetic_{m}.yaml")) + .collect(), + model_files, + document: crate::ossie::Document { + version: "0.1.1".to_string(), + dialects: vec![], + vendors: vec![], + semantic_model: models, + }, + }; + cell.def.semantic = Some(crate::ossie::bind::SemanticIndex::build( + record, + &routes.iter().map(|(_, e)| e.clone()).collect::>(), + )); + let (state, _store) = + build_state(cell, /* no_data */ false, &cell_yaml, "local", "").expect("build state"); + app(state, DEFAULT_MAX_CONCURRENCY) + } + + /// The motivating measurement, reproduced on a hermetic fixture: on a + /// 40-export, 30-column, one-dataset-per-export synthetic cell, + /// `?view=index` and `?model=` are both well under 10% of the full + /// document's bytes — the whole point of the projection. + #[test] + fn view_index_and_model_are_under_a_tenth_of_the_full_document_on_a_synthetic_cell() { + let router = synthetic_router(40, 30); + rt().block_on(async { + let (status, full_body) = get(&router, "/context", None).await; + assert_eq!(status, StatusCode::OK, "{full_body}"); + let full_bytes = full_body.len(); + + let (status, index_body) = get(&router, "/context?view=index", None).await; + assert_eq!(status, StatusCode::OK, "{index_body}"); + let index_bytes = index_body.len(); + + let (status, model_body) = + get(&router, "/context?model=synthetic_model_0", None).await; + assert_eq!(status, StatusCode::OK, "{model_body}"); + let model_bytes = model_body.len(); + + eprintln!( + "synthetic 40-export/30-column cell: full={full_bytes}B \ + view=index={index_bytes}B ({:.1}%) model=synthetic_model_0={model_bytes}B ({:.1}%)", + 100.0 * index_bytes as f64 / full_bytes as f64, + 100.0 * model_bytes as f64 / full_bytes as f64, + ); + assert!( + (index_bytes as f64) < 0.10 * full_bytes as f64, + "view=index ({index_bytes}B) must be under 10% of full ({full_bytes}B)" + ); + assert!( + (model_bytes as f64) < 0.10 * full_bytes as f64, + "?model= ({model_bytes}B) must be under 10% of full ({full_bytes}B)" + ); + }); + } + #[test] fn model_query_param_returns_the_full_model_with_a_distinct_etag() { let router = router_with_semantic(); @@ -4370,6 +4703,175 @@ pub(in crate::serve) mod smoke { assert!(body.contains("`include`"), "{body}"); assert!(body.contains("`terms`"), "{body}"); assert!(body.contains("`model`"), "{body}"); + assert!(body.contains("`view`"), "{body}"); + }); + } + + // --- item 2: `?view=index` ---------------------------------------------- + + #[test] + fn view_index_is_200_with_a_distinct_etag_and_projects_exports() { + let router = router_with(|_| {}); + rt().block_on(async { + let (status, body) = get(&router, "/context?view=index", None).await; + assert_eq!(status, StatusCode::OK, "{body}"); + let v: serde_json::Value = serde_json::from_str(&body).unwrap(); + let e = &v["exports"][0]; + assert_eq!(e["route"], "orders_daily@2"); + assert_eq!(e["schema"], serde_json::json!({}), "{v}"); + assert!(e.get("check").is_none(), "{v}"); + assert!(e.get("probe").is_none(), "{v}"); + let columns: Vec<&str> = e["columns"] + .as_array() + .unwrap() + .iter() + .map(|c| c.as_str().unwrap()) + .collect(); + assert_eq!(columns, vec!["order_date", "region", "revenue"]); + assert_eq!(v["index_request"], "context?view=index"); + + let (_, h_plain, _) = get_with_headers(&router, "/context", &[]).await; + let (_, h_index, _) = get_with_headers(&router, "/context?view=index", &[]).await; + let etag_plain = h_plain + .get(header::ETAG) + .unwrap() + .to_str() + .unwrap() + .to_string(); + let etag_index = h_index + .get(header::ETAG) + .unwrap() + .to_str() + .unwrap() + .to_string(); + assert_ne!(etag_plain, etag_index); + assert!(etag_index.contains("~index"), "{etag_index}"); + + let (status, _, _) = get_with_headers( + &router, + "/context?view=index", + &[("if-none-match", &etag_index)], + ) + .await; + assert_eq!(status, StatusCode::NOT_MODIFIED); + }); + } + + #[test] + fn view_bogus_is_400_naming_the_vocabulary() { + let router = router_with(|_| {}); + rt().block_on(async { + let (status, body) = get(&router, "/context?view=bogus", None).await; + assert_eq!(status, StatusCode::BAD_REQUEST, "{body}"); + assert!(body.contains("unknown `view`"), "{body}"); + assert!(body.contains("full"), "{body}"); + assert!(body.contains("index"), "{body}"); + }); + } + + #[test] + fn view_index_on_the_route_door_is_400() { + let router = router_with(|_| {}); + rt().block_on(async { + let (status, body) = get(&router, "/context/orders_daily@2?view=index", None).await; + assert_eq!(status, StatusCode::BAD_REQUEST, "{body}"); + assert!(body.contains("whole-cell projection"), "{body}"); + }); + } + + // --- item 3: `?model=`/`?terms=` without a route also project ---------- + + /// `?model=` and `?terms=` (no route) carry the index projection + /// automatically — no `schema`, no `semantic[]` bodies — while + /// `/context/?terms=` keeps that route's full export (ADR 0017 + /// §2's composition holds). + #[test] + fn model_and_terms_without_a_route_project_exports_but_the_route_door_keeps_the_full_export() { + let model_router = router_with_semantic(); + let terms_router = router_definitions(|_| {}); + rt().block_on(async { + let (status, body) = get(&model_router, "/context?model=invoice", None).await; + assert_eq!(status, StatusCode::OK, "{body}"); + let v: serde_json::Value = serde_json::from_str(&body).unwrap(); + assert_eq!(v["semantic_model"]["name"], "invoice"); + let e = &v["exports"][0]; + assert_eq!(e["schema"], serde_json::json!({}), "{v}"); + assert!(e.get("semantic").is_none(), "{v}"); + assert!( + e["semantic_datasets"] + .as_array() + .unwrap() + .contains(&serde_json::json!("invoice/orders_daily")), + "{v}" + ); + + let (status, body) = get(&terms_router, "/context?terms=net_revenue", None).await; + assert_eq!(status, StatusCode::OK, "{body}"); + let v: serde_json::Value = serde_json::from_str(&body).unwrap(); + assert_eq!(v["exports"][0]["schema"], serde_json::json!({}), "{v}"); + + // The route door is exempt: `/context/?terms=` still + // carries that route's full schema. + let (status, body) = + get(&terms_router, "/context/orders@1?terms=net_revenue", None).await; + assert_eq!(status, StatusCode::OK, "{body}"); + let v: serde_json::Value = serde_json::from_str(&body).unwrap(); + assert_eq!( + v["exports"][0]["schema"].as_object().unwrap().len(), + 2, + "{v}" + ); + assert!(v["exports"][0]["schema"]["revenue"].is_object(), "{v}"); + }); + } + + // --- item 4: `?include=check` ------------------------------------------- + + /// The served default withholds `check.columns`; `?include=check` inlines + /// it, flips `included`, and gets its own `ETag` variant. + #[test] + fn include_check_inlines_columns_and_flips_included_and_etag() { + let router = all_never_verified_router_with(|_| {}); + rt().block_on(async { + let (status, body) = get(&router, "/context", None).await; + assert_eq!(status, StatusCode::OK, "{body}"); + let v: serde_json::Value = serde_json::from_str(&body).unwrap(); + let check = &v["exports"][0]["check"]; + assert!(check.is_object(), "{v}"); + assert!(check.get("columns").is_none(), "{v}"); + assert_eq!(check["rows"], 2, "the rollup stays: {v}"); + assert_eq!(v["included"], serde_json::json!([])); + + let (status, body) = get(&router, "/context?include=check", None).await; + assert_eq!(status, StatusCode::OK, "{body}"); + let v: serde_json::Value = serde_json::from_str(&body).unwrap(); + assert!(v["exports"][0]["check"]["columns"]["id"].is_object(), "{v}"); + assert_eq!(v["included"], serde_json::json!(["check"])); + + let (_, h_plain, _) = get_with_headers(&router, "/context", &[]).await; + let (_, h_check, _) = get_with_headers(&router, "/context?include=check", &[]).await; + let etag_plain = h_plain + .get(header::ETAG) + .unwrap() + .to_str() + .unwrap() + .to_string(); + let etag_check = h_check + .get(header::ETAG) + .unwrap() + .to_str() + .unwrap() + .to_string(); + assert_ne!(etag_plain, etag_check); + assert!(etag_check.contains("~check"), "{etag_check}"); + + let (status, _, _) = get_with_headers( + &router, + "/context?include=check", + &[("if-none-match", &etag_check)], + ) + .await; + assert_eq!(status, StatusCode::NOT_MODIFIED); }); } @@ -4491,6 +4993,13 @@ pub(in crate::serve) mod smoke { /// `false`), so there is no "routes should be here but aren't" to /// explain; only a *server* that could have mounted routes and /// didn't needs the sentence. + /// - `exports[].check.columns` / `included`'s `"check"` entry (item + /// 4): portable always inlines the census when one exists — there + /// is no `--no-check` flag, a file cannot be re-requested — while + /// hosted's default withholds it (`?include=check` opts in). This + /// test never requests `include=check` on the hosted side, so + /// unlike the docs asymmetry below it can't be resolved by + /// comparing like-for-like; it is always stripped. /// /// Everything else — `status`, `grain_verified`, `declared` /// (including `query`, unconditional since commit 4), @@ -4547,12 +5056,19 @@ pub(in crate::serve) mod smoke { for e in exports.iter_mut() { if let Some(e) = e.as_object_mut() { e.remove("probe"); + if let Some(check) = e.get_mut("check").and_then(|c| c.as_object_mut()) + { + check.remove("columns"); + } } } } if let Some(notes) = obj.get_mut("notes").and_then(|n| n.as_array_mut()) { notes.retain(|n| n.as_str() != Some(crate::context::NOTE_NO_ROUTES_MOUNTED)); } + if let Some(included) = obj.get_mut("included").and_then(|i| i.as_array_mut()) { + included.retain(|s| s.as_str() != Some("check")); + } } assert_eq!( diff --git a/src/serve/openapi.rs b/src/serve/openapi.rs index 72f50dc..a282dbe 100644 --- a/src/serve/openapi.rs +++ b/src/serve/openapi.rs @@ -139,14 +139,16 @@ fn terms_vocabulary( v } -/// `/context`'s `include`, `terms`, and `model` parameters, documented from -/// `INCLUDE_SECTIONS` (`serve::INCLUDE_SECTIONS`), the cell's glossary plus +/// `/context`'s `include`, `terms`, `model`, and `view` parameters, +/// documented from `INCLUDE_SECTIONS`/`VIEW_SECTIONS` (`serve:: +/// INCLUDE_SECTIONS`/`serve::VIEW_SECTIONS`), the cell's glossary plus /// Ossie lookup, and the bound semantic model index respectively — the /// exact vocabularies `validate_context_query`/`resolve_terms`/ /// `resolve_model` enforce, so none can drift from what's generated here -/// (ADR 0013 §8, ADR 0017 §6, ADR 0018 §7). +/// (ADR 0013 §8, ADR 0017 §6, ADR 0018 §7, item 2). fn context_path_item(terms_vocabulary: &[Value], model_vocabulary: &[Value]) -> Value { let sections: Vec = super::INCLUDE_SECTIONS.iter().map(|s| json!(s)).collect(); + let views: Vec = super::VIEW_SECTIONS.iter().map(|s| json!(s)).collect(); json!({ "get": { "summary": "The cell's context document (ADR 0012)", @@ -160,12 +162,15 @@ fn context_path_item(terms_vocabulary: &[Value], model_vocabulary: &[Value]) -> "style": "form", "explode": false, "description": "Comma-separated optional sections to inline. Omit for the \ - default document; `docs` inlines every declared docs page. \ - Each section named in the response's `included` array is \ - inlined ON THE RECORDS it belongs to — `include=docs` echoes \ - `\"included\": [\"docs\"]` and sets `content` on each \ - `docs[]` entry. `included` holds section names, never the \ - content itself.", + default document; `docs` inlines every declared docs page; \ + `check` inlines the per-column census on every bound \ + export's `check`. Each section named in the response's \ + `included` array is inlined ON THE RECORDS it belongs to — \ + `include=docs` echoes `\"included\": [\"docs\"]` and sets \ + `content` on each `docs[]` entry; `include=check` echoes \ + `\"included\": [\"check\"]` and sets `columns` on each \ + `exports[].check`. `included` holds section names, never \ + the content itself.", "schema": { "type": "array", "items": { "type": "string", "enum": sections } } }, { @@ -178,7 +183,11 @@ fn context_path_item(terms_vocabulary: &[Value], model_vocabulary: &[Value]) -> `definitions[]` and, under `include=docs`, `docs[]` to \ those terms' `definition:` pages only. Matching is \ case-insensitive; an unmatched token is echoed in \ - `missing_terms`, never a 400 or 404.", + `missing_terms`, never a 400 or 404. On this door (never \ + `/context/{route}`), also projects `exports[]` to the \ + index shape (item 3) — the same projection `?view=index` \ + applies — since asking for a handful of terms has no reason \ + to pay for every export's full body too.", "schema": { "type": "array", "items": { "type": "string", "enum": terms_vocabulary } } }, @@ -189,8 +198,24 @@ fn context_path_item(terms_vocabulary: &[Value], model_vocabulary: &[Value]) -> "description": "One Apache Ossie semantic model name (ADR 0018 §7) — \ returns that model in full as `semantic_model`, composable \ with `terms`. A whole-cell view: not accepted on \ - `/context/{route}` (400).", + `/context/{route}` (400). Also projects `exports[]` to the \ + index shape (item 3), same reason as `terms`.", "schema": { "type": "string", "enum": model_vocabulary } + }, + { + "name": "view", + "in": "query", + "required": false, + "description": "`full` (default) or `index` (item 2). `index` projects \ + every `exports[]` entry to identity, claims, and \ + affordances (`ExportDoc::to_index`) — no `schema`, `check`, \ + `probe`, or `semantic[]` bodies; declared column names ride \ + `columns[]` and bound Ossie datasets ride \ + `semantic_datasets[]` instead. Composes with `include`, \ + `terms`, and `model`. Not accepted on `/context/{route}` \ + (400) — a single-export door is already the index's whole \ + point.", + "schema": { "type": "string", "enum": views } } ], "responses": { @@ -201,9 +226,9 @@ fn context_path_item(terms_vocabulary: &[Value], model_vocabulary: &[Value]) -> "304": { "description": "not modified (If-None-Match matched the current ETag \ for the requested variant)" }, "400": { "description": "unknown query parameter; an unrecognized/empty \ - `include` section; or a `terms`/`model` token that's \ + `include` section; a `terms`/`model` token that's \ empty, out of grammar, or (terms) exceeds the token \ - cap" }, + cap; or an unrecognized `view`" }, "401": { "description": "missing or unknown bearer token (cell has access.roles)" }, "403": { "description": "cell is not shareable, or the token's roles do not \ include an allowed role" }, @@ -224,7 +249,7 @@ fn context_schema() -> Value { "type": "object", "required": ["datamk_context", "cell", "status", "grain_verified", "exports", "upstreams", "definitions", "missing_terms", "definitions_request", "docs", - "include_request", "data", "notes", "included"], + "include_request", "index_request", "data", "notes", "included"], "description": "One level, no regions. A fact is a CLAIM iff its record carries \ `from` (who said it); a fact is a MEASUREMENT iff it sits in a \ block with a timestamp (`build`, `source_check`, `freshness`, an \ @@ -308,6 +333,9 @@ fn context_schema() -> Value { "description": "Present only when `included` contains `docs`." } }}}, "include_request": { "type": "string" }, + "index_request": { "type": "string", + "description": "The affordance to fetch the index projection (item 2) — \ + `context?view=index`, relative to the document's own URL." }, "semantic_models": { "type": "array", "description": "Apache Ossie semantic-model index (ADR 0018 §7) — always \ present, `[]` without a bound `semantic_model:`. Full detail \ @@ -557,7 +585,17 @@ fn export_schema() -> Value { "semantic": { "type": "array", "items": semantic_dataset_schema(), "description": "Every Apache Ossie dataset bound to this route (ADR 0018 §7) \ — `[]` without a bound semantic model, or when nothing binds \ - to this route." } + to this route." }, + "columns": { "type": "array", "items": { "type": "string" }, + "description": "`?view=index`/`?model=`/`?terms=` (without a route) only \ + (item 2/3): declared column names, in order, no type, unit, \ + description, or `from` — `schema`'s replacement in the index \ + projection. Empty (omitted) on the full document." }, + "semantic_datasets": { "type": "array", "items": { "type": "string" }, + "description": "`?view=index`/`?model=`/`?terms=` (without a route) only: \ + `model/dataset` names bound to this route — `semantic[]`'s \ + replacement in the index projection. Empty (omitted) on the \ + full document." } } }) } @@ -654,7 +692,9 @@ fn column_census_schema() -> Value { that, `distinct_over_50: true` and no values. \ `top_values` is withheld under --no-data. Absent on a \ materialized export and on a record that predates \ - this measurement.", + this measurement — and, served, unless `?include=check` \ + asked for it (item 4); the portable `datamk context` \ + artifact always inlines it when present.", "additionalProperties": { "type": "object", "required": ["null_rows"], @@ -980,6 +1020,7 @@ mod tests { "definitions_request", "docs", "include_request", + "index_request", "semantic_models", "semantic_matches", "build", @@ -1022,6 +1063,9 @@ mod tests { export["check"]["properties"]["check"]["enum"], json!(["grain_unique", "schema"]) ); + // item 2: the index projection's replacements for `schema`/`semantic[]`. + assert!(export.get("columns").is_some()); + assert!(export.get("semantic_datasets").is_some()); // The measurement beside the freshness claim is documented, and the // claim's own description points at it. assert!(export.get("freshness_observed").is_some()); @@ -1198,7 +1242,7 @@ mod tests { let params = doc["paths"]["/context"]["get"]["parameters"] .as_array() .unwrap(); - assert_eq!(params.len(), 3, "{params:?}"); + assert_eq!(params.len(), 4, "{params:?}"); let include = ¶ms[0]; assert_eq!(include["name"], "include"); assert_eq!(include["style"], "form"); @@ -1212,6 +1256,18 @@ mod tests { assert_eq!(enumerated, super::super::INCLUDE_SECTIONS.to_vec()); assert_eq!(params[2]["name"], "model"); + // item 2: `view` is documented from the same shared-vocabulary + // discipline as `include`. + let view = ¶ms[3]; + assert_eq!(view["name"], "view"); + let view_enum: Vec<&str> = view["schema"]["enum"] + .as_array() + .unwrap() + .iter() + .map(|v| v.as_str().unwrap()) + .collect(); + assert_eq!(view_enum, super::super::VIEW_SECTIONS.to_vec()); + let responses = &doc["paths"]["/context"]["get"]["responses"]; for code in ["200", "304", "400", "401", "403", "404"] { assert!(responses.get(code).is_some(), "missing response {code}"); diff --git a/test/fixtures/context_v4_golden.json b/test/fixtures/context_v4_golden.json index 82cae2a..4d53332 100644 --- a/test/fixtures/context_v4_golden.json +++ b/test/fixtures/context_v4_golden.json @@ -71,6 +71,7 @@ "definitions_request": "context?terms=[,]&include=docs", "docs": [], "include_request": "context?include=docs", + "index_request": "context?view=index", "semantic_models": [], "semantic_matches": [], "build": {