diff --git a/AGENTS.md b/AGENTS.md index 5a30ab0a..8695721f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,15 +33,15 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime. - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level. - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Strict insert and upsert both route through the sealed exact-`id`, filter-bearing adapter; bare Lance Append is test-only. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. Mutation/Load keeps one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before recovery arm; mutation update scans stream into the remaining table budget after pending-key shadowing, with blob sizes checked before payload reads. It then arms an identity-bearing recovery-v9 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, and publishes under the same token. Unrelated retryable pre-effect authority movement may fully reprepare without changing logical mode. A proven effect-free strict conflict returns `KeyConflict` only after a fresh exact-ID probe; no exact match triggers bounded full strict-mode reprepare, while an effect-free upsert conflict also fully reprepares. Any earlier effect or ambiguity returns `RecoveryRequired` with the sidecar retained. Strict read-set conflicts return `ReadSetChanged`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary. -- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL; internal schema v12 moved that hidden path to recovery-v14 `StreamEnrollmentV2`, which selects exact enrollment/binding ledger receipts with the physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB of logical dense-slice Arrow data), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and the bounded fold mechanics. Cheap raw bounds and exact post-tombstone validation run before recovery; resident-producing served put ownership follows bounded preprocessing/inflight → root MemWAL opportunity shared → profile shared → table admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans claim, seal, drain proof, both fold effects, and publication. Admission, replay, and fold all charge `ArrayData::get_slice_memory_size`; fold densifies retained arrays. Backing-buffer capacity and physical allocation are not the limit, while isolated fold RSS remains a 384-MiB remeasurement tripwire. The 8,192-row high-entropy near-cap closure cell remains green. Internal schema v9 added the private B2 compare-and-chain core: canonical payload and token digests, grammar-impossible trusted hidden row metadata (`__omnigraph_stream_v1$`), same-generation token overlays, and a manifest-selected graph-global `_stream_tokens.lance` authority. Exact base/token lookups and recovery validation materialize at most the requested rows plus one and cap both Arrow batch bytes and cumulative retained bytes. Admission recaptures lifecycle/binding/HEAD authority only after acquiring shared admission and the same-key queue, so a stale provisional capture cannot authorize a WAL put. Under lifecycle-v3, every cold opener and ordinary fold first publishes an exact recovery-v14 writer claim whose immutable attempt/terminal receipts authenticate the bounded WAL suffix and full current-generation LWW projection. `StreamFoldV2` and `StreamDrainFold` own exact pre-minted base/token transactions, bind the selected current claim and authenticated tail, and publish the two pointers, one lifecycle successor, and graph lineage together. Quiescence is restartable `OPEN → DRAINING → SEALED`: it supports never-written empty lanes, folds non-empty lanes, reuses an already-flushed exact cut after restart, and emits one immutable terminal management receipt. V13's recovery-v15 `StreamResume` privately consumes an exact sealed proof or guarded drain state, owns the higher-epoch claim plus terminal claim/management receipts, and alone publishes the next `OPEN` row. Receipt-first replay is idempotent, and named-branch or stale-binding authority fails before the claim. V14's recovery-v16 `StreamSealedEnsureIndices` layers exact prior/next SEALED rows over recovery-v8's CreateIndex plan; under `stream_manage` and the retained checked runtime it publishes every productive index pointer, both HEAD witnesses, recomputed empty proof, and lifecycle revision in one CAS. V15's recovery-v17 `StreamSealedOptimize` separately owns Optimize's internally committing maintenance plan and exact achieved HEADs, then publishes productive table pointers with recomputed `SEALED` proofs in one CAS. Neither checked maintenance path writes a token row, advances a management-receipt chain, or accepts a caller operation ID; true no-work remains effect-free. V16's recovery-v18 `StreamRebind` consumes one exact `SEALED` proof, creates a fresh enrollment and empty shard namespace, appends immutable binding and fence-only claim receipts, and publishes a new exact `SEALED` proof without admitting a writer or put. A separate recovery-v15 resume is required to open the fresh scope. This physical rebind keeps the accepted schema unchanged. Productive SchemaApply remains refused on any enrolled graph in EXP; schema changes use a checked sealed/retired export, initialize a fresh graph with the desired schema, and load there, never over the enrolled source. V17's recovery-v19 `StreamAuthorityRetirement` owns the exact immutable retirement-receipt transaction and sole lineage-neutral `DISABLED → RETIRED` publication; it never moves graph or branch heads and its selected receipt/profile chain replaces `RecoveryAudit`. V18's recovery-v20 `StreamCorrection` owns one exact blocked generation cut, a pre-minted base effect, and one combined token-successor plus correction/management-receipt effect; recovery may recreate only the missing token participant after the exact base effect and publishes the two pointers, `DRAINING` successor, lineage, and optional PRESENT attribution together. V19's recovery-v21 `DeadLetterFold` owns one conditionally created canonical object plus exact base and token effects, accepts mixed and all-diverted outcomes, and publishes versioned fold attribution with current token-schema-v3 `DEAD_LETTERED` evidence. Recovery-v21 also carries the three-disposition retirement owner; recovery-v19 retains its historical two-disposition meaning. Ambient EnsureIndices and Optimize remain refused for enrolled tables. Historical recovery-v10 enrollment, recovery-v12 lifecycle-v2 fold, and older resume/maintenance/rebind scaffolds keep their old wire meanings and are refused rather than synthesized. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all profile is active**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and untouched through recovery/reopen; provider exhaustion is loud. The core remains crate-private behind feature-gated, doc-hidden test seams. The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary is registered, and embedded `Omnigraph::stream_status` exposes a durable manifest-only projection. V11's protocol-v2 checked cluster authority remains unchanged: `ENABLED` carries its active fold delegation, `DISABLING` carries a drain-only continuation plan, and `RETIRED` is receipt/cut-bound and fail-closed. Recovery-v13 remains exactly `StreamProfileChange`. Profile apply enforces `stream_manage` under both the currently applied and desired graph policies through the cluster-state CAS. Enabled-profile Mutation/Load/delete require the exact checked served runtime, while BranchMerge remains closed under both `ENABLED` and `DISABLING` even with that runtime. The narrow cluster-only `stream retire-for-rebuild plan|confirm` command is the sole active retirement surface and permanently freezes a verified `WITHDRAWN | DEAD_LETTERED` cut for export/rebuild. Checked offline disable is the sole supported production quiescence owner. F7a activates one graph-native mixed node/edge served-ingest bridge at `POST /graphs/{graph_id}/stream/ingest` plus remote client/CLI/OpenAPI parity. A strong graph-authority ETag gates body ownership; the engine resolves declarations, lazily prepares private lanes, and returns only ordered graph-logical results. Public lane enrollment, general resume/abort, rebind, and lifecycle/maintenance/status transports remain inactive; exact-terminal served export remains the F6b5 exception. The stopped/offline cluster-only `stream block show|correct` and `stream dead-letter list|export` surfaces are the narrow active inspection/correction exceptions; no hidden seam is a supported operator escape hatch. +- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL; internal schema v12 moved that hidden path to recovery-v14 `StreamEnrollmentV2`, which selects exact enrollment/binding ledger receipts with the physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB of logical dense-slice Arrow data), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and the bounded fold mechanics. Cheap raw bounds and exact post-tombstone validation run before recovery; resident-producing served put ownership follows bounded preprocessing/inflight → root MemWAL opportunity shared → profile shared → table admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans claim, seal, drain proof, both fold effects, and publication. Admission, replay, and fold all charge `ArrayData::get_slice_memory_size`; fold densifies retained arrays. Backing-buffer capacity and physical allocation are not the limit, while isolated fold RSS remains a 384-MiB remeasurement tripwire. The 8,192-row high-entropy near-cap closure cell remains green. Internal schema v9 added the private B2 compare-and-chain core: canonical payload and token digests, grammar-impossible trusted hidden row metadata (`__omnigraph_stream_v1$`), same-generation token overlays, and a manifest-selected graph-global `_stream_tokens.lance` authority. Exact base/token lookups and recovery validation materialize at most the requested rows plus one and cap both Arrow batch bytes and cumulative retained bytes. Admission recaptures lifecycle/binding/HEAD authority only after acquiring shared admission and the same-key queue, so a stale provisional capture cannot authorize a WAL put. Under lifecycle-v3, every cold opener and ordinary fold first publishes an exact recovery-v14 writer claim whose immutable attempt/terminal receipts authenticate the bounded WAL suffix and full current-generation LWW projection. `StreamFoldV2` and `StreamDrainFold` own exact pre-minted base/token transactions, bind the selected current claim and authenticated tail, and publish the two pointers, one lifecycle successor, and graph lineage together. Quiescence is restartable `OPEN → DRAINING → SEALED`: it supports never-written empty lanes, folds non-empty lanes, reuses an already-flushed exact cut after restart, and emits one immutable terminal management receipt. V13's recovery-v15 `StreamResume` privately consumes an exact sealed proof or guarded drain state, owns the higher-epoch claim plus terminal claim/management receipts, and alone publishes the next `OPEN` row. Receipt-first replay is idempotent, and named-branch or stale-binding authority fails before the claim. V14's recovery-v16 `StreamSealedEnsureIndices` layers exact prior/next SEALED rows over recovery-v8's CreateIndex plan; under `stream_manage` and the retained checked runtime it publishes every productive index pointer, both HEAD witnesses, recomputed empty proof, and lifecycle revision in one CAS. V15's recovery-v17 `StreamSealedOptimize` separately owns Optimize's internally committing maintenance plan and exact achieved HEADs, then publishes productive table pointers with recomputed `SEALED` proofs in one CAS. Neither checked maintenance path writes a token row, advances a management-receipt chain, or accepts a caller operation ID; true no-work remains effect-free. V16's recovery-v18 `StreamRebind` consumes one exact `SEALED` proof, creates a fresh enrollment and empty shard namespace, appends immutable binding and fence-only claim receipts, and publishes a new exact `SEALED` proof without admitting a writer or put. A separate recovery-v15 resume is required to open the fresh scope. This physical rebind keeps the accepted schema unchanged. Productive SchemaApply remains refused on any enrolled graph in EXP; schema changes use a checked sealed/retired export, initialize a fresh graph with the desired schema, and load there, never over the enrolled source. V17's recovery-v19 `StreamAuthorityRetirement` owns the exact immutable retirement-receipt transaction and sole lineage-neutral `DISABLED → RETIRED` publication; it never moves graph or branch heads and its selected receipt/profile chain replaces `RecoveryAudit`. V18's recovery-v20 `StreamCorrection` owns one exact blocked generation cut, a pre-minted base effect, and one combined token-successor plus correction/management-receipt effect; recovery may recreate only the missing token participant after the exact base effect and publishes the two pointers, `DRAINING` successor, lineage, and optional PRESENT attribution together. V19's recovery-v21 `DeadLetterFold` owns one conditionally created canonical object plus exact base and token effects, accepts mixed and all-diverted outcomes, and publishes versioned fold attribution with current token-schema-v3 `DEAD_LETTERED` evidence. Recovery-v21 also carries the three-disposition retirement owner; recovery-v19 retains its historical two-disposition meaning. Ambient EnsureIndices and Optimize remain refused for enrolled tables. Historical recovery-v10 enrollment, recovery-v12 lifecycle-v2 fold, and older resume/maintenance/rebind scaffolds keep their old wire meanings and are refused rather than synthesized. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all profile is active**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and untouched through recovery/reopen; provider exhaustion is loud. The core remains crate-private behind feature-gated, doc-hidden test seams. The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary is registered, and embedded `Omnigraph::stream_status` exposes a durable manifest-only projection. V11's protocol-v2 checked cluster authority remains unchanged: `ENABLED` carries its active fold delegation, `DISABLING` carries a drain-only continuation plan, and `RETIRED` is receipt/cut-bound and fail-closed. Recovery-v13 remains exactly `StreamProfileChange`. Profile apply enforces `stream_manage` under both the currently applied and desired graph policies through the cluster-state CAS. Enabled-profile Mutation/Load/delete require the exact checked served runtime, while BranchMerge remains closed under both `ENABLED` and `DISABLING` even with that runtime. The narrow cluster-only `stream retire-for-rebuild plan|confirm` command is the sole active retirement surface and permanently freezes a verified `WITHDRAWN | DEAD_LETTERED` cut for export/rebuild. Checked offline disable is the sole supported production quiescence owner. F7a activates one graph-native mixed node/edge served-ingest bridge at `POST /graphs/{graph_id}/stream/ingest` plus remote client/CLI/OpenAPI parity. A strong graph-authority ETag gates body ownership; the engine resolves declarations, lazily prepares private lanes, and returns only ordered graph-logical results. F7b exposes F6b6's checked read-only cut at `GET /graphs/{graph_id}/stream/status` and `stream status` through a graph-logical projection that omits every table/dataset/lane/binding/shard/generation/Lance/recovery identity. Public lane enrollment, general resume/abort, rebind, and lifecycle/maintenance transports remain inactive; exact-terminal served export remains the F6b5 exception. The stopped/offline cluster-only `stream block show|correct` and `stream dead-letter list|export` surfaces are the narrow active inspection/correction exceptions; no hidden seam is a supported operator escape hatch. - **RFC-026 F7a lifecycle boundary**: graph firehose ingress can lazily prepare an absent declaration or continue an existing `OPEN` lane. It never implicitly resumes a `SEALED` lane; that state returns graph-level `stream_authority_changed` until a public checked resume exists, or the operator takes the documented retire/export/rebuild exit. - **RFC-026 retired-export provenance**: the immutable retirement receipt binds the whole frozen graph cut. Every retired JSONL export pairs that root receipt with a selected `branch_member` witness containing the canonical branch name, exact Lance branch identifier, graph head, manifest version, `table_witness_digest`, and a recomputable `branch_member_digest`. The export's `source_schema_ir_hash`, exact `ordered_branch_member_digests`, and `selected_member_index` let load prove membership and recompute the receipt's `export_cut_digest`. The source schema hash commits the retired source cut; it is not required to equal the fresh target graph identity, whose compatibility remains ordinary loader validation. Load imports no live stream authority. - **RFC-026 B2 preprocessing bound**: before blob materialization or canonical encoding, the private adapter reserves a 128-MiB worst-case envelope (original 32-MiB Arrow row + possible 32-MiB replacement + 64-MiB canonical payload) and an inflight slot. The private profile admits exactly two envelopes (256 MiB root-wide), preserving the two-caller stale-authority race without unbounded preprocessing. The slot transfers into queued/worker ownership; scratch releases after digest derivation. Pressure fails effect-free as typed `stream_b2_preprocessing_bytes`; this is process-memory admission, not retained-storage GC or quota. - **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster `, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart. - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table). -- **RFC-026 hidden F4/F5/F6 status**: F4 owns the complete caller-shaped authorized JSON/NDJSON and bodyless lazy-enrollment prepare path. Format-neutral F5a adds one weakly root-scoped automatic supervisor for `OPEN` lanes. Detached ownership schedules a coalesced timer wake immediately after physical put invocation, including caller cancellation or eventual `AckUnknown`; passive readiness discards no-effect wakes, and capacity pressure makes the same entry urgent. Cold start derives backlog from manifest plus authenticated MemWAL authority, while finite rounds visit nodes before edges with a carried round-robin cursor inside each immutable-identity cohort. F5b0 extends that exact-`ENABLED`, checked-runtime owner to restart and continue unblocked `DRAINING(goal = SEALED)` lanes through the existing recovery-v14 quiesce adapter. The same format-neutral cut makes checked offline `cluster apply` publish `DISABLING` before work, derive its finite lane set from that manifest cut, visit nodes then edges in immutable-identity order one lane at a time, continue `OPEN` and goal-`SEALED` drains, and recovery-v14-adopt an existing `OPEN_AFTER_FOLD` drain without minting a second occurrence. A selected `DataBlock` parks apply loudly at the durable `DISABLING` revision; stopped/offline correction followed by the same apply retry resumes the stored plan. F5b adds the v19/recovery-v21 terminal path: deterministic conflict partitioning can publish visible winners and one bounded dead-letter object, including a marker-only base transaction when every candidate is diverted; exact retries return the current terminal result, while a fresh ordinary successor naming that terminal predecessor restores `PRESENT`. Stopped/offline `cluster stream dead-letter list|export` inspects only manifest-selected current authority, and retirement now accepts `WITHDRAWN | DEAD_LETTERED`. The resident and offline owners retain their non-overlap boundary. After listener bind the cluster server starts every selected graph supervisor; graceful shutdown fences the root MemWAL opportunity exclusively and then the profile gate exclusively, drops both before joining the drivers concurrently under one bounded deadline and retains any timed-out live owner. F6a adds one typed failpoints-only process-local advisory driver snapshot and one hidden in-process composed acceptance spanning prepare, NDJSON, automatic mixed terminal fold, stopped/offline list/export, corrected successor, driver restart, clean shutdown ownership, and offline disable. F6b1 adds a distinct lower/engine checked served-export authority and one doc-hidden, move-only `StreamExportCut`. A managed exact `DISABLED | RETIRED` row binds directly; when the streaming row was legitimately removed, exact graph/state evidence binds only if the engine proves `RETIRED` or enrolled `DISABLED`. Retirement confirmation CAS-converges a managed row to `RETIRED` and its exact revision, while refresh preserves declaration identity and treats `RETIRED` as satisfying `streaming: false`. Ambient enrolled `DISABLED` export refuses. The receipt-verified ambient `RETIRED` direct-export bridge remains, while F6b5 routes served export through the checked cut. Checked capture non-waitingly owns the sole root slot, closes profile/admission/schema/branch/token/table gates, prevalidates filters and terminal authority, and freezes the accepted catalog, selected snapshot's exact Lance table versions, and retired provenance. It then drops all writer gates while the cut retains its checked process authority and root slot through consuming output, so a later writer cannot retarget the bytes. Branch create/create-from/delete, cleanup, schema apply, and supported whole-root deletion acquire the same slot non-waitingly, excluding path/version reuse for the cut's lifetime. The temporary `RETIRED` bridge also retains that slot through output. Refusal is before output; a storage or writer failure after output starts remains that stream error. Pending triggers are not a durable backlog, stopped driver state is not offline authority, and public durable `StreamStatus` remains manifest-only. F6b1 changed no format or recovery grammar; F6b5 subsequently activated the existing HTTP/remote-client/CLI/OpenAPI export route. F7a now activates only the graph-native served row route and remote command over the same hidden runtime, with no new coordinator, format, or recovery grammar. Public status/driver and lifecycle transports remain inactive; F6b2 now has green SIGTERM/shared shutdown, sequential OS-process recovery, frozen-round node/edge fairness, rebind/re-enable/reopen/resume, combined maintenance, fresh-target import, and legacy writer-refusal cells and is implemented. F6b3 adds exact manifest-selected token-index coverage probes plus a fixed-cardinality cost harness: zero-lane profile cycles grow immutable token-ledger receipt history while graph-manifest history also advances, then one enrolled conflicting key produces one logical terminal entry at every depth. The fast local 1/8-cycle cell and ignored local/configured-RustFS 1/8/32/128 sweeps record per-sample coverage, page bytes, and cumulative advisory whole-process RSS. Fresh-handle hit/miss plus the first terminal page and same-handle warm hit/miss plus repeat terminal pages report token-read counts, total table-store read bytes, manifest reads/bytes, adapter-operation counts, and per-sample warm/repeat p50 plus max-of-eight; graph open precedes those windows, so this is not a cold-provider-cache claim. The instrument does not query receipt keys and fails if a measured operation writes authority, touches MemWAL/base rows, lists prefixes, or fetches the dead-letter payload object. This remains the historical F6b3 uncovered-tail evidence. F6b7 adds a paired failpoints-only content-identical selected-index cut, current-token and profile-receipt lookup measurements, and maintenance-cost accounting. Its configured-RustFS result is a bounded NO-GO only for the uncompacted profile-cycle fixture, so it schedules no standalone production reconciler. F6b4 closes production dead-letter envelope/peak-RSS evidence. F6b5 adds pre-header checked cut capture, incremental exact-version scans, strict 64-KiB chunks, a two-chunk queue, complete per-response/process queue reservation, body-plus-producer lease ownership, terminal-frame cut retention, backpressure, and disconnect-safe release. F6b6 implements one engine-internal checked read-only operational cut: exact profile-specific authority; one writer-unfenced immutable token/base, bounded-sample, coverage, and lifecycle-ledger preflight; then short root/profile/lane fences for mutable physical/recovery witnesses; an explicitly advisory driver projection; typed movement/busy refusal; a complete pending-sidecar inventory inside a hard advisory envelope of 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, 32 MiB per sidecar body, and 32 MiB of cumulative bodies; and honest unavailable states for exactly recovery-owned physical movement, cold replay, flushed LWW accounting, and exact oldest-uncovered age. Exceeding any envelope bound is a typed resource refusal, never partial status. It neither heals recovery nor opens/claims/seals/folds a writer; the public status remains manifest-only. F6b8 separately closes resume-to-driver ownership without a format change: the root producer permit transfers through detached installation, an urgent trigger is armed before release, and exact empty-owner housekeeping precedes the unchanged node-before-edge round. No active path produces reserved `AuthorityBlock`, so its repair remains fail-closed. -- **RFC-026 F6b2/F6b8 implemented acceptance scope**: this no-format slice has green Unix `SIGTERM` through the shared graceful-shutdown path, sequential OS-process recovery, a frozen-round node/edge non-starvation proof, terminal-disable physical rebind → re-enable → reopen → explicit resume → exactly-once ingest/fold, combined `quiesce → EnsureIndices → Optimize → resume`, checked-cut fresh-target import, and legacy Mutation/Load/delete, `load_file`, and corresponding `_as` refusal under `ENABLED` and interrupted `DISABLING`. Resident-producing served puts acquire bounded preprocessing/inflight → root MemWAL opportunity shared → profile shared → table admission; the driver holds root opportunity exclusive across one frozen round and then takes profile/admission per candidate. Producer and round permits retain the `MemWalWorkerRegistry` `Arc`, preventing weak-root fence ABA. Shutdown fences root exclusive and then profile exclusive, drops both, and joins the driver. F6b8 closes the empty-resume handoff: resume transfers its non-clone root producer permit into detached installation and retained retirement, arms an urgent trigger before release, and the driver retires only exact empty owners under lane-exclusive authority before preserving the ordinary node-before-edge round. Driver-first, caller-cancelled resume-first, cross-lane slot reuse, and clean-shutdown cells are green; the broader post-claim install/retirement-failure matrix remains later F6 work. Productive SchemaApply remains refused on enrolled graphs; EXP schema changes use checked export/init/load into a fresh graph, while physical rebind preserves accepted schema. F6b7 now owns the paired uncovered/reconciled token decision instrument; public operational-status and lifecycle/maintenance SDK/HTTP/CLI/OpenAPI parity remain later work; F7a graph-native served row ingest and exact-terminal served export are active, and F6b6's checked operational-status core is engine-internal. +- **RFC-026 hidden F4/F5/F6 status**: F4 owns the complete caller-shaped authorized JSON/NDJSON and bodyless lazy-enrollment prepare path. Format-neutral F5a adds one weakly root-scoped automatic supervisor for `OPEN` lanes. Detached ownership schedules a coalesced timer wake immediately after physical put invocation, including caller cancellation or eventual `AckUnknown`; passive readiness discards no-effect wakes, and capacity pressure makes the same entry urgent. Cold start derives backlog from manifest plus authenticated MemWAL authority, while finite rounds visit nodes before edges with a carried round-robin cursor inside each immutable-identity cohort. F5b0 extends that exact-`ENABLED`, checked-runtime owner to restart and continue unblocked `DRAINING(goal = SEALED)` lanes through the existing recovery-v14 quiesce adapter. The same format-neutral cut makes checked offline `cluster apply` publish `DISABLING` before work, derive its finite lane set from that manifest cut, visit nodes then edges in immutable-identity order one lane at a time, continue `OPEN` and goal-`SEALED` drains, and recovery-v14-adopt an existing `OPEN_AFTER_FOLD` drain without minting a second occurrence. A selected `DataBlock` parks apply loudly at the durable `DISABLING` revision; stopped/offline correction followed by the same apply retry resumes the stored plan. F5b adds the v19/recovery-v21 terminal path: deterministic conflict partitioning can publish visible winners and one bounded dead-letter object, including a marker-only base transaction when every candidate is diverted; exact retries return the current terminal result, while a fresh ordinary successor naming that terminal predecessor restores `PRESENT`. Stopped/offline `cluster stream dead-letter list|export` inspects only manifest-selected current authority, and retirement now accepts `WITHDRAWN | DEAD_LETTERED`. The resident and offline owners retain their non-overlap boundary. After listener bind the cluster server starts every selected graph supervisor; graceful shutdown fences the root MemWAL opportunity exclusively and then the profile gate exclusively, drops both before joining the drivers concurrently under one bounded deadline and retains any timed-out live owner. F6a adds one typed failpoints-only process-local advisory driver snapshot and one hidden in-process composed acceptance spanning prepare, NDJSON, automatic mixed terminal fold, stopped/offline list/export, corrected successor, driver restart, clean shutdown ownership, and offline disable. F6b1 adds a distinct lower/engine checked served-export authority and one doc-hidden, move-only `StreamExportCut`. A managed exact `DISABLED | RETIRED` row binds directly; when the streaming row was legitimately removed, exact graph/state evidence binds only if the engine proves `RETIRED` or enrolled `DISABLED`. Retirement confirmation CAS-converges a managed row to `RETIRED` and its exact revision, while refresh preserves declaration identity and treats `RETIRED` as satisfying `streaming: false`. Ambient enrolled `DISABLED` export refuses. The receipt-verified ambient `RETIRED` direct-export bridge remains, while F6b5 routes served export through the checked cut. Checked capture non-waitingly owns the sole root slot, closes profile/admission/schema/branch/token/table gates, prevalidates filters and terminal authority, and freezes the accepted catalog, selected snapshot's exact Lance table versions, and retired provenance. It then drops all writer gates while the cut retains its checked process authority and root slot through consuming output, so a later writer cannot retarget the bytes. Branch create/create-from/delete, cleanup, schema apply, and supported whole-root deletion acquire the same slot non-waitingly, excluding path/version reuse for the cut's lifetime. The temporary `RETIRED` bridge also retains that slot through output. Refusal is before output; a storage or writer failure after output starts remains that stream error. Pending triggers are not a durable backlog, stopped driver state is not offline authority, and public durable `StreamStatus` remains manifest-only. F6b1 changed no format or recovery grammar; F6b5 subsequently activated the existing HTTP/remote-client/CLI/OpenAPI export route. F7a now activates only the graph-native served row route and remote command over the same hidden runtime, with no new coordinator, format, or recovery grammar. F7b exposes F6b6's checked cut through a graph-redacted read-only HTTP/OpenAPI route and remote CLI while the ambient manifest-only status remains unchanged; lifecycle/maintenance transports remain inactive; F6b2 now has green SIGTERM/shared shutdown, sequential OS-process recovery, frozen-round node/edge fairness, rebind/re-enable/reopen/resume, combined maintenance, fresh-target import, and legacy writer-refusal cells and is implemented. F6b3 adds exact manifest-selected token-index coverage probes plus a fixed-cardinality cost harness: zero-lane profile cycles grow immutable token-ledger receipt history while graph-manifest history also advances, then one enrolled conflicting key produces one logical terminal entry at every depth. The fast local 1/8-cycle cell and ignored local/configured-RustFS 1/8/32/128 sweeps record per-sample coverage, page bytes, and cumulative advisory whole-process RSS. Fresh-handle hit/miss plus the first terminal page and same-handle warm hit/miss plus repeat terminal pages report token-read counts, total table-store read bytes, manifest reads/bytes, adapter-operation counts, and per-sample warm/repeat p50 plus max-of-eight; graph open precedes those windows, so this is not a cold-provider-cache claim. The instrument does not query receipt keys and fails if a measured operation writes authority, touches MemWAL/base rows, lists prefixes, or fetches the dead-letter payload object. This remains the historical F6b3 uncovered-tail evidence. F6b7 adds a paired failpoints-only content-identical selected-index cut, current-token and profile-receipt lookup measurements, and maintenance-cost accounting. Its configured-RustFS result is a bounded NO-GO only for the uncompacted profile-cycle fixture, so it schedules no standalone production reconciler. F6b4 closes production dead-letter envelope/peak-RSS evidence. F6b5 adds pre-header checked cut capture, incremental exact-version scans, strict 64-KiB chunks, a two-chunk queue, complete per-response/process queue reservation, body-plus-producer lease ownership, terminal-frame cut retention, backpressure, and disconnect-safe release. F6b6 implements one engine-internal checked read-only operational cut: exact profile-specific authority; one writer-unfenced immutable token/base, bounded-sample, coverage, and lifecycle-ledger preflight; then short root/profile/lane fences for mutable physical/recovery witnesses; an explicitly advisory driver projection; typed movement/busy refusal; a complete pending-sidecar inventory inside a hard advisory envelope of 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, 32 MiB per sidecar body, and 32 MiB of cumulative bodies; and honest unavailable states for exactly recovery-owned physical movement, cold replay, flushed LWW accounting, and exact oldest-uncovered age. Exceeding any envelope bound is a typed resource refusal, never partial status. It neither heals recovery nor opens/claims/seals/folds a writer; the public status remains manifest-only. F6b8 separately closes resume-to-driver ownership without a format change: the root producer permit transfers through detached installation, an urgent trigger is armed before release, and exact empty-owner housekeeping precedes the unchanged node-before-edge round. No active path produces reserved `AuthorityBlock`, so its repair remains fail-closed. +- **RFC-026 F6b2/F6b8 implemented acceptance scope**: this no-format slice has green Unix `SIGTERM` through the shared graceful-shutdown path, sequential OS-process recovery, a frozen-round node/edge non-starvation proof, terminal-disable physical rebind → re-enable → reopen → explicit resume → exactly-once ingest/fold, combined `quiesce → EnsureIndices → Optimize → resume`, checked-cut fresh-target import, and legacy Mutation/Load/delete, `load_file`, and corresponding `_as` refusal under `ENABLED` and interrupted `DISABLING`. Resident-producing served puts acquire bounded preprocessing/inflight → root MemWAL opportunity shared → profile shared → table admission; the driver holds root opportunity exclusive across one frozen round and then takes profile/admission per candidate. Producer and round permits retain the `MemWalWorkerRegistry` `Arc`, preventing weak-root fence ABA. Shutdown fences root exclusive and then profile exclusive, drops both, and joins the driver. F6b8 closes the empty-resume handoff: resume transfers its non-clone root producer permit into detached installation and retained retirement, arms an urgent trigger before release, and the driver retires only exact empty owners under lane-exclusive authority before preserving the ordinary node-before-edge round. Driver-first, caller-cancelled resume-first, cross-lane slot reuse, and clean-shutdown cells are green; the broader post-claim install/retirement-failure matrix remains later F6 work. Productive SchemaApply remains refused on enrolled graphs; EXP schema changes use checked export/init/load into a fresh graph, while physical rebind preserves accepted schema. F6b7 now owns the paired uncovered/reconciled token decision instrument; lifecycle/maintenance and direct-SDK status parity remain later work; F7a graph-native served row ingest, F6b5 exact-terminal served export, and F7b graph-redacted HTTP/OpenAPI/remote-CLI status are active over F6b6's checked operational core. - **RFC-026 F6b3/F6b7 token-index evidence**: failpoints-only instruments compare manifest-selected current-token and profile-receipt lookups before and after a content-identical selected-index refresh. The bounded RustFS NO-GO schedules no @@ -283,7 +283,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift | | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: they publish only logical data/schema effects, leaving physical intent pending. `ensure_indices` first runs the roll-forward-only recovery barrier before base capture or planning, then materializes every declared-but-missing artifact for one table through one staged mixed CreateIndex transaction under its identity-bearing recovery-v9 authority/lineage/delta envelope; it continues to report untrainable Vector columns as pending. `Armed` is rollback-only, `EffectsConfirmed` rolls forward only while captured authority holds, and first-touch refs carry exact identity. The private v14 checked-runtime main-only seam instead uses recovery-v16 when any productive table is enrolled: all such lanes must be exactly `SEALED`, and one CAS publishes every table pointer with its recomputed proof and next lifecycle revision. V15 adds the separate recovery-v17 checked `SEALED` Optimize bridge for bounded internally committing compaction/index-maintenance effects. Neither path creates a token receipt or caller operation occurrence; ambient EnsureIndices and Optimize remain refused for enrolled tables. | | Strict insert / upsert ingestion | ✅ transaction conflict filters + uncommitted fragment staging | Internal schema v6 introduced the explicit logical mode and v9 preserves it. General strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint the same optional certificate only from its completed effect statistics. Mutation/Load remains one transaction per table, capped before arm at 8,192 rows / 32 MiB. BranchMerge's proven all-new route accepts only a complete certificate chain plus final source/target native-incarnation checks; it stages bounded fragments with `InsertBuilder`, commits them as exact-`id` filtered `Update` transactions, and performs zero target preflights, target merge joins, or committed Appends. Missing or malformed proof falls back to the general ordered diff. Raw Lance writers are outside the supported graph-writer topology, and the certificate is an internal, non-cryptographic capability. The final five-pair production gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes). | -| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v9 introduced exact identity-keyed physical binding, current-HEAD witness, lifecycle state-v2, per-shard epoch floor, stream-config v3, and the manifest-selected `_stream_tokens.lance` pointer; v10 added the graph-global `stream_profile` singleton and v11 added checked protocol-v2 profile authority. V12 activates lifecycle-v3 fixed-size binding/management/claim chain references plus authenticated WAL-tail authority. Its hidden recovery-v14 path enrolls one empty unsharded shard, recovery-covers every cold writer claim, authenticates each bounded WAL suffix and full active-generation LWW projection, folds exact base/token participants, and quiesces empty or non-empty lanes through restartable `OPEN → DRAINING → SEALED`. V13's recovery-v15 path adds revision-fenced `SEALED → OPEN` resume and guarded `DRAINING → OPEN` abort, claiming a higher epoch and publishing exact terminal claim/management receipts. V14's recovery-v16 path adds the checked-runtime, main-only, same-binding `SEALED` EnsureIndices bridge; v15's recovery-v17 path adds the distinct checked `SEALED` Optimize bridge; v16's recovery-v18 path adds crate-private physical rebind from one exact `SEALED` binding scope to a fresh empty scope that remains `SEALED`; v17's recovery-v19 path adds irreversible stopped/offline authority retirement for an exact `DISABLED`, all-`SEALED`, settled-recovery graph with at least one current `WITHDRAWN` token; v18's recovery-v20 path adds stopped/offline, exact-cut `DataBlock` correction with durable correction/management receipts and explicit `WITHDRAWN` sequencing authority; v19's recovery-v21 path adds deterministic bounded mixed/all-diverted terminal folds, current token-schema-v3 `DEAD_LETTERED`, and retirement over `WITHDRAWN | DEAD_LETTERED`. A restart after terminal claim reclaims with a higher epoch; a restart after physical seal reuses the receipt-bound flushed cut instead of invalidating it. The private B1 worker remains one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB of logical dense-slice Arrow data with watcher-plus-post-fence acknowledgement; backing-buffer capacity and physical RSS are not admission authority. Private B2 admission adds canonical payload/token digests, trusted hidden metadata, same-key compare-and-chain/idempotency classification, same-generation overlays, and authority recapture after shared admission. The private B2a profile remains unbounded retain-all: no OmniGraph byte/object/file/history quota, no canonical durable `_mem_wal` deletion, typed provider failure, and inert retained orphan residue. The 1/8/32/128 local/RustFS instrument remains advisory. It performs no generation GC or fresh reads. Profile change is cluster-apply-only under explicit offline confirmation and the persisted state lock; `stream_manage` is checked against both the currently applied and desired graph policies. Enabled-profile Mutation/Load/delete require the exact checked cluster serving runtime, while BranchMerge remains closed under both `ENABLED` and `DISABLING` even with that runtime. The exact-`ENABLED` resident owner now also continues unblocked goal-`SEALED` drains, while checked offline disable persists `DISABLING`, drains its finite manifest-derived node-before-edge lane cut serially, adopts `OPEN_AFTER_FOLD` through recovery-v14, and parks on `DataBlock` until correction plus apply retry. `RETIRED` decodes fail-closed and leaves only query/status/export on the source. F6b1's hidden checked path re-proves terminal authority and freezes exact export versions. Ordinary enrolled `DISABLED` ambient export refuses; the receipt-verified ambient `RETIRED` direct-export bridge remains, while F6b5 routes served export through the checked cut. Both paths serialize against destructive root control with the same nonwaiting slot. The cluster-only `stream retire-for-rebuild plan|confirm` handshake is active. Checked offline disable is the sole supported production quiescence owner. F6a adds a failpoints-only advisory driver snapshot and hidden in-process candidate-runtime composition without changing public durable status. F7a activates graph-native served row ingestion with remote-client/CLI/OpenAPI parity while keeping physical lane enrollment private. Its strong graph-authority precondition is derived from existing manifest/runtime evidence, and it reuses the current lazy prepare and resident driver without a new format or coordinator. General resume/abort, rebind, and lifecycle/maintenance/status transport contracts remain inactive; exact-terminal served export remains the F6b5 exception, while cluster-only `stream block show|correct` and `stream dead-letter list|export` are the active correction/inspection surfaces. F6b3 adds exact-selected uncovered-tail current-token hit/miss and terminal-page cost evidence without a format or production surface. F6b7 adds the paired failpoints-only selected-index refresh and receipt-key/current-token decision evidence without format, recovery, or production behavior. F6b5 performs pre-header authority/filter/limit validation, incrementally scans exact pinned Lance versions with approximate targets, emits independently owned 64-KiB chunks through a two-chunk queue, reserves the complete per-response queue envelope from one true process-wide budget, and retains both the queue lease and move-only cut through disconnect or a terminal frame. F6b7's uncompacted-profile-cycle bounded NO-GO schedules no standalone production token-index reconciler; public status and remaining guardrail acceptance stay open. | +| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v9 introduced exact identity-keyed physical binding, current-HEAD witness, lifecycle state-v2, per-shard epoch floor, stream-config v3, and the manifest-selected `_stream_tokens.lance` pointer; v10 added the graph-global `stream_profile` singleton and v11 added checked protocol-v2 profile authority. V12 activates lifecycle-v3 fixed-size binding/management/claim chain references plus authenticated WAL-tail authority. Its hidden recovery-v14 path enrolls one empty unsharded shard, recovery-covers every cold writer claim, authenticates each bounded WAL suffix and full active-generation LWW projection, folds exact base/token participants, and quiesces empty or non-empty lanes through restartable `OPEN → DRAINING → SEALED`. V13's recovery-v15 path adds revision-fenced `SEALED → OPEN` resume and guarded `DRAINING → OPEN` abort, claiming a higher epoch and publishing exact terminal claim/management receipts. V14's recovery-v16 path adds the checked-runtime, main-only, same-binding `SEALED` EnsureIndices bridge; v15's recovery-v17 path adds the distinct checked `SEALED` Optimize bridge; v16's recovery-v18 path adds crate-private physical rebind from one exact `SEALED` binding scope to a fresh empty scope that remains `SEALED`; v17's recovery-v19 path adds irreversible stopped/offline authority retirement for an exact `DISABLED`, all-`SEALED`, settled-recovery graph with at least one current `WITHDRAWN` token; v18's recovery-v20 path adds stopped/offline, exact-cut `DataBlock` correction with durable correction/management receipts and explicit `WITHDRAWN` sequencing authority; v19's recovery-v21 path adds deterministic bounded mixed/all-diverted terminal folds, current token-schema-v3 `DEAD_LETTERED`, and retirement over `WITHDRAWN | DEAD_LETTERED`. A restart after terminal claim reclaims with a higher epoch; a restart after physical seal reuses the receipt-bound flushed cut instead of invalidating it. The private B1 worker remains one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB of logical dense-slice Arrow data with watcher-plus-post-fence acknowledgement; backing-buffer capacity and physical RSS are not admission authority. Private B2 admission adds canonical payload/token digests, trusted hidden metadata, same-key compare-and-chain/idempotency classification, same-generation overlays, and authority recapture after shared admission. The private B2a profile remains unbounded retain-all: no OmniGraph byte/object/file/history quota, no canonical durable `_mem_wal` deletion, typed provider failure, and inert retained orphan residue. The 1/8/32/128 local/RustFS instrument remains advisory. It performs no generation GC or fresh reads. Profile change is cluster-apply-only under explicit offline confirmation and the persisted state lock; `stream_manage` is checked against both the currently applied and desired graph policies. Enabled-profile Mutation/Load/delete require the exact checked cluster serving runtime, while BranchMerge remains closed under both `ENABLED` and `DISABLING` even with that runtime. The exact-`ENABLED` resident owner now also continues unblocked goal-`SEALED` drains, while checked offline disable persists `DISABLING`, drains its finite manifest-derived node-before-edge lane cut serially, adopts `OPEN_AFTER_FOLD` through recovery-v14, and parks on `DataBlock` until correction plus apply retry. `RETIRED` decodes fail-closed and leaves only query/status/export on the source. F6b1's hidden checked path re-proves terminal authority and freezes exact export versions. Ordinary enrolled `DISABLED` ambient export refuses; the receipt-verified ambient `RETIRED` direct-export bridge remains, while F6b5 routes served export through the checked cut. Both paths serialize against destructive root control with the same nonwaiting slot. The cluster-only `stream retire-for-rebuild plan|confirm` handshake is active. Checked offline disable is the sole supported production quiescence owner. F6a adds a failpoints-only advisory driver snapshot and hidden in-process candidate-runtime composition without changing public durable status. F7a activates graph-native served row ingestion with remote-client/CLI/OpenAPI parity while keeping physical lane enrollment private. Its strong graph-authority precondition is derived from existing manifest/runtime evidence, and it reuses the current lazy prepare and resident driver without a new format or coordinator. F7b exposes F6b6's checked cut through graph-redacted HTTP/OpenAPI and remote CLI status. General resume/abort, rebind, lifecycle/maintenance transport, and direct-SDK status contracts remain inactive; exact-terminal served export remains the F6b5 exception, while cluster-only `stream block show|correct` and `stream dead-letter list|export` are the active correction/inspection surfaces. F6b3 adds exact-selected uncovered-tail current-token hit/miss and terminal-page cost evidence without a format or production surface. F6b7 adds the paired failpoints-only selected-index refresh and receipt-key/current-token decision evidence without format, recovery, or production behavior. F6b5 performs pre-header authority/filter/limit validation, incrementally scans exact pinned Lance versions with approximate targets, emits independently owned 64-KiB chunks through a two-chunk queue, reserves the complete per-response queue envelope from one true process-wide budget, and retains both the queue lease and move-only cut through disconnect or a terminal frame. F6b7's uncompacted-profile-cycle bounded NO-GO schedules no standalone production token-index reconciler; direct-SDK status and remaining guardrail acceptance stay open. | | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema | | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops | | Hybrid ranking | — | `rrf(...)` Reciprocal Rank Fusion in one runtime | diff --git a/crates/omnigraph-api-types/src/lib.rs b/crates/omnigraph-api-types/src/lib.rs index f4534b0c..79c0e505 100644 --- a/crates/omnigraph-api-types/src/lib.rs +++ b/crates/omnigraph-api-types/src/lib.rs @@ -3,7 +3,12 @@ //! and one engine-result -> DTO mapping per verb. Plain serde/utoipa //! types; no transport, no server internals. -use omnigraph::db::{GraphCommit, MergeOutcome, ReadTarget, SchemaApplyResult, Snapshot}; +use omnigraph::db::{ + GraphCommit, GraphStreamDeclaration, GraphStreamDeclarationStatus, + GraphStreamDriverErrorStatus, GraphStreamDriverStatus, GraphStreamOperationalStatus, + GraphStreamPendingStatus, GraphStreamRebuildBlocker, GraphStreamRebuildStatus, + GraphStreamTokenCounts, MergeOutcome, ReadTarget, SchemaApplyResult, Snapshot, +}; use omnigraph::error::{MergeConflict, MergeConflictKind}; use omnigraph::loader::{LoadMode, LoadResult}; use omnigraph_compiler::SchemaMigrationStep; @@ -633,6 +638,600 @@ pub struct StreamIngestLineOutput { pub blocking_status: Option, } +/// Stable graph-level streaming profile state. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(rename_all = "snake_case")] +pub enum StreamProfileModeOutput { + Disabled, + Enabled, + Disabling, + Retired, +} + +/// Logical schema declaration with streaming state. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamDeclarationOutput { + pub kind: StreamIngestKindOutput, + #[serde(rename = "type")] + pub type_name: String, +} + +/// Stable graph-level lifecycle state for one logical declaration. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(rename_all = "snake_case")] +pub enum StreamLifecycleOutput { + Open, + Draining, + Sealed, +} + +/// Active drain operation for one logical declaration. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamDrainStatusOutput { + pub goal: String, + pub phase: String, + pub initiated_at: i64, +} + +/// Current strict validation block for one logical declaration. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamStrictBlockStatusOutput { + pub kind: String, + pub violation_code: String, +} + +/// Most recent durable fold summary, with physical operation and generation +/// coordinates removed. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamLastFoldStatusOutput { + pub outcome: String, + pub input_rows: u64, + pub input_bytes: u64, + pub visible_rows: u64, + pub visible_bytes: u64, + pub recorded_at: i64, +} + +/// Pending acknowledged work for one logical declaration. +/// +/// Counts are returned only when the checked status cut can observe them +/// without claiming a writer or advancing replay state. An unavailable result +/// names only the graph-safe reason classes; it never exposes a shard, +/// generation, dataset, or recovery operation. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(tag = "state", rename_all = "snake_case")] +pub enum StreamPendingStatusOutput { + Exact { + rows: u64, + arrow_bytes: u64, + batches: u64, + }, + Unavailable { + cold_replay: bool, + flushed: bool, + recovery: bool, + }, +} + +/// Checked status for one logical node or edge declaration. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamDeclarationStatusOutput { + #[serde(flatten)] + pub declaration: StreamDeclarationOutput, + pub lifecycle: StreamLifecycleOutput, + pub lifecycle_revision: u64, + #[serde(skip_serializing_if = "Option::is_none")] + pub drain: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub strict_block: Option, + pub pending: StreamPendingStatusOutput, + #[serde(skip_serializing_if = "Option::is_none")] + pub last_fold: Option, +} + +/// Current graph-wide sequencing-authority counts. Generic status deliberately +/// omits the sampled logical IDs and per-key stream tokens used by the +/// internal proof. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamTokenCountsOutput { + pub present: u64, + pub withdrawn: u64, + pub dead_lettered: u64, +} + +/// Process-local fold-driver run state. Driver health is advisory and never +/// substitutes for durable lifecycle or recovery authority. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(rename_all = "snake_case")] +pub enum StreamDriverStateOutput { + Stopped, + Running, + Stopping, + Failed, +} + +/// Redacted most-recent driver error. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamDriverErrorOutput { + pub kind: String, + #[serde(skip_serializing_if = "Option::is_none")] + pub retry_in_ms: Option, +} + +/// Advisory health of the fold driver serving this graph. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamDriverStatusOutput { + pub scope: String, + pub authoritative: bool, + pub state: StreamDriverStateOutput, + pub pending_count: u64, + pub published_open_folds: u64, + #[serde(skip_serializing_if = "Option::is_none")] + pub last_completion_kind: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub last_error: Option, +} + +/// One graph-safe reason an export/import rebuild is not currently allowed. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(tag = "reason", rename_all = "snake_case")] +pub enum StreamRebuildBlockerOutput { + ProfileNotTerminal, + DeclarationNotSealed { + #[serde(flatten)] + declaration: StreamDeclarationOutput, + }, + StrictBlock { + #[serde(flatten)] + declaration: StreamDeclarationOutput, + }, + PendingWork { + #[serde(flatten)] + declaration: StreamDeclarationOutput, + }, + PendingWorkUnavailable { + #[serde(flatten)] + declaration: StreamDeclarationOutput, + }, + RecoveryPending { + count: u64, + }, + TerminalTokenAuthority { + withdrawn_count: u64, + dead_lettered_count: u64, + }, +} + +/// Whether the checked graph cut can be rebuilt without discarding streaming +/// sequencing authority. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamRebuildStatusOutput { + pub ready: bool, + pub blockers: Vec, +} + +/// One coherent, graph-redacted operational status cut. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +pub struct StreamStatusOutput { + pub manifest_version: u64, + pub profile_mode: StreamProfileModeOutput, + pub profile_revision: u64, + /// Logical node/edge declarations whose streaming state has initialized. + /// Absence here does not mean the graph schema has no declarations. + pub enrolled_declarations: Vec, + pub token_counts: StreamTokenCountsOutput, + pub recovery_pending_count: u64, + pub driver: StreamDriverStatusOutput, + pub rebuild: StreamRebuildStatusOutput, +} + +fn stream_profile_mode_output( + value: &str, +) -> std::result::Result { + match value { + "DISABLED" => Ok(StreamProfileModeOutput::Disabled), + "ENABLED" => Ok(StreamProfileModeOutput::Enabled), + "DISABLING" => Ok(StreamProfileModeOutput::Disabling), + "RETIRED" => Ok(StreamProfileModeOutput::Retired), + _ => Err("unknown stream profile mode"), + } +} + +fn stream_declaration_output( + value: GraphStreamDeclaration, +) -> std::result::Result { + let kind = match value.kind { + "node" => StreamIngestKindOutput::Node, + "edge" => StreamIngestKindOutput::Edge, + _ => return Err("unknown stream declaration kind"), + }; + Ok(StreamDeclarationOutput { + kind, + type_name: value.type_name, + }) +} + +fn stream_lifecycle_output( + value: &str, +) -> std::result::Result { + match value { + "OPEN" => Ok(StreamLifecycleOutput::Open), + "DRAINING" => Ok(StreamLifecycleOutput::Draining), + "SEALED" => Ok(StreamLifecycleOutput::Sealed), + _ => Err("unknown stream lifecycle"), + } +} + +fn stream_declaration_status_output( + value: GraphStreamDeclarationStatus, +) -> std::result::Result { + let declaration = stream_declaration_output(value.declaration)?; + let lifecycle = stream_lifecycle_output(value.lifecycle)?; + let drain = value.drain.map(|drain| StreamDrainStatusOutput { + goal: drain.goal.to_ascii_lowercase(), + phase: drain.phase.to_ascii_lowercase(), + initiated_at: drain.initiated_at, + }); + let strict_block = value + .strict_block + .map(|block| StreamStrictBlockStatusOutput { + kind: block.kind.to_ascii_lowercase(), + violation_code: block.violation_code, + }); + let pending = match value.pending { + GraphStreamPendingStatus::Exact { + rows, + arrow_bytes, + batches, + } => StreamPendingStatusOutput::Exact { + rows, + arrow_bytes, + batches, + }, + GraphStreamPendingStatus::Unavailable { + cold_replay, + flushed, + recovery, + } => StreamPendingStatusOutput::Unavailable { + cold_replay, + flushed, + recovery, + }, + }; + let last_fold = value.last_fold.map(|fold| StreamLastFoldStatusOutput { + outcome: fold.outcome.to_ascii_lowercase(), + input_rows: fold.input_rows, + input_bytes: fold.input_bytes, + visible_rows: fold.visible_rows, + visible_bytes: fold.visible_bytes, + recorded_at: fold.recorded_at, + }); + Ok(StreamDeclarationStatusOutput { + declaration, + lifecycle, + lifecycle_revision: value.lifecycle_revision, + drain, + strict_block, + pending, + last_fold, + }) +} + +fn stream_driver_status_output( + value: GraphStreamDriverStatus, +) -> std::result::Result { + let state = match value.state { + "STOPPED" => StreamDriverStateOutput::Stopped, + "RUNNING" => StreamDriverStateOutput::Running, + "STOPPING" => StreamDriverStateOutput::Stopping, + "FAILED" => StreamDriverStateOutput::Failed, + _ => return Err("unknown stream driver state"), + }; + let last_error = value + .last_error + .map( + |GraphStreamDriverErrorStatus { kind, retry_in_ms }| StreamDriverErrorOutput { + kind: kind.to_ascii_lowercase(), + retry_in_ms, + }, + ); + Ok(StreamDriverStatusOutput { + scope: value.scope.to_ascii_lowercase(), + authoritative: value.authoritative, + state, + pending_count: value.pending_count, + published_open_folds: value.published_open_folds, + last_completion_kind: value + .last_completion_kind + .map(|kind| kind.to_ascii_lowercase()), + last_error, + }) +} + +fn stream_rebuild_blocker_output( + value: GraphStreamRebuildBlocker, +) -> std::result::Result { + Ok(match value { + GraphStreamRebuildBlocker::ProfileNotTerminal => { + StreamRebuildBlockerOutput::ProfileNotTerminal + } + GraphStreamRebuildBlocker::DeclarationNotSealed { declaration } => { + StreamRebuildBlockerOutput::DeclarationNotSealed { + declaration: stream_declaration_output(declaration)?, + } + } + GraphStreamRebuildBlocker::StrictBlock { declaration } => { + StreamRebuildBlockerOutput::StrictBlock { + declaration: stream_declaration_output(declaration)?, + } + } + GraphStreamRebuildBlocker::PendingWork { declaration } => { + StreamRebuildBlockerOutput::PendingWork { + declaration: stream_declaration_output(declaration)?, + } + } + GraphStreamRebuildBlocker::PendingWorkUnavailable { declaration } => { + StreamRebuildBlockerOutput::PendingWorkUnavailable { + declaration: stream_declaration_output(declaration)?, + } + } + GraphStreamRebuildBlocker::RecoveryPending { count } => { + StreamRebuildBlockerOutput::RecoveryPending { count } + } + GraphStreamRebuildBlocker::TerminalTokenAuthority { + withdrawn_count, + dead_lettered_count, + } => StreamRebuildBlockerOutput::TerminalTokenAuthority { + withdrawn_count, + dead_lettered_count, + }, + }) +} + +fn stream_rebuild_status_output( + value: GraphStreamRebuildStatus, +) -> std::result::Result { + Ok(StreamRebuildStatusOutput { + ready: value.ready, + blockers: value + .blockers + .into_iter() + .map(stream_rebuild_blocker_output) + .collect::, _>>()?, + }) +} + +/// Convert the engine's graph-redacted checked cut into the stable HTTP/CLI +/// shape. This mapping is deliberately explicit: adding a physical field to +/// the engine bridge cannot make it appear on the wire by accident. +pub fn stream_status_output( + value: GraphStreamOperationalStatus, +) -> std::result::Result { + let enrolled_declarations = value + .enrolled_declarations + .into_iter() + .map(stream_declaration_status_output) + .collect::, _>>()?; + let GraphStreamTokenCounts { + present, + withdrawn, + dead_lettered, + } = value.token_counts; + Ok(StreamStatusOutput { + manifest_version: value.manifest_version, + profile_mode: stream_profile_mode_output(value.profile_mode)?, + profile_revision: value.profile_revision, + enrolled_declarations, + token_counts: StreamTokenCountsOutput { + present, + withdrawn, + dead_lettered, + }, + recovery_pending_count: value.recovery_pending_count, + driver: stream_driver_status_output(value.driver)?, + rebuild: stream_rebuild_status_output(value.rebuild)?, + }) +} + +#[cfg(test)] +mod stream_status_tests { + use omnigraph::db::{ + GraphStreamDrainStatus, GraphStreamLastFoldStatus, GraphStreamStrictBlockStatus, + }; + + use super::*; + + fn complete_graph_status(profile_mode: &'static str) -> GraphStreamOperationalStatus { + let declaration = GraphStreamDeclaration { + kind: "edge", + type_name: "Knows".to_string(), + }; + GraphStreamOperationalStatus { + manifest_version: 21, + profile_mode, + profile_revision: 8, + enrolled_declarations: vec![GraphStreamDeclarationStatus { + declaration: declaration.clone(), + lifecycle: "DRAINING", + lifecycle_revision: 13, + drain: Some(GraphStreamDrainStatus { + goal: "SEALED", + phase: "FOLDING", + initiated_at: 1_700_000_000, + }), + strict_block: Some(GraphStreamStrictBlockStatus { + kind: "DATA_BLOCK", + violation_code: "OG-CARD-MIN".to_string(), + }), + pending: GraphStreamPendingStatus::Unavailable { + cold_replay: true, + flushed: true, + recovery: true, + }, + last_fold: Some(GraphStreamLastFoldStatus { + outcome: "PUBLISHED".to_string(), + input_rows: 9, + input_bytes: 90, + visible_rows: 7, + visible_bytes: 70, + recorded_at: 1_700_000_010, + }), + }], + token_counts: GraphStreamTokenCounts { + present: 3, + withdrawn: 2, + dead_lettered: 1, + }, + recovery_pending_count: 4, + driver: GraphStreamDriverStatus { + scope: "GRAPH", + authoritative: false, + state: "FAILED", + pending_count: 5, + published_open_folds: 6, + last_completion_kind: Some("FOLD_PUBLISHED"), + last_error: Some(GraphStreamDriverErrorStatus { + kind: "RETRYABLE", + retry_in_ms: Some(250), + }), + }, + rebuild: GraphStreamRebuildStatus { + ready: false, + blockers: vec![ + GraphStreamRebuildBlocker::ProfileNotTerminal, + GraphStreamRebuildBlocker::DeclarationNotSealed { + declaration: declaration.clone(), + }, + GraphStreamRebuildBlocker::StrictBlock { + declaration: declaration.clone(), + }, + GraphStreamRebuildBlocker::PendingWork { + declaration: declaration.clone(), + }, + GraphStreamRebuildBlocker::PendingWorkUnavailable { declaration }, + GraphStreamRebuildBlocker::RecoveryPending { count: 4 }, + GraphStreamRebuildBlocker::TerminalTokenAuthority { + withdrawn_count: 2, + dead_lettered_count: 1, + }, + ], + }, + } + } + + #[test] + fn complete_graph_status_maps_without_physical_identity() { + let output = stream_status_output(complete_graph_status("DISABLING")).unwrap(); + assert_eq!(output.profile_mode, StreamProfileModeOutput::Disabling); + assert_eq!(output.enrolled_declarations.len(), 1); + let declaration = &output.enrolled_declarations[0]; + assert_eq!(declaration.declaration.kind, StreamIngestKindOutput::Edge); + assert_eq!(declaration.declaration.type_name, "Knows"); + assert_eq!(declaration.lifecycle, StreamLifecycleOutput::Draining); + assert_eq!(declaration.drain.as_ref().unwrap().phase, "folding"); + assert_eq!( + declaration.strict_block.as_ref().unwrap().violation_code, + "OG-CARD-MIN" + ); + assert!(matches!( + declaration.pending, + StreamPendingStatusOutput::Unavailable { + cold_replay: true, + flushed: true, + recovery: true, + } + )); + assert_eq!(declaration.last_fold.as_ref().unwrap().visible_rows, 7); + assert_eq!(output.token_counts.dead_lettered, 1); + assert_eq!(output.recovery_pending_count, 4); + assert_eq!(output.driver.state, StreamDriverStateOutput::Failed); + assert_eq!( + output.driver.last_error.as_ref().unwrap().retry_in_ms, + Some(250) + ); + assert_eq!(output.rebuild.blockers.len(), 7); + assert!(output.rebuild.blockers.iter().any(|blocker| matches!( + blocker, + StreamRebuildBlockerOutput::TerminalTokenAuthority { + withdrawn_count: 2, + dead_lettered_count: 1, + } + ))); + + let json = serde_json::to_string(&output).unwrap(); + assert!(json.contains("enrolled_declarations")); + for forbidden in [ + "table_key", + "table_id", + "dataset", + "binding", + "shard", + "epoch", + "generation", + "lance", + "receipt", + "operation_id", + "recovery_id", + ] { + assert!( + !json.contains(forbidden), + "wire status leaked private vocabulary: {forbidden}" + ); + } + } + + #[test] + fn every_current_profile_mode_maps_explicitly() { + for (raw, expected) in [ + ("DISABLED", StreamProfileModeOutput::Disabled), + ("ENABLED", StreamProfileModeOutput::Enabled), + ("DISABLING", StreamProfileModeOutput::Disabling), + ("RETIRED", StreamProfileModeOutput::Retired), + ] { + assert_eq!( + stream_status_output(complete_graph_status(raw)) + .unwrap() + .profile_mode, + expected + ); + } + } + + #[test] + fn unknown_driver_state_fails_closed() { + let status = GraphStreamOperationalStatus { + manifest_version: 1, + profile_mode: "ENABLED", + profile_revision: 2, + enrolled_declarations: Vec::new(), + token_counts: GraphStreamTokenCounts { + present: 0, + withdrawn: 0, + dead_lettered: 0, + }, + recovery_pending_count: 0, + driver: GraphStreamDriverStatus { + scope: "GRAPH", + authoritative: true, + state: "FUTURE_STATE", + pending_count: 0, + published_open_folds: 0, + last_completion_kind: None, + last_error: None, + }, + rebuild: GraphStreamRebuildStatus { + ready: false, + blockers: vec![GraphStreamRebuildBlocker::ProfileNotTerminal], + }, + }; + + assert_eq!( + stream_status_output(status), + Err("unknown stream driver state") + ); + } +} + #[derive(Debug, Clone, Deserialize, IntoParams)] pub struct SnapshotQuery { pub branch: Option, diff --git a/crates/omnigraph-cli/src/cli.rs b/crates/omnigraph-cli/src/cli.rs index 58bbccf4..02c17b0f 100644 --- a/crates/omnigraph-cli/src/cli.rs +++ b/crates/omnigraph-cli/src/cli.rs @@ -16,7 +16,7 @@ pub(crate) const DEFAULT_BEARER_TOKEN_ENV: &str = "OMNIGRAPH_BEARER_TOKEN"; COMMANDS BY CAPABILITY:\n \ any — run against a graph, served (--server / --profile) or embedded (--store / a \ URI): query, mutate, load, branch, snapshot, export, commit, schema show/apply.\n \ -served — require a server: stream ingest (graph scope) and graphs (registry scope).\n \ +served — require a server: stream ingest/status (graph scope) and graphs (registry scope).\n \ direct — direct storage access; reject --server (init, optimize, repair, cleanup, \ schema plan, lint).\n \ control — manage or inspect a cluster (cluster via --config; policy & queries via \ @@ -388,6 +388,12 @@ pub(crate) enum StreamCommand { #[arg(long, value_name = "TOKEN")] graph_token: Option, }, + /// Inspect the served graph's checked streaming health and readiness. + Status { + /// Emit the complete graph-logical status as JSON. + #[arg(long)] + json: bool, + }, } #[derive(Debug, Subcommand)] diff --git a/crates/omnigraph-cli/src/client.rs b/crates/omnigraph-cli/src/client.rs index d536ac48..f43477eb 100644 --- a/crates/omnigraph-cli/src/client.rs +++ b/crates/omnigraph-cli/src/client.rs @@ -33,8 +33,9 @@ use omnigraph_api_types::{ BranchMergeOutput, BranchMergeRequest, ChangeOutput, CommitListOutput, CommitOutput, ErrorOutput, ExportRequest, GraphListResponse, IngestOutput, IngestRequest, InvokeStoredQueryRequest, ReadOutput, - ReadRequest, SchemaApplyOutput, SchemaApplyRequest, SchemaOutput, SnapshotOutput, commit_output, - ingest_output, read_output, schema_apply_output, snapshot_payload, + ReadRequest, SchemaApplyOutput, SchemaApplyRequest, SchemaOutput, SnapshotOutput, + StreamStatusOutput, commit_output, ingest_output, read_output, schema_apply_output, + snapshot_payload, }; use omnigraph_compiler::catalog::Catalog; use reqwest::Method; @@ -322,6 +323,34 @@ impl GraphClient { cli_as: Option<&str>, profile: Option<&str>, store: Option<&str>, + ) -> Result { + Self::resolve_selected_served_graph("stream ingest", server, graph, cli_as, profile, store) + .await + } + + /// Resolve the served graph selected by read-only `stream status`. + /// Like ingest, status has no flat-server fallback: the checked runtime is + /// owned by one graph selected explicitly or by operator configuration. + pub(crate) async fn resolve_stream_status( + server: Option<&str>, + graph: Option<&str>, + profile: Option<&str>, + store: Option<&str>, + ) -> Result { + Self::resolve_selected_served_graph("stream status", server, graph, None, profile, store) + .await + } + + /// Shared graph-selection owner for the served-only stream family. + /// `command` is threaded only into the observable missing-graph error; + /// keeping the ingest spelling here preserves its existing contract. + async fn resolve_selected_served_graph( + command: &str, + server: Option<&str>, + graph: Option<&str>, + cli_as: Option<&str>, + profile: Option<&str>, + store: Option<&str>, ) -> Result { let scope = crate::scope::resolve_scope( &crate::operator::load_operator_config()?, @@ -337,7 +366,7 @@ impl GraphClient { )?; if scope.graph.is_none() { bail!( - "`stream ingest` requires one selected graph; pass --graph with \ + "`{command}` requires one selected graph; pass --graph with \ --server, or configure default_graph on the selected server profile" ); } @@ -1016,6 +1045,32 @@ impl GraphClient { Ok(()) } + /// `stream status` — obtain one checked, graph-logical operational cut + /// from the selected served graph. The wire DTO is deliberately redacted; + /// the CLI never receives physical lane, dataset, or recovery identities. + pub(crate) async fn stream_operational_status(&self) -> Result { + match self { + GraphClient::Remote { + http, + base_url, + token, + } => { + remote_json( + http, + Method::GET, + remote_url(base_url, &["stream", "status"], &[])?, + None, + token.as_deref(), + ) + .await + } + GraphClient::Embedded { .. } => bail!( + "internal error: `stream status` reached an embedded client — stream status \ + addressing always resolves a server" + ), + } + } + /// `export` — stream the branch as JSONL into `writer`. The streaming /// shape (a `W: Write`, not a returned DTO) is why this lands in 3c /// rather than 3b. Opens WITHOUT policy (like reads), so it is reached @@ -1395,7 +1450,74 @@ mod tests { } #[tokio::test] - async fn stream_ingest_requires_and_resolves_a_selected_graph_without_network_io() { + async fn stream_status_gets_the_selected_graph_and_decodes_the_redacted_cut() { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base_url = format!("http://{}/graphs/knowledge", listener.local_addr().unwrap()); + let server = tokio::spawn(async move { + let (mut stream, _) = listener.accept().await.unwrap(); + let request = read_http_request(&mut stream).await; + assert!( + request + .head + .starts_with("GET /graphs/knowledge/stream/status HTTP/1.1") + ); + assert!( + request + .head + .to_ascii_lowercase() + .contains("authorization: bearer status-token") + ); + assert!(request.body.is_empty()); + write_response( + &mut stream, + "200 OK", + &[("Content-Type", "application/json")], + br#"{ + "manifest_version":7, + "profile_mode":"enabled", + "profile_revision":3, + "enrolled_declarations":[{ + "kind":"node", + "type":"Person", + "lifecycle":"open", + "lifecycle_revision":2, + "pending":{"state":"exact","rows":1,"arrow_bytes":64,"batches":1} + }], + "token_counts":{"present":1,"withdrawn":0,"dead_lettered":0}, + "recovery_pending_count":0, + "driver":{ + "scope":"checked_runtime", + "authoritative":false, + "state":"running", + "pending_count":1, + "published_open_folds":0 + }, + "rebuild":{"ready":false,"blockers":[{"reason":"profile_not_terminal"}]} + }"#, + ) + .await; + }); + + let client = GraphClient::Remote { + http: reqwest::Client::new(), + base_url, + token: Some("status-token".to_string()), + }; + let output = client.stream_operational_status().await.unwrap(); + assert_eq!(output.manifest_version, 7); + assert_eq!(output.profile_revision, 3); + assert_eq!(output.enrolled_declarations.len(), 1); + assert_eq!( + output.enrolled_declarations[0].declaration.type_name, + "Person" + ); + assert_eq!(output.token_counts.present, 1); + assert!(!output.rebuild.ready); + server.await.unwrap(); + } + + #[tokio::test] + async fn stream_commands_require_and_resolve_a_selected_graph_without_network_io() { let error = match GraphClient::resolve_stream_ingest( Some("http://127.0.0.1:9"), None, @@ -1420,5 +1542,27 @@ mod tests { .await .unwrap(); assert_eq!(client.uri(), "http://127.0.0.1:9/graphs/knowledge"); + + let error = + match GraphClient::resolve_stream_status(Some("http://127.0.0.1:9"), None, None, None) + .await + { + Ok(_) => panic!("stream status accepted a server without a selected graph"), + Err(error) => error.to_string(), + }; + assert!( + error.contains("`stream status` requires one selected graph"), + "{error}" + ); + + let client = GraphClient::resolve_stream_status( + Some("http://127.0.0.1:9"), + Some("knowledge"), + None, + None, + ) + .await + .unwrap(); + assert_eq!(client.uri(), "http://127.0.0.1:9/graphs/knowledge"); } } diff --git a/crates/omnigraph-cli/src/main.rs b/crates/omnigraph-cli/src/main.rs index dbaea807..4160767a 100644 --- a/crates/omnigraph-cli/src/main.rs +++ b/crates/omnigraph-cli/src/main.rs @@ -9,7 +9,9 @@ use omnigraph::error::OmniError; use omnigraph::loader::LoadMode; use omnigraph_api_types::{ ChangeOutput, CommitOutput, ErrorOutput, IngestOutput, ReadOutput, SchemaApplyOutput, - SnapshotTableOutput, + SnapshotTableOutput, StreamDriverStateOutput, StreamIngestKindOutput, StreamLifecycleOutput, + StreamPendingStatusOutput, StreamProfileModeOutput, StreamRebuildBlockerOutput, + StreamStatusOutput, }; use omnigraph_cluster::{ ApplyOptions, ApplyOutput, ApproveOutput, DiagnosticSeverity, ForceUnlockOutput, PlanOutput, @@ -746,6 +748,17 @@ async fn main() -> Result<()> { .stream_ingest(&data, graph_token.as_deref(), &mut stdout) .await?; } + StreamCommand::Status { json } => { + let client = client::GraphClient::resolve_stream_status( + cli.server.as_deref(), + cli.graph.as_deref(), + cli.profile.as_deref(), + cli.store.as_deref(), + ) + .await?; + let output = client.stream_operational_status().await?; + finish_stream_status(&output, json)?; + } }, Command::Query { name, diff --git a/crates/omnigraph-cli/src/output.rs b/crates/omnigraph-cli/src/output.rs index 22c29815..8d17130f 100644 --- a/crates/omnigraph-cli/src/output.rs +++ b/crates/omnigraph-cli/src/output.rs @@ -2,6 +2,7 @@ //! main.rs in the modularization). use super::*; +use std::fmt::Write as _; #[derive(Debug, Serialize)] pub(crate) struct LoadOutput { @@ -166,6 +167,219 @@ pub(crate) fn print_json(value: &T) -> Result<()> { Ok(()) } +fn stream_profile_mode_label(mode: StreamProfileModeOutput) -> &'static str { + match mode { + StreamProfileModeOutput::Disabled => "disabled", + StreamProfileModeOutput::Enabled => "enabled", + StreamProfileModeOutput::Disabling => "disabling", + StreamProfileModeOutput::Retired => "retired", + } +} + +fn stream_lifecycle_label(lifecycle: StreamLifecycleOutput) -> &'static str { + match lifecycle { + StreamLifecycleOutput::Open => "open", + StreamLifecycleOutput::Draining => "draining", + StreamLifecycleOutput::Sealed => "sealed", + } +} + +fn stream_driver_state_label(state: StreamDriverStateOutput) -> &'static str { + match state { + StreamDriverStateOutput::Stopped => "stopped", + StreamDriverStateOutput::Running => "running", + StreamDriverStateOutput::Stopping => "stopping", + StreamDriverStateOutput::Failed => "failed", + } +} + +fn stream_declaration_label(kind: StreamIngestKindOutput, type_name: &str) -> String { + let kind = match kind { + StreamIngestKindOutput::Node => "node", + StreamIngestKindOutput::Edge => "edge", + }; + format!("{kind} {type_name}") +} + +fn render_stream_rebuild_blocker(rendered: &mut String, blocker: &StreamRebuildBlockerOutput) { + let declaration = + |kind: StreamIngestKindOutput, type_name: &str| stream_declaration_label(kind, type_name); + let result = match blocker { + StreamRebuildBlockerOutput::ProfileNotTerminal => { + writeln!(rendered, " - streaming profile is not terminal") + } + StreamRebuildBlockerOutput::DeclarationNotSealed { declaration: item } => writeln!( + rendered, + " - {} is not sealed", + declaration(item.kind, &item.type_name) + ), + StreamRebuildBlockerOutput::StrictBlock { declaration: item } => writeln!( + rendered, + " - {} has a strict validation block", + declaration(item.kind, &item.type_name) + ), + StreamRebuildBlockerOutput::PendingWork { declaration: item } => writeln!( + rendered, + " - {} has pending work", + declaration(item.kind, &item.type_name) + ), + StreamRebuildBlockerOutput::PendingWorkUnavailable { declaration: item } => writeln!( + rendered, + " - {} pending work is unavailable", + declaration(item.kind, &item.type_name) + ), + StreamRebuildBlockerOutput::RecoveryPending { count } => { + writeln!(rendered, " - {count} recovery operation(s) pending") + } + StreamRebuildBlockerOutput::TerminalTokenAuthority { + withdrawn_count, + dead_lettered_count, + } => writeln!( + rendered, + " - terminal sequencing authority: {withdrawn_count} withdrawn, \ + {dead_lettered_count} dead-lettered" + ), + }; + result.expect("writing stream status to a String cannot fail"); +} + +pub(crate) fn render_stream_status_human(output: &StreamStatusOutput) -> String { + let mut rendered = String::new(); + writeln!( + rendered, + "stream profile: {} (revision {}, manifest {})", + stream_profile_mode_label(output.profile_mode), + output.profile_revision, + output.manifest_version + ) + .expect("writing stream status to a String cannot fail"); + if output.enrolled_declarations.is_empty() { + writeln!( + rendered, + " enrolled declarations: none (streaming state initializes lazily on first ingest)" + ) + .expect("writing stream status to a String cannot fail"); + } else { + writeln!(rendered, " enrolled declarations:") + .expect("writing stream status to a String cannot fail"); + for status in &output.enrolled_declarations { + let declaration = + stream_declaration_label(status.declaration.kind, &status.declaration.type_name); + let pending = match &status.pending { + StreamPendingStatusOutput::Exact { + rows, + arrow_bytes, + batches, + } => format!("{rows} row(s), {arrow_bytes} Arrow byte(s), {batches} batch(es)"), + StreamPendingStatusOutput::Unavailable { + cold_replay, + flushed, + recovery, + } => { + let mut reasons = Vec::new(); + if *cold_replay { + reasons.push("cold replay"); + } + if *flushed { + reasons.push("flushed state"); + } + if *recovery { + reasons.push("recovery"); + } + if reasons.is_empty() { + "unavailable".to_string() + } else { + format!("unavailable ({})", reasons.join(", ")) + } + } + }; + writeln!( + rendered, + " {declaration}: {} (revision {}), pending {pending}", + stream_lifecycle_label(status.lifecycle), + status.lifecycle_revision + ) + .expect("writing stream status to a String cannot fail"); + if let Some(block) = status.strict_block.as_ref() { + writeln!( + rendered, + " strict block: {} / {}", + block.kind, block.violation_code + ) + .expect("writing stream status to a String cannot fail"); + } + if let Some(drain) = status.drain.as_ref() { + writeln!(rendered, " drain: {} / {}", drain.goal, drain.phase) + .expect("writing stream status to a String cannot fail"); + } + } + } + writeln!( + rendered, + " sequencing authority: {} present, {} withdrawn, {} dead-lettered", + output.token_counts.present, + output.token_counts.withdrawn, + output.token_counts.dead_lettered + ) + .expect("writing stream status to a String cannot fail"); + writeln!( + rendered, + " recovery: {} pending", + output.recovery_pending_count + ) + .expect("writing stream status to a String cannot fail"); + writeln!( + rendered, + " driver: {} ({}; {} pending, {} open fold(s) published)", + stream_driver_state_label(output.driver.state), + if output.driver.authoritative { + "authoritative" + } else { + "advisory" + }, + output.driver.pending_count, + output.driver.published_open_folds + ) + .expect("writing stream status to a String cannot fail"); + if let Some(error) = output.driver.last_error.as_ref() { + let result = match error.retry_in_ms { + Some(retry_in_ms) => { + writeln!( + rendered, + " last error: {} (retry in {retry_in_ms} ms)", + error.kind + ) + } + None => writeln!(rendered, " last error: {}", error.kind), + }; + result.expect("writing stream status to a String cannot fail"); + } + if output.rebuild.ready { + writeln!(rendered, " rebuild: ready") + .expect("writing stream status to a String cannot fail"); + } else { + writeln!(rendered, " rebuild: blocked") + .expect("writing stream status to a String cannot fail"); + for blocker in &output.rebuild.blockers { + render_stream_rebuild_blocker(&mut rendered, blocker); + } + } + rendered +} + +pub(crate) fn print_stream_status_human(output: &StreamStatusOutput) { + print!("{}", render_stream_status_human(output)); +} + +pub(crate) fn finish_stream_status(output: &StreamStatusOutput, json: bool) -> Result<()> { + if json { + print_json(output) + } else { + print_stream_status_human(output); + Ok(()) + } +} + pub(crate) fn print_cluster_validate_human(output: &ValidateOutput) { if output.ok { println!( @@ -1230,11 +1444,16 @@ pub(crate) fn resolve_table_render_options() -> ReadRenderOptions { #[cfg(test)] mod tests { + use omnigraph_api_types::{ + StreamDriverErrorOutput, StreamDriverStateOutput, StreamDriverStatusOutput, + StreamProfileModeOutput, StreamRebuildBlockerOutput, StreamRebuildStatusOutput, + StreamStatusOutput, StreamTokenCountsOutput, + }; use omnigraph_compiler::schema::ast::Annotation; use omnigraph_compiler::schema::parser::parse_schema; use std::collections::BTreeMap; - use super::render_annotations; + use super::{render_annotations, render_stream_status_human}; #[test] fn render_annotations_quotes_values_so_embed_round_trips() { @@ -1266,4 +1485,63 @@ mod tests { parsed.err() ); } + + #[test] + fn stream_status_human_output_stays_graph_logical() { + let output = StreamStatusOutput { + manifest_version: 42, + profile_mode: StreamProfileModeOutput::Retired, + profile_revision: 7, + enrolled_declarations: Vec::new(), + token_counts: StreamTokenCountsOutput { + present: 3, + withdrawn: 1, + dead_lettered: 2, + }, + recovery_pending_count: 4, + driver: StreamDriverStatusOutput { + scope: "graph".to_string(), + authoritative: false, + state: StreamDriverStateOutput::Failed, + pending_count: 5, + published_open_folds: 6, + last_completion_kind: Some("folded".to_string()), + last_error: Some(StreamDriverErrorOutput { + kind: "retryable".to_string(), + retry_in_ms: Some(250), + }), + }, + rebuild: StreamRebuildStatusOutput { + ready: false, + blockers: vec![StreamRebuildBlockerOutput::TerminalTokenAuthority { + withdrawn_count: 1, + dead_lettered_count: 2, + }], + }, + }; + + let rendered = render_stream_status_human(&output); + assert!(rendered.contains("stream profile: retired (revision 7, manifest 42)")); + assert!(rendered.contains( + "enrolled declarations: none (streaming state initializes lazily on first ingest)" + )); + assert!( + rendered.contains("driver: failed (advisory; 5 pending, 6 open fold(s) published)") + ); + assert!(rendered.contains("terminal sequencing authority: 1 withdrawn, 2 dead-lettered")); + for forbidden in [ + "dataset", + "table_key", + "lane", + "binding", + "shard", + "generation", + "recovery_id", + ] { + assert!( + !rendered.contains(forbidden), + "human output leaked private vocabulary: {forbidden}" + ); + } + } } diff --git a/crates/omnigraph-cli/src/planes.rs b/crates/omnigraph-cli/src/planes.rs index d4456385..e570d85b 100644 --- a/crates/omnigraph-cli/src/planes.rs +++ b/crates/omnigraph-cli/src/planes.rs @@ -149,11 +149,11 @@ fn flag_applies(flag: ScopeFlag, capability: Capability, cmd: &Command) -> bool let graph_ok = accepts_graph_selector(cmd); match flag { // Served addressing always needs a server. `graphs list` uses the bare - // registry scope; `stream ingest` selects one graph on that server. + // registry scope; the `stream` family selects one graph on that server. ScopeFlag::Server => matches!(capability, Any | Served), ScopeFlag::Cluster => cluster_ok, // The one graph selector across scopes: a served graph (`any` or the - // served-only `stream ingest`), or a cluster graph on verbs that take + // served-only `stream` family), or a cluster graph on verbs that take // cluster addressing. The other served-only command, `graphs list`, // rejects it because that command IS the registry enumeration. ScopeFlag::Graph => match capability { @@ -209,8 +209,8 @@ fn flag_applies(flag: ScopeFlag, capability: Capability, cmd: &Command) -> bool } /// The capability a subcommand needs, derived from its `Plane` (the exhaustive -/// classifier) plus the Data→Served refinements: graph-scoped `stream ingest` -/// and registry-scoped `graphs` are both remote-only. +/// classifier) plus the Data→Served refinements: graph-scoped `stream` +/// commands and registry-scoped `graphs` are remote-only. /// /// This reflects *current enforced behavior*, so messages stay truthful: /// `queries`/`policy` read a cluster's applied state (`Control`). @@ -345,6 +345,7 @@ pub(crate) fn command_label(cmd: &Command) -> &'static str { }, Command::Stream { command } => match command { StreamCommand::Ingest { .. } => "stream ingest", + StreamCommand::Status { .. } => "stream status", }, } } @@ -479,8 +480,8 @@ mod tests { // capability, both cluster_ok refinements of `direct` (optimize vs // init) and of `control` (queries vs cluster), plus the graph-only // stream-control selectors. Served commands split by scope: `graphs` - // is registry-only, while `stream ingest` consumes --graph. Neither - // accepts --store/--as. + // is registry-only, while the `stream` family consumes --graph. + // Neither accepts --store/--as. let parse = |args: &[&str]| Cli::try_parse_from(args).unwrap().command; // (command, [server, cluster, graph, store, as, profile]) let rows = [ @@ -496,6 +497,10 @@ mod tests { parse(&["omnigraph", "stream", "ingest"]), [true, false, true, false, false, true], ), + ( + parse(&["omnigraph", "stream", "status"]), + [true, false, true, false, false, true], + ), ( parse(&["omnigraph", "optimize", "g.omni"]), [false, true, true, true, false, true], @@ -570,9 +575,10 @@ mod tests { fn command_capability_classifies_representative_verbs() { let cap = |args: &[&str]| command_capability(&Cli::try_parse_from(args).unwrap().command); // Both Data→Served refinements are explicit: one registry-scoped and - // one graph-scoped. + // one graph-scoped family. assert_eq!(cap(&["omnigraph", "graphs", "list"]), Capability::Served); assert_eq!(cap(&["omnigraph", "stream", "ingest"]), Capability::Served); + assert_eq!(cap(&["omnigraph", "stream", "status"]), Capability::Served); assert_eq!(cap(&["omnigraph", "alias", "who"]), Capability::Local); assert_eq!( cap(&["omnigraph", "optimize", "graph.omni"]), @@ -644,7 +650,7 @@ mod tests { } #[test] - fn stream_ingest_is_served_only_before_input_dispatch() { + fn stream_commands_are_served_only_before_dispatch() { let cli = Cli::try_parse_from([ "omnigraph", "stream", @@ -682,6 +688,37 @@ mod tests { } other => panic!("expected stream ingest, got {other:?}"), } + + let cli = Cli::try_parse_from([ + "omnigraph", + "stream", + "status", + "--store", + "file:///must-not-open.omni", + ]) + .unwrap(); + let error = guard_addressing(&cli).unwrap_err().to_string(); + assert!(error.contains("`stream status` is a served command")); + assert!(error.contains("--store addresses")); + + let cli = Cli::try_parse_from([ + "omnigraph", + "stream", + "status", + "--server", + "http://server.invalid:9", + "--graph", + "knowledge", + "--json", + ]) + .unwrap(); + guard_addressing(&cli).unwrap(); + match cli.command { + Command::Stream { + command: StreamCommand::Status { json }, + } => assert!(json), + other => panic!("expected stream status, got {other:?}"), + } } #[test] diff --git a/crates/omnigraph-server/src/handlers.rs b/crates/omnigraph-server/src/handlers.rs index 291f290e..9dd65bb7 100644 --- a/crates/omnigraph-server/src/handlers.rs +++ b/crates/omnigraph-server/src/handlers.rs @@ -718,6 +718,63 @@ fn graph_stream_etag(authority_token: &str) -> std::result::Result, + Extension(handle): Extension>, + actor: Option>, +) -> std::result::Result<([(HeaderName, HeaderValue); 1], Json), ApiError> { + authorize_request( + actor.as_ref().map(|Extension(actor)| actor), + handle.policy.as_deref(), + PolicyRequest { + action: PolicyAction::Read, + branch: None, + target_branch: None, + }, + )?; + let _process_observation = Arc::clone(&state.stream_status_gate) + .try_acquire_owned() + .map_err(|_| { + ApiError::service_unavailable( + "another graph stream status observation is already in progress; retry", + ) + })?; + let status = handle + .engine + .capture_served_graph_stream_status() + .await + .map_err(ApiError::from_graph_stream_status)?; + let output = stream_status_output(status) + .map_err(|_| ApiError::internal("graph stream status projection failed"))?; + Ok(( + [(CACHE_CONTROL, HeaderValue::from_static("no-store"))], + Json(output), + )) +} + fn require_graph_stream_content_type( headers: &axum::http::HeaderMap, ) -> std::result::Result<(), ApiError> { diff --git a/crates/omnigraph-server/src/lib.rs b/crates/omnigraph-server/src/lib.rs index 9af9e103..f7a511db 100644 --- a/crates/omnigraph-server/src/lib.rs +++ b/crates/omnigraph-server/src/lib.rs @@ -22,7 +22,7 @@ use crate::queries::{QueryRegistry, check, format_check_breakages}; use std::collections::{BTreeMap, HashMap, HashSet}; use std::fs; use std::path::PathBuf; -use std::sync::Arc; +use std::sync::{Arc, OnceLock}; use api::{ BranchCreateOutput, BranchCreateRequest, BranchDeleteOutput, BranchListOutput, @@ -30,8 +30,8 @@ use api::{ CommitListQuery, ErrorCode, ErrorOutput, ExportRequest, GraphInfo, GraphListResponse, HealthOutput, IngestOutput, IngestRequest, InvokeStoredQueryRequest, InvokeStoredQueryResponse, QueriesCatalogOutput, QueryRequest, ReadOutput, ReadRequest, SchemaApplyOutput, - SchemaApplyRequest, SchemaOutput, SnapshotQuery, StreamIngestChallenge, ingest_output, - schema_apply_output, snapshot_payload, + SchemaApplyRequest, SchemaOutput, SnapshotQuery, StreamIngestChallenge, StreamStatusOutput, + ingest_output, schema_apply_output, snapshot_payload, stream_status_output, }; pub use auth::{AWS_SECRET_ENV, EnvOrFileTokenSource, TokenSource, resolve_token_source}; use axum::body::{Body, Bytes}; @@ -63,6 +63,7 @@ use sha2::{Digest, Sha256}; use std::io::{self, Write}; use subtle::ConstantTimeEq; use tokio::net::TcpListener; +use tokio::sync::Semaphore; use tower_http::trace::TraceLayer; use tracing::{error, info, warn}; use tracing_subscriber::EnvFilter; @@ -73,6 +74,13 @@ use utoipa::openapi::security::{Http, HttpAuthScheme, SecurityScheme}; type BearerTokenHash = [u8; 32]; +const STREAM_STATUS_PROCESS_MAX_INFLIGHT: usize = 1; + +fn process_stream_status_gate() -> Arc { + static GATE: OnceLock> = OnceLock::new(); + Arc::clone(GATE.get_or_init(|| Arc::new(Semaphore::new(STREAM_STATUS_PROCESS_MAX_INFLIGHT)))) +} + /// Machine-readable stdout record emitted after the HTTP listener owns its /// requested address. In particular, this exposes the OS-selected port for a /// `--bind 127.0.0.1:0` process without a reserve-and-rebind race. @@ -100,6 +108,7 @@ fn hash_bearer_token(token: &str) -> BearerTokenHash { #[allow(deprecated)] handlers::server_read, handlers::server_query, handlers::server_export, + handlers::server_stream_status, handlers::server_stream_ingest, #[allow(deprecated)] handlers::server_change, handlers::server_mutate, @@ -287,6 +296,11 @@ pub struct AppState { /// Bounded process-wide ownership for queued served-export bytes. The /// response body and detached producer jointly retain each reservation. export_transport: export_transport::ExportTransport, + /// One process-wide checked-status observation. Each accepted observation + /// may retain its complete bounded recovery inventory while scanning exact + /// token/base authority, so per-graph slots alone would not bound a + /// multi-graph serving process. + stream_status_gate: Arc, } #[derive(Clone)] @@ -631,6 +645,7 @@ impl AppState { server_policy: None, stream_fold_drivers: StreamFoldDrivers::default(), export_transport: export_transport::ExportTransport::with_defaults(), + stream_status_gate: process_stream_status_gate(), } } @@ -659,6 +674,7 @@ impl AppState { server_policy: server_policy.map(Arc::new), stream_fold_drivers: StreamFoldDrivers::default(), export_transport: export_transport::ExportTransport::with_defaults(), + stream_status_gate: process_stream_status_gate(), }) } @@ -837,6 +853,20 @@ impl ApiError { } } + fn service_unavailable(message: impl Into) -> Self { + Self { + status: StatusCode::SERVICE_UNAVAILABLE, + code: None, + message: message.into(), + merge_conflicts: Vec::new(), + manifest_conflict: None, + read_set_conflict: None, + key_conflict: None, + resource_limit: None, + recovery_required: None, + } + } + /// HTTP 429 Too Many Requests — actor exceeded their per-actor /// admission cap (count or byte budget). Clients should respect the /// `Retry-After` header. Mapped from `RejectReason::InFlightCountExceeded` @@ -1117,6 +1147,38 @@ impl ApiError { translated.recovery_required = None; translated } + + /// Translate checked operational-status failures without exposing the + /// physical member, authority binding, storage path, or recovery identity + /// that produced them. The status operation returns no partial cut. + fn from_graph_stream_status(err: OmniError) -> Self { + match err { + OmniError::StreamStatusBusy { .. } | OmniError::StreamStatusChanged { .. } => { + Self::service_unavailable( + "graph stream status could not obtain a stable cut; retry", + ) + } + OmniError::StreamingRequiresClusterRuntime { .. } + | OmniError::StreamingAuthorityMismatch { .. } => Self::conflict( + "graph stream status is unavailable for the current serving authority", + ), + OmniError::ResourceLimitExceeded { + resource: _, + limit, + actual, + } => Self::resource_limit( + format!( + "graph stream status observation limit exceeded: actual {actual}, limit {limit}" + ), + api::ResourceLimitOutput { + resource: "graph_stream_status_observation".to_string(), + limit, + actual, + }, + ), + _ => Self::internal("graph stream status failed"), + } + } } fn summarize_merge_conflicts(conflicts: &[api::MergeConflictOutput]) -> String { @@ -1181,6 +1243,23 @@ impl IntoResponse for ApiError { mod api_error_tests { use super::*; + #[test] + fn checked_stream_status_has_one_process_wide_observation_slot() { + let gate = process_stream_status_gate(); + let held = Arc::clone(&gate) + .try_acquire_owned() + .expect("the first checked status observation must acquire"); + assert!( + Arc::clone(&gate).try_acquire_owned().is_err(), + "another graph must not start a second bounded preflight in this process" + ); + drop(held); + let reacquired = gate + .try_acquire_owned() + .expect("dropping status ownership must release the process slot"); + drop(reacquired); + } + #[tokio::test] async fn recovery_required_503_omits_closed_error_code() { let response = ApiError::from_omni(OmniError::RecoveryRequired { @@ -1356,6 +1435,71 @@ mod api_error_tests { assert!(!error.error.contains("private checked-runtime binding")); } + #[tokio::test] + async fn graph_stream_status_errors_redact_physical_cut_evidence() { + let cases = [ + ( + OmniError::StreamStatusBusy { + phase: "private immutable scan".to_string(), + }, + StatusCode::SERVICE_UNAVAILABLE, + "private immutable scan", + ), + ( + OmniError::StreamStatusChanged { + member: "private shard generation".to_string(), + }, + StatusCode::SERVICE_UNAVAILABLE, + "private shard generation", + ), + ( + OmniError::StreamingAuthorityMismatch { + reason: "private binding receipt".to_string(), + }, + StatusCode::CONFLICT, + "private binding receipt", + ), + ( + OmniError::Lance("s3://private-bucket/private-table".to_string()), + StatusCode::INTERNAL_SERVER_ERROR, + "private-bucket", + ), + ]; + + for (engine_error, expected_status, private_evidence) in cases { + let response = ApiError::from_graph_stream_status(engine_error).into_response(); + assert_eq!(response.status(), expected_status); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let error: ErrorOutput = serde_json::from_slice(&body).unwrap(); + assert!(!error.error.contains(private_evidence), "{}", error.error); + assert!(error.merge_conflicts.is_empty()); + assert!(error.manifest_conflict.is_none()); + assert!(error.read_set_conflict.is_none()); + assert!(error.key_conflict.is_none()); + assert!(error.resource_limit.is_none()); + assert!(error.recovery_required.is_none()); + } + + let response = ApiError::from_graph_stream_status(OmniError::ResourceLimitExceeded { + resource: "private table-key inventory".to_string(), + limit: 32, + actual: 33, + }) + .into_response(); + assert_eq!(response.status(), StatusCode::PAYLOAD_TOO_LARGE); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let error: ErrorOutput = serde_json::from_slice(&body).unwrap(); + assert!(!error.error.contains("private table-key inventory")); + let details = error.resource_limit.expect("public status resource limit"); + assert_eq!(details.resource, "graph_stream_status_observation"); + assert_eq!(details.limit, 32); + assert_eq!(details.actual, 33); + } + #[tokio::test] async fn stream_management_conflicts_serialize_as_409() { let cases = [ @@ -1467,6 +1611,7 @@ pub fn build_app(state: AppState) -> Router { let per_graph_protected = Router::new() .route("/snapshot", get(server_snapshot)) .route("/export", post(server_export)) + .route("/stream/status", get(server_stream_status)) .route("/stream/ingest", post(server_stream_ingest)) // /read and /change are kept indefinitely for back-compat; // their handlers carry #[deprecated] so the OpenAPI operation is diff --git a/crates/omnigraph-server/tests/auth_policy.rs b/crates/omnigraph-server/tests/auth_policy.rs index 330ffce8..5cb8f6e9 100644 --- a/crates/omnigraph-server/tests/auth_policy.rs +++ b/crates/omnigraph-server/tests/auth_policy.rs @@ -69,6 +69,19 @@ async fn protected_routes_require_bearer_token() { Some(omnigraph_server::api::ErrorCode::Unauthorized) ); + let status_response = app + .clone() + .oneshot( + Request::builder() + .uri(g("/stream/status")) + .method(Method::GET) + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(status_response.status(), StatusCode::UNAUTHORIZED); + let stream_body_polled = Arc::new(AtomicBool::new(false)); let body_probe = Arc::clone(&stream_body_polled); let response = app diff --git a/crates/omnigraph-server/tests/multi_graph.rs b/crates/omnigraph-server/tests/multi_graph.rs index dfe16c77..0bc504e7 100644 --- a/crates/omnigraph-server/tests/multi_graph.rs +++ b/crates/omnigraph-server/tests/multi_graph.rs @@ -13,8 +13,9 @@ use futures::StreamExt; use omnigraph::db::Omnigraph; use omnigraph::loader::{LoadMode, load_jsonl}; use omnigraph_server::api::{ - ChangeRequest, ErrorOutput, ExportRequest, QueryRequest, ReadRequest, StreamIngestChallenge, - StreamIngestKindOutput, StreamIngestLineOutput, StreamIngestStatusOutput, + ChangeRequest, ErrorOutput, ExportRequest, QueryRequest, ReadRequest, StreamDriverStateOutput, + StreamIngestChallenge, StreamIngestKindOutput, StreamIngestLineOutput, + StreamIngestStatusOutput, StreamProfileModeOutput, StreamStatusOutput, }; use omnigraph_server::{AppState, build_app}; use serde_json::Value; @@ -590,6 +591,37 @@ graphs: async fn cluster_boot_installs_enabled_stream_runtime_authority() { let (_temp, state, engine) = enabled_stream_state(Vec::new()).await; let app = build_app(state); + let status_response = app + .clone() + .oneshot( + Request::builder() + .method(Method::GET) + .uri("/graphs/knowledge/stream/status") + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(status_response.status(), StatusCode::OK); + assert_eq!( + status_response.headers().get(CACHE_CONTROL), + Some(&HeaderValue::from_static("no-store")) + ); + let status: StreamStatusOutput = serde_json::from_slice( + &to_bytes(status_response.into_body(), usize::MAX) + .await + .unwrap(), + ) + .unwrap(); + assert_eq!(status.profile_mode, StreamProfileModeOutput::Enabled); + assert!(status.enrolled_declarations.is_empty()); + assert_eq!(status.token_counts.present, 0); + assert_eq!(status.token_counts.withdrawn, 0); + assert_eq!(status.token_counts.dead_lettered, 0); + assert_eq!(status.recovery_pending_count, 0); + assert_eq!(status.driver.state, StreamDriverStateOutput::Running); + assert!(!status.rebuild.ready); + let (status, body) = json_response( &app, Request::builder() @@ -903,6 +935,57 @@ query streamed_edge() { ); tokio::task::yield_now().await; } + + let (status_code, status_body) = json_response( + &app, + Request::builder() + .method(Method::GET) + .uri("/graphs/knowledge/stream/status") + .body(Body::empty()) + .unwrap(), + ) + .await; + assert_eq!(status_code, StatusCode::OK, "{status_body}"); + let redacted = status_body.to_string(); + for forbidden in [ + "table_key", + "stable_table_id", + "table_incarnation_id", + "stream_incarnation_id", + "enrollment_id", + "binding_scope_id", + "shard_id", + "writer_epoch", + "generation", + "dataset", + "operation_id", + "block_token", + "drain_id", + "initiating_actor", + "graph_commit_id", + ] { + assert!( + !redacted.contains(forbidden), + "graph status leaked forbidden physical/opaque field '{forbidden}': {status_body}" + ); + } + let status: StreamStatusOutput = serde_json::from_value(status_body).unwrap(); + assert_eq!(status.token_counts.present, 3); + assert_eq!(status.token_counts.withdrawn, 0); + assert_eq!(status.token_counts.dead_lettered, 0); + let declarations = status + .enrolled_declarations + .iter() + .map(|status| { + ( + status.declaration.kind, + status.declaration.type_name.as_str(), + ) + }) + .collect::>(); + assert!(declarations.contains(&(StreamIngestKindOutput::Node, "Person"))); + assert!(declarations.contains(&(StreamIngestKindOutput::Edge, "Knows"))); + engine .shutdown_stream_fold_driver() .await @@ -920,6 +1003,7 @@ async fn authenticated_default_deny_refuses_stream_ingest_before_body_ownership( let app = build_app(state); let body_polled = Arc::new(AtomicBool::new(false)); let response = app + .clone() .oneshot( Request::builder() .method(Method::POST) @@ -944,6 +1028,22 @@ async fn authenticated_default_deny_refuses_stream_ingest_before_body_ownership( engine.stream_status().await.unwrap().tables.is_empty(), "a default-denied request cannot lazily enroll a private lane" ); + let (status, body) = json_response( + &app, + Request::builder() + .method(Method::GET) + .uri("/graphs/knowledge/stream/status") + .header("authorization", "Bearer stream-secret") + .body(Body::empty()) + .unwrap(), + ) + .await; + assert_eq!( + status, + StatusCode::OK, + "read-authorized status must not require stream_manage: {body}" + ); + assert_eq!(body["profile_mode"], "enabled"); engine .shutdown_stream_fold_driver() .await @@ -1014,7 +1114,29 @@ graphs: ) .await .expect("server startup consumes terminal binding into checked authority"); - let response = build_app(state) + let app = build_app(state); + let status_response = app + .clone() + .oneshot( + Request::builder() + .method(Method::GET) + .uri("/graphs/knowledge/stream/status") + .body(Body::empty()) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(status_response.status(), StatusCode::OK); + let status: StreamStatusOutput = serde_json::from_slice( + &to_bytes(status_response.into_body(), usize::MAX) + .await + .unwrap(), + ) + .unwrap(); + assert_eq!(status.profile_mode, StreamProfileModeOutput::Disabled); + assert!(status.enrolled_declarations.is_empty()); + + let response = app .oneshot( Request::builder() .method(Method::POST) diff --git a/crates/omnigraph-server/tests/openapi.rs b/crates/omnigraph-server/tests/openapi.rs index 02429216..ef9c022d 100644 --- a/crates/omnigraph-server/tests/openapi.rs +++ b/crates/omnigraph-server/tests/openapi.rs @@ -170,6 +170,7 @@ const EXPECTED_PATHS: &[&str] = &[ "/graphs/{graph_id}/read", "/graphs/{graph_id}/query", "/graphs/{graph_id}/export", + "/graphs/{graph_id}/stream/status", "/graphs/{graph_id}/stream/ingest", "/graphs/{graph_id}/change", "/graphs/{graph_id}/mutate", @@ -236,6 +237,25 @@ fn openapi_export_is_post() { assert!(doc["paths"]["/graphs/{graph_id}/export"]["post"].is_object()); } +#[test] +fn graph_stream_status_documents_checked_json_cut() { + let doc = openapi_json(); + let operation = &doc["paths"]["/graphs/{graph_id}/stream/status"]["get"]; + assert!(operation.is_object()); + assert_eq!( + operation["responses"]["200"]["content"]["application/json"]["schema"]["$ref"], + "#/components/schemas/StreamStatusOutput" + ); + assert!(operation["responses"]["200"]["headers"]["Cache-Control"].is_object()); + for status in ["401", "403", "409", "413", "500", "503"] { + assert_eq!( + operation["responses"][status]["content"]["application/json"]["schema"]["$ref"], + "#/components/schemas/ErrorOutput", + "stream status {status} must use ErrorOutput" + ); + } +} + #[test] fn graph_stream_ingest_documents_ndjson_and_token_preconditions() { let doc = openapi_json(); @@ -461,11 +481,26 @@ const EXPECTED_SCHEMAS: &[&str] = &[ "SchemaApplyRequest", "SnapshotOutput", "SnapshotTableOutput", + "StreamDeclarationOutput", + "StreamDeclarationStatusOutput", + "StreamDrainStatusOutput", + "StreamDriverErrorOutput", + "StreamDriverStateOutput", + "StreamDriverStatusOutput", "StreamIngestChallenge", "StreamIngestKindOutput", "StreamIngestLineOutput", "StreamIngestScopeOutput", "StreamIngestStatusOutput", + "StreamLastFoldStatusOutput", + "StreamLifecycleOutput", + "StreamPendingStatusOutput", + "StreamProfileModeOutput", + "StreamRebuildBlockerOutput", + "StreamRebuildStatusOutput", + "StreamStatusOutput", + "StreamStrictBlockStatusOutput", + "StreamTokenCountsOutput", ]; #[test] @@ -860,6 +895,7 @@ fn protected_endpoints_reference_bearer_token_security() { ("/graphs/{graph_id}/load", "post"), ("/graphs/{graph_id}/ingest", "post"), ("/graphs/{graph_id}/export", "post"), + ("/graphs/{graph_id}/stream/status", "get"), ("/graphs/{graph_id}/stream/ingest", "post"), ("/graphs/{graph_id}/snapshot", "get"), ("/graphs/{graph_id}/branches", "get"), @@ -1234,6 +1270,7 @@ async fn auth_mode_spec_has_security_on_protected_operations() { let protected_paths = [ ("/graphs/{graph_id}/read", "post"), ("/graphs/{graph_id}/change", "post"), + ("/graphs/{graph_id}/stream/status", "get"), ("/graphs/{graph_id}/stream/ingest", "post"), ("/graphs/{graph_id}/snapshot", "get"), ("/graphs/{graph_id}/branches", "get"), @@ -1314,6 +1351,7 @@ const EXPECTED_CLUSTER_PATHS: &[&str] = &[ "/graphs/{graph_id}/snapshot", "/graphs/{graph_id}/read", "/graphs/{graph_id}/export", + "/graphs/{graph_id}/stream/status", "/graphs/{graph_id}/stream/ingest", "/graphs/{graph_id}/change", "/graphs/{graph_id}/schema", @@ -1389,6 +1427,7 @@ async fn multi_mode_openapi_drops_flat_protected_paths() { "/snapshot", "/read", "/export", + "/stream/status", "/stream/ingest", "/change", "/schema", @@ -1586,6 +1625,7 @@ async fn served_spec_always_nests_under_cluster_prefix() { "/read", "/query", "/export", + "/stream/status", "/stream/ingest", "/change", "/mutate", diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index e7599a45..52a9b7fa 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -25,7 +25,13 @@ pub use omnigraph::{ StreamingProfileResult, TableCleanupStats, TableOptimizeStats, TableRepairStats, }; #[doc(hidden)] -pub use omnigraph::{GraphStreamChunkSource, GraphStreamIngestHandle, GraphStreamIngestStart}; +pub use omnigraph::{ + GraphStreamChunkSource, GraphStreamDeclaration, GraphStreamDeclarationStatus, + GraphStreamDrainStatus, GraphStreamDriverErrorStatus, GraphStreamDriverStatus, + GraphStreamIngestHandle, GraphStreamIngestStart, GraphStreamLastFoldStatus, + GraphStreamOperationalStatus, GraphStreamPendingStatus, GraphStreamRebuildBlocker, + GraphStreamRebuildStatus, GraphStreamStrictBlockStatus, GraphStreamTokenCounts, +}; pub(crate) use omnigraph::{ DeferredTableFork, StreamAuthorityRetirementExportProvenance, WriteAuthorityToken, WriteTxn, }; diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index e9eee235..8b7e2529 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -103,6 +103,13 @@ pub use stream_status::{ StreamTablePhysicalOperationalStatus, StreamTerminalTokenOperationalStatus, StreamTokenIndexCoverageStatus, StreamTokenLedgerOperationalStatus, }; +#[doc(hidden)] +pub use stream_status::{ + GraphStreamDeclaration, GraphStreamDeclarationStatus, GraphStreamDrainStatus, + GraphStreamDriverErrorStatus, GraphStreamDriverStatus, GraphStreamLastFoldStatus, + GraphStreamOperationalStatus, GraphStreamPendingStatus, GraphStreamRebuildBlocker, + GraphStreamRebuildStatus, GraphStreamStrictBlockStatus, GraphStreamTokenCounts, +}; pub use stream_status::{StreamStatus, StreamTableStatus}; pub use table_ops::PendingIndex; pub(crate) use table_ops::{DeferredTableFork, OpenedForMutation}; diff --git a/crates/omnigraph/src/db/omnigraph/stream_profile.rs b/crates/omnigraph/src/db/omnigraph/stream_profile.rs index 145e1b7c..411d2394 100644 --- a/crates/omnigraph/src/db/omnigraph/stream_profile.rs +++ b/crates/omnigraph/src/db/omnigraph/stream_profile.rs @@ -268,7 +268,6 @@ impl Omnigraph { /// for this profile state: the served writer, the served terminal-export /// slot, or the checked offline cluster-apply continuation for DISABLING. /// Ambient handles retain the manifest-only `stream_status` projection. - #[allow(dead_code)] // Production-compiled F6b6 status is attached by F7. pub(super) fn validate_stream_operational_status_authority( &self, profile: &StreamProfileEntry, diff --git a/crates/omnigraph/src/db/omnigraph/stream_status.rs b/crates/omnigraph/src/db/omnigraph/stream_status.rs index 14284868..e99114db 100644 --- a/crates/omnigraph/src/db/omnigraph/stream_status.rs +++ b/crates/omnigraph/src/db/omnigraph/stream_status.rs @@ -19,9 +19,9 @@ //! against exact selected versions without blocking writers. A short second //! phase closes every selected lane, reads the mutable Lance shard/recovery //! witnesses without healing them, then rereads authority before release. It -//! is engine-internal until F7 gives the shape a reviewed transport contract. +//! remains engine-internal; F7b exposes only its graph-redacted projection. -use std::collections::{BTreeMap, HashMap}; +use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::str::FromStr; use std::time::Duration; @@ -384,6 +384,451 @@ pub struct StreamOperationalStatus { pub rebuild: StreamRebuildReadiness, } +/// Graph-logical declaration selected by one checked stream status cut. +/// +/// This is deliberately the only declaration identity carried across the +/// served boundary. The physical table identity and current table key remain +/// inside the engine. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)] +pub struct GraphStreamDeclaration { + /// `node` or `edge`. + pub kind: &'static str, + /// Current accepted-schema name from the exact status cut. + pub type_name: String, +} + +/// Logical drain state for one graph declaration. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamDrainStatus { + pub goal: &'static str, + pub phase: &'static str, + pub initiated_at: i64, +} + +/// Logical strict-block state for one graph declaration. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamStrictBlockStatus { + pub kind: &'static str, + pub violation_code: String, +} + +/// Aggregate pending work for one graph declaration. +/// +/// Exact counts are returned only when every selected physical member has an +/// exact observation. An unavailable member makes the aggregate unavailable; +/// the booleans retain the actionable class without exposing a shard, +/// generation, recovery owner, or storage coordinate. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum GraphStreamPendingStatus { + Exact { + rows: u64, + arrow_bytes: u64, + batches: u64, + }, + Unavailable { + cold_replay: bool, + flushed: bool, + recovery: bool, + }, +} + +/// Last graph-visible fold summary, with physical generation and operation +/// identities intentionally removed. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamLastFoldStatus { + pub outcome: String, + pub input_rows: u64, + pub input_bytes: u64, + pub visible_rows: u64, + pub visible_bytes: u64, + pub recorded_at: i64, +} + +/// Checked operational status for one accepted graph declaration. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamDeclarationStatus { + pub declaration: GraphStreamDeclaration, + pub lifecycle: &'static str, + pub lifecycle_revision: u64, + pub drain: Option, + pub strict_block: Option, + pub pending: GraphStreamPendingStatus, + pub last_fold: Option, +} + +/// Graph-global current sequencing-authority counts. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamTokenCounts { + pub present: u64, + pub withdrawn: u64, + pub dead_lettered: u64, +} + +/// Sanitized process-local driver error summary. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamDriverErrorStatus { + pub kind: &'static str, + pub retry_in_ms: Option, +} + +/// Advisory driver summary. It intentionally carries no trigger, event, or +/// declaration identity and no raw error message. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamDriverStatus { + pub scope: &'static str, + pub authoritative: bool, + pub state: &'static str, + pub pending_count: u64, + pub published_open_folds: u64, + pub last_completion_kind: Option<&'static str>, + pub last_error: Option, +} + +/// Graph-safe reason a logical rebuild is not currently admissible. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum GraphStreamRebuildBlocker { + ProfileNotTerminal, + DeclarationNotSealed { + declaration: GraphStreamDeclaration, + }, + StrictBlock { + declaration: GraphStreamDeclaration, + }, + PendingWork { + declaration: GraphStreamDeclaration, + }, + PendingWorkUnavailable { + declaration: GraphStreamDeclaration, + }, + RecoveryPending { + count: u64, + }, + TerminalTokenAuthority { + withdrawn_count: u64, + dead_lettered_count: u64, + }, +} + +/// Graph-safe rebuild decision from the checked status cut. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamRebuildStatus { + pub ready: bool, + pub blockers: Vec, +} + +/// One checked, graph-redacted operational status cut for the served F7 +/// boundary. +/// +/// The richer internal value remains private because it contains table, +/// binding, shard, generation, Lance-version, receipt, and recovery identities. +#[doc(hidden)] +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct GraphStreamOperationalStatus { + pub manifest_version: u64, + pub profile_mode: &'static str, + pub profile_revision: u64, + pub enrolled_declarations: Vec, + pub token_counts: GraphStreamTokenCounts, + pub recovery_pending_count: u64, + pub driver: GraphStreamDriverStatus, + pub rebuild: GraphStreamRebuildStatus, +} + +fn graph_stream_declaration(table_key: &str) -> Result { + let (kind, type_name) = if let Some(type_name) = table_key.strip_prefix("node:") { + ("node", type_name) + } else if let Some(type_name) = table_key.strip_prefix("edge:") { + ("edge", type_name) + } else { + return Err(OmniError::manifest_internal(format!( + "checked graph stream status found invalid table key '{table_key}'" + ))); + }; + if type_name.is_empty() { + return Err(OmniError::manifest_internal( + "checked graph stream status found an empty declaration name", + )); + } + Ok(GraphStreamDeclaration { + kind, + type_name: type_name.to_string(), + }) +} + +fn checked_graph_status_add(total: &mut u64, value: u64, label: &str) -> Result<()> { + *total = total.checked_add(value).ok_or_else(|| { + OmniError::manifest_internal(format!( + "checked graph stream status overflowed aggregate {label}" + )) + })?; + Ok(()) +} + +fn graph_stream_pending_status( + physical: &StreamTablePhysicalOperationalStatus, +) -> Result { + let StreamTablePhysicalOperationalStatus::Observed { shards } = physical else { + return Ok(GraphStreamPendingStatus::Unavailable { + cold_replay: false, + flushed: false, + recovery: true, + }); + }; + if shards.is_empty() { + return Err(OmniError::manifest_internal( + "checked graph stream status found no physical member for an enrolled declaration", + )); + } + + let mut rows = 0_u64; + let mut arrow_bytes = 0_u64; + let mut batches = 0_u64; + let mut cold_replay = false; + let mut flushed = false; + for shard in shards { + match shard.pending { + StreamPendingGenerationStatus::Exact { + rows: member_rows, + arrow_bytes: member_arrow_bytes, + batches: member_batches, + .. + } => { + checked_graph_status_add(&mut rows, member_rows, "pending rows")?; + checked_graph_status_add( + &mut arrow_bytes, + member_arrow_bytes, + "pending Arrow bytes", + )?; + let member_batches = u64::try_from(member_batches).map_err(|_| { + OmniError::manifest_internal( + "checked graph stream status could not represent pending batch count", + ) + })?; + checked_graph_status_add(&mut batches, member_batches, "pending batches")?; + } + StreamPendingGenerationStatus::UnavailableColdReplay { .. } => cold_replay = true, + StreamPendingGenerationStatus::UnavailableFlushed { .. } => flushed = true, + } + } + if cold_replay || flushed { + Ok(GraphStreamPendingStatus::Unavailable { + cold_replay, + flushed, + recovery: false, + }) + } else { + Ok(GraphStreamPendingStatus::Exact { + rows, + arrow_bytes, + batches, + }) + } +} + +fn graph_stream_declaration_status( + table: &StreamTableOperationalStatus, +) -> Result { + Ok(GraphStreamDeclarationStatus { + declaration: graph_stream_declaration(&table.durable.table_key)?, + lifecycle: table.durable.lifecycle, + lifecycle_revision: table.durable.lifecycle_revision, + drain: table.drain.as_ref().map(|drain| GraphStreamDrainStatus { + goal: drain.goal, + phase: drain.phase, + initiated_at: drain.initiated_at, + }), + strict_block: table + .strict_block + .as_ref() + .map(|block| GraphStreamStrictBlockStatus { + kind: block.kind, + violation_code: block.violation_code.clone(), + }), + pending: graph_stream_pending_status(&table.physical)?, + last_fold: table + .last_fold + .as_ref() + .map(|fold| GraphStreamLastFoldStatus { + outcome: fold.outcome.clone(), + input_rows: fold.input_rows, + input_bytes: fold.input_bytes, + visible_rows: fold.visible_rows, + visible_bytes: fold.visible_bytes, + recorded_at: fold.recorded_at, + }), + }) +} + +fn graph_stream_rebuild_status( + rebuild: StreamRebuildReadiness, + declarations_by_table_key: &BTreeMap, + recovery_pending_count: u64, +) -> Result { + let declaration = |table_key: &str| { + declarations_by_table_key + .get(table_key) + .cloned() + .ok_or_else(|| { + OmniError::manifest_internal(format!( + "checked graph stream rebuild reason referenced unknown declaration '{table_key}'" + )) + }) + }; + let mut blockers = Vec::with_capacity(rebuild.reasons.len()); + let mut declaration_reasons = BTreeSet::new(); + let mut recovery_reason_count = 0_u64; + for reason in rebuild.reasons { + let projected = match reason { + StreamRebuildBlockReason::ProfileNotTerminal { .. } => { + Some(GraphStreamRebuildBlocker::ProfileNotTerminal) + } + StreamRebuildBlockReason::LaneNotSealed { table_key } => declaration_reasons + .insert(("not_sealed", table_key.clone())) + .then(|| declaration(&table_key)) + .transpose()? + .map(|declaration| GraphStreamRebuildBlocker::DeclarationNotSealed { declaration }), + StreamRebuildBlockReason::StrictBlock { table_key } => declaration_reasons + .insert(("strict_block", table_key.clone())) + .then(|| declaration(&table_key)) + .transpose()? + .map(|declaration| GraphStreamRebuildBlocker::StrictBlock { declaration }), + StreamRebuildBlockReason::PendingGeneration { table_key } => declaration_reasons + .insert(("pending", table_key.clone())) + .then(|| declaration(&table_key)) + .transpose()? + .map(|declaration| GraphStreamRebuildBlocker::PendingWork { declaration }), + StreamRebuildBlockReason::PendingGenerationUnavailable { table_key } => { + declaration_reasons + .insert(("pending_unavailable", table_key.clone())) + .then(|| declaration(&table_key)) + .transpose()? + .map( + |declaration| GraphStreamRebuildBlocker::PendingWorkUnavailable { + declaration, + }, + ) + } + StreamRebuildBlockReason::RelevantRecovery { .. } => { + recovery_reason_count = recovery_reason_count.checked_add(1).ok_or_else(|| { + OmniError::manifest_internal( + "checked graph stream status overflowed recovery blocker count", + ) + })?; + if recovery_reason_count == 1 { + Some(GraphStreamRebuildBlocker::RecoveryPending { + count: recovery_pending_count, + }) + } else { + None + } + } + StreamRebuildBlockReason::TerminalTokenAuthority { + withdrawn_count, + dead_lettered_count, + } => Some(GraphStreamRebuildBlocker::TerminalTokenAuthority { + withdrawn_count, + dead_lettered_count, + }), + }; + if let Some(projected) = projected { + blockers.push(projected); + } + } + if recovery_reason_count != recovery_pending_count { + return Err(OmniError::manifest_internal(format!( + "checked graph stream status found {recovery_pending_count} recovery operation(s) but {recovery_reason_count} rebuild blocker(s)" + ))); + } + Ok(GraphStreamRebuildStatus { + ready: rebuild.ready, + blockers, + }) +} + +fn project_graph_stream_operational_status( + status: StreamOperationalStatus, +) -> Result { + let StreamOperationalStatus { + cut_manifest_version, + durable, + tables, + token_ledger, + relevant_recovery, + driver, + rebuild, + } = status; + let enrolled_declarations = tables + .iter() + .map(graph_stream_declaration_status) + .collect::>>()?; + let declarations_by_table_key = tables + .iter() + .zip(&enrolled_declarations) + .map(|(table, declaration)| { + ( + table.durable.table_key.clone(), + declaration.declaration.clone(), + ) + }) + .collect::>(); + if declarations_by_table_key.len() != enrolled_declarations.len() { + return Err(OmniError::manifest_internal( + "checked graph stream status found duplicate declaration aliases", + )); + } + let recovery_pending_count = u64::try_from(relevant_recovery.len()).map_err(|_| { + OmniError::manifest_internal( + "checked graph stream status could not represent recovery operation count", + ) + })?; + let pending_count = u64::try_from(driver.pending_triggers.len()).map_err(|_| { + OmniError::manifest_internal( + "checked graph stream status could not represent driver pending count", + ) + })?; + let rebuild = + graph_stream_rebuild_status(rebuild, &declarations_by_table_key, recovery_pending_count)?; + Ok(GraphStreamOperationalStatus { + manifest_version: cut_manifest_version, + profile_mode: durable.profile_mode, + profile_revision: durable.profile_revision, + enrolled_declarations, + token_counts: GraphStreamTokenCounts { + present: token_ledger.present_count, + withdrawn: token_ledger.withdrawn_count, + dead_lettered: token_ledger.dead_lettered_count, + }, + recovery_pending_count, + driver: GraphStreamDriverStatus { + scope: driver.scope, + authoritative: driver.authoritative, + state: driver.state, + pending_count, + published_open_folds: driver.published_open_folds, + last_completion_kind: driver.last_completion.as_ref().map(|event| event.kind), + last_error: driver + .last_error + .as_ref() + .map(|error| GraphStreamDriverErrorStatus { + kind: error.kind, + retry_in_ms: error.retry_in_ms, + }), + }, + rebuild, + }) +} + /// Project every durable field from one already-resolved immutable snapshot. /// /// Keeping the projection behind one snapshot parameter makes the atomicity @@ -1410,9 +1855,9 @@ impl Omnigraph { }) } - /// Checked, read-only full stream status for the later F7 transport. - /// Existing callers continue to use the non-blocking manifest-only - /// [`Self::stream_status`]. + /// Checked, read-only full stream status underlying the F7b graph-safe + /// served projection. Existing callers continue to use the non-blocking + /// manifest-only [`Self::stream_status`]. pub(crate) async fn stream_operational_status(&self) -> Result { self.stream_operational_status_with_deadlines( None, @@ -1422,6 +1867,17 @@ impl Omnigraph { .await } + /// Capture one checked operational cut and project it onto graph-logical + /// declarations for the served F7 status boundary. + /// + /// The returned value intentionally cannot identify a table dataset, + /// physical binding, receipt, shard, epoch, generation, Lance version, or + /// recovery sidecar. The complete physical cut remains engine-private. + #[doc(hidden)] + pub async fn capture_served_graph_stream_status(&self) -> Result { + project_graph_stream_operational_status(self.stream_operational_status().await?) + } + /// Checked DISABLING observation for the stopped cluster-apply owner. The /// capability retains the real cluster lock for the complete cut. pub(crate) async fn stream_operational_status_for_apply( @@ -1442,6 +1898,15 @@ impl Omnigraph { preflight_deadline: Duration, cut_deadline: Duration, ) -> Result { + let _observation = self + .write_queue() + .try_acquire_stream_status_observation() + .ok_or_else(|| OmniError::StreamStatusBusy { + phase: "another checked status observation is already in progress".to_string(), + })?; + crate::failpoints::maybe_fail( + crate::failpoints::names::STREAM_STATUS_POST_OBSERVATION_ADMISSION, + )?; let preflight = tokio::time::timeout( preflight_deadline, self.prepare_stream_operational_status(offline_apply), @@ -1612,6 +2077,64 @@ impl Omnigraph { mod tests { use super::*; + #[test] + fn graph_rebuild_projection_collapses_physical_reason_multiplicity() { + let declaration = GraphStreamDeclaration { + kind: "node", + type_name: "Person".to_string(), + }; + let declarations = BTreeMap::from([("node:Person".to_string(), declaration)]); + let mut reasons = Vec::new(); + for _ in 0..3 { + reasons.extend([ + StreamRebuildBlockReason::LaneNotSealed { + table_key: "node:Person".to_string(), + }, + StreamRebuildBlockReason::StrictBlock { + table_key: "node:Person".to_string(), + }, + StreamRebuildBlockReason::PendingGeneration { + table_key: "node:Person".to_string(), + }, + StreamRebuildBlockReason::PendingGenerationUnavailable { + table_key: "node:Person".to_string(), + }, + ]); + } + let projected = graph_stream_rebuild_status( + StreamRebuildReadiness { + ready: false, + reasons, + }, + &declarations, + 0, + ) + .unwrap(); + + assert_eq!(projected.blockers.len(), 4); + assert_eq!( + projected + .blockers + .iter() + .filter(|reason| matches!(reason, GraphStreamRebuildBlocker::PendingWork { .. })) + .count(), + 1, + "physical shard multiplicity must not appear on the graph wire" + ); + assert_eq!( + projected + .blockers + .iter() + .filter(|reason| matches!( + reason, + GraphStreamRebuildBlocker::PendingWorkUnavailable { .. } + )) + .count(), + 1, + "unavailable physical members must collapse to one logical blocker" + ); + } + #[test] fn resident_binding_validation_rejects_same_identity_stale_physical_keys() { let identity = TableIdentity::new(101, 103).unwrap(); diff --git a/crates/omnigraph/src/db/write_queue.rs b/crates/omnigraph/src/db/write_queue.rs index 83581cb7..e14f2f31 100644 --- a/crates/omnigraph/src/db/write_queue.rs +++ b/crates/omnigraph/src/db/write_queue.rs @@ -127,6 +127,16 @@ pub(crate) struct StreamExportDestructivePermit { _permit: OwnedRwLockReadGuard<()>, } +/// Non-cloneable ownership of the sole checked operational-status observation +/// for one graph root. The immutable preflight can scan token/base authority +/// for up to its full deadline, so allowing parallel observations would turn +/// a bounded request into unbounded aggregate IO and retained memory. +#[must_use = "dropping the permit releases stream-status observation"] +pub(crate) struct StreamStatusObservationPermit { + _manager: Arc, + _permit: OwnedMutexGuard<()>, +} + impl StreamAdmissionKey { /// Bind one immutable table lifetime to its resolved physical Lance ref. pub(crate) fn for_resolved_ref(identity: TableIdentity, physical_ref: Option<&str>) -> Self { @@ -195,6 +205,12 @@ pub(crate) struct WriteQueueManager { /// Cooperative controls that can remove/reuse exact paths or versions own /// it shared, so they remain mutually concurrent but cannot race a cut. stream_export_gate: Arc>, + /// Root-wide checked operational-status admission. + /// + /// The expensive immutable preflight intentionally takes no writer gate. + /// This separate non-waiting slot keeps one root from running several + /// bounded scans concurrently while preserving writer availability. + stream_status_gate: Arc>, } impl WriteQueueManager { @@ -400,6 +416,20 @@ impl WriteQueueManager { }) } + /// Try to reserve the sole checked operational-status observation. + /// + /// Refusal is immediate. A caller must not queue another 60-second + /// immutable scan behind the current one or run those scans in parallel. + pub(crate) fn try_acquire_stream_status_observation( + self: &Arc, + ) -> Option { + let permit = Arc::clone(&self.stream_status_gate).try_lock_owned().ok()?; + Some(StreamStatusObservationPermit { + _manager: Arc::clone(self), + _permit: permit, + }) + } + /// Acquire several graph-branch control gates in one deterministic order. /// /// Native branch create-from reads a source ref and mutates a target ref, so @@ -721,7 +751,7 @@ mod tests { } #[tokio::test] - async fn stream_export_gate_is_root_shared_non_waiting_and_released_on_drop() { + async fn stream_export_and_status_gates_are_root_shared_non_waiting_and_release_on_drop() { let root = format!("memory://stream-export-slot/{}", ulid::Ulid::new()); let first = WriteQueueManager::for_root(&root); let second = WriteQueueManager::for_root(&root); @@ -778,6 +808,29 @@ mod tests { .try_acquire_stream_export_cut() .expect("dropping the cut owner must release the root gate"); drop(reacquired); + + let status = reopened_again + .try_acquire_stream_status_observation() + .expect("the first checked status observation must acquire its root slot"); + assert!( + WriteQueueManager::for_root(&root) + .try_acquire_stream_status_observation() + .is_none(), + "a second checked status observation must refuse across handles" + ); + drop(reopened_again); + let status_reopened = WriteQueueManager::for_root(&root); + assert!( + status_reopened + .try_acquire_stream_status_observation() + .is_none(), + "the status permit must keep the weakly registered root manager alive" + ); + drop(status); + let status_reacquired = status_reopened + .try_acquire_stream_status_observation() + .expect("dropping checked status must release its root slot"); + drop(status_reacquired); } #[tokio::test] diff --git a/crates/omnigraph/src/failpoints.rs b/crates/omnigraph/src/failpoints.rs index 5b1172be..8ca780fe 100644 --- a/crates/omnigraph/src/failpoints.rs +++ b/crates/omnigraph/src/failpoints.rs @@ -202,6 +202,12 @@ pub mod names { /// immutable cut while every writer gate is still held. Concurrency tests /// park here to prove writers wait only until capture returns. pub const STREAM_EXPORT_POST_CUT_CAPTURE: &str = "stream_export.post_cut_capture"; + /// A checked operational-status observer owns the root-scoped observation + /// slot but has not started its immutable preflight. Tests park the first + /// observer here to prove later observers refuse immediately and the slot + /// is released after completion or cancellation. + pub const STREAM_STATUS_POST_OBSERVATION_ADMISSION: &str = + "stream_status.post_observation_admission"; /// The resident stream driver has frozen and ordered one finite /// manifest-derived round, but has not attempted its first lane. Tests use /// this boundary to add a new node trigger and prove it cannot enter ahead diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index e299e884..92b7b995 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -383,6 +383,10 @@ const READ_ONLY_SURFACES: &[(&str, &str)] = &[ // admission while it observes physical authority, but never resolves // recovery, opens/claims a writer, or publishes anything. ("db/omnigraph/stream_status.rs", "stream_status"), + ( + "db/omnigraph/stream_status.rs", + "capture_served_graph_stream_status", + ), ( "db/omnigraph/stream_status.rs", "failpoint_stream_operational_status_for_test", @@ -2397,6 +2401,146 @@ fn served_stream_export_cut_is_hidden_move_only_and_non_forgeable() { ); } +#[test] +fn served_graph_stream_status_bridge_has_only_graph_logical_fields() { + let path = engine_src_root().join("db/omnigraph/stream_status.rs"); + let contents = std::fs::read_to_string(&path) + .unwrap_or_else(|error| panic!("failed to read {}: {error}", path.display())); + let ast = parse_rust_source(&contents, "db/omnigraph/stream_status.rs"); + let expected_types = BTreeSet::from([ + "GraphStreamDeclaration".to_string(), + "GraphStreamDeclarationStatus".to_string(), + "GraphStreamDrainStatus".to_string(), + "GraphStreamDriverErrorStatus".to_string(), + "GraphStreamDriverStatus".to_string(), + "GraphStreamLastFoldStatus".to_string(), + "GraphStreamOperationalStatus".to_string(), + "GraphStreamPendingStatus".to_string(), + "GraphStreamRebuildBlocker".to_string(), + "GraphStreamRebuildStatus".to_string(), + "GraphStreamStrictBlockStatus".to_string(), + "GraphStreamTokenCounts".to_string(), + ]); + let allowed_fields = BTreeSet::from([ + "arrow_bytes", + "authoritative", + "batches", + "blockers", + "cold_replay", + "count", + "dead_lettered", + "dead_lettered_count", + "declaration", + "drain", + "driver", + "enrolled_declarations", + "flushed", + "goal", + "initiated_at", + "input_bytes", + "input_rows", + "kind", + "last_completion_kind", + "last_error", + "last_fold", + "lifecycle", + "lifecycle_revision", + "manifest_version", + "outcome", + "pending", + "pending_count", + "phase", + "present", + "profile_mode", + "profile_revision", + "published_open_folds", + "ready", + "recorded_at", + "rebuild", + "recovery", + "recovery_pending_count", + "retry_in_ms", + "rows", + "scope", + "state", + "strict_block", + "token_counts", + "type_name", + "violation_code", + "visible_bytes", + "visible_rows", + "withdrawn", + "withdrawn_count", + ]); + let mut seen_types = BTreeSet::new(); + let mut bridge_count = 0; + + let check_fields = |type_name: &str, fields: &syn::Fields| { + for field in fields { + let Some(name) = field.ident.as_ref().map(ToString::to_string) else { + continue; + }; + assert!( + allowed_fields.contains(name.as_str()), + "served graph status type {type_name} exposes unreviewed field '{name}'" + ); + } + }; + + for item in &ast.items { + match item { + Item::Struct(item) if item.ident.to_string().starts_with("GraphStream") => { + let name = item.ident.to_string(); + seen_types.insert(name.clone()); + assert!(matches!(item.vis, Visibility::Public(_))); + assert!(has_doc_hidden(&item.attrs)); + check_fields(&name, &item.fields); + } + Item::Enum(item) if item.ident.to_string().starts_with("GraphStream") => { + let name = item.ident.to_string(); + seen_types.insert(name.clone()); + assert!(matches!(item.vis, Visibility::Public(_))); + assert!(has_doc_hidden(&item.attrs)); + for variant in &item.variants { + check_fields(&name, &variant.fields); + } + } + Item::Impl(item) if item.trait_.is_none() && is_omnigraph_type(&item.self_ty) => { + for member in &item.items { + let syn::ImplItem::Fn(function) = member else { + continue; + }; + if function.sig.ident != "capture_served_graph_stream_status" { + continue; + } + bridge_count += 1; + assert!(matches!(function.vis, Visibility::Public(_))); + assert!(function.sig.asyncness.is_some()); + assert!(has_doc_hidden(&function.attrs)); + assert!(return_type_contains_identifier( + &function.sig.output, + "GraphStreamOperationalStatus" + )); + assert!(!return_type_contains_identifier( + &function.sig.output, + "StreamOperationalStatus" + )); + } + } + _ => {} + } + } + + assert_eq!( + seen_types, expected_types, + "graph status type registry drifted" + ); + assert_eq!( + bridge_count, 1, + "there must be exactly one checked served graph status bridge" + ); +} + #[test] fn stream_driver_runtime_status_remains_advisory_and_failpoints_only() { let path = engine_src_root().join("db/omnigraph/stream_driver.rs"); diff --git a/crates/omnigraph/tests/memwal_stream.rs b/crates/omnigraph/tests/memwal_stream.rs index 7bbbc009..aa8cb737 100644 --- a/crates/omnigraph/tests/memwal_stream.rs +++ b/crates/omnigraph/tests/memwal_stream.rs @@ -39,10 +39,11 @@ use object_store::{ UploadPart, }; use omnigraph::db::{ - CleanupPolicyOptions, GraphStreamChunkSource, GraphStreamIngestStart, Omnigraph, ReadTarget, - StreamAuthorityRetirementPlan, StreamDataCorrectionAction, StreamDataCorrectionRequest, - StreamOldestUncoveredAgeStatus, StreamPendingGenerationStatus, StreamRebuildBlockReason, - StreamTablePhysicalOperationalStatus, StreamTableStatus, StreamTokenIndexCoverageStatus, + CleanupPolicyOptions, GraphStreamChunkSource, GraphStreamIngestStart, GraphStreamPendingStatus, + GraphStreamRebuildBlocker, Omnigraph, ReadTarget, StreamAuthorityRetirementPlan, + StreamDataCorrectionAction, StreamDataCorrectionRequest, StreamOldestUncoveredAgeStatus, + StreamPendingGenerationStatus, StreamRebuildBlockReason, StreamTablePhysicalOperationalStatus, + StreamTableStatus, StreamTokenIndexCoverageStatus, }; use omnigraph::error::OmniError; use omnigraph::failpoints::{ScopedFailPoint, names}; @@ -12671,6 +12672,72 @@ async fn checked_operational_status_reports_one_coherent_read_only_physical_cut( "the advisory due-in countdown may move, but status must not add driver work" ); + let graph_status = db + .capture_served_graph_stream_status() + .await + .expect("the served bridge must reuse the same checked status owner"); + assert_eq!(graph_status.manifest_version, manifest_before); + assert_eq!(graph_status.profile_mode, "ENABLED"); + assert_eq!( + graph_status.profile_revision, + status.durable.profile_revision + ); + assert_eq!(graph_status.enrolled_declarations.len(), 1); + let declaration = &graph_status.enrolled_declarations[0]; + assert_eq!(declaration.declaration.kind, "node"); + assert_eq!(declaration.declaration.type_name, "Person"); + assert_eq!(declaration.lifecycle, "OPEN"); + assert_eq!( + declaration.lifecycle_revision, + status.tables[0].durable.lifecycle_revision + ); + assert!(declaration.drain.is_none()); + assert!(declaration.strict_block.is_none()); + assert!(declaration.last_fold.is_none()); + assert!(matches!( + declaration.pending, + GraphStreamPendingStatus::Exact { + rows: 1, + arrow_bytes, + batches: 1, + } if arrow_bytes > 0 + )); + assert_eq!(graph_status.token_counts.present, 0); + assert_eq!(graph_status.token_counts.withdrawn, 0); + assert_eq!(graph_status.token_counts.dead_lettered, 0); + assert_eq!(graph_status.recovery_pending_count, 0); + assert_eq!(graph_status.driver.state, status.driver.state); + assert_eq!( + graph_status.driver.pending_count, + u64::try_from(status.driver.pending_triggers.len()).unwrap() + ); + assert!(!graph_status.rebuild.ready); + assert!( + graph_status + .rebuild + .blockers + .iter() + .any(|blocker| matches!(blocker, GraphStreamRebuildBlocker::ProfileNotTerminal)) + ); + assert!(graph_status.rebuild.blockers.iter().any(|blocker| matches!( + blocker, + GraphStreamRebuildBlocker::PendingWork { declaration } + if declaration.kind == "node" && declaration.type_name == "Person" + ))); + assert!( + graph_status + .rebuild + .blockers + .iter() + .all(|blocker| !matches!(blocker, GraphStreamRebuildBlocker::RecoveryPending { .. })), + "zero internal sidecars must remain zero graph-level recovery blockers" + ); + assert_eq!( + db.version_of(ReadTarget::branch("main")).await.unwrap(), + manifest_before, + "the served graph projection must remain read-only" + ); + let busy = db .failpoint_stream_operational_status_for_test(Duration::ZERO) .await @@ -12693,6 +12760,34 @@ async fn operational_status_times_out_only_the_blocked_authority_cut_and_cancels let _scenario = FailScenario::setup(); let (dir, db) = init_enrolled_served_with_schema(STREAM_SCHEMA).await; let manifest_before = db.version_of(ReadTarget::branch("main")).await.unwrap(); + + let admitted = + helpers::failpoint::Rendezvous::park_first(names::STREAM_STATUS_POST_OBSERVATION_ADMISSION); + let first_db = Arc::clone(&db); + let first = tokio::spawn(async move { + first_db + .failpoint_stream_operational_status_for_test(Duration::from_secs(10)) + .await + }); + admitted.wait_until_reached().await; + let overlapping = db + .failpoint_stream_operational_status_for_test(Duration::from_secs(10)) + .await + .expect_err("a second root observation must refuse without waiting"); + assert!(matches!( + overlapping, + OmniError::StreamStatusBusy { ref phase } + if phase == "another checked status observation is already in progress" + )); + admitted.release(); + first + .await + .expect("the admitted observation task must remain live") + .expect("the admitted observation must complete after release"); + db.failpoint_stream_operational_status_for_test(Duration::from_secs(10)) + .await + .expect("a completed observation must release its root slot"); + let (acquired_tx, acquired_rx) = tokio::sync::oneshot::channel(); let (release_tx, release_rx) = tokio::sync::oneshot::channel(); let blocker_db = Arc::clone(&db); diff --git a/docs/dev/canon.md b/docs/dev/canon.md index 3bd3994c..f58160e9 100644 --- a/docs/dev/canon.md +++ b/docs/dev/canon.md @@ -572,10 +572,11 @@ state-lock-held, stopped/offline cluster adapter can show and correct one exact `REPLACE`/`WITHDRAW`. V19 adds the similarly stopped/offline `cluster stream dead-letter list|export` inspector over manifest-selected current authority. Explicit production enrollment, general lifecycle control, -`SEALED -> OPEN`, `AuthorityBlock` repair, every public row surface, public -operational-status transport, and SDK/HTTP/OpenAPI row/lifecycle/maintenance -parity remain inactive. F6b6 implements the checked read-only operational cut -internally: `ENABLED` uses runtime ownership, terminal `DISABLED | RETIRED` +`SEALED -> OPEN`, `AuthorityBlock` repair, direct SDK stream ingress/status, +and SDK/HTTP/OpenAPI lifecycle/maintenance parity remain inactive. F7b exposes +F6b6's checked read-only operational cut through one graph-redacted served +HTTP/OpenAPI route and remote CLI command. `ENABLED` uses runtime ownership, +terminal `DISABLED | RETIRED` uses served-export ownership, and `DISABLING` uses explicit checked cluster- apply status authority. Immutable token/base parity, its bounded sample, coverage, and selected receipt proofs run once without writer gates; the short @@ -584,8 +585,13 @@ inside the hard status envelope is reported and blocks rebuild; exceeding any discovery bound refuses the whole status. Only an exact canonical-main participant outcome can explain physical movement as unavailable rather than a false change. Cold replay, flushed LWW projection accounting, and exact oldest- -uncovered age are unavailable. The public status remains nonblocking and -manifest-only. F6b7 separately adds a failpoints-only paired selected token- +uncovered age are unavailable. The embedded `Omnigraph::stream_status` +projection remains nonblocking and manifest-only; the served F7b projection +exposes graph declarations, aggregate counts, logical blockers, and advisory +driver state without table, lane, binding, dataset, writer, generation, +operation, actor, or commit identifiers. It is read-only, non-cacheable, and +fails closed rather than returning a partial or invented cut. F6b7 separately +adds a failpoints-only paired selected token- index decision instrument; its content-identical reconciled cut owns no recovery sidecar, and its uncompacted-profile-cycle bounded NO-GO schedules no standalone production maintenance. F6b5 @@ -1103,10 +1109,10 @@ on the proposer.) initializer for recoverable empty enrollment; private Phase B1 now consumes its bounded data/ack/replay mechanics rather than building one, and private B2 adds durable compare-and-chain/token-fold authority over those primitives. - Public row admission, lifecycle mutation, and operational-status transport - remain inactive; the Cedar vocabulary, enablement flip, and embedded - manifest-only status are active, while F6b6's checked operational cut remains - engine-internal. + Direct SDK row admission/status and lifecycle mutation remain inactive; the + Cedar vocabulary, enablement flip, and embedded manifest-only status are + active, while F7b exposes F6b6's checked cut only as a graph-redacted served + status route and remote CLI command. The selected profile forbids OmniGraph from deleting raw `_mem_wal` paths and accepts monotonic storage plus loud provider exhaustion; a future managed-reclamation profile would require the missing operation to @@ -1227,9 +1233,11 @@ production quiescence owner. F7a exposes one graph-native mixed node/edge NDJSON route plus remote client/CLI/OpenAPI parity over the existing lazy private-lane prepare and resident driver; it adds no coordinator or format. Explicit public lane enrollment, general lifecycle control, `AuthorityBlock` -repair, operational-status transport, and SDK/HTTP/OpenAPI -lifecycle/maintenance parity remain inactive. F6b6's internal checked status core observes one read-only authority -cut and retains honest unavailable states for cold replay, flushed LWW +repair, direct SDK stream ingress/status, and SDK/HTTP/OpenAPI +lifecycle/maintenance parity remain inactive. F7b exposes F6b6's checked +status core through a graph-redacted served HTTP/OpenAPI route and remote CLI +command. The core observes one read-only authority cut and retains honest +unavailable states for cold replay, flushed LWW projection, exact canonical-main recovery-owned physical movement, and oldest uncovered age; all pending sidecars inside the accepted envelope remain visible and rebuild- blocking. That envelope permits 256 matching direct `.json` sidecars, 256 @@ -1281,8 +1289,8 @@ selected-index decision evidence for current-token and receipt-key lookup work; its uncompacted-profile-cycle bounded NO-GO schedules no standalone production reconciler. F6b4 closes isolated dead-letter -envelope evidence, F6b5 closes served export, and F6b6 closes the internal -checked operational-status core while leaving its public transport inactive. +envelope evidence, F6b5 closes served export, F6b6 closes the internal checked +operational-status core, and F7b activates its graph-safe served transport. Dead-letter inspection/export, future `AuthorityBlock` repair, and retirement remain cluster/offline-only, with no replay mutation or public @@ -1305,7 +1313,7 @@ resource budgets. | **R4: Manifest authority access grows with commit count.** Current-state resolution folds history; a selective index does not by itself bound the complete physical read. | Medium | `optimize` compacts internal tables (keeps periodically-optimized shipped paths flat where separately cost-gated). RFC-024 Gate A rejected durable heads because representative RustFS latest-manifest reads/bytes grow despite flat exact-BTREE row/range work. RFC-025 Gate 0 independently rejected checkpoint-registry activation: at local 10→1,000 on RC.1, uncompacted reconciled work and the eight-fragment tail stay flat, but compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; exact-show bytes and operation counts also grow. Both RFCs are research-blocked; v8 retains the journal fold and internal-table *cleanup* remains deferred behind the resurrection watermark. | | **R5: Schema identity corruption or alias/identity drift.** Internal schema v5 introduced stable IDs/incarnation as durable authority; v6, v7, and v8 preserve them. | Medium | Open/init validate the SchemaIR domain and exact bidirectional IR↔manifest identity/path/alias contract; every active recovery envelope carries the identity pair; zero, duplicate, missing, or mismatched identity fails closed. | | **R6: Merge cost at divergence** — full-width classification and history-growing manifest folds. | Medium | Coherent coordinator scans plus retained probe handles reduced the pre-slice measured depth-5/depth-80 baseline from 59/651 manifest reads to 40/410 and cap the common fast-forward route at three internal opens and three scans, but the uncompacted-history slope remains. `merge_cost.rs` keeps both facts visible; O(delta) merge is blocked on a real deletion-delta source **(RFC-027)**; fragment adoption is **(draft RFC-0001)**. | -| **R7: The public graph firehose has a deliberately narrow operating envelope** — F7a serves absent/`OPEN` lanes in the single-writer checked runtime, while lifecycle/status/maintenance transport and overlapping-process ownership remain incomplete. | Medium | MemWAL is the strategic substrate. The bounded worker provides watcher-plus-post-fence acknowledgement and conservative replay; its legal near-cap shape closes under logical dense-slice accounting. Schema v9–v18 establish compare-and-chain tokens, checked profile/lifecycle authority, maintenance/rebind, retirement, and exact DataBlock correction. Current v19/recovery-v21 adds bounded terminal folds, current `DEAD_LETTERED` authority, ordinary successors, cluster/offline inspection/export, and three-disposition retirement. F7a activates graph-native served row ingress by reusing that core and the resident driver; it adds no new coordinator or format. F6b5 separately activates exact-terminal served export; F6b6 implements checked operational status internally; F6b7 adds paired failpoints-only selected token-index evidence and records the uncompacted-profile-cycle bounded NO-GO. Exact evidence remains in [RFC-026](../rfcs/0026-memwal-streaming-ingest.md) and the [testing map](testing.md). Public resume/rebind, operational-status and maintenance transport, and the remaining guardrail acceptance stay open. Managed reclamation and a whole-root history budget are optional later work; a public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | +| **R7: The public graph firehose has a deliberately narrow operating envelope** — F7a serves absent/`OPEN` lanes in the single-writer checked runtime and F7b serves a graph-redacted checked status cut, while lifecycle/maintenance transport and overlapping-process ownership remain incomplete. | Medium | MemWAL is the strategic substrate. The bounded worker provides watcher-plus-post-fence acknowledgement and conservative replay; its legal near-cap shape closes under logical dense-slice accounting. Schema v9–v18 establish compare-and-chain tokens, checked profile/lifecycle authority, maintenance/rebind, retirement, and exact DataBlock correction. Current v19/recovery-v21 adds bounded terminal folds, current `DEAD_LETTERED` authority, ordinary successors, cluster/offline inspection/export, and three-disposition retirement. F7a activates graph-native served row ingress by reusing that core and the resident driver; it adds no new coordinator or format. F6b5 separately activates exact-terminal served export; F6b6 implements checked operational status internally; F7b exposes only its logical graph projection over HTTP/OpenAPI and the remote CLI; F6b7 adds paired failpoints-only selected token-index evidence and records the uncompacted-profile-cycle bounded NO-GO. Exact evidence remains in [RFC-026](../rfcs/0026-memwal-streaming-ingest.md) and the [testing map](testing.md). Public resume/rebind and maintenance transport, direct SDK parity, and the remaining guardrail acceptance stay open. Managed reclamation and a whole-root history budget are optional later work; a public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | | **R8: Some operations lack enforced memory/time budgets.** | Medium | Known gap, narrowed and accepted for RFC-023. Its direct-substrate instrument rejected the first whole-delta fenced adopt (~447 MB peak at 100K × 256 versus ~74 MB Append), and the first corrected production 10K series failed at 30.0× / 108,625,920 bytes overhead; both negative results remain evidence. Mutation/Load now refuses a keyed table above 8,192 rows / 32 MiB before arm, while BranchMerge uses a recovery-enrolled chain with the same per-chunk bounds and a 1,024-transaction ceiling. The inductive certificate route removes the general diff, temporary delta, target preflight, and target join without weakening that chain. Final five-pair production medians passed at 31/8 ms (3.875×) for 10K and 136/35 ms (~3.886×) for 100K; maximum signed paired RSS overheads were 24,297,472 and 32,604,160 bytes. Inclusive row/transaction ceilings, byte refusal (including materialized blobs), operation-wide validation retention, exact source/target incarnation revalidation, second-generation certificate composition, and both between-chunk recovery directions are pinned; other operations still need explicit bounds. | | **R9: Local-FS conditional-write emulation** (`write_text_if_match` check-then-act gap). | Low | All current callers sit behind the cluster lock protocol; S3 uses true conditional puts; close before admitting any lock-free caller. | | **R10: Doc/spec drift as the system grows** — this document included. | Low | Maintenance contract (same-PR doc updates, `check-agents-md.sh` link CI, "don't lie" stale markers); this canon defers to area docs by construction. | @@ -1365,8 +1373,9 @@ Live design questions, each owned by an RFC or a known gap — not a wishlist: export, F6b6's internal checked operational status, and F6b7's failpoints-only paired token-index decision instrument are implemented. Its uncompacted- profile-cycle bounded NO-GO schedules no standalone production reconciler; - public status/driver transport and the F6 - guardrail remainder stay later. + F7b's graph-redacted served status transport is implemented; direct-SDK + status, lifecycle/maintenance transport, and the F6 guardrail remainder stay + later. No active producer creates `AuthorityBlock`, so its repair stays fail-closed. RFC-024's heads, RFC-025's retention, and later RFC-026 phases remain @@ -1401,7 +1410,7 @@ The plan of record is the RFC-022…028 family (all under | [0023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | Substrate-native keyed-write fencing via Lance's unenforced-PK filter; fleet/format activation barrier | **Implemented** (2026-07-15) | | [0024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | Materialized head-row research; the first exact-BTREE candidate bounded scan work but failed the full latest-manifest/object-byte cost gate | **Research blocked** | | [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags; current in-manifest registry lookup rejected by Gate 0 | **Research-blocked** | -| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: bounded watcher-plus-post-fence acknowledgement, compare-and-chain token authority, unbounded retain-all, checked profile/lifecycle control, recovery-v15–v20 maintenance/rebind/retirement/correction, and v19/recovery-v21 bounded terminal dead-letter folds plus three-disposition retirement. Hidden lifecycle/rebind/dead-letter, narrow cluster/offline inspection/correction/rebuild exits, F6b5 exact-terminal served export, F6b6 internal checked operational status, and the F6b7 failpoints-only token-index decision instrument (uncompacted-profile-cycle bounded NO-GO; no standalone production reconciler) are implemented; F7a graph-native served row ingress is active, while public lane enrollment/lifecycle and status transports remain inactive | **Draft; hidden protocol through F5b, F6 evidence, and F7a graph row activation implemented; remaining control/status activation stays staged** | +| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: bounded watcher-plus-post-fence acknowledgement, compare-and-chain token authority, unbounded retain-all, checked profile/lifecycle control, recovery-v15–v20 maintenance/rebind/retirement/correction, and v19/recovery-v21 bounded terminal dead-letter folds plus three-disposition retirement. Hidden lifecycle/rebind/dead-letter, narrow cluster/offline inspection/correction/rebuild exits, F6b5 exact-terminal served export, F6b6 checked operational status, F7b's graph-redacted HTTP/OpenAPI/remote-CLI projection, and the F6b7 failpoints-only token-index decision instrument (uncompacted-profile-cycle bounded NO-GO; no standalone production reconciler) are implemented; F7a graph-native served row ingress is active, while public lane enrollment/lifecycle and maintenance transports remain inactive | **Draft; hidden protocol through F5b, F6 evidence, F7a graph row activation, and F7b graph status transport implemented; remaining lifecycle/maintenance activation stays staged** | | [0027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | O(delta) merge classification from row-version lineage | Research-blocked | Deliberately split, not one mega-format: identity, key fencing, head rows, diff --git a/docs/dev/firehose-path-specs.md b/docs/dev/firehose-path-specs.md index ed280a83..d6233f68 100644 --- a/docs/dev/firehose-path-specs.md +++ b/docs/dev/firehose-path-specs.md @@ -53,9 +53,10 @@ receipt work runs once without writer gates and the short cut repeats only mutable witnesses. Only an exact canonical-main recovery participant outcome can explain physical movement as an unavailable projection rather than a movement error. Cold-replay and flushed-LWW pending accounting plus -exact oldest-uncovered-token age remain explicit unavailable values; the public -`stream_status` stays nonblocking and manifest-only, and its -CLI/HTTP/OpenAPI/SDK transport remains in the F7 remainder. F6b7 supplies +exact oldest-uncovered-token age remain explicit unavailable values. The public +embedded `stream_status` stays nonblocking and manifest-only; F7b now exposes a +separate graph-redacted checked projection through served HTTP/OpenAPI and the +remote CLI, while direct-SDK parity remains later work. F6b7 supplies covered/reconciled decision evidence; its configured-RustFS result is a bounded NO-GO only for the uncompacted profile- cycle fixture, so no standalone production reconciler is scheduled. F6b8 closes @@ -282,10 +283,11 @@ authority; it is not an import or replay surface. A typed failpoints-only snapshot now exposes process-local driver scheduling evidence to tests; it is explicitly advisory, and its pending triggers are not a durable backlog. Public durable `StreamStatus` remains manifest-only. F7a activates one -graph-native served row surface; public driver status and lifecycle/maintenance -surfaces remain inactive. F6b6 adds a -separate engine-internal checked operational cut over physical, token, -recovery, advisory-driver, and rebuild evidence. It has no public transport; +graph-native served row surface; F7b activates a graph-redacted read-only +projection of driver/recovery/rebuild health; lifecycle/maintenance surfaces +remain inactive. F6b6 adds a separate engine-internal checked operational cut +over physical, token, recovery, advisory-driver, and rebuild evidence. Its raw +physical form has no public transport; `DISABLING` uses explicit checked cluster-apply status authority. Every sidecar within the hard status envelope is visible and rebuild-blocking; exceeding any discovery bound refuses the whole status. Only exact canonical-main recovery @@ -334,7 +336,8 @@ byte/capacity/timing and isolated peak-RSS acceptance described below. | ~~F6b8 resume/driver handoff~~ | Compile-enforced root-producer-permit transfer into detached resume installation, urgent trigger-before-release, exact empty-owner housekeeping before the unchanged node-before-edge round, cancellation-safe shutdown, and cross-lane root-slot reuse | no format or recovery change | implemented behind existing hidden lifecycle/driver seams; broader retirement-failure matrix remains in F6 | | **F6b remainder** | Remaining guardrail matrix, including F6b8's post-claim install/retirement-failure cells; token-index evidence reopens at greater depth, after a Lance/index-grammar change, or before considering graph-manifest-compacted / checked-Optimize-coupled maintenance | — | later | | ~~F7a served graph ingress~~ | One graph-native mixed node/edge NDJSON route, strong graph-authority precondition, remote client/CLI, OpenAPI, and direct-mode refusals over the existing checked runtime and resident driver | no format or recovery change | implemented; no public table/lane selector or management surface | -| **F7 remainder** | Graph-level lifecycle, maintenance, operational-status transport, and their SDK/HTTP/remote-CLI/OpenAPI parity | — | only after their remaining F6 cells pass; export is already the F6b5 exception | +| ~~F7b served graph status~~ | One checked graph-logical operational cut, read authorization, HTTP/remote-CLI/OpenAPI parity, and structural redaction of physical/recovery identity | no format or recovery change | implemented over F6b6; ambient SDK status remains manifest-only and no lifecycle writer is exposed | +| **F7 remainder** | Graph-level lifecycle and maintenance transports plus their SDK/HTTP/remote-CLI/OpenAPI parity | — | only after their remaining F6 cells pass; export and read-only status are the existing exceptions | These are dependency milestones, not mandates for giant PRs. Keep each PR reviewable behind the hidden seam: the next lifecycle tranche may land receipts, @@ -1946,8 +1949,9 @@ uncovered-tail current-token instrument. F6b4 separately closes the isolated dead-letter envelope evidence and F6b5 closes bounded served export. F6b7 adds the paired failpoints-only covered/reconciled decision instrument. Its uncompacted-profile-cycle bounded NO-GO schedules no standalone production -reconciler. Public operational-status transport and the -remaining guardrails still keep F6 open and F7 forbidden. +reconciler. At that slice boundary, public operational-status transport and the +remaining guardrails still kept F6 open and F7 forbidden; F7a and F7b later +activated the proved graph row and graph-redacted status surfaces. ### Implemented F6b1 checked immutable export-cut subset @@ -1990,9 +1994,9 @@ measurement, or public status. F6b3 subsequently closed the exact-selected uncovered-tail token instrument and F6b5 subsequently closed the bounded transport. F6b7 subsequently added paired failpoints-only covered/reconciled decision evidence without a production maintenance path; its uncompacted- -profile-cycle bounded NO-GO schedules no standalone reconciler. Public -operational-status transport and the -remaining correctness/performance matrix stay in the F6b remainder/F7 boundary. +profile-cycle bounded NO-GO schedules no standalone reconciler. F7b later +activates the graph-redacted checked-status transport; the remaining +correctness/performance matrix stays in the F6b remainder/F7 boundary. ### Implemented F6b5 bounded served-export subset @@ -2028,8 +2032,9 @@ all data frames. A stalled receiver therefore backpressures production; body drop wakes cancellation and drops both cut and permit; completion and post-header error also release them. A producer that disappears without a terminal frame becomes a body error rather than a false clean EOF. This slice -changes no manifest, token, recovery, or storage grammar and activates no row -ingress, lifecycle, maintenance, or public status surface. +changes no manifest, token, recovery, or storage grammar and itself activates +no row ingress, lifecycle, maintenance, or public status surface; F7b later +activates status without changing those grammars. ### Implemented F6b6 checked operational-status core @@ -2037,8 +2042,9 @@ F6b6 adds one engine-internal `stream_operational_status` operation for the mode-appropriate checked runtime, export, or apply owner. It is distinct from the public `Omnigraph::stream_status`: the public method remains a cheap, nonblocking -projection of one manifest snapshot. No CLI, HTTP, OpenAPI, remote-client, or -ambient SDK contract exposes the operational shape; F7 owns that transport. +projection of one manifest snapshot. F7b now exposes only a graph-redacted +projection of the checked shape through HTTP/OpenAPI and remote CLI. The raw +cut and ambient SDK contract remain internal/manifest-only. The checked operation first runs the expensive immutable work—token/base parity, its bounded terminal sample, lookup-index coverage, and selected @@ -2119,7 +2125,7 @@ fresh graph initialization with the desired schema, and ordinary load there. Physical rebind preserves accepted schema. F6b5 closes bounded stream-aware served export; F6b7 closes the paired failpoints-only token-index decision instrument with a bounded NO-GO for the uncompacted profile-cycle fixture. -Public operational-status transport and the other +F7b later closes graph-redacted operational-status transport; the other served/public surfaces remain later F6b/F7 work. F6b4 separately closes the isolated dead-letter envelope evidence. @@ -2279,8 +2285,9 @@ cargo test -p omnigraph-engine --features failpoints --test memwal_stream_cost f sidecar is reported and blocks rebuild; a sidecar-explained physical move is unavailable rather than `StreamStatusChanged`. A reconciliation error appears only after measured evidence has scheduled a reconciler. - Public `Omnigraph::stream_status` remains manifest-only, and F7 still owns - CLI/HTTP/OpenAPI/SDK transport for the operational shape. Cluster-only + Public `Omnigraph::stream_status` remains manifest-only. F7b owns the graph- + redacted CLI/HTTP/OpenAPI transport; direct SDK and raw operational transport + remain inactive. Cluster-only list/export continues to revalidate each current terminal row. - **Shutdown**: the F5 supervisor protocol is wired into multi-graph server shutdown. F6a composes clean in-process shutdown ownership; F6b2 owns the @@ -2480,12 +2487,12 @@ Keep CI sustainable: only after an isolated harness demonstrates measured empty-runner and warm p95 within its proposed budget. -**Stopping after F6 is safe:** row ingress, remaining lifecycle/maintenance, -and operational-status transport remain behind the internal activation seam; -F6b5's exact-terminal served export is the narrow public exception. F7 is -forbidden until every required F6 cell is green. F7's row/control HTTP/remote -capabilities, DTOs, authorization, and direct-refusal tests co-land with those -surfaces and must pass before that activation PR merges. +**Stopping after F6 was safe:** at that boundary row ingress, lifecycle/ +maintenance, and operational-status transport remained behind the internal +activation seam; F6b5's exact-terminal served export was the narrow public +exception. F7a and F7b later activate only the proved graph row and graph- +redacted status surfaces. Remaining F7 control capabilities, DTOs, +authorization, and direct-refusal tests must still co-land with each surface. --- @@ -2499,9 +2506,10 @@ The engine resolves each declaration and reuses the existing lazy private-lane prepare, checked runtime, bounded request owner, and resident fold driver. It adds no coordinator, recovery owner, persisted authority, or format strand. -F6b5's export arm is already active, and F6b6's checked status core remains -internal. Lifecycle, status, and maintenance transport stay in the F7 -remainder and must not be exposed by weakening that cut. Raw physical +F6b5's export arm is already active. F7b exposes F6b6's checked status core +through a graph-logical read-only projection at HTTP and remote CLI while the +raw physical cut remains internal. Lifecycle and maintenance transport stay in +the F7 remainder and must not be exposed by weakening that cut. Raw physical operations never become ambient `Omnigraph` writers in this cluster-only profile. F2 already landed the profile adapter and `cluster apply --confirm-stream-offline`; F7 does not restage that control. @@ -2509,7 +2517,7 @@ profile. F2 already landed the profile adapter and | Capability | Owned cluster runtime | HTTP | Remote client / CLI | |---|---|---|---| | graph ingest | checked graph authority then the existing hidden core | `POST /graphs/{graph_id}/stream/ingest` (NDJSON in/out); missing `If-Match` returns a bodyless `428` challenge | `stream ingest --data [--graph-token ]` performs the challenge before opening input | -| status | F6b6 checked exclusive-cut status | not exposed in F7a | later graph-level `stream status` | +| status | F6b6 checked exclusive-cut status behind the F7b graph-redacted bridge | `GET /graphs/{graph_id}/stream/status` | `stream status [--json]` | | fold | internal resident driver; explicit operator fold remains private | not exposed in F7a | later graph-level management surface | | quiesce | capability-bound quiesce remains private | not exposed in F7a | later graph-level management surface | | resume / abort | capability-bound resume remains private | not exposed in F7a | later graph-level management surface | @@ -2592,8 +2600,9 @@ typed JSON before headers; exact Lance versions scan incrementally using approximate batch targets and feed a strict bounded chunk queue; a stalled receiver backpressures production; and completion, disconnect, and error release every reservation. The HTTP/remote- -CLI/OpenAPI export cells co-land with those limits. F7 still owns public row -ingress, lifecycle, maintenance, status, and their SDK/HTTP/CLI/OpenAPI parity. +CLI/OpenAPI export cells co-land with those limits. F7a later activates public +graph row ingress and F7b graph-redacted status; F7 still owns lifecycle, +maintenance, direct SDK status, and their remaining transport parity. The export artifact may initialize only a fresh target through the normal cluster workflow; it is never loaded back over the enrolled source. @@ -2711,8 +2720,8 @@ F6b6 adds the checked read-only operational cut. F6b7 adds the failpoints-only paired token-index decision instrument and records a bounded NO-GO for the uncompacted profile-cycle fixture, so the F6b remainder owns only the remaining guardrails. F7a activates the proved graph-ingest composition without exposing -its private lanes; the F7 remainder later owns graph-level control/status -surfaces. +its private lanes; F7b activates the graph-redacted checked status projection; +the F7 remainder owns graph-level lifecycle and maintenance surfaces. This ordering makes every intermediate merge safe: diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index acf3f42d..caf218ff 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -251,7 +251,7 @@ shared side and remain concurrent with one another. | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | | Keyed writes | Every current-format node/edge table declares exact non-null physical `id` as Lance's unenforced PK from creation, using the v6-introduced and later-preserved exact-`id` fence. General production strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint it only when completed effect statistics prove one attempt inserted every source row with zero updates, deletes, or skipped duplicates. Certificate admission is optional: BranchMerge uses the shortcut only when every transaction in the complete contiguous source interval carries v1 and its persisted operation is the full pure-insert `Update` shape (exact parent, no removed or updated fragments, nonempty new fragments with `physical_rows`, no field or generation rewrites, `RewriteRows`, exact-`id` filter, full nested schema preorder, and matching physical-row total). It then rechecks both source and target native ref incarnations and passes owned batches through an opaque capability whose one production mint site is structurally guarded. The proven publisher stages immutable fragments with `InsertBuilder`, replaces the uncommitted Append operation with that filter-bearing `Update`, and performs zero target preflights, target merge joins, or committed Appends. Missing, cleaned, unknown, or malformed proof uses the general ordered diff. Mutation/Load remains one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before arm; BranchMerge keeps its bounded v4 chain and exact-recovery limits. Raw Lance graph writers are unsupported, and the certificate is an internal non-cryptographic capability rather than an authenticity mechanism. The final production cost gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes) | [RFC-023](../rfcs/0023-key-conflict-fencing.md), [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | -| Streaming ingest | RFC-026 adopts Lance MemWAL and remains Draft; F7a graph-native served row ingress is active while public management remains inactive. V7–v9 established enrollment, the bounded worker, compare-and-chain token authority, and exact base/token fold; v10–v18 added checked profile/lifecycle authority plus recovery-owned profile, claim, drain, resume, maintenance, rebind, retirement, and DataBlock correction. Current v19/token-schema-v3/recovery-v21 adds deterministic mixed/all-diverted folds, one bounded dead-letter object, current `DEAD_LETTERED` sequencing, ordinary-successor correction, selected-token stopped/offline inspection/export, and three-disposition retirement. F4/F5 provide the hidden caller path, resident folding, deterministic disable, and terminal diversion. F6a–F6b4 provide composition, process/lifecycle, token-cost, and production dead-letter-envelope evidence. F6b1's checked exact-version move-only export cut blocks current terminal authority and owns one nonwaiting root slot; F6b5 now connects it to the existing served HTTP/remote-CLI/OpenAPI export route with pre-header validation, incremental exact-version scans using approximate Lance targets, strict 64-KiB chunks, a complete queue-envelope/process queue reservation under a deadline, backpressure, and disconnect-safe release. The queue reservation is not a whole-response or RSS bound. F6b6 adds an engine-internal checked read-only operational cut over physical lane, token, recovery, advisory-driver, and rebuild evidence. Expensive immutable token/base parity, the bounded sample, index coverage, and selected lifecycle-ledger proofs run once without writer gates; a short second phase validates the exact selected manifest/recovery cut and repeats only mutable witnesses under root/profile/lane fences. It binds the exact runtime owner for `ENABLED`, served-export owner for terminal `DISABLED | RETIRED`, or explicit checked cluster-apply status owner for `DISABLING`. Within the hard status envelope, every pending recovery sidecar is reported and rebuild-blocking; exceeding any discovery bound refuses the whole status; only exact canonical-main recovery ownership makes physical movement explicit unavailable evidence, while overlap or unexplained movement is `StreamStatusChanged`. Cold replay, flushed LWW projection accounting, and exact oldest-uncovered age are unavailable rather than inferred. The public nonblocking `stream_status` remains manifest-only and no CLI/HTTP/OpenAPI/SDK operational-status transport exists. Embedded/direct export of an enrolled graph still refuses; the served graph must be exact terminal `DISABLED | RETIRED`, and `RETIRED` export carries verified provenance. Checked offline disable remains the supported production quiescence owner; same-schema rebind requires terminal `DISABLED` and exact `SEALED` authority, while enrolled schema evolution uses checked export into a fresh graph. F7a exposes one graph-only mixed node/edge NDJSON route and remote command behind a strong derived graph-authority ETag, reusing lazy private-lane preparation and the resident driver without a new coordinator or format. Public lane enrollment, general lifecycle/rebind, operational-status transport, fresh read, and generation GC remain inactive. F6b7 adds a failpoints-only paired selected-index decision instrument for current-token and receipt-key work. Its configured-RustFS result is a bounded NO-GO only for the uncompacted profile-cycle fixture, so no standalone production token-index reconciler is scheduled; the remaining guardrail matrix stays open. No active path produces reserved `AuthorityBlock`, so its repair stays fail-closed. | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | +| Streaming ingest | RFC-026 adopts Lance MemWAL and remains Draft; F7a graph-native served row ingress is active while public management remains inactive. V7–v9 established enrollment, the bounded worker, compare-and-chain token authority, and exact base/token fold; v10–v18 added checked profile/lifecycle authority plus recovery-owned profile, claim, drain, resume, maintenance, rebind, retirement, and DataBlock correction. Current v19/token-schema-v3/recovery-v21 adds deterministic mixed/all-diverted folds, one bounded dead-letter object, current `DEAD_LETTERED` sequencing, ordinary-successor correction, selected-token stopped/offline inspection/export, and three-disposition retirement. F4/F5 provide the hidden caller path, resident folding, deterministic disable, and terminal diversion. F6a–F6b4 provide composition, process/lifecycle, token-cost, and production dead-letter-envelope evidence. F6b1's checked exact-version move-only export cut blocks current terminal authority and owns one nonwaiting root slot; F6b5 now connects it to the existing served HTTP/remote-CLI/OpenAPI export route with pre-header validation, incremental exact-version scans using approximate Lance targets, strict 64-KiB chunks, a complete queue-envelope/process queue reservation under a deadline, backpressure, and disconnect-safe release. The queue reservation is not a whole-response or RSS bound. F6b6 adds an engine-internal checked read-only operational cut over physical lane, token, recovery, advisory-driver, and rebuild evidence. Expensive immutable token/base parity, the bounded sample, index coverage, and selected lifecycle-ledger proofs run once without writer gates; a short second phase validates the exact selected manifest/recovery cut and repeats only mutable witnesses under root/profile/lane fences. It binds the exact runtime owner for `ENABLED`, served-export owner for terminal `DISABLED | RETIRED`, or explicit checked cluster-apply status owner for `DISABLING`. Within the hard status envelope, every pending recovery sidecar is reported and rebuild-blocking; exceeding any discovery bound refuses the whole status; only exact canonical-main recovery ownership makes physical movement explicit unavailable evidence, while overlap or unexplained movement is `StreamStatusChanged`. Cold replay, flushed LWW projection accounting, and exact oldest-uncovered age are unavailable rather than inferred. The public nonblocking embedded `stream_status` remains manifest-only. F7b exposes the checked cut only through a doc-hidden graph-redacted engine bridge, `GET /graphs/{graph_id}/stream/status`, remote `stream status`, and OpenAPI; the wire omits physical and recovery identities. Embedded/direct export of an enrolled graph still refuses; the served graph must be exact terminal `DISABLED | RETIRED`, and `RETIRED` export carries verified provenance. Checked offline disable remains the supported production quiescence owner; same-schema rebind requires terminal `DISABLED` and exact `SEALED` authority, while enrolled schema evolution uses checked export into a fresh graph. F7a exposes one graph-only mixed node/edge NDJSON route and remote command behind a strong derived graph-authority ETag, reusing lazy private-lane preparation and the resident driver without a new coordinator or format. Public lane enrollment, general lifecycle/rebind, fresh read, and generation GC remain inactive. F6b7 adds a failpoints-only paired selected-index decision instrument for current-token and receipt-key work. Its configured-RustFS result is a bounded NO-GO only for the uncompacted profile-cycle fixture, so no standalone production token-index reconciler is scheduled; the remaining guardrail matrix stays open. No active path produces reserved `AuthorityBlock`, so its repair stays fail-closed. | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | | Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Under schema/source-target/all-table gates, each control uses one operation-local post-gate manifest/namespace capture rather than refreshing the handle-local coordinator around table-gate acquisition; successful ref movement explicitly invalidates derived read caches. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. A target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks. Public create/create-from/delete also acquire the stream-export root slot nonwaitingly, preventing native named-branch delete/recreate ABA while an immutable cut is live | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one identity-bearing v9 recovery envelope; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. Optimize's payload remains a bounded maintenance adapter rather than an exact caller-minted Lance transaction proof, so it is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/0022-unified-write-path.md) | @@ -493,15 +493,15 @@ them explicit. ordinary load there. Never load the artifact over the enrolled source, and never describe physical rebind as changing accepted schema. Public/production rebind remains inactive. - Public enrollment, general resume/abort and lifecycle control, - row admission, operational-status transport, and transport parity remain + Public enrollment and general resume/abort and lifecycle control remain required before a public profile. The `stream_ingest` / `stream_manage` Cedar vocabulary is already registered, `stream_manage` gates the enablement writer, and the embedded SDK already exposes durable manifest-only status; a `GraphHistoryBudget` or retained-storage watermark belongs only to a future - bounded/managed profile, not this one. The public/served row and lifecycle - CLI/HTTP/OpenAPI surfaces remain inactive; the stopped/offline retirement + bounded/managed profile, not this one. F7a's graph-native served row path and + F7b's graph-redacted checked status path are active; public lane enrollment + and lifecycle/maintenance CLI/HTTP/OpenAPI surfaces remain inactive. The stopped/offline retirement exact `DataBlock` show/correct, and current dead-letter list/export commands are active emergency controls. Hidden F4 caller-shaped ingress/prepare, format-neutral F5a automatic @@ -534,8 +534,10 @@ them explicit. is still typed `StreamStatusChanged`. Resident accounting and a verified- empty `SEALED` proof can be exact, but cold replay and flushed LWW projection accounting are explicit unavailable values. Exact oldest-uncovered-token age - is also unavailable. The operation remains read-only, and its public - CLI/HTTP/OpenAPI/SDK transport is F7 work. + is also unavailable. The operation remains read-only. F7b exposes only its + graph-redacted checked projection through served HTTP/OpenAPI and the remote + CLI; the embedded public method remains manifest-only and direct-SDK parity + remains later work. F6b1 adds managed-exact or unmanaged-terminal lower/engine served-export authority and a doc-hidden, move-only immutable cut. An unmanaged bind is accepted only for `RETIRED` or enrolled `DISABLED`; retirement confirmation diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md index dfdcd049..6e3215e5 100644 --- a/docs/dev/rfc-022-027-architecture-review.md +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -12,12 +12,12 @@ v14/recovery-v16 `SEALED` EnsureIndices, v15/recovery-v17 `SEALED` Optimize, v16/recovery-v18 checked offline physical rebind, v17/recovery-v19 terminal authority retirement/export, v18/recovery-v20 stopped/offline DataBlock correction, and v19/recovery-v21 terminal dead-letter folding plus -three-disposition retirement implemented; public row streaming/enrollment, -general lifecycle/rebind control, -row/lifecycle/status transport surfaces inactive; F6b6 internal checked status -implemented +three-disposition retirement implemented; F7a graph-native served row ingress +and F7b graph-redacted checked status transport active; public lane enrollment, +general lifecycle/rebind control, direct SDK status, and lifecycle/maintenance +transport remain inactive **Date:** 2026-07-11 -**Last updated:** 2026-08-03 +**Last updated:** 2026-08-04 **Audience:** RFC authors, engine/storage maintainers, and release reviewers **Reviewed against:** OmniGraph 0.8.1; Lance 9.0.0-beta.15 at `f24e42c11a742581365e1cbe17c906ea2dac1bc6`; full Lance transaction, @@ -93,9 +93,10 @@ v19/token-schema-v3/recovery-v21 adds deterministic terminal diversion and three-disposition retirement. Checked offline disable is the sole supported production quiescence owner; public/production rebind remains inactive. Historical recovery-v12 fold keeps its wire meaning and is refused rather than -reinterpreted. Production/public enrollment, general resume/abort, public -status transport, and product parity remain specified, required, and inactive; -F6b6's checked read-only operational core is implemented internally. Its +reinterpreted. Production/public enrollment, general resume/abort, direct SDK +status, and lifecycle/maintenance parity remain specified, required, and +inactive. F7b exposes a graph-redacted HTTP/OpenAPI/remote-CLI projection of +F6b6's checked read-only operational core. Its complete pending-sidecar view is bounded to 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, @@ -563,8 +564,9 @@ receipt-bearing export; v18/recovery-v20 adds stopped/offline exact DataBlock correction; current v19/token-schema-v3/recovery-v21 adds deterministic mixed/all-diverted terminal folding and three-disposition retirement while leaving public/production rebind inactive. RFC-026 remains draft; F7a exposes -one graph-native served row route, while lane enrollment/control, rebind, and -operational-status transport remain inactive. +one graph-native served row route and F7b exposes graph-redacted checked status, +while lane enrollment/control, rebind, maintenance transport, and direct SDK +status remain inactive. Enrollment creates persistent MemWAL metadata and `stream_state` changes the correctness preconditions for schema, branch, maintenance, and data operations. @@ -596,10 +598,11 @@ the selected retain-all profile. The implemented private common-B2 core assigns internal schema v9, stream-config v3, stream-state protocol v2, and recovery-v12 to trusted hidden row metadata, canonical payload/token digests, the manifest-selected current-token participant, and exact base-plus-token fold -publication. Genuine v8↔v9 refusal/rebuild evidence is green. Explicit public -enrollment/general lifecycle control, public status transport, and product -parity remain inactive; F6b6's checked operational core is internal and checked -offline disable is the supported quiescence owner. Internal +publication. Genuine v8↔v9 refusal/rebuild evidence is green. F7a's graph-native +served row bridge and F7b's graph-redacted checked status transport are active; +explicit public lane enrollment/general lifecycle control, direct-SDK status, +and remaining product parity stay inactive. F6b6's richer physical cut remains +internal, and checked offline disable is the supported quiescence owner. Internal schema v11/profile-v2 and recovery-v13 own checked profile changes and their exact receipt. V12/lifecycle-v3 and recovery-v14 own the hidden enrollment, claim, ordinary/drain fold, and terminal management diff --git a/docs/dev/testing.md b/docs/dev/testing.md index b827b19a..0e338bdf 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -30,6 +30,16 @@ no replay after 412, while `planes::tests` pins served graph addressing and direct-mode refusal before dispatch. No test exposes a table/lane selector or adds a lifecycle, status, or maintenance route. +**F7b amendment:** checked operational status is now the second narrow served +stream surface. `memwal_stream.rs` extends the existing F6b6 owner with the +graph-logical projection and no-effect proof; raw table, binding, shard, +generation, token-sample, and recovery identities stay engine-private. +Server route/auth tests own `GET /graphs/{graph_id}/stream/status`, read-policy +authorization, redaction, checked-runtime ownership, and retryable refusal; +`openapi.rs` pins the generated contract. CLI in-source client and plane tests +own selected-graph GET/decode, served-only addressing, and JSON/human output. +No F7b test exposes or mutates lifecycle, maintenance, or rebind authority. + The engine's `tests/` is the principal coverage surface; most graph-shaped behavior is exercised there. ## CI control-plane tests @@ -755,9 +765,10 @@ evidence, or maintenance/rebind/resume composition. Later F6b2 closes the named acceptance cells, F6b3 closes the uncovered-tail token-cost harness, and F6b7 closes the paired decision evidence with a bounded standalone-reconciler NO-GO through 260 exact uncovered fragments for the uncompacted profile-cycle -fixture. Public operational-status transport and the remaining guardrails keep -F7 closed; graph-manifest-compacted or checked-Optimize-coupled reconciliation -requires fresh evidence. +fixture. At that boundary, public operational-status transport and the +remaining guardrails kept F7 closed; F7b now covers the graph-redacted served +status surface. Graph-manifest-compacted or checked-Optimize-coupled +reconciliation requires fresh evidence. ### RFC-026 F6b1 checked export-cut evidence ownership @@ -844,8 +855,10 @@ cumulative bodies. Crossing any one bound must refuse the whole status; the bounded path never returns a truncated inventory. `memwal_stream.rs::operational_status_times_out_only_the_blocked_authority_cut_and_cancels_cleanly` -proves the immutable preflight can succeed while a deterministic root owner -blocks only the short authority cut. The cut returns `StreamStatusBusy` for +first parks one admitted observer and proves a second observer on the same root +immediately returns `StreamStatusBusy`, then proves completion releases the +slot. It also proves the immutable preflight can succeed while a deterministic +root owner blocks only the short authority cut. The cut returns `StreamStatusBusy` for `exclusive authority cut` without manifest movement, cancellation retains no partial gate ownership, and a later status plus ordinary writer both progress. `memwal_stream.rs::operational_status_terminal_sample_is_the_first_eight_current_keys_and_marks_more` @@ -898,8 +911,11 @@ all remain `StreamStatusChanged`. Worker-registry tests likewise use the full identity/enrollment/shard key and cover non-vacant Active, Opening, and Retiring entries so a stale physical binding cannot disappear behind identity-only projection. -There are deliberately no CLI/HTTP/OpenAPI/SDK parity or generated-OpenAPI -tests for this shape until F7 supplies that transport contract. +F7b extends this owner rather than duplicating the physical proof. The engine +test pins the graph-logical projection and no manifest movement. Server and +CLI owners pin the read-authorized HTTP/OpenAPI/remote contract and a forbidden +wire-key allowlist. The ambient SDK still exposes only manifest-only +`stream_status`; the checked method remains a doc-hidden served bridge. ### RFC-026 implemented F6b2 acceptance scope @@ -947,9 +963,10 @@ F6b2 deliberately does **not** accept in-place productive SchemaApply on an enrolled graph. Schema-change acceptance is a separate checked sealed/retired export → initialize fresh graph with the desired schema → ordinary load workflow; physical rebind keeps the accepted schema unchanged. Covered/ -reconciled token evidence is owned by F6b7; public operational-status transport -and the remaining served row/control parity remain later F6/F7 owners. F6b4 -separately closes the isolated dead-letter envelope evidence. +reconciled token evidence is owned by F6b7; implemented F7b owns graph-redacted +checked operational-status transport, while direct SDK status and remaining +lifecycle/maintenance parity remain later F6/F7 owners. F6b4 separately closes the +isolated dead-letter envelope evidence. ### RFC-026 implemented F6b3/F6b7 selected-token cost evidence diff --git a/docs/dev/writing-path-state-of-affairs.md b/docs/dev/writing-path-state-of-affairs.md index 4814328c..81149752 100644 --- a/docs/dev/writing-path-state-of-affairs.md +++ b/docs/dev/writing-path-state-of-affairs.md @@ -116,8 +116,9 @@ authority. Checked offline apply is the one active quiescence owner: it freezes admission, drains a finite manifest-selected lane cut, and resumes that stored plan after restart. F7a activates one graph-native served-ingress route and remote CLI/OpenAPI contract; declaration routing and lazy lane enrollment stay -private. Public lane enrollment, general lifecycle resume/abort/rebind, and -maintenance/status transports remain inactive; +private. F7b activates one graph-redacted checked status route and remote CLI. +Public lane enrollment, general lifecycle resume/abort/rebind, maintenance +transport, and direct-SDK status remain inactive; retirement, `stream block show|correct`, and `stream dead-letter list|export` are the narrow cluster-only CLI exceptions. @@ -126,10 +127,10 @@ The RFC remains Draft. Row admission is reachable through the checked served graph bridge at `POST /graphs/{graph_id}/stream/ingest`; its engine mechanics remain private and feature-gated tests own their fault matrix. The graph-scoped Cedar vocabulary, public manifest-only status, checked profile-control -boundary, and engine-internal checked operational-status cut exist, but there -is still no `@stream` syntax, public lane-enrollment/fold API, enrolled-lane -drain/resume/correction surface, operational-status transport, or fresh-read -mode. +boundary, and engine-internal checked operational-status cut exist. F7b exposes +the cut's graph-redacted served projection; there is still no `@stream` syntax, +public lane-enrollment/fold API, enrolled-lane drain/resume/correction surface, +direct-SDK checked status, or fresh-read mode. This page answers four practical questions: @@ -313,7 +314,7 @@ These operations do not create a side door around the protocol: | Recovery | Same-process handles share ordered queues for the same canonical local root or identical normalized opaque remote/custom URI. Stream-admission domains are acquired outside profile-shared → schema → branch → stream-token → table gates where applicable. Mutation/load, SchemaApply, BranchMerge, EnsureIndices, Optimize, active v14–v21 stream operations, StreamProfileChange, and `refresh` heal roll-forward-only; Repair and Cleanup refuse pending recovery, while branch controls use specialized barriers. Read-write open performs the quiesced full sweep, including v14 enrollment/claim/fold/terminal-receipt, v15 resume/abort, v16 SEALED EnsureIndices, v17 SEALED Optimize, v18 SEALED physical rebind, historical v19 two-disposition retirement, v20 exact `DataBlock` correction, v21 terminal fold/three-disposition retirement, and v13 profile-receipt completion; read-only open never repairs and refuses unresolved stream recovery. Historical recovery-v10 enrollment, v11 B1 fold, v12 lifecycle-v2 fold, and v14 resume/maintenance/retirement/rebind are refused. A resolved graph-lineage intent is audited internally with the original actor when present; retirement instead uses its selected immutable receipt/profile chain and emits no graph lineage or `RecoveryAudit`. No-effect cleanup need not create graph lineage. | | RFC-026 Phase A (v7 foundation) | Internal schema v7 introduced identity-keyed lifecycle rows and exact empty main/unsharded enrollment. At that Phase-A-only boundary, any lifecycle row—including `SEALED`—rejected base-table, schema, maintenance, repair-adoption, and recovery effects under a process-local admission lease because no drain/fold witness-update adapter existed. Native branch create/delete alone could proceed at `SEALED` because it did not move table HEAD; `OPEN`/`DRAINING` refused it. Enrollment and open-time validation rejected named-branch overlap and any uncovered lifecycle/MemWAL mismatch. Current v19 preserves these foundation guarantees while adding the guarded lifecycle-v3 claim/fold/quiesce/resume transitions and explicit SEALED EnsureIndices/Optimize/rebind bridges. | | RFC-026 Phase B1 (historical private core) | Internal schema v8/config-v2 added one root-scoped, cross-handle serialized worker and a hard-bounded 8,192-row/32-MiB logical dense-slice Arrow no-roll generation. Watcher success proves durability; a clean `DurableBatchAck` additionally requires the same `ShardWriter::check_fenced()` to succeed immediately afterward. Fence loss, epoch-read failure, owner-task failure, or deadline ambiguity is post-invocation `AckUnknown` plus worker retirement. Reopen/replay is conservative and exact drain proof precedes quiesced abort. Gate R0's deterministic legal high-entropy near-cap cell exposed sparse scanner arrays; logical-slice charging plus dense copies repaired it. Physical RSS is evidence only. V9 preserves those worker/closure mechanics, but recovery-v11 itself is historical only. | -| RFC-026 B2 token/fold core + profile/lifecycle authority + F7a graph ingress | Internal schema v9/config-v3/state-v2 adds canonical payload/token digests, hidden trusted row metadata, exact idempotency and compare-and-chain classification, same-generation token overlays, and post-admission authority recapture. `_stream_tokens.lance` is graph-global durable sequencing state selected only by its manifest witness. V11/profile-v2 adds checked stopped/offline and runtime ownership, resumable `DISABLING`, fail-closed `RETIRED`, and exact recovery-v13 profile receipts. V12/lifecycle-v3 replaces inline histories with fixed-size ledger heads; recovery-v14 covers enrollment, writer claims, ordinary/drain folds, and terminal management receipts. V13–v20 add private resume, SEALED maintenance/rebind, authority retirement, and DataBlock correction. Current v19/token-schema-v3/recovery-v21 adds deterministic terminal diversion and extends retirement to `WITHDRAWN | DEAD_LETTERED`. The private one-lane core can cycle `OPEN → DRAINING → SEALED → OPEN`; checked offline apply is the supported `OPEN/DRAINING → SEALED` owner. F7a exposes graph-native served row admission over absent or `OPEN` lanes while keeping lane enrollment private. Public lane enrollment, resume/rebind control, status transport, and maintenance transport remain inactive. F6b5 separately activates exact-terminal served HTTP/remote-CLI/OpenAPI export with bounded transport ownership. F6b6 implements an engine-internal checked read-only operational cut with explicit checked `DISABLING` authority and complete pending-sidecar inventory inside a hard advisory envelope: 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, 32 MiB per sidecar body, and 32 MiB of cumulative bodies. Exceeding any bound is a typed refusal, never partial status. The cut retains honest recovery/unavailable physical evidence and unavailable cold-replay/flushed/oldest-age fields. Retirement, DataBlock inspection/correction, and selected current dead-letter list/export remain narrow cluster-only CLI controls. F6 owns remaining measurements and guardrail acceptance. | +| RFC-026 B2 token/fold core + profile/lifecycle authority + F7a graph ingress | Internal schema v9/config-v3/state-v2 adds canonical payload/token digests, hidden trusted row metadata, exact idempotency and compare-and-chain classification, same-generation token overlays, and post-admission authority recapture. `_stream_tokens.lance` is graph-global durable sequencing state selected only by its manifest witness. V11/profile-v2 adds checked stopped/offline and runtime ownership, resumable `DISABLING`, fail-closed `RETIRED`, and exact recovery-v13 profile receipts. V12/lifecycle-v3 replaces inline histories with fixed-size ledger heads; recovery-v14 covers enrollment, writer claims, ordinary/drain folds, and terminal management receipts. V13–v20 add private resume, SEALED maintenance/rebind, authority retirement, and DataBlock correction. Current v19/token-schema-v3/recovery-v21 adds deterministic terminal diversion and extends retirement to `WITHDRAWN | DEAD_LETTERED`. The private one-lane core can cycle `OPEN → DRAINING → SEALED → OPEN`; checked offline apply is the supported `OPEN/DRAINING → SEALED` owner. F7a exposes graph-native served row admission over absent or `OPEN` lanes while keeping lane enrollment private. F7b exposes a graph-redacted read-only projection of F6b6's checked cut over HTTP/OpenAPI and the remote CLI. Public lane enrollment, resume/rebind control, maintenance transport, and direct-SDK status remain inactive. F6b5 separately activates exact-terminal served HTTP/remote-CLI/OpenAPI export with bounded transport ownership. F6b6 implements the underlying checked read-only operational cut with explicit checked `DISABLING` authority and complete pending-sidecar inventory inside a hard advisory envelope: 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, 32 MiB per sidecar body, and 32 MiB of cumulative bodies. Exceeding any bound is a typed refusal, never partial status. The cut retains honest recovery/unavailable physical evidence and unavailable cold-replay/flushed/oldest-age fields. Retirement, DataBlock inspection/correction, and selected current dead-letter list/export remain narrow cluster-only CLI controls. F6 owns remaining measurements and guardrail acceptance. | | Lance access | One process-wide `ObjectStoreRegistry` reuses clients. Each `Omnigraph` handle owns its cached data-table `Session`; one process-wide zero-cache control `Session` opens mutable tips. Only the object-store registry is shared between the data and control sessions. This is “cache the past, never the present,” not one global cached session. | | Maintenance | EnsureIndices stages exact missing-index transactions. Its private checked-runtime recovery-v16 path can refresh an enrolled table only at exact `SEALED`; ambient calls remain refused. Optimize coordinates graph-wide compaction/index work under one bounded recovery envelope; its private recovery-v17 path applies the same checked-runtime, exact-`SEALED` pointer/proof refresh while ambient calls remain refused for enrolled tables. Recovery-v18 separately rebinds an exact `SEALED` lane without opening it. Periodic optimize compacts `__manifest`, but unmaintained history-dependent paths are not globally flat. | @@ -338,7 +339,7 @@ retry rules, recovery classification, and the full owned limits. | [023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | **Implemented** | Internal schema v6 introduced exact-`id` PK metadata, closed keyed routing, typed conflicts, bounded replay, rebuild/refusal, and accepted performance evidence; v19 preserves that contract. | | [024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | **Research-blocked** | The first in-manifest BTREE candidate has a specified logical contract and flat indexed row/range work, but fails the complete physical-I/O gate. No head rows or heads format are active. | | [025 — Checkpoint retention](../rfcs/0025-checkpoint-retention.md) | **Research-blocked** | Lance tag/pin semantics pass, but the proposed in-manifest registry access shape is not history-flat after compaction. No checkpoint rows, `ogcp_` production tags, API, or cleanup integration are active. | -| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; graph-native served row ingress, hidden lane/fold/lifecycle core, narrow cluster/offline controls, exact-terminal served export, and internal checked status implemented** | Schema v9 introduced config-v3/state-v2 token authority plus recovery-v12 exact base+token fold. V11–v21 add checked profile/lifecycle ownership, recovery-covered claims/quiesce/resume/maintenance/rebind/correction/retirement, deterministic terminal folds, exact retry/ordinary successor, and selected-token inspection. Historical recovery payloads retain their exact grammar. F6b3/F6b7 provide uncovered and failpoints-only paired reconciled token-index evidence; F6b7 records a bounded standalone-reconciler NO-GO for the uncompacted profile-cycle fixture, so recovery-owned maintenance remains open only to new evidence (including graph-manifest-compacted/checked-Optimize coupling). F6b4 closes the dead-letter envelope; F6b5 connects the move-only exact-terminal cut to existing HTTP/remote-CLI/OpenAPI export with strict chunk/queue ownership; F6b6 implements the checked read-only operational core internally. Public status transport and remaining guardrails stay open. Checked offline disable is the supported production quiescence owner. F7a activates one graph-level mixed node/edge NDJSON row route and remote CLI/OpenAPI contract over the existing lazy private-lane prepare and resident driver. Public lane enrollment, general resume/abort/rebind, and served lifecycle/maintenance/status surfaces remain inactive. B2b managed reclamation remains optional future work. | +| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; graph-native served row ingress and graph-redacted status, hidden lane/fold/lifecycle core, narrow cluster/offline controls, and exact-terminal served export implemented** | Schema v9 introduced config-v3/state-v2 token authority plus recovery-v12 exact base+token fold. V11–v21 add checked profile/lifecycle ownership, recovery-covered claims/quiesce/resume/maintenance/rebind/correction/retirement, deterministic terminal folds, exact retry/ordinary successor, and selected-token inspection. Historical recovery payloads retain their exact grammar. F6b3/F6b7 provide uncovered and failpoints-only paired reconciled token-index evidence; F6b7 records a bounded standalone-reconciler NO-GO for the uncompacted profile-cycle fixture, so recovery-owned maintenance remains open only to new evidence (including graph-manifest-compacted/checked-Optimize coupling). F6b4 closes the dead-letter envelope; F6b5 connects the move-only exact-terminal cut to existing HTTP/remote-CLI/OpenAPI export with strict chunk/queue ownership; F6b6 implements the checked read-only operational core; F7b exposes its graph-redacted HTTP/OpenAPI/remote-CLI projection. Direct-SDK status and remaining guardrails stay open. Checked offline disable is the supported production quiescence owner. F7a activates one graph-level mixed node/edge NDJSON row route and remote CLI/OpenAPI contract over the existing lazy private-lane prepare and resident driver. Public lane enrollment, general resume/abort/rebind, served lifecycle/maintenance surfaces, and direct-SDK status remain inactive. B2b managed reclamation remains optional future work. | | [027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | **Research-blocked** | The desired O(delta) classifier and fallback contract are specified. Selective live-row and deletion-delta discovery are not yet bounded, so `OrderedTableCursor` remains the correctness path. | | [028 — Stable schema identity](../rfcs/0028-stable-schema-identity.md) | **Implemented** | Rename-stable IDs, table incarnation, identity-derived paths, schema/recovery integration, and strict rebuild activation were introduced in v5 and remain active in v19. | @@ -568,9 +569,11 @@ sequence/idempotency conflicts, exact retry, same-generation chaining, post-admission stale-authority refusal, exact two-participant recovery, and durable fold attribution. The historical final-v8-source → CURRENT cell proves strict two-way refusal and rebuild fidelity while excluding hidden trusted metadata -from export. The RFC remains Draft and public row, lifecycle, maintenance, and -physical/driver-status transports remain inactive. F6b5 is the narrow -transport exception: the existing served HTTP/remote-CLI/OpenAPI export route +from export. The RFC remains Draft. F7a activates graph-native served rows and +F7b activates a graph-redacted checked driver/rebuild-status projection; +lifecycle, maintenance, direct-SDK status, and raw physical-status transports +remain inactive. F6b5 is the other narrow transport exception: the existing +served HTTP/remote-CLI/OpenAPI export route accepts an exact cluster-served terminal cut with bounded transport ownership. F6b6 implements the checked operational-status core behind an engine-internal seam. Its full immutable token/base and receipt preflight runs without writer diff --git a/docs/rfcs/0026-memwal-streaming-ingest.md b/docs/rfcs/0026-memwal-streaming-ingest.md index 280fd3f7..84b8662d 100644 --- a/docs/rfcs/0026-memwal-streaming-ingest.md +++ b/docs/rfcs/0026-memwal-streaming-ingest.md @@ -58,7 +58,8 @@ unavailable rather than falsely inconsistent. Cold-replay pending accounting, flushed LWW projection accounting, and exact oldest-uncovered-token age are reported as unavailable rather than inferred. The already-public `Omnigraph::stream_status` remains the nonblocking manifest-only projection; -CLI/HTTP/OpenAPI/SDK transport for the operational shape remains in the F7 +F7b exposes a separate graph-redacted checked projection through served +HTTP/OpenAPI and the remote CLI. Direct SDK transport remains in the F7 remainder. F6b8 closes the format-neutral resume-to-driver ownership handoff: resume transfers its non-clone root producer permit into detached writer installation @@ -71,12 +72,20 @@ work. F7a activates graph-native mixed node/edge row streaming through one checked-runtime HTTP/remote-client/CLI/OpenAPI surface and strong graph- authority precondition. It reuses hidden lazy enrollment and the resident driver, exposes no lane selector or physical evidence, and changes no format -or recovery grammar. +or recovery grammar. F7b activates one read-only, non-cacheable graph status +route plus remote CLI over F6b6's checked cut. Its DTO includes logical +declarations whose streaming state has initialized, aggregate token/recovery/pending evidence, logical rebuild +blockers, and advisory driver state. It omits table/lane/binding/dataset/writer/ +generation identities and opaque operation, actor, control-token, and graph- +commit identifiers. An unmanaged graph or changed/busy cut fails closed; the +route never invents or truncates evidence. One nonwaiting observation slot per +graph root and one per serving process bound concurrent immutable scans. It changes no format, recovery +grammar, coordinator, or lifecycle authority. Public lane enrollment, general lifecycle/rebind verbs, `AuthorityBlock` repair, the standalone production token-index reconciler deferred by F6b7's uncompacted- profile-cycle bounded NO-GO, the F6b-remainder -guardrail acceptance, and SDK/HTTP/OpenAPI lifecycle, operational-status, or -maintenance surfaces remain inactive. +guardrail acceptance, and SDK/HTTP/OpenAPI lifecycle or maintenance surfaces +remain inactive. Direct SDK operational-status parity also remains inactive. Stream-aware served export remains the separate terminal-cut transport. Retirement, DataBlock show/correct, and current dead-letter list/export are exposed only by narrow stopped/offline cluster controls. @@ -278,8 +287,9 @@ manifest-only. This slice adds no format/recovery or public API/SDK/HTTP/CLI/ OpenAPI contract and does not complete F6. Later F6b2 closes the named process, fairness, and maintenance/rebind/resume cells, and F6b3 closes the uncovered- tail current-token hit/miss and terminal-page instrument. F6b7 later adds the -paired token-index decision instrument; public operational-status transport and -the remaining guardrails keep F7 forbidden; +paired token-index decision instrument; at that boundary operational-status +transport and the remaining guardrails kept F7 forbidden. F7a/F7b later +activated graph row ingress and graph-redacted checked status; F6b4 separately closes the isolated dead-letter envelope evidence and F6b5 closes bounded served export. **F6b1 checked immutable export-cut slice implemented:** 2026-08-02 — an exact @@ -448,8 +458,8 @@ flushed LWW projection is `UnavailableFlushed`. Token-index uncovered counts are exact when Lance exposes coverage, but oldest-uncovered age is explicitly unavailable because the selected cut has no exact fragment-creation timestamp. The existing public `Omnigraph::stream_status` remains manifest-only and -nonblocking. This slice adds no CLI/HTTP/OpenAPI/SDK transport; F7 owns that -wire contract. +nonblocking. This F6b6 slice added no CLI/HTTP/OpenAPI/SDK transport; the +implemented F7b slice now owns the served graph-redacted wire contract. **F6b7 selected token-index decision instrument implemented:** 2026-08-03 — the F6b3 fixture now takes paired observations over the same logical authority. It measures current-token and profile-management-receipt hit/miss work on the @@ -489,6 +499,21 @@ the fence transfer, a lower-sorted cold tail publishing in the same first round without a driver error, and shutdown waiting for detached ownership. The broader post-claim install/retirement-failure matrix remains in F6. This slice changes no persisted or wire grammar and activates no public surface. +**F7b graph-safe operational status implemented:** 2026-08-04 — the served +`GET /graphs/{graph_id}/stream/status` route and remote `stream status` CLI +project F6b6's checked cut at graph level. The response contains logical node/ +edge declarations whose streaming state has initialized, lifecycle and drain phase, aggregate token/recovery/pending +evidence, logical rebuild blockers, and advisory driver state. It contains no +table key, stable/incarnation/binding/enrollment/shard/writer/generation or +dataset identity, and no opaque operation, actor, control-token, or graph- +commit identifier. The route uses graph `read` authorization, sends +`Cache-Control: no-store`, and returns typed pre-response refusals for missing +checked authority, a changing/busy cut, or an exceeded observation bound. It +never heals recovery, mutates lifecycle, creates evidence for an unmanaged +graph, or returns a partial inventory. One nonwaiting observation slot per +graph root and one per serving process bounds concurrent immutable scans. The embedded manifest-only status API +is unchanged. This slice changes no format, recovery grammar, coordinator, or +Lance operation. **Author track:** Maintainer design series **Depends on:** [RFC-022](0022-unified-write-path.md)'s unified write and generic recovery-sidecar protocol, plus @@ -1003,16 +1028,18 @@ v17/recovery-v19 adds the cluster-only stopped/offline authority-retirement and receipt-bearing export exit; v18/recovery-v20 adds the separate stopped/offline exact `DataBlock` show/correct exit; and current v19/token-schema-v3/ recovery-v21 adds deterministic terminal diversion plus three-disposition -retirement. None exposes a production -row caller or maintenance transport. The +retirement. Those format slices themselves expose no production row caller or +maintenance transport; F7a later activates the graph row caller without a +format change. The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary and embedded manifest-only read-only status are active under §4.7; v11 profile mutation additionally requires checked cluster-control/runtime ownership. -Supported enrollment/quiesce/rebind, general lifecycle control, -`AuthorityBlock` repair, public operational-status transport, public row -admission, and transport parity remain future gates. The checked read-only -operational-status core itself is implemented internally by F6b6. +Supported explicit enrollment/quiesce/rebind, general lifecycle control, +`AuthorityBlock` repair, direct SDK status, and maintenance transport parity +remain future gates. F7a activates graph-native served row admission, while F7b +exposes a graph-redacted HTTP/OpenAPI/remote-CLI projection of F6b6's checked +read-only operational-status core. **B2a unbounded retain-all** is the selected first profile: it deletes no MemWAL object and performs no physical-storage admission or accounting. **B2b** is the deferred managed-reclamation profile through the @@ -1959,12 +1986,14 @@ continue to the sealed proof. This rule is identical in §8 and the B2 gates. The full B2 status contract starts from one bounded current lifecycle row and its manifest-selected ledger references plus a bounded cut-consistent physical observation. It reports only current receipt identifiers and summaries; no -public receipt-history scan or pagination contract is exposed. The -experimental §4.7 slice currently exposes only the manifest projection through embedded -`Omnigraph::stream_status`; it takes no admission lease, reads no physical -shard witness, and has no CLI/HTTP/OpenAPI surface. F6b6 implements the checked -read-only operational core described below behind an engine-internal seam; it -does not change that public method or add a CLI/HTTP/OpenAPI/SDK transport. +public receipt-history scan or pagination contract is exposed. The embedded API +continues to expose only the manifest projection through +`Omnigraph::stream_status`; it takes no admission lease and reads no physical +shard witness. F6b6 implements the checked read-only operational core described +below behind an engine-internal seam. F7b leaves that embedded public method +unchanged and exposes only a graph-redacted projection of the checked cut at +`GET /graphs/{graph_id}/stream/status`, in OpenAPI, and through the remote +`stream status` CLI. Direct-SDK checked status remains inactive. In the supported full B2 profile, status first proves the expensive immutable token/base and lifecycle-ledger evidence against one manifest-selected cut @@ -3050,15 +3079,15 @@ future management verbs pass back as their compare token. It is read-only, takes no admission lease, resolves no recovery, and deliberately omits physical observations. F6b6 later added a separate engine-internal checked operational cut with physical, token, recovery, advisory-driver, and rebuild -evidence plus typed movement/deadline refusal. It remains read-only and has no -public transport; exact cold-replay pending accounting and oldest uncovered + evidence plus typed movement/deadline refusal. It remains read-only; exact cold-replay pending accounting and oldest uncovered token age are explicit unavailable values, as is flushed LWW projection accounting. `DISABLING` uses explicit checked cluster-apply status authority. All pending sidecars inside the hard status envelope are reported and rebuild- blocking; exceeding any discovery bound refuses the whole status. Only an exact canonical-main recovery participant outcome makes physical movement an -unavailable projection rather than a change. All CLI/HTTP/OpenAPI operational-status -surfaces remain inactive. The private P2 prepare proof has no row + unavailable projection rather than a change. F7b exposes only its graph- + redacted served HTTP/OpenAPI and remote-CLI projection; direct-SDK checked + status remains inactive. The private P2 prepare proof has no row body or product surface; P3–P7 remain unimplemented, and enabling the profile activates no ingest surface. Ambient embedded SDK and direct-store callers cannot mint the checked authority required to mutate it. @@ -3916,8 +3945,9 @@ incrementally using approximate batch targets and feed a strict bounded chunk queue; a stalled receiver backpressures production; and completion, disconnect, and error release every reservation. The queue/root limits and preflight/stall/disconnect handler cells -co-land with HTTP/remote-CLI/OpenAPI export parity. F7 still owns public row -ingress, lifecycle, maintenance, status, and the rest of their served parity. +co-land with HTTP/remote-CLI/OpenAPI export parity. F7a later activates public +graph row ingress and F7b graph-redacted checked status; F7 still owns +lifecycle, maintenance, direct SDK parity, and the rest of their served parity. The resulting artifact may initialize a fresh target through normal cluster control, never load over the enrolled source. @@ -4620,8 +4650,10 @@ MemTable positions or WAL cursor statistics. This row/fold design is active in v9; v18 adds the separate stopped/offline exact `DataBlock` correction exit, and current v19 adds terminal dead-letter evidence plus versioned attribution. Public row exposure, general lifecycle control, `AuthorityBlock` repair, -operational-status transport, SDK/HTTP/OpenAPI parity, and the remaining F6 -acceptance evidence remain later gates. The checked status core is internal. +direct SDK checked-status parity, and the remaining F6 acceptance evidence +remain later gates. Graph-redacted operational status is active through the +served HTTP/OpenAPI/CLI surface; the checked physical status core remains +internal. The registered Cedar vocabulary and embedded manifest-only status do not widen the private row/fold seam. @@ -4866,10 +4898,11 @@ those limits wherever the tier is exposed. Phase B1 keeps row/fold observability internal. The later §4.7 slice exposes only durable manifest authority through embedded `Omnigraph::stream_status`; -it does not expose physical worker state or create a CLI/HTTP/OpenAPI status -contract. F6b6 adds a separate checked operational observation internally, -but the public method remains manifest-only and no CLI/HTTP/OpenAPI/SDK -transport exists. The private implementation exposes test seams at its durability, +it does not expose physical worker state. F6b6 adds a separate checked +operational observation internally. F7b keeps the public embedded method +manifest-only and exposes a graph-redacted checked HTTP/OpenAPI/remote-CLI +projection; direct-SDK checked status remains inactive. The private +implementation exposes test seams at its durability, replay, fencing, resource, cut, fold, visibility, and recovery boundaries. The 2026-07-21 dense-scan repair and near-cap/RSS cell in §12.3 re-prove closure for the widest admitted shape. That evidence is not a public latency SLO, @@ -5728,10 +5761,11 @@ diagnostics only and enforce no product threshold. This B2a result itself added no schema or product surface. The subsequent private B2-common row/fold slice activated schema v9; F3f later added the narrow -stopped/offline DataBlock exit, and F7a later activated the graph-native served -row caller plus HTTP/remote-CLI/OpenAPI parity. General lifecycle control, -`AuthorityBlock` repair, and operational-status transport remain §12.6 work. -F6b6 implements the checked operational core behind an internal seam; the +stopped/offline DataBlock exit, F7a later activated the graph-native served row +caller plus HTTP/remote-CLI/OpenAPI parity, and F7b activated the graph-redacted +checked status route and remote CLI. General lifecycle control, +`AuthorityBlock` repair, direct SDK status, and maintenance transport remain +§12.6 work. F6b6 implements the checked operational core; the authorization/manifest-status slice shipped earlier under §4.7. ### 12.6 Private B2-common implementation and remaining public/B2b gates @@ -5747,10 +5781,11 @@ exact stopped/offline `DataBlock` show/correct with recovery-v20, and F5b adds current `DEAD_LETTERED`, selected-token inspection/export, ordinary successors, and three-disposition retirement through recovery-v21. F7a activates the selected profile's graph-native served row bridge, lazy private prepare, -cancellation ownership, and HTTP/remote-CLI/OpenAPI parity. Explicit lane -enrollment, general lifecycle controls, `AuthorityBlock` repair, public -operational-status transport, and embedded/direct SDK row ingress remain -inactive. F6b6 implements the checked operational-status core internally. +cancellation ownership, and HTTP/remote-CLI/OpenAPI parity. F7b activates the +graph-redacted checked operational-status HTTP/OpenAPI route and remote CLI. +Explicit lane enrollment, general lifecycle controls, `AuthorityBlock` repair, +direct SDK stream ingress/status, and maintenance transport remain inactive. +F6b6 implements the underlying checked operational-status core. Cold-replay and flushed-LWW accounting plus exact oldest-uncovered age are explicitly unavailable. `DISABLING` uses explicit checked cluster- apply status authority; all within-envelope sidecars are reported and rebuild- @@ -5838,11 +5873,12 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. token version rather than object prefixes. F6b1 lets exact terminal cluster/ server boot mint a checked export-only authority and exposes only a doc- hidden immutable engine cut. F6b5 routes that cut through the existing served - HTTP/remote-CLI/OpenAPI export surface; it adds no status fields. The remaining - minimum controls—public operational-status transport, explicit fold, + HTTP/remote-CLI/OpenAPI export surface; it adds no status fields. F7b exposes + a graph-redacted HTTP/OpenAPI/remote-CLI view of F6b6's checked read-only + operational-status core. The remaining minimum controls—explicit fold, persistent quiesce, resume/abort-drain, rebuild execution, general data - correction, and authority repair—remain inactive. F6b6 implements the - checked read-only operational-status core internally. + correction, and authority repair—remain inactive. Direct SDK checked status + also remains inactive. Embedded durable-only status is already active. F6a adds a separate typed, failpoints-only process-local advisory driver snapshot for tests; it does not add fields to that durable projection, and pending triggers are not backlog. @@ -6144,8 +6180,8 @@ remain concurrent with one another. | R0 | production-neutral retained-growth/source audit; current-object census; referenced-cut retry; legal high-entropy near-cap materialize/fold cell; no schema, public caller, or deletion | **Historical bounded-retention no-go 2026-07-20; disposition amended 2026-07-21 (§0.2/§12.4):** RC.1 still exposes neither a complete reserve-first physical envelope/receipt nor a durable cross-open randomized-attempt cap. Those facts prohibit a finite storage promise but do not block selected unbounded retain-all. The formerly red widest cell is now green locally and on the configured-RustFS CI path; current-object observations remain advisory retention evidence, not provider billing/accounting | | B2a | selected unbounded retain-all/no-GC profile on stock Lance | **Private gate implemented 2026-07-21 (§12.5):** no OmniGraph byte/object/file/history quota; zero canonical `_mem_wal` deletion; complete/partial provider residue remains retained, unreferenced, and untouched below its root through retry/reopen; provider failures are loud; local/configured-RustFS history sweeps are advisory. This gate itself activated no schema or product surface; the later private B2-common slice activates v9 | | B2b | candidate managed-reclamation retention profile | Inactive. Requires the Lance-owned durable inspect/plan/execute + receipt, post-success fencing, bounded checkpoint/inventory/accounting, local/RustFS enforced-bound validation, and the profile-specific crash matrix (§4.5.2/§12.6). Passing it alone activates no product surface | -| B2-common | schema v9/config-v3/state-v2, compare-and-chain token/attribution, graph-global token authority, recovery-v12 base+token fold; then explicit enrollment, revision-fenced lifecycle/correction/full status, SDK row/control methods, HTTP, CLI, and OpenAPI | **Private row/fold subset implemented 2026-07-22 (§11/§12.6):** canonical digests, hidden attribution, stale-authority revalidation after shared admission, same-generation chains, exact two-participant recovery/publication, durable fold attribution, retain-all, and genuine v8↔v9 refusal/rebuild are green. Explicit production enrollment, general lifecycle mutation, public row admission, cancellation/shutdown, API compatibility, and transport parity remain inactive. The later F6b6 slice implements the checked read-only operational-status core internally; its public transport remains inactive. The Cedar vocabulary, embedded manifest-only status, and narrow stopped/offline F3f DataBlock correction shipped in later EXP slices. `GraphHistoryBudget` belongs only to a future bounded/managed profile | -| EXP | experimental cluster-only activation of the §4.7 profile: offline capability-bound enablement, lazy enrollment, caller-supplied vectors, terminal per-key dead letter plus correction, irreversible authority retirement for fresh-root rebuild, SEALED maintenance/rebind, starvation-free serial folding, and graph-native served ingress | **Selected 2026-07-27 and amended through F7a (§4.7); F3a–F3f, hidden F4, F5a/F5b0/F5b, F6a–F6b8 evidence subsets, and F7a graph ingress are implemented.** Current v19/token-schema-v3/recovery-v21 publishes deterministic mixed/all-diverted folds, current `DEAD_LETTERED` authority, exact retry/ordinary successor, stopped/offline inspection, and three-disposition retirement. F6b1 freezes an exact-terminal move-only cut; F6b5 connects it to existing served HTTP/remote-client/CLI/OpenAPI export with incremental exact-version scans using approximate Lance targets, strict 64-KiB chunks, complete queue-envelope reservation, pre-header typed refusal, backpressure, and disconnect-safe body-plus-producer ownership. F6b6 adds the engine-internal checked operational cut with explicit checked `DISABLING` cluster-apply status authority. F6b7 adds a paired failpoints-only exact-selected token-index decision instrument without recovery or production maintenance. Within the hard status envelope it reports every sidecar as rebuild-blocking, while an over-bound discovery refuses the whole status; it makes only exact sidecar-owned base-HEAD movement physically unavailable, and reports cold-replay/flushed-LWW accounting plus exact oldest-uncovered age as unavailable; the public manifest-only status is unchanged. F7a exposes one graph-only mixed node/edge NDJSON route and remote command over the existing checked runtime, hidden lazy enrollment, and resident fold driver. A strong graph-authority ETag gates body ownership; results are graph-logical and redact all table/lane/binding evidence. Public lane enrollment, lifecycle/rebind control, served maintenance/status transport, and unreachable `AuthorityBlock` repair remain inactive. F6b7's bounded NO-GO applies only to the uncompacted profile-cycle fixture and schedules no standalone production token-index reconciler; remeasurement begins beyond 260 uncovered fragments, after a Lance/index-grammar change, or before considering graph-manifest-compacted or checked-Optimize-coupled maintenance. | +| B2-common | schema v9/config-v3/state-v2, compare-and-chain token/attribution, graph-global token authority, recovery-v12 base+token fold; then explicit enrollment, revision-fenced lifecycle/correction/full status, SDK row/control methods, HTTP, CLI, and OpenAPI | **Private row/fold subset implemented 2026-07-22 (§11/§12.6):** canonical digests, hidden attribution, stale-authority revalidation after shared admission, same-generation chains, exact two-participant recovery/publication, durable fold attribution, retain-all, and genuine v8↔v9 refusal/rebuild are green. Explicit production enrollment and general lifecycle mutation remain inactive. Later EXP slices activate graph-native served row ingress (F7a) and the graph-redacted checked HTTP/OpenAPI/remote-CLI status projection (F7b); direct SDK status and lifecycle/maintenance parity remain inactive. The Cedar vocabulary, embedded manifest-only status, and narrow stopped/offline F3f DataBlock correction also shipped in EXP slices. `GraphHistoryBudget` belongs only to a future bounded/managed profile | +| EXP | experimental cluster-only activation of the §4.7 profile: offline capability-bound enablement, lazy enrollment, caller-supplied vectors, terminal per-key dead letter plus correction, irreversible authority retirement for fresh-root rebuild, SEALED maintenance/rebind, starvation-free serial folding, graph-native served ingress, and graph-redacted served status | **Selected 2026-07-27 and amended through F7b (§4.7); F3a–F3f, hidden F4, F5a/F5b0/F5b, F6a–F6b8 evidence subsets, F7a graph ingress, and F7b graph status are implemented.** Current v19/token-schema-v3/recovery-v21 publishes deterministic mixed/all-diverted folds, current `DEAD_LETTERED` authority, exact retry/ordinary successor, stopped/offline inspection, and three-disposition retirement. F6b1 freezes an exact-terminal move-only cut; F6b5 connects it to existing served HTTP/remote-client/CLI/OpenAPI export with incremental exact-version scans using approximate Lance targets, strict 64-KiB chunks, complete queue-envelope reservation, pre-header typed refusal, backpressure, and disconnect-safe body-plus-producer ownership. F6b6 adds the checked operational cut with explicit checked `DISABLING` cluster-apply status authority. F6b7 adds a paired failpoints-only exact-selected token-index decision instrument without recovery or production maintenance. Within the hard status envelope status reports every sidecar as rebuild-blocking, while an over-bound discovery refuses the whole cut; it makes only exact sidecar-owned base-HEAD movement physically unavailable and reports cold-replay/flushed-LWW accounting plus exact oldest-uncovered age as unavailable; the public manifest-only status is unchanged. F7a exposes one graph-only mixed node/edge NDJSON route and remote command over the existing checked runtime, hidden lazy enrollment, and resident fold driver. A strong graph-authority ETag gates body ownership; results are graph-logical and redact all table/lane/binding evidence. F7b exposes the logical checked status cut with `read` authorization and `no-store`, omitting physical and opaque control identities. Public lane enrollment, lifecycle/rebind control, served maintenance, direct SDK parity, and unreachable `AuthorityBlock` repair remain inactive. F6b7's bounded NO-GO applies only to the uncompacted profile-cycle fixture and schedules no standalone production token-index reconciler; remeasurement begins beyond 260 uncovered fragments, after a Lance/index-grammar change, or before considering graph-manifest-compacted or checked-Optimize-coupled maintenance. | | C | restart-stable reject-row identity, atomic dead letter, richer status, and evidence-backed configurable bounds | reject crash matrix; reject-retention proof; backpressure and RSS/latency evidence. The §4.7 profile pulls a bounded object-form dead-letter subset forward using the §4.1 token as reject identity | | D | automatic operation drain, broader schema/branch/upgrade integration, and orchestrated rematerialization rebind beyond P7's explicit bridge | two-coordinator race, old/new physical-binding crash matrix, and format-transition suite | | E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion | diff --git a/docs/user/cli/reference.md b/docs/user/cli/reference.md index dea1efa8..8ac92edd 100644 --- a/docs/user/cli/reference.md +++ b/docs/user/cli/reference.md @@ -12,6 +12,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po | `load` | bulk load a branch, local or remote (`--mode overwrite\|append\|merge` is **required** — overwrite is destructive, so there is no default). Without `--from` the target branch must exist; `--from ` forks a missing `--branch` from `` first | | `ingest` | deprecated alias of `load --from ` (defaults: `--from main --mode merge`); prints a one-line warning to stderr | | `stream ingest` | experimental graph-native firehose; served-only. `stream ingest [--data ] [--graph-token ]` streams mixed node/edge NDJSON to `/graphs/{graph_id}/stream/ingest`, performs at most one missing-token preflight before opening the body, and emits ordered NDJSON acknowledgement lines. `--data` defaults to `-` (stdin). It never targets a table or direct `--store` | +| `stream status` | checked operational status for one served streaming graph. `stream status [--json]` reports graph/profile readiness, lifecycle revisions for logical node/edge declarations whose streaming state has initialized, pending-work availability, terminal-authority counts, driver/recovery health, and rebuild blockers. A declaration absent from `enrolled_declarations` may still exist in the graph schema. It never exposes or accepts a table, dataset, lane, binding, shard, or Lance identifier | | `query ` (alias: `read`) | run a read query. **Catalog lane** (default): `` is a stored query invoked **by name** from the served catalog (served-only — address with `--server`/`--profile`; the verb asserts the query is a read). **Ad-hoc lane**: with `--query ` or `-e`/`--query-string `, runs that source (the positional `` then selects which query in it). No positional graph URI — address via `--store`/`--server`/`--profile`. `read` is the deprecated previous name (one-line stderr warning) | | `mutate ` (alias: `change`) | run a mutation query; same catalog (by-name, served-only, verb asserts mutation) / ad-hoc (`--query`/`-e`) lanes as `query`. `change` is the deprecated previous name (one-line stderr warning) | | `alias [args]` | invoke an operator alias — a read-only personal binding (under `aliases:` in `~/.omnigraph/config.yaml`) to a stored query on a named server (replaces the removed `--alias` flag; stored mutations are rejected before execution) | @@ -37,7 +38,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po Every command declares the **capability** it needs — what it requires to reach a graph — which determines the addressing flags that apply: - **`any`** — `query`, `mutate`, `load`, `ingest`, `branch *`, `snapshot`, `export`, `commit *`, `schema show`, `schema apply`. Run against a graph **served (via a server) or embedded (direct against a store)**: accept a positional `file://`/`s3://` URI, `--server ` (+ `--graph ` for multi-graph servers), `--store `, or `--profile `. A remote server is addressed with `--server` — a positional `http(s)://` URI does **not** dispatch to one. -- **`served`** — `graphs list` and `stream ingest`. Both require a server. `graphs list` addresses the graph *registry* (the bare server URL), while `stream ingest` requires one selected graph through `--server` / `--profile` plus `--graph` or a configured default. Direct `--store` and client-supplied `--as` are rejected for firehose ingest; the server resolves the authenticated actor. +- **`served`** — `graphs list`, `stream ingest`, and `stream status`. All require a server. `graphs list` addresses the graph *registry* (the bare server URL), while the `stream` commands require one selected graph through `--server` / `--profile` plus `--graph` or a configured default. Direct `--store` is rejected for both stream commands; client-supplied `--as` is additionally rejected because the server resolves the authenticated actor. - **`direct`** — `init`, `optimize`, `repair`, `cleanup`, `schema plan`, `lint`. Need **direct storage access** (`file://` / `s3://`), never through a server. They accept a positional `URI`, but **not** `--server`, and a remote (`http(s)://`) URI is rejected. `optimize` / `repair` / `cleanup` additionally accept **`--cluster --graph `** (`--cluster` is a cluster directory or storage-root URI, named via `clusters:` in `~/.omnigraph/config.yaml` or a literal root), which resolves the graph's storage URI from the served cluster state (so you needn't know the `/graphs/.omni` layout). `--graph` is the one graph selector across all scopes — on these three verbs it picks the cluster graph; on the other `direct` verbs it does not apply. `--as` does not apply to any `direct` verb — maintenance records no actor. - **`control`** — `cluster *` via `--config `; `policy *` and `queries *` via `--cluster ` or a cluster profile. - **`local`** — `alias`, `embed`, `login`, `logout`, `profile`, `version`. Address no explicit graph scope. diff --git a/docs/user/operations/errors.md b/docs/user/operations/errors.md index 624e6ad9..8969f34d 100644 --- a/docs/user/operations/errors.md +++ b/docs/user/operations/errors.md @@ -56,6 +56,15 @@ returns **412** without a replacement token, also before body polling. A non-`application/x-ndjson` request returns **415**. None of these outcomes enrolls a lane or invokes MemWAL. +- Checked stream status may return a redacted **503** when its bounded + observation cannot obtain one coherent authority cut (`StreamStatusBusy` or + `StreamStatusChanged`). The response intentionally omits the private gate, + physical member, and recovery identity that moved; retry the read. A **409** + means the served graph does not own checked operational-status authority for + its current profile, including an unmanaged graph that never established + checked serving authority. A **413** means the bounded observation inventory exceeded + its hard envelope; the server refuses the whole response instead of returning + partial recovery evidence. None of these responses mutates or heals the graph. - `StreamRetirementPlanChanged` — graph/profile/lifecycle/token authority moved between retirement plan and confirm. No retirement effect is accepted; rerun the plan and review the new digest. diff --git a/docs/user/operations/server.md b/docs/user/operations/server.md index e0116ff4..851e3673 100644 --- a/docs/user/operations/server.md +++ b/docs/user/operations/server.md @@ -223,6 +223,29 @@ F7a admits an absent declaration lane (prepared lazily) or an existing `OPEN` lane. It does not expose resume: a `SEALED` lane after disable/re-enable returns `stream_authority_changed` until a later graph-level resume surface exists. +`GET /graphs/{graph_id}/stream/status` returns one checked, read-only +operational cut for a cluster-served streaming graph. The response is +graph-native: it reports the profile and revision, logical node/edge type +lifecycle and compare revision for declarations whose streaming state has been +initialized, bounded pending-work availability, current +terminal-authority counts, advisory driver health, pending recovery count, and +rebuild readiness. It never exposes table or dataset keys, stable table ids, +stream bindings, shards, epochs, generations, Lance versions, recovery-object +identities, or storage paths. The response carries `Cache-Control: no-store`. + +Status is authorized like other graph read metadata; it does not require a +`stream_manage` grant. An unmanaged graph that has never established checked +streaming authority returns **409** instead of synthesizing full physical +evidence from its manifest-only state. If the bounded observation cannot obtain +one stable cut, the route returns a redacted, retryable **503**. Status never +heals recovery or publishes graph, token, lifecycle, or Lance state. +Absence from `enrolled_declarations` means streaming state has not yet been +initialized for that accepted-schema declaration; it does not mean the graph +schema lacks the declaration. To bound immutable status scans, at most one +checked observation runs per graph storage root and at most one runs across a +server process. An overlapping request refuses immediately with the same +redacted, retryable **503**. + The existing `/export` route also streams `application/x-ndjson`; other routes remain buffered JSON. Export authorization, relevant checked recovery settlement/validation, branch/filter validation, and stream-authority validation all finish before the diff --git a/docs/user/reference/constants.md b/docs/user/reference/constants.md index 522de1ef..281ec45f 100644 --- a/docs/user/reference/constants.md +++ b/docs/user/reference/constants.md @@ -33,6 +33,7 @@ | Served graph-ingest authority precondition | strong opaque `ETag` / `If-Match`; one missing-token challenge, no automatic stale-token replacement | `POST /graphs/{id}/stream/ingest` derives the token from existing graph identity, accepted schema, streaming-profile revision, and fold delegation. Missing/stale checks finish before request-body polling and add no persisted authority | | Served graph-ingest run coalescing | `GRAPH_STREAM_RUN_MAX_COALESCE_DELAY = 50 ms` | public observable, fixed maximum dwell while collecting more same-declaration rows after one non-full request run contains a complete row; new arrivals do not reset the deadline. Capacity, declaration change, and EOF still flush immediately. This is not an end-to-end acknowledgement SLO: storage, authority, and backpressure begin after submission and may add latency | | Served graph-ingest request envelope | `1` request per actor; `2` per graph root; `134,217,728` reserved envelope bytes per request (`268,435,456` root-wide); `8,192` retained result statuses; `33,554,432` bytes per input chunk and raw line | acquired only after an exact graph token and before body polling. The permit follows the detached request owner through any invoked tail; saturation is HTTP 413. These are process-memory/ownership admission bounds, not retained-storage quota or a promise that one request allocates the full reservation | +| Served graph operational-status observation | `1` in-flight observation per graph root and `1` per serving process; `60 s` immutable preflight; `5 s` authority cut; terminal sample `8`; at most `256` matching recovery sidecars plus `256` irrelevant objects encountered; `4 MiB` cumulative URI bytes; `32 MiB` per sidecar and cumulative sidecar bodies | `GET /graphs/{id}/stream/status` reuses the checked F6b6 read-only cut. Both admission slots refuse immediately with retryable HTTP 503, so per-request bounds also bound aggregate process work. Crossing an inventory bound refuses the whole response rather than presenting partial recovery evidence. The wire response redacts the internal terminal sample, physical identities, and recovery-object identities | | Private RFC-026 B2 token-projection ceiling | `B2_MAX_TOKEN_PROJECTION_ARROW_BYTES = 33,554,432` (32 MiB) | exact winning token projection and cumulative exact-authority lookup retention | | Private RFC-026 B2 recovery-JSON ceiling | `B2_MAX_TOKEN_RECOVERY_JSON_BYTES = 33,554,432` (32 MiB) | bounded token rows embedded in one stream-fold recovery sidecar; lifecycle-v3 recovery-v14 preserves the same logical token-plan bound | | Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | optimize/cleanup | diff --git a/openapi.json b/openapi.json index b9d67d4d..396107df 100644 --- a/openapi.json +++ b/openapi.json @@ -1928,6 +1928,112 @@ ] } }, + "/graphs/{graph_id}/stream/status": { + "get": { + "tags": [ + "streaming" + ], + "summary": "Return one coherent, graph-redacted streaming operational cut.", + "description": "This is graph-wide read metadata, so Cedar authorizes it with `read`\nrather than the mutation-capable `stream_manage` action. The engine bridge\nalready removes every table, lane, binding, shard, epoch, generation,\ndataset, and recovery identity before the transport receives the value.", + "operationId": "cluster_stream_status", + "parameters": [ + { + "name": "graph_id", + "in": "path", + "description": "Graph id to route the request to.", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "Checked graph-level streaming operational status", + "headers": { + "Cache-Control": { + "schema": { + "type": "string" + }, + "description": "Always `no-store`; the response contains live advisory state and compare revisions" + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/StreamStatusOutput" + } + } + } + }, + "401": { + "description": "Unauthorized", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "403": { + "description": "Forbidden", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "409": { + "description": "The current server does not own checked status authority for this profile state", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "413": { + "description": "The bounded status inventory exceeded its hard observation envelope", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "500": { + "description": "Checked status failed without exposing physical diagnostics", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "503": { + "description": "A stable checked cut could not be obtained; retry", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + } + }, + "security": [ + { + "bearer_token": [] + } + ] + } + }, "/healthz": { "get": { "tags": [ @@ -3051,6 +3157,180 @@ } } }, + "StreamDeclarationOutput": { + "type": "object", + "description": "Logical schema declaration with streaming state.", + "required": [ + "kind", + "type" + ], + "properties": { + "kind": { + "$ref": "#/components/schemas/StreamIngestKindOutput" + }, + "type": { + "type": "string" + } + } + }, + "StreamDeclarationStatusOutput": { + "allOf": [ + { + "$ref": "#/components/schemas/StreamDeclarationOutput" + }, + { + "type": "object", + "required": [ + "lifecycle", + "lifecycle_revision", + "pending" + ], + "properties": { + "drain": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/StreamDrainStatusOutput" + } + ] + }, + "last_fold": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/StreamLastFoldStatusOutput" + } + ] + }, + "lifecycle": { + "$ref": "#/components/schemas/StreamLifecycleOutput" + }, + "lifecycle_revision": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "pending": { + "$ref": "#/components/schemas/StreamPendingStatusOutput" + }, + "strict_block": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/StreamStrictBlockStatusOutput" + } + ] + } + } + } + ], + "description": "Checked status for one logical node or edge declaration." + }, + "StreamDrainStatusOutput": { + "type": "object", + "description": "Active drain operation for one logical declaration.", + "required": [ + "goal", + "phase", + "initiated_at" + ], + "properties": { + "goal": { + "type": "string" + }, + "initiated_at": { + "type": "integer", + "format": "int64" + }, + "phase": { + "type": "string" + } + } + }, + "StreamDriverErrorOutput": { + "type": "object", + "description": "Redacted most-recent driver error.", + "required": [ + "kind" + ], + "properties": { + "kind": { + "type": "string" + }, + "retry_in_ms": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + } + } + }, + "StreamDriverStateOutput": { + "type": "string", + "description": "Process-local fold-driver run state. Driver health is advisory and never\nsubstitutes for durable lifecycle or recovery authority.", + "enum": [ + "stopped", + "running", + "stopping", + "failed" + ] + }, + "StreamDriverStatusOutput": { + "type": "object", + "description": "Advisory health of the fold driver serving this graph.", + "required": [ + "scope", + "authoritative", + "state", + "pending_count", + "published_open_folds" + ], + "properties": { + "authoritative": { + "type": "boolean" + }, + "last_completion_kind": { + "type": [ + "string", + "null" + ] + }, + "last_error": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/StreamDriverErrorOutput" + } + ] + }, + "pending_count": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "published_open_folds": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "scope": { + "type": "string" + }, + "state": { + "$ref": "#/components/schemas/StreamDriverStateOutput" + } + } + }, "StreamIngestChallenge": { "type": "object", "description": "Effect-free precondition challenge for graph-native streaming ingest.\n\nThe same value is returned as a strong `ETag` response header. Clients\nretry the request with that tag in `If-Match`; this convenience copy keeps\nthe graph authority token distinct from the per-row sequencing\n[`StreamIngestLineOutput::stream_token`].", @@ -3209,6 +3489,389 @@ "recovery_required", "stream_retry_required" ] + }, + "StreamLastFoldStatusOutput": { + "type": "object", + "description": "Most recent durable fold summary, with physical operation and generation\ncoordinates removed.", + "required": [ + "outcome", + "input_rows", + "input_bytes", + "visible_rows", + "visible_bytes", + "recorded_at" + ], + "properties": { + "input_bytes": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "input_rows": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "outcome": { + "type": "string" + }, + "recorded_at": { + "type": "integer", + "format": "int64" + }, + "visible_bytes": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "visible_rows": { + "type": "integer", + "format": "int64", + "minimum": 0 + } + } + }, + "StreamLifecycleOutput": { + "type": "string", + "description": "Stable graph-level lifecycle state for one logical declaration.", + "enum": [ + "open", + "draining", + "sealed" + ] + }, + "StreamPendingStatusOutput": { + "oneOf": [ + { + "type": "object", + "required": [ + "rows", + "arrow_bytes", + "batches", + "state" + ], + "properties": { + "arrow_bytes": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "batches": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "rows": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "state": { + "type": "string", + "enum": [ + "exact" + ] + } + } + }, + { + "type": "object", + "required": [ + "cold_replay", + "flushed", + "recovery", + "state" + ], + "properties": { + "cold_replay": { + "type": "boolean" + }, + "flushed": { + "type": "boolean" + }, + "recovery": { + "type": "boolean" + }, + "state": { + "type": "string", + "enum": [ + "unavailable" + ] + } + } + } + ], + "description": "Pending acknowledged work for one logical declaration.\n\nCounts are returned only when the checked status cut can observe them\nwithout claiming a writer or advancing replay state. An unavailable result\nnames only the graph-safe reason classes; it never exposes a shard,\ngeneration, dataset, or recovery operation." + }, + "StreamProfileModeOutput": { + "type": "string", + "description": "Stable graph-level streaming profile state.", + "enum": [ + "disabled", + "enabled", + "disabling", + "retired" + ] + }, + "StreamRebuildBlockerOutput": { + "oneOf": [ + { + "type": "object", + "required": [ + "reason" + ], + "properties": { + "reason": { + "type": "string", + "enum": [ + "profile_not_terminal" + ] + } + } + }, + { + "allOf": [ + { + "$ref": "#/components/schemas/StreamDeclarationOutput" + }, + { + "type": "object", + "required": [ + "reason" + ], + "properties": { + "reason": { + "type": "string", + "enum": [ + "declaration_not_sealed" + ] + } + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/components/schemas/StreamDeclarationOutput" + }, + { + "type": "object", + "required": [ + "reason" + ], + "properties": { + "reason": { + "type": "string", + "enum": [ + "strict_block" + ] + } + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/components/schemas/StreamDeclarationOutput" + }, + { + "type": "object", + "required": [ + "reason" + ], + "properties": { + "reason": { + "type": "string", + "enum": [ + "pending_work" + ] + } + } + } + ] + }, + { + "allOf": [ + { + "$ref": "#/components/schemas/StreamDeclarationOutput" + }, + { + "type": "object", + "required": [ + "reason" + ], + "properties": { + "reason": { + "type": "string", + "enum": [ + "pending_work_unavailable" + ] + } + } + } + ] + }, + { + "type": "object", + "required": [ + "count", + "reason" + ], + "properties": { + "count": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "reason": { + "type": "string", + "enum": [ + "recovery_pending" + ] + } + } + }, + { + "type": "object", + "required": [ + "withdrawn_count", + "dead_lettered_count", + "reason" + ], + "properties": { + "dead_lettered_count": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "reason": { + "type": "string", + "enum": [ + "terminal_token_authority" + ] + }, + "withdrawn_count": { + "type": "integer", + "format": "int64", + "minimum": 0 + } + } + } + ], + "description": "One graph-safe reason an export/import rebuild is not currently allowed." + }, + "StreamRebuildStatusOutput": { + "type": "object", + "description": "Whether the checked graph cut can be rebuilt without discarding streaming\nsequencing authority.", + "required": [ + "ready", + "blockers" + ], + "properties": { + "blockers": { + "type": "array", + "items": { + "$ref": "#/components/schemas/StreamRebuildBlockerOutput" + } + }, + "ready": { + "type": "boolean" + } + } + }, + "StreamStatusOutput": { + "type": "object", + "description": "One coherent, graph-redacted operational status cut.", + "required": [ + "manifest_version", + "profile_mode", + "profile_revision", + "enrolled_declarations", + "token_counts", + "recovery_pending_count", + "driver", + "rebuild" + ], + "properties": { + "driver": { + "$ref": "#/components/schemas/StreamDriverStatusOutput" + }, + "enrolled_declarations": { + "type": "array", + "items": { + "$ref": "#/components/schemas/StreamDeclarationStatusOutput" + }, + "description": "Logical node/edge declarations whose streaming state has initialized.\nAbsence here does not mean the graph schema has no declarations." + }, + "manifest_version": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "profile_mode": { + "$ref": "#/components/schemas/StreamProfileModeOutput" + }, + "profile_revision": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "rebuild": { + "$ref": "#/components/schemas/StreamRebuildStatusOutput" + }, + "recovery_pending_count": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "token_counts": { + "$ref": "#/components/schemas/StreamTokenCountsOutput" + } + } + }, + "StreamStrictBlockStatusOutput": { + "type": "object", + "description": "Current strict validation block for one logical declaration.", + "required": [ + "kind", + "violation_code" + ], + "properties": { + "kind": { + "type": "string" + }, + "violation_code": { + "type": "string" + } + } + }, + "StreamTokenCountsOutput": { + "type": "object", + "description": "Current graph-wide sequencing-authority counts. Generic status deliberately\nomits the sampled logical IDs and per-key stream tokens used by the\ninternal proof.", + "required": [ + "present", + "withdrawn", + "dead_lettered" + ], + "properties": { + "dead_lettered": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "present": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "withdrawn": { + "type": "integer", + "format": "int64", + "minimum": 0 + } + } } }, "securitySchemes": {