From 0f38726584ed96f750a483c47057926c343b5bd8 Mon Sep 17 00:00:00 2001 From: aaltshuler Date: Tue, 4 Aug 2026 15:34:41 +0300 Subject: [PATCH 1/4] feat(streaming): activate served graph firehose --- AGENTS.md | 9 +- Cargo.lock | 1 + crates/omnigraph-api-types/src/lib.rs | 90 ++++ crates/omnigraph-cli/Cargo.toml | 5 +- crates/omnigraph-cli/src/cli.rs | 21 +- crates/omnigraph-cli/src/client.rs | 506 +++++++++++++++++- crates/omnigraph-cli/src/main.rs | 18 + crates/omnigraph-cli/src/planes.rs | 92 +++- crates/omnigraph-server/src/handlers.rs | 187 +++++++ crates/omnigraph-server/src/lib.rs | 216 +++++++- crates/omnigraph-server/tests/auth_policy.rs | 25 +- crates/omnigraph-server/tests/data_routes.rs | 66 ++- crates/omnigraph-server/tests/multi_graph.rs | 400 +++++++++++++- crates/omnigraph-server/tests/openapi.rs | 55 ++ crates/omnigraph/src/db/mod.rs | 2 + crates/omnigraph/src/db/omnigraph.rs | 2 + .../src/db/omnigraph/stream_ingest.rs | 49 +- .../src/db/omnigraph/stream_ndjson.rs | 288 +++++++++- .../src/db/omnigraph/stream_request.rs | 7 +- crates/omnigraph/tests/forbidden_apis.rs | 6 + crates/omnigraph/tests/memwal_stream.rs | 234 +++++++- docs/dev/canon.md | 25 +- docs/dev/firehose-path-specs.md | 94 ++-- docs/dev/invariants.md | 15 +- docs/dev/lance-memwal-pr.md | 23 +- docs/dev/rfc-022-027-architecture-review.md | 8 +- docs/dev/testing.md | 43 +- docs/dev/versioning.md | 15 +- docs/dev/wal-options.md | 4 +- docs/dev/wal-thinking.md | 26 +- docs/dev/writes.md | 11 +- docs/dev/writing-path-state-of-affairs.md | 26 +- docs/releases/v0.10.0.md | 35 +- docs/rfcs/0026-memwal-streaming-ingest.md | 444 +++++++-------- docs/rfcs/0028-stable-schema-identity.md | 8 +- docs/user/cli/reference.md | 3 +- docs/user/clusters/config.md | 7 +- docs/user/clusters/index.md | 21 +- docs/user/operations/errors.md | 13 +- docs/user/operations/policy.md | 2 +- docs/user/operations/server.md | 65 ++- docs/user/operations/upgrade.md | 12 +- docs/user/reference/constants.md | 3 + openapi.json | 314 +++++++++++ 44 files changed, 3021 insertions(+), 475 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 55ce9993..5a30ab0a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -33,14 +33,15 @@ OmniGraph is a typed property-graph engine built as a coordination layer over ma - **Multi-modal querying**: vector ANN (`nearest`), full-text (`search`/`fuzzy`/`match_text`/`bm25`), Reciprocal Rank Fusion (`rrf`), and graph traversal (`Expand`, anti-join `not { … }`) in one runtime. - **Branches and commits across the whole graph**: Git-style — every successful publish appends to a commit DAG; merges are three-way at the row level. - **Atomic per-query writes**: `mutate_as` and `load` accumulate insert/update batches into an in-memory `MutationStaging.pending` per touched table. Strict insert and upsert both route through the sealed exact-`id`, filter-bearing adapter; bare Lance Append is test-only. Their RFC-022 adapter resolves or rejects relevant recovery intents before base capture, captures `(native branch id, exact graph_head, schema identity)`, then rechecks recovery and authority under schema → branch → table gates. Mutation/Load keeps one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before recovery arm; mutation update scans stream into the remaining table budget after pending-key shadowing, with blob sizes checked before payload reads. It then arms an identity-bearing recovery-v9 sidecar containing exact Lance transaction identities + pre-minted lineage, commits each table with zero transparent conflict retries, confirms the achieved effects, and publishes under the same token. Unrelated retryable pre-effect authority movement may fully reprepare without changing logical mode. A proven effect-free strict conflict returns `KeyConflict` only after a fresh exact-ID probe; no exact match triggers bounded full strict-mode reprepare, while an effect-free upsert conflict also fully reprepares. Any earlier effect or ambiguity returns `RecoveryRequired` with the sidecar retained. Strict read-set conflicts return `ReadSetChanged`. Deletes stage through the same path. D₂ at parse time remains the constructive (insert/update) XOR destructive (delete) boundary. -- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL; internal schema v12 moved that hidden path to recovery-v14 `StreamEnrollmentV2`, which selects exact enrollment/binding ledger receipts with the physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB of logical dense-slice Arrow data), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and the bounded fold mechanics. Cheap raw bounds and exact post-tombstone validation run before recovery; resident-producing served put ownership follows bounded preprocessing/inflight → root MemWAL opportunity shared → profile shared → table admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans claim, seal, drain proof, both fold effects, and publication. Admission, replay, and fold all charge `ArrayData::get_slice_memory_size`; fold densifies retained arrays. Backing-buffer capacity and physical allocation are not the limit, while isolated fold RSS remains a 384-MiB remeasurement tripwire. The 8,192-row high-entropy near-cap closure cell remains green. Internal schema v9 added the private B2 compare-and-chain core: canonical payload and token digests, grammar-impossible trusted hidden row metadata (`__omnigraph_stream_v1$`), same-generation token overlays, and a manifest-selected graph-global `_stream_tokens.lance` authority. Exact base/token lookups and recovery validation materialize at most the requested rows plus one and cap both Arrow batch bytes and cumulative retained bytes. Admission recaptures lifecycle/binding/HEAD authority only after acquiring shared admission and the same-key queue, so a stale provisional capture cannot authorize a WAL put. Under lifecycle-v3, every cold opener and ordinary fold first publishes an exact recovery-v14 writer claim whose immutable attempt/terminal receipts authenticate the bounded WAL suffix and full current-generation LWW projection. `StreamFoldV2` and `StreamDrainFold` own exact pre-minted base/token transactions, bind the selected current claim and authenticated tail, and publish the two pointers, one lifecycle successor, and graph lineage together. Quiescence is restartable `OPEN → DRAINING → SEALED`: it supports never-written empty lanes, folds non-empty lanes, reuses an already-flushed exact cut after restart, and emits one immutable terminal management receipt. V13's recovery-v15 `StreamResume` privately consumes an exact sealed proof or guarded drain state, owns the higher-epoch claim plus terminal claim/management receipts, and alone publishes the next `OPEN` row. Receipt-first replay is idempotent, and named-branch or stale-binding authority fails before the claim. V14's recovery-v16 `StreamSealedEnsureIndices` layers exact prior/next SEALED rows over recovery-v8's CreateIndex plan; under `stream_manage` and the retained checked runtime it publishes every productive index pointer, both HEAD witnesses, recomputed empty proof, and lifecycle revision in one CAS. V15's recovery-v17 `StreamSealedOptimize` separately owns Optimize's internally committing maintenance plan and exact achieved HEADs, then publishes productive table pointers with recomputed `SEALED` proofs in one CAS. Neither checked maintenance path writes a token row, advances a management-receipt chain, or accepts a caller operation ID; true no-work remains effect-free. V16's recovery-v18 `StreamRebind` consumes one exact `SEALED` proof, creates a fresh enrollment and empty shard namespace, appends immutable binding and fence-only claim receipts, and publishes a new exact `SEALED` proof without admitting a writer or put. A separate recovery-v15 resume is required to open the fresh scope. This physical rebind keeps the accepted schema unchanged. Productive SchemaApply remains refused on any enrolled graph in EXP; schema changes use a checked sealed/retired export, initialize a fresh graph with the desired schema, and load there, never over the enrolled source. V17's recovery-v19 `StreamAuthorityRetirement` owns the exact immutable retirement-receipt transaction and sole lineage-neutral `DISABLED → RETIRED` publication; it never moves graph or branch heads and its selected receipt/profile chain replaces `RecoveryAudit`. V18's recovery-v20 `StreamCorrection` owns one exact blocked generation cut, a pre-minted base effect, and one combined token-successor plus correction/management-receipt effect; recovery may recreate only the missing token participant after the exact base effect and publishes the two pointers, `DRAINING` successor, lineage, and optional PRESENT attribution together. V19's recovery-v21 `DeadLetterFold` owns one conditionally created canonical object plus exact base and token effects, accepts mixed and all-diverted outcomes, and publishes versioned fold attribution with current token-schema-v3 `DEAD_LETTERED` evidence. Recovery-v21 also carries the three-disposition retirement owner; recovery-v19 retains its historical two-disposition meaning. Ambient EnsureIndices and Optimize remain refused for enrolled tables. Historical recovery-v10 enrollment, recovery-v12 lifecycle-v2 fold, and older resume/maintenance/rebind scaffolds keep their old wire meanings and are refused rather than synthesized. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all profile is active**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and untouched through recovery/reopen; provider exhaustion is loud. The core remains crate-private behind feature-gated, doc-hidden test seams. The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary is registered, and embedded `Omnigraph::stream_status` exposes a durable manifest-only projection. V11's protocol-v2 checked cluster authority remains unchanged: `ENABLED` carries its active fold delegation, `DISABLING` carries a drain-only continuation plan, and `RETIRED` is receipt/cut-bound and fail-closed. Recovery-v13 remains exactly `StreamProfileChange`. Profile apply enforces `stream_manage` under both the currently applied and desired graph policies through the cluster-state CAS. Enabled-profile Mutation/Load/delete require the exact checked served runtime, while BranchMerge remains closed under both `ENABLED` and `DISABLING` even with that runtime. The narrow cluster-only `stream retire-for-rebuild plan|confirm` command is the sole active retirement surface and permanently freezes a verified `WITHDRAWN | DEAD_LETTERED` cut for export/rebuild. Checked offline disable is the sole supported production quiescence owner. Public row ingest/enrollment, general resume/abort, rebind, and lifecycle/maintenance/status transports remain inactive; exact-terminal served HTTP/remote-client/CLI/OpenAPI export is the narrow F6b5 exception. The stopped/offline cluster-only `stream block show|correct` and `stream dead-letter list|export` surfaces are the narrow active inspection/correction exceptions; no hidden seam is a supported operator escape hatch. +- **RFC-026 private streaming core**: Phase A's main-only adapter enrolls one exact-`id` table into one empty unsharded Lance MemWAL; internal schema v12 moved that hidden path to recovery-v14 `StreamEnrollmentV2`, which selects exact enrollment/binding ledger receipts with the physical binding and `OPEN` lifecycle. Phase B1 adds a root-singleflight worker for one no-roll generation (8,192 rows / 32 MiB of logical dense-slice Arrow data), watcher success plus the same writer's post-durability `check_fenced()` before clean acknowledgement, conservative replay/fold-only recovery, and the bounded fold mechanics. Cheap raw bounds and exact post-tombstone validation run before recovery; resident-producing served put ownership follows bounded preprocessing/inflight → root MemWAL opportunity shared → profile shared → table admission → same-key queue → worker mode; cold replay installs exact fold-only accounting; exclusive admission spans claim, seal, drain proof, both fold effects, and publication. Admission, replay, and fold all charge `ArrayData::get_slice_memory_size`; fold densifies retained arrays. Backing-buffer capacity and physical allocation are not the limit, while isolated fold RSS remains a 384-MiB remeasurement tripwire. The 8,192-row high-entropy near-cap closure cell remains green. Internal schema v9 added the private B2 compare-and-chain core: canonical payload and token digests, grammar-impossible trusted hidden row metadata (`__omnigraph_stream_v1$`), same-generation token overlays, and a manifest-selected graph-global `_stream_tokens.lance` authority. Exact base/token lookups and recovery validation materialize at most the requested rows plus one and cap both Arrow batch bytes and cumulative retained bytes. Admission recaptures lifecycle/binding/HEAD authority only after acquiring shared admission and the same-key queue, so a stale provisional capture cannot authorize a WAL put. Under lifecycle-v3, every cold opener and ordinary fold first publishes an exact recovery-v14 writer claim whose immutable attempt/terminal receipts authenticate the bounded WAL suffix and full current-generation LWW projection. `StreamFoldV2` and `StreamDrainFold` own exact pre-minted base/token transactions, bind the selected current claim and authenticated tail, and publish the two pointers, one lifecycle successor, and graph lineage together. Quiescence is restartable `OPEN → DRAINING → SEALED`: it supports never-written empty lanes, folds non-empty lanes, reuses an already-flushed exact cut after restart, and emits one immutable terminal management receipt. V13's recovery-v15 `StreamResume` privately consumes an exact sealed proof or guarded drain state, owns the higher-epoch claim plus terminal claim/management receipts, and alone publishes the next `OPEN` row. Receipt-first replay is idempotent, and named-branch or stale-binding authority fails before the claim. V14's recovery-v16 `StreamSealedEnsureIndices` layers exact prior/next SEALED rows over recovery-v8's CreateIndex plan; under `stream_manage` and the retained checked runtime it publishes every productive index pointer, both HEAD witnesses, recomputed empty proof, and lifecycle revision in one CAS. V15's recovery-v17 `StreamSealedOptimize` separately owns Optimize's internally committing maintenance plan and exact achieved HEADs, then publishes productive table pointers with recomputed `SEALED` proofs in one CAS. Neither checked maintenance path writes a token row, advances a management-receipt chain, or accepts a caller operation ID; true no-work remains effect-free. V16's recovery-v18 `StreamRebind` consumes one exact `SEALED` proof, creates a fresh enrollment and empty shard namespace, appends immutable binding and fence-only claim receipts, and publishes a new exact `SEALED` proof without admitting a writer or put. A separate recovery-v15 resume is required to open the fresh scope. This physical rebind keeps the accepted schema unchanged. Productive SchemaApply remains refused on any enrolled graph in EXP; schema changes use a checked sealed/retired export, initialize a fresh graph with the desired schema, and load there, never over the enrolled source. V17's recovery-v19 `StreamAuthorityRetirement` owns the exact immutable retirement-receipt transaction and sole lineage-neutral `DISABLED → RETIRED` publication; it never moves graph or branch heads and its selected receipt/profile chain replaces `RecoveryAudit`. V18's recovery-v20 `StreamCorrection` owns one exact blocked generation cut, a pre-minted base effect, and one combined token-successor plus correction/management-receipt effect; recovery may recreate only the missing token participant after the exact base effect and publishes the two pointers, `DRAINING` successor, lineage, and optional PRESENT attribution together. V19's recovery-v21 `DeadLetterFold` owns one conditionally created canonical object plus exact base and token effects, accepts mixed and all-diverted outcomes, and publishes versioned fold attribution with current token-schema-v3 `DEAD_LETTERED` evidence. Recovery-v21 also carries the three-disposition retirement owner; recovery-v19 retains its historical two-disposition meaning. Ambient EnsureIndices and Optimize remain refused for enrolled tables. Historical recovery-v10 enrollment, recovery-v12 lifecycle-v2 fold, and older resume/maintenance/rebind scaffolds keep their old wire meanings and are refused rather than synthesized. Post-invocation ambiguity is `AckUnknown`; unresolved recovery blocks progress. The topology remains main-only, unsharded, one resident writer and one live writer process, with no fresh reads or generation GC. The private **B2a unbounded retain-all profile is active**: OmniGraph imposes no retained-byte, object-count, file-count, or history quota and never deletes a canonical durable `_mem_wal` object. Lance may clean only its losing `.binpb.tmp.` atomic-CAS staging. Complete and partial unreferenced generation residue stays non-authoritative and untouched through recovery/reopen; provider exhaustion is loud. The core remains crate-private behind feature-gated, doc-hidden test seams. The graph-scoped `stream_ingest` / `stream_manage` Cedar vocabulary is registered, and embedded `Omnigraph::stream_status` exposes a durable manifest-only projection. V11's protocol-v2 checked cluster authority remains unchanged: `ENABLED` carries its active fold delegation, `DISABLING` carries a drain-only continuation plan, and `RETIRED` is receipt/cut-bound and fail-closed. Recovery-v13 remains exactly `StreamProfileChange`. Profile apply enforces `stream_manage` under both the currently applied and desired graph policies through the cluster-state CAS. Enabled-profile Mutation/Load/delete require the exact checked served runtime, while BranchMerge remains closed under both `ENABLED` and `DISABLING` even with that runtime. The narrow cluster-only `stream retire-for-rebuild plan|confirm` command is the sole active retirement surface and permanently freezes a verified `WITHDRAWN | DEAD_LETTERED` cut for export/rebuild. Checked offline disable is the sole supported production quiescence owner. F7a activates one graph-native mixed node/edge served-ingest bridge at `POST /graphs/{graph_id}/stream/ingest` plus remote client/CLI/OpenAPI parity. A strong graph-authority ETag gates body ownership; the engine resolves declarations, lazily prepares private lanes, and returns only ordered graph-logical results. Public lane enrollment, general resume/abort, rebind, and lifecycle/maintenance/status transports remain inactive; exact-terminal served export remains the F6b5 exception. The stopped/offline cluster-only `stream block show|correct` and `stream dead-letter list|export` surfaces are the narrow active inspection/correction exceptions; no hidden seam is a supported operator escape hatch. +- **RFC-026 F7a lifecycle boundary**: graph firehose ingress can lazily prepare an absent declaration or continue an existing `OPEN` lane. It never implicitly resumes a `SEALED` lane; that state returns graph-level `stream_authority_changed` until a public checked resume exists, or the operator takes the documented retire/export/rebuild exit. - **RFC-026 retired-export provenance**: the immutable retirement receipt binds the whole frozen graph cut. Every retired JSONL export pairs that root receipt with a selected `branch_member` witness containing the canonical branch name, exact Lance branch identifier, graph head, manifest version, `table_witness_digest`, and a recomputable `branch_member_digest`. The export's `source_schema_ir_hash`, exact `ordered_branch_member_digests`, and `selected_member_index` let load prove membership and recompute the receipt's `export_cut_digest`. The source schema hash commits the retired source cut; it is not required to equal the fresh target graph identity, whose compatibility remains ordinary loader validation. Load imports no live stream authority. - **RFC-026 B2 preprocessing bound**: before blob materialization or canonical encoding, the private adapter reserves a 128-MiB worst-case envelope (original 32-MiB Arrow row + possible 32-MiB replacement + 64-MiB canonical payload) and an inflight slot. The private profile admits exactly two envelopes (256 MiB root-wide), preserving the two-caller stale-authority race without unbounded preprocessing. The slot transfers into queued/worker ownership; scratch releases after digest derivation. Pressure fails effect-free as typed `stream_b2_preprocessing_bytes`; this is process-memory admission, not retained-storage GC or quota. - **HTTP server**: Axum + utoipa OpenAPI, bearer auth (SHA-256 hashed, optional AWS Secrets Manager). Cedar policy enforcement is engine-wide — every `_as` writer calls `Omnigraph::enforce(action, scope, actor)`, so HTTP, CLI, and embedded SDK consumers all hit the same gate. **Cluster-only boot** (RFC-011): the server always boots from a cluster directory (`--cluster `, RFC-005) and serves N graphs (N ≥ 1) under multi-graph routes (`/graphs/{graph_id}/...` + read-only `GET /graphs` enumeration); there are no single-graph flat routes and no positional-URI boot. Per-graph + server-level Cedar policies. Runtime add/remove (`POST /graphs`, `DELETE /graphs/{id}`) is not exposed — operators run `cluster apply` and restart. - **CLI** with two-surface config (RFC-007/008): the team-owned cluster directory (`cluster.yaml`) plus the per-operator `~/.omnigraph/config.yaml` (servers, clusters, credentials, actor, profiles, aliases, defaults). Graphs are addressed via `--store`/`--server`/`--cluster`/`--profile`/operator defaults (RFC-011). Multi-format output (json/jsonl/csv/kv/table). -- **RFC-026 hidden F4/F5/F6 status**: F4 owns the complete caller-shaped authorized JSON/NDJSON and bodyless lazy-enrollment prepare path. Format-neutral F5a adds one weakly root-scoped automatic supervisor for `OPEN` lanes. Detached ownership schedules a coalesced timer wake immediately after physical put invocation, including caller cancellation or eventual `AckUnknown`; passive readiness discards no-effect wakes, and capacity pressure makes the same entry urgent. Cold start derives backlog from manifest plus authenticated MemWAL authority, while finite rounds visit nodes before edges with a carried round-robin cursor inside each immutable-identity cohort. F5b0 extends that exact-`ENABLED`, checked-runtime owner to restart and continue unblocked `DRAINING(goal = SEALED)` lanes through the existing recovery-v14 quiesce adapter. The same format-neutral cut makes checked offline `cluster apply` publish `DISABLING` before work, derive its finite lane set from that manifest cut, visit nodes then edges in immutable-identity order one lane at a time, continue `OPEN` and goal-`SEALED` drains, and recovery-v14-adopt an existing `OPEN_AFTER_FOLD` drain without minting a second occurrence. A selected `DataBlock` parks apply loudly at the durable `DISABLING` revision; stopped/offline correction followed by the same apply retry resumes the stored plan. F5b adds the v19/recovery-v21 terminal path: deterministic conflict partitioning can publish visible winners and one bounded dead-letter object, including a marker-only base transaction when every candidate is diverted; exact retries return the current terminal result, while a fresh ordinary successor naming that terminal predecessor restores `PRESENT`. Stopped/offline `cluster stream dead-letter list|export` inspects only manifest-selected current authority, and retirement now accepts `WITHDRAWN | DEAD_LETTERED`. The resident and offline owners retain their non-overlap boundary. After listener bind the cluster server starts every selected graph supervisor; graceful shutdown fences the root MemWAL opportunity exclusively and then the profile gate exclusively, drops both before joining the drivers concurrently under one bounded deadline and retains any timed-out live owner. F6a adds one typed failpoints-only process-local advisory driver snapshot and one hidden in-process composed acceptance spanning prepare, NDJSON, automatic mixed terminal fold, stopped/offline list/export, corrected successor, driver restart, clean shutdown ownership, and offline disable. F6b1 adds a distinct lower/engine checked served-export authority and one doc-hidden, move-only `StreamExportCut`. A managed exact `DISABLED | RETIRED` row binds directly; when the streaming row was legitimately removed, exact graph/state evidence binds only if the engine proves `RETIRED` or enrolled `DISABLED`. Retirement confirmation CAS-converges a managed row to `RETIRED` and its exact revision, while refresh preserves declaration identity and treats `RETIRED` as satisfying `streaming: false`. Ambient enrolled `DISABLED` export refuses. The receipt-verified ambient `RETIRED` direct-export bridge remains, while F6b5 routes served export through the checked cut. Checked capture non-waitingly owns the sole root slot, closes profile/admission/schema/branch/token/table gates, prevalidates filters and terminal authority, and freezes the accepted catalog, selected snapshot's exact Lance table versions, and retired provenance. It then drops all writer gates while the cut retains its checked process authority and root slot through consuming output, so a later writer cannot retarget the bytes. Branch create/create-from/delete, cleanup, schema apply, and supported whole-root deletion acquire the same slot non-waitingly, excluding path/version reuse for the cut's lifetime. The temporary `RETIRED` bridge also retains that slot through output. Refusal is before output; a storage or writer failure after output starts remains that stream error. Pending triggers are not a durable backlog, stopped driver state is not offline authority, and public durable `StreamStatus` remains manifest-only. F6b1 changed no format or recovery grammar; F6b5 subsequently activated the existing HTTP/remote-client/CLI/OpenAPI export route without activating a row or lifecycle-control route. Public status/driver transport and every public row/lifecycle transport remain inactive; F6b2 now has green SIGTERM/shared shutdown, sequential OS-process recovery, frozen-round node/edge fairness, rebind/re-enable/reopen/resume, combined maintenance, fresh-target import, and legacy writer-refusal cells and is implemented. F6b3 adds exact manifest-selected token-index coverage probes plus a fixed-cardinality cost harness: zero-lane profile cycles grow immutable token-ledger receipt history while graph-manifest history also advances, then one enrolled conflicting key produces one logical terminal entry at every depth. The fast local 1/8-cycle cell and ignored local/configured-RustFS 1/8/32/128 sweeps record per-sample coverage, page bytes, and cumulative advisory whole-process RSS. Fresh-handle hit/miss plus the first terminal page and same-handle warm hit/miss plus repeat terminal pages report token-read counts, total table-store read bytes, manifest reads/bytes, adapter-operation counts, and per-sample warm/repeat p50 plus max-of-eight; graph open precedes those windows, so this is not a cold-provider-cache claim. The instrument does not query receipt keys and fails if a measured operation writes authority, touches MemWAL/base rows, lists prefixes, or fetches the dead-letter payload object. This remains the historical F6b3 uncovered-tail evidence. F6b7 adds a paired failpoints-only content-identical selected-index cut, current-token and profile-receipt lookup measurements, and maintenance-cost accounting. Its configured-RustFS result is a bounded NO-GO only for the uncompacted profile-cycle fixture, so it schedules no standalone production reconciler. F6b4 closes production dead-letter envelope/peak-RSS evidence. F6b5 adds pre-header checked cut capture, incremental exact-version scans, strict 64-KiB chunks, a two-chunk queue, complete per-response/process queue reservation, body-plus-producer lease ownership, terminal-frame cut retention, backpressure, and disconnect-safe release. F6b6 implements one engine-internal checked read-only operational cut: exact profile-specific authority; one writer-unfenced immutable token/base, bounded-sample, coverage, and lifecycle-ledger preflight; then short root/profile/lane fences for mutable physical/recovery witnesses; an explicitly advisory driver projection; typed movement/busy refusal; a complete pending-sidecar inventory inside a hard advisory envelope of 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, 32 MiB per sidecar body, and 32 MiB of cumulative bodies; and honest unavailable states for exactly recovery-owned physical movement, cold replay, flushed LWW accounting, and exact oldest-uncovered age. Exceeding any envelope bound is a typed resource refusal, never partial status. It neither heals recovery nor opens/claims/seals/folds a writer; the public status remains manifest-only. F6b8 separately closes resume-to-driver ownership without a format change: the root producer permit transfers through detached installation, an urgent trigger is armed before release, and exact empty-owner housekeeping precedes the unchanged node-before-edge round. No active path produces reserved `AuthorityBlock`, so its repair remains fail-closed. -- **RFC-026 F6b2/F6b8 implemented acceptance scope**: this no-format slice has green Unix `SIGTERM` through the shared graceful-shutdown path, sequential OS-process recovery, a frozen-round node/edge non-starvation proof, terminal-disable physical rebind → re-enable → reopen → explicit resume → exactly-once ingest/fold, combined `quiesce → EnsureIndices → Optimize → resume`, checked-cut fresh-target import, and legacy Mutation/Load/delete, `load_file`, and corresponding `_as` refusal under `ENABLED` and interrupted `DISABLING`. Resident-producing served puts acquire bounded preprocessing/inflight → root MemWAL opportunity shared → profile shared → table admission; the driver holds root opportunity exclusive across one frozen round and then takes profile/admission per candidate. Producer and round permits retain the `MemWalWorkerRegistry` `Arc`, preventing weak-root fence ABA. Shutdown fences root exclusive and then profile exclusive, drops both, and joins the driver. F6b8 closes the empty-resume handoff: resume transfers its non-clone root producer permit into detached installation and retained retirement, arms an urgent trigger before release, and the driver retires only exact empty owners under lane-exclusive authority before preserving the ordinary node-before-edge round. Driver-first, caller-cancelled resume-first, cross-lane slot reuse, and clean-shutdown cells are green; the broader post-claim install/retirement-failure matrix remains later F6 work. Productive SchemaApply remains refused on enrolled graphs; EXP schema changes use checked export/init/load into a fresh graph, while physical rebind preserves accepted schema. F6b7 now owns the paired uncovered/reconciled token decision instrument; public operational-status transport and served row/lifecycle/maintenance SDK/HTTP/CLI/OpenAPI parity remain later work; exact-terminal served export is active, and F6b6's checked operational-status core is engine-internal. +- **RFC-026 hidden F4/F5/F6 status**: F4 owns the complete caller-shaped authorized JSON/NDJSON and bodyless lazy-enrollment prepare path. Format-neutral F5a adds one weakly root-scoped automatic supervisor for `OPEN` lanes. Detached ownership schedules a coalesced timer wake immediately after physical put invocation, including caller cancellation or eventual `AckUnknown`; passive readiness discards no-effect wakes, and capacity pressure makes the same entry urgent. Cold start derives backlog from manifest plus authenticated MemWAL authority, while finite rounds visit nodes before edges with a carried round-robin cursor inside each immutable-identity cohort. F5b0 extends that exact-`ENABLED`, checked-runtime owner to restart and continue unblocked `DRAINING(goal = SEALED)` lanes through the existing recovery-v14 quiesce adapter. The same format-neutral cut makes checked offline `cluster apply` publish `DISABLING` before work, derive its finite lane set from that manifest cut, visit nodes then edges in immutable-identity order one lane at a time, continue `OPEN` and goal-`SEALED` drains, and recovery-v14-adopt an existing `OPEN_AFTER_FOLD` drain without minting a second occurrence. A selected `DataBlock` parks apply loudly at the durable `DISABLING` revision; stopped/offline correction followed by the same apply retry resumes the stored plan. F5b adds the v19/recovery-v21 terminal path: deterministic conflict partitioning can publish visible winners and one bounded dead-letter object, including a marker-only base transaction when every candidate is diverted; exact retries return the current terminal result, while a fresh ordinary successor naming that terminal predecessor restores `PRESENT`. Stopped/offline `cluster stream dead-letter list|export` inspects only manifest-selected current authority, and retirement now accepts `WITHDRAWN | DEAD_LETTERED`. The resident and offline owners retain their non-overlap boundary. After listener bind the cluster server starts every selected graph supervisor; graceful shutdown fences the root MemWAL opportunity exclusively and then the profile gate exclusively, drops both before joining the drivers concurrently under one bounded deadline and retains any timed-out live owner. F6a adds one typed failpoints-only process-local advisory driver snapshot and one hidden in-process composed acceptance spanning prepare, NDJSON, automatic mixed terminal fold, stopped/offline list/export, corrected successor, driver restart, clean shutdown ownership, and offline disable. F6b1 adds a distinct lower/engine checked served-export authority and one doc-hidden, move-only `StreamExportCut`. A managed exact `DISABLED | RETIRED` row binds directly; when the streaming row was legitimately removed, exact graph/state evidence binds only if the engine proves `RETIRED` or enrolled `DISABLED`. Retirement confirmation CAS-converges a managed row to `RETIRED` and its exact revision, while refresh preserves declaration identity and treats `RETIRED` as satisfying `streaming: false`. Ambient enrolled `DISABLED` export refuses. The receipt-verified ambient `RETIRED` direct-export bridge remains, while F6b5 routes served export through the checked cut. Checked capture non-waitingly owns the sole root slot, closes profile/admission/schema/branch/token/table gates, prevalidates filters and terminal authority, and freezes the accepted catalog, selected snapshot's exact Lance table versions, and retired provenance. It then drops all writer gates while the cut retains its checked process authority and root slot through consuming output, so a later writer cannot retarget the bytes. Branch create/create-from/delete, cleanup, schema apply, and supported whole-root deletion acquire the same slot non-waitingly, excluding path/version reuse for the cut's lifetime. The temporary `RETIRED` bridge also retains that slot through output. Refusal is before output; a storage or writer failure after output starts remains that stream error. Pending triggers are not a durable backlog, stopped driver state is not offline authority, and public durable `StreamStatus` remains manifest-only. F6b1 changed no format or recovery grammar; F6b5 subsequently activated the existing HTTP/remote-client/CLI/OpenAPI export route. F7a now activates only the graph-native served row route and remote command over the same hidden runtime, with no new coordinator, format, or recovery grammar. Public status/driver and lifecycle transports remain inactive; F6b2 now has green SIGTERM/shared shutdown, sequential OS-process recovery, frozen-round node/edge fairness, rebind/re-enable/reopen/resume, combined maintenance, fresh-target import, and legacy writer-refusal cells and is implemented. F6b3 adds exact manifest-selected token-index coverage probes plus a fixed-cardinality cost harness: zero-lane profile cycles grow immutable token-ledger receipt history while graph-manifest history also advances, then one enrolled conflicting key produces one logical terminal entry at every depth. The fast local 1/8-cycle cell and ignored local/configured-RustFS 1/8/32/128 sweeps record per-sample coverage, page bytes, and cumulative advisory whole-process RSS. Fresh-handle hit/miss plus the first terminal page and same-handle warm hit/miss plus repeat terminal pages report token-read counts, total table-store read bytes, manifest reads/bytes, adapter-operation counts, and per-sample warm/repeat p50 plus max-of-eight; graph open precedes those windows, so this is not a cold-provider-cache claim. The instrument does not query receipt keys and fails if a measured operation writes authority, touches MemWAL/base rows, lists prefixes, or fetches the dead-letter payload object. This remains the historical F6b3 uncovered-tail evidence. F6b7 adds a paired failpoints-only content-identical selected-index cut, current-token and profile-receipt lookup measurements, and maintenance-cost accounting. Its configured-RustFS result is a bounded NO-GO only for the uncompacted profile-cycle fixture, so it schedules no standalone production reconciler. F6b4 closes production dead-letter envelope/peak-RSS evidence. F6b5 adds pre-header checked cut capture, incremental exact-version scans, strict 64-KiB chunks, a two-chunk queue, complete per-response/process queue reservation, body-plus-producer lease ownership, terminal-frame cut retention, backpressure, and disconnect-safe release. F6b6 implements one engine-internal checked read-only operational cut: exact profile-specific authority; one writer-unfenced immutable token/base, bounded-sample, coverage, and lifecycle-ledger preflight; then short root/profile/lane fences for mutable physical/recovery witnesses; an explicitly advisory driver projection; typed movement/busy refusal; a complete pending-sidecar inventory inside a hard advisory envelope of 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, 32 MiB per sidecar body, and 32 MiB of cumulative bodies; and honest unavailable states for exactly recovery-owned physical movement, cold replay, flushed LWW accounting, and exact oldest-uncovered age. Exceeding any envelope bound is a typed resource refusal, never partial status. It neither heals recovery nor opens/claims/seals/folds a writer; the public status remains manifest-only. F6b8 separately closes resume-to-driver ownership without a format change: the root producer permit transfers through detached installation, an urgent trigger is armed before release, and exact empty-owner housekeeping precedes the unchanged node-before-edge round. No active path produces reserved `AuthorityBlock`, so its repair remains fail-closed. +- **RFC-026 F6b2/F6b8 implemented acceptance scope**: this no-format slice has green Unix `SIGTERM` through the shared graceful-shutdown path, sequential OS-process recovery, a frozen-round node/edge non-starvation proof, terminal-disable physical rebind → re-enable → reopen → explicit resume → exactly-once ingest/fold, combined `quiesce → EnsureIndices → Optimize → resume`, checked-cut fresh-target import, and legacy Mutation/Load/delete, `load_file`, and corresponding `_as` refusal under `ENABLED` and interrupted `DISABLING`. Resident-producing served puts acquire bounded preprocessing/inflight → root MemWAL opportunity shared → profile shared → table admission; the driver holds root opportunity exclusive across one frozen round and then takes profile/admission per candidate. Producer and round permits retain the `MemWalWorkerRegistry` `Arc`, preventing weak-root fence ABA. Shutdown fences root exclusive and then profile exclusive, drops both, and joins the driver. F6b8 closes the empty-resume handoff: resume transfers its non-clone root producer permit into detached installation and retained retirement, arms an urgent trigger before release, and the driver retires only exact empty owners under lane-exclusive authority before preserving the ordinary node-before-edge round. Driver-first, caller-cancelled resume-first, cross-lane slot reuse, and clean-shutdown cells are green; the broader post-claim install/retirement-failure matrix remains later F6 work. Productive SchemaApply remains refused on enrolled graphs; EXP schema changes use checked export/init/load into a fresh graph, while physical rebind preserves accepted schema. F6b7 now owns the paired uncovered/reconciled token decision instrument; public operational-status and lifecycle/maintenance SDK/HTTP/CLI/OpenAPI parity remain later work; F7a graph-native served row ingest and exact-terminal served export are active, and F6b6's checked operational-status core is engine-internal. - **RFC-026 F6b3/F6b7 token-index evidence**: failpoints-only instruments compare manifest-selected current-token and profile-receipt lookups before and after a content-identical selected-index refresh. The bounded RustFS NO-GO schedules no @@ -282,7 +283,7 @@ omnigraph policy explain --cluster ./company-brain --graph knowledge --actor act | Cleanup (`cleanup_old_versions`) | ✅ | `omnigraph cleanup` derives requested `--keep` / `--older-than` cutoffs from each table's available versions; Lance refs plus OmniGraph's live-lazy-branch and recovery floors may retain additional versions. It fails closed on unopenable pins, recovery intent, or uncovered main-table HEAD drift | | BTREE / inverted (FTS) / vector indexes | ✅ | `@index`/`@key` declares intent; the physical index is derived state that never fails a logical op. Built per column through one chokepoint (`build_indices_on_dataset_for_catalog`, type-dispatched by `node_prop_index_kind`: enum + orderable scalar → BTREE, free-text String → FTS, Vector → vector); idempotent; lazy across branches. **Schema apply and mutation/load build no indexes inline**: they publish only logical data/schema effects, leaving physical intent pending. `ensure_indices` first runs the roll-forward-only recovery barrier before base capture or planning, then materializes every declared-but-missing artifact for one table through one staged mixed CreateIndex transaction under its identity-bearing recovery-v9 authority/lineage/delta envelope; it continues to report untrainable Vector columns as pending. `Armed` is rollback-only, `EffectsConfirmed` rolls forward only while captured authority holds, and first-touch refs carry exact identity. The private v14 checked-runtime main-only seam instead uses recovery-v16 when any productive table is enrolled: all such lanes must be exactly `SEALED`, and one CAS publishes every table pointer with its recomputed proof and next lifecycle revision. V15 adds the separate recovery-v17 checked `SEALED` Optimize bridge for bounded internally committing compaction/index-maintenance effects. Neither path creates a token receipt or caller operation occurrence; ambient EnsureIndices and Optimize remain refused for enrolled tables. | | Strict insert / upsert ingestion | ✅ transaction conflict filters + uncommitted fragment staging | Internal schema v6 introduced the explicit logical mode and v9 preserves it. General strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint the same optional certificate only from its completed effect statistics. Mutation/Load remains one transaction per table, capped before arm at 8,192 rows / 32 MiB. BranchMerge's proven all-new route accepts only a complete certificate chain plus final source/target native-incarnation checks; it stages bounded fragments with `InsertBuilder`, commits them as exact-`id` filtered `Update` transactions, and performs zero target preflights, target merge joins, or committed Appends. Missing or malformed proof falls back to the general ordered diff. Raw Lance writers are outside the supported graph-writer topology, and the certificate is an internal, non-cryptographic capability. The final five-pair production gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes). | -| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v9 introduced exact identity-keyed physical binding, current-HEAD witness, lifecycle state-v2, per-shard epoch floor, stream-config v3, and the manifest-selected `_stream_tokens.lance` pointer; v10 added the graph-global `stream_profile` singleton and v11 added checked protocol-v2 profile authority. V12 activates lifecycle-v3 fixed-size binding/management/claim chain references plus authenticated WAL-tail authority. Its hidden recovery-v14 path enrolls one empty unsharded shard, recovery-covers every cold writer claim, authenticates each bounded WAL suffix and full active-generation LWW projection, folds exact base/token participants, and quiesces empty or non-empty lanes through restartable `OPEN → DRAINING → SEALED`. V13's recovery-v15 path adds revision-fenced `SEALED → OPEN` resume and guarded `DRAINING → OPEN` abort, claiming a higher epoch and publishing exact terminal claim/management receipts. V14's recovery-v16 path adds the checked-runtime, main-only, same-binding `SEALED` EnsureIndices bridge; v15's recovery-v17 path adds the distinct checked `SEALED` Optimize bridge; v16's recovery-v18 path adds crate-private physical rebind from one exact `SEALED` binding scope to a fresh empty scope that remains `SEALED`; v17's recovery-v19 path adds irreversible stopped/offline authority retirement for an exact `DISABLED`, all-`SEALED`, settled-recovery graph with at least one current `WITHDRAWN` token; v18's recovery-v20 path adds stopped/offline, exact-cut `DataBlock` correction with durable correction/management receipts and explicit `WITHDRAWN` sequencing authority; v19's recovery-v21 path adds deterministic bounded mixed/all-diverted terminal folds, current token-schema-v3 `DEAD_LETTERED`, and retirement over `WITHDRAWN | DEAD_LETTERED`. A restart after terminal claim reclaims with a higher epoch; a restart after physical seal reuses the receipt-bound flushed cut instead of invalidating it. The private B1 worker remains one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB of logical dense-slice Arrow data with watcher-plus-post-fence acknowledgement; backing-buffer capacity and physical RSS are not admission authority. Private B2 admission adds canonical payload/token digests, trusted hidden metadata, same-key compare-and-chain/idempotency classification, same-generation overlays, and authority recapture after shared admission. The private B2a profile remains unbounded retain-all: no OmniGraph byte/object/file/history quota, no canonical durable `_mem_wal` deletion, typed provider failure, and inert retained orphan residue. The 1/8/32/128 local/RustFS instrument remains advisory. It performs no generation GC or fresh reads. Profile change is cluster-apply-only under explicit offline confirmation and the persisted state lock; `stream_manage` is checked against both the currently applied and desired graph policies. Enabled-profile Mutation/Load/delete require the exact checked cluster serving runtime, while BranchMerge remains closed under both `ENABLED` and `DISABLING` even with that runtime. The exact-`ENABLED` resident owner now also continues unblocked goal-`SEALED` drains, while checked offline disable persists `DISABLING`, drains its finite manifest-derived node-before-edge lane cut serially, adopts `OPEN_AFTER_FOLD` through recovery-v14, and parks on `DataBlock` until correction plus apply retry. `RETIRED` decodes fail-closed and leaves only query/status/export on the source. F6b1's hidden checked path re-proves terminal authority and freezes exact export versions. Ordinary enrolled `DISABLED` ambient export refuses; the receipt-verified ambient `RETIRED` direct-export bridge remains, while F6b5 routes served export through the checked cut. Both paths serialize against destructive root control with the same nonwaiting slot. The cluster-only `stream retire-for-rebuild plan|confirm` handshake is active. Checked offline disable is the sole supported production quiescence owner. F6a adds a failpoints-only advisory driver snapshot and hidden in-process candidate-runtime composition without changing public durable status. Public production enrollment/row ingestion, general resume/abort, rebind, and lifecycle/maintenance/status transport contracts remain inactive; exact-terminal served HTTP/remote-client/CLI/OpenAPI export is the narrow F6b5 exception, while cluster-only `stream block show|correct` and `stream dead-letter list|export` are the active correction/inspection surfaces. F6b3 adds exact-selected uncovered-tail current-token hit/miss and terminal-page cost evidence without a format or production surface. F6b7 adds the paired failpoints-only selected-index refresh and receipt-key/current-token decision evidence without format, recovery, or production behavior. F6b5 performs pre-header authority/filter/limit validation, incrementally scans exact pinned Lance versions with approximate targets, emits independently owned 64-KiB chunks through a two-chunk queue, reserves the complete per-response queue envelope from one true process-wide budget, and retains both the queue lease and move-only cut through disconnect or a terminal frame. F6b7's uncompacted-profile-cycle bounded NO-GO schedules no standalone production token-index reconciler; public status and remaining guardrail acceptance stay open. | +| MemWAL streaming foundation | ✅ MemWAL system index, durable WAL generations, epoch-fenced shard writer | Internal schema v9 introduced exact identity-keyed physical binding, current-HEAD witness, lifecycle state-v2, per-shard epoch floor, stream-config v3, and the manifest-selected `_stream_tokens.lance` pointer; v10 added the graph-global `stream_profile` singleton and v11 added checked protocol-v2 profile authority. V12 activates lifecycle-v3 fixed-size binding/management/claim chain references plus authenticated WAL-tail authority. Its hidden recovery-v14 path enrolls one empty unsharded shard, recovery-covers every cold writer claim, authenticates each bounded WAL suffix and full active-generation LWW projection, folds exact base/token participants, and quiesces empty or non-empty lanes through restartable `OPEN → DRAINING → SEALED`. V13's recovery-v15 path adds revision-fenced `SEALED → OPEN` resume and guarded `DRAINING → OPEN` abort, claiming a higher epoch and publishing exact terminal claim/management receipts. V14's recovery-v16 path adds the checked-runtime, main-only, same-binding `SEALED` EnsureIndices bridge; v15's recovery-v17 path adds the distinct checked `SEALED` Optimize bridge; v16's recovery-v18 path adds crate-private physical rebind from one exact `SEALED` binding scope to a fresh empty scope that remains `SEALED`; v17's recovery-v19 path adds irreversible stopped/offline authority retirement for an exact `DISABLED`, all-`SEALED`, settled-recovery graph with at least one current `WITHDRAWN` token; v18's recovery-v20 path adds stopped/offline, exact-cut `DataBlock` correction with durable correction/management receipts and explicit `WITHDRAWN` sequencing authority; v19's recovery-v21 path adds deterministic bounded mixed/all-diverted terminal folds, current token-schema-v3 `DEAD_LETTERED`, and retirement over `WITHDRAWN | DEAD_LETTERED`. A restart after terminal claim reclaims with a higher epoch; a restart after physical seal reuses the receipt-bound flushed cut instead of invalidating it. The private B1 worker remains one root-singleflight, no-roll generation capped at 8,192 rows / 32 MiB of logical dense-slice Arrow data with watcher-plus-post-fence acknowledgement; backing-buffer capacity and physical RSS are not admission authority. Private B2 admission adds canonical payload/token digests, trusted hidden metadata, same-key compare-and-chain/idempotency classification, same-generation overlays, and authority recapture after shared admission. The private B2a profile remains unbounded retain-all: no OmniGraph byte/object/file/history quota, no canonical durable `_mem_wal` deletion, typed provider failure, and inert retained orphan residue. The 1/8/32/128 local/RustFS instrument remains advisory. It performs no generation GC or fresh reads. Profile change is cluster-apply-only under explicit offline confirmation and the persisted state lock; `stream_manage` is checked against both the currently applied and desired graph policies. Enabled-profile Mutation/Load/delete require the exact checked cluster serving runtime, while BranchMerge remains closed under both `ENABLED` and `DISABLING` even with that runtime. The exact-`ENABLED` resident owner now also continues unblocked goal-`SEALED` drains, while checked offline disable persists `DISABLING`, drains its finite manifest-derived node-before-edge lane cut serially, adopts `OPEN_AFTER_FOLD` through recovery-v14, and parks on `DataBlock` until correction plus apply retry. `RETIRED` decodes fail-closed and leaves only query/status/export on the source. F6b1's hidden checked path re-proves terminal authority and freezes exact export versions. Ordinary enrolled `DISABLED` ambient export refuses; the receipt-verified ambient `RETIRED` direct-export bridge remains, while F6b5 routes served export through the checked cut. Both paths serialize against destructive root control with the same nonwaiting slot. The cluster-only `stream retire-for-rebuild plan|confirm` handshake is active. Checked offline disable is the sole supported production quiescence owner. F6a adds a failpoints-only advisory driver snapshot and hidden in-process candidate-runtime composition without changing public durable status. F7a activates graph-native served row ingestion with remote-client/CLI/OpenAPI parity while keeping physical lane enrollment private. Its strong graph-authority precondition is derived from existing manifest/runtime evidence, and it reuses the current lazy prepare and resident driver without a new format or coordinator. General resume/abort, rebind, and lifecycle/maintenance/status transport contracts remain inactive; exact-terminal served export remains the F6b5 exception, while cluster-only `stream block show|correct` and `stream dead-letter list|export` are the active correction/inspection surfaces. F6b3 adds exact-selected uncovered-tail current-token hit/miss and terminal-page cost evidence without a format or production surface. F6b7 adds the paired failpoints-only selected-index refresh and receipt-key/current-token decision evidence without format, recovery, or production behavior. F6b5 performs pre-header authority/filter/limit validation, incrementally scans exact pinned Lance versions with approximate targets, emits independently owned 64-KiB chunks through a two-chunk queue, reserves the complete per-response queue envelope from one true process-wide budget, and retains both the queue lease and move-only cut through disconnect or a terminal frame. F6b7's uncompacted-profile-cycle bounded NO-GO schedules no standalone production token-index reconciler; public status and remaining guardrail acceptance stay open. | | Vector search | ✅ | `nearest()` query op; embedding pipeline (Gemini / OpenAI clients); `@embed` in schema | | Full-text search | ✅ | `search/fuzzy/match_text/bm25` query ops | | Hybrid ranking | — | `rrf(...)` Reciprocal Rank Fusion in one runtime | diff --git a/Cargo.lock b/Cargo.lock index 00864f06..83ee2c3b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5026,6 +5026,7 @@ dependencies = [ "assert_cmd", "clap", "color-eyre", + "futures", "lance", "lance-index", "omnigraph-api-types", diff --git a/crates/omnigraph-api-types/src/lib.rs b/crates/omnigraph-api-types/src/lib.rs index bba7ec8c..f4534b0c 100644 --- a/crates/omnigraph-api-types/src/lib.rs +++ b/crates/omnigraph-api-types/src/lib.rs @@ -543,6 +543,96 @@ pub struct ExportRequest { pub table_keys: Vec, } +/// Effect-free precondition challenge for graph-native streaming ingest. +/// +/// The same value is returned as a strong `ETag` response header. Clients +/// retry the request with that tag in `If-Match`; this convenience copy keeps +/// the graph authority token distinct from the per-row sequencing +/// [`StreamIngestLineOutput::stream_token`]. +#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] +pub struct StreamIngestChallenge { + pub graph_token: String, +} + +/// Logical declaration kind selected by one graph-native stream row. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(rename_all = "snake_case")] +pub enum StreamIngestKindOutput { + Node, + Edge, +} + +/// Whether one stream result applies to a single row or blocks the graph-wide +/// remainder of the request. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(rename_all = "snake_case")] +pub enum StreamIngestScopeOutput { + Row, + Graph, +} + +/// Stable, graph-logical status vocabulary for one streaming-ingest line. +/// +/// Several private physical-authority transitions intentionally collapse to +/// `stream_authority_changed`; the transport never exposes lane, binding, +/// shard, epoch, generation, dataset, or recovery-sidecar identity. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize, ToSchema)] +#[serde(rename_all = "snake_case")] +pub enum StreamIngestStatusOutput { + Durable, + AckUnknown, + AlreadyDurable, + Withdrawn, + DeadLettered, + Invalid, + StreamInputTooLarge, + StreamAuthorityChanged, + StreamSequenceConflict, + StreamIdempotencyConflict, + StreamFoldRequired, + StreamBackpressure, + RecoveryRequired, + StreamRetryRequired, +} + +/// One ordered, newline-delimited result from graph-native streaming ingest. +/// +/// Every field is caller-logical or directly actionable retry evidence. +/// Physical table and MemWAL identities are deliberately absent. +#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] +pub struct StreamIngestLineOutput { + pub ordinal: u64, + pub status: StreamIngestStatusOutput, + pub scope: StreamIngestScopeOutput, + #[serde(skip_serializing_if = "Option::is_none")] + pub kind: Option, + #[serde(rename = "type", skip_serializing_if = "Option::is_none")] + pub type_name: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub write_id: Option, + /// Confirmed per-row sequencing token. This is not the graph-level ETag. + #[serde(skip_serializing_if = "Option::is_none")] + pub stream_token: Option, + /// Candidate token whose durability acknowledgement is unknown. + #[serde(skip_serializing_if = "Option::is_none")] + pub unconfirmed_candidate_token: Option, + /// Current per-row token returned with a sequencing or terminal outcome. + #[serde(skip_serializing_if = "Option::is_none")] + pub current_token: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub message: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub limit: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub actual: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub blocking_ordinal: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub blocking_status: Option, +} + #[derive(Debug, Clone, Deserialize, IntoParams)] pub struct SnapshotQuery { pub branch: Option, diff --git a/crates/omnigraph-cli/Cargo.toml b/crates/omnigraph-cli/Cargo.toml index a0ab134e..e4747eb5 100644 --- a/crates/omnigraph-cli/Cargo.toml +++ b/crates/omnigraph-cli/Cargo.toml @@ -24,8 +24,9 @@ color-eyre = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } serde_yaml = { workspace = true } -tokio = { workspace = true } -reqwest = { workspace = true, features = ["blocking"] } +tokio = { workspace = true, features = ["fs", "io-std", "io-util"] } +reqwest = { workspace = true, features = ["blocking", "stream"] } +futures = { workspace = true } [dev-dependencies] assert_cmd = "2" diff --git a/crates/omnigraph-cli/src/cli.rs b/crates/omnigraph-cli/src/cli.rs index c454a0a2..58bbccf4 100644 --- a/crates/omnigraph-cli/src/cli.rs +++ b/crates/omnigraph-cli/src/cli.rs @@ -16,7 +16,7 @@ pub(crate) const DEFAULT_BEARER_TOKEN_ENV: &str = "OMNIGRAPH_BEARER_TOKEN"; COMMANDS BY CAPABILITY:\n \ any — run against a graph, served (--server / --profile) or embedded (--store / a \ URI): query, mutate, load, branch, snapshot, export, commit, schema show/apply.\n \ -served — require a server (registry scope, --server/--profile only): graphs.\n \ +served — require a server: stream ingest (graph scope) and graphs (registry scope).\n \ direct — direct storage access; reject --server (init, optimize, repair, cleanup, \ schema plan, lint).\n \ control — manage or inspect a cluster (cluster via --config; policy & queries via \ @@ -247,6 +247,11 @@ pub(crate) enum Command { #[command(subcommand)] command: GraphsCommand, }, + /// Stream graph rows durably through a served firehose. + Stream { + #[command(subcommand)] + command: StreamCommand, + }, // ── Storage / local graph ops ── direct storage or local files; reject --server. /// Initialize a new graph from a schema @@ -371,6 +376,20 @@ pub(crate) enum Command { Version, } +#[derive(Debug, Subcommand)] +pub(crate) enum StreamCommand { + /// Stream newline-delimited graph rows to the served graph. + Ingest { + /// NDJSON input path, or `-` for stdin. + #[arg(long, value_name = "PATH|-", default_value = "-")] + data: PathBuf, + /// Opaque graph-ingest eligibility token. Omit it to obtain one with a + /// bodyless preflight before the input is opened. + #[arg(long, value_name = "TOKEN")] + graph_token: Option, + }, +} + #[derive(Debug, Subcommand)] pub(crate) enum ProfileCommand { /// List the profiles defined in ~/.omnigraph/config.yaml. diff --git a/crates/omnigraph-cli/src/client.rs b/crates/omnigraph-cli/src/client.rs index b887e2fd..d536ac48 100644 --- a/crates/omnigraph-cli/src/client.rs +++ b/crates/omnigraph-cli/src/client.rs @@ -22,9 +22,11 @@ //! Same one-body-two-impls collapse, less ceremony. use std::io::Write; +use std::path::Path; use color_eyre::Result; -use color_eyre::eyre::bail; +use color_eyre::eyre::{bail, eyre}; +use futures::stream; use omnigraph::db::{Omnigraph, ReadTarget}; use omnigraph_api_types::{ BranchCreateOutput, BranchCreateRequest, BranchDeleteOutput, BranchListOutput, @@ -36,7 +38,9 @@ use omnigraph_api_types::{ }; use omnigraph_compiler::catalog::Catalog; use reqwest::Method; +use reqwest::header::{ACCEPT, CONTENT_TYPE, ETAG, HeaderValue, IF_MATCH}; use serde_json::Value; +use tokio::io::{AsyncRead, AsyncReadExt}; use crate::cli::CliLoadMode; use crate::helpers::{ @@ -47,6 +51,74 @@ use crate::helpers::{ }; use crate::output::{LoadOutput, load_output_from_result, load_output_from_tables}; +const NDJSON_CONTENT_TYPE: &str = "application/x-ndjson"; +const STREAM_INPUT_CHUNK_BYTES: usize = 64 * 1024; + +fn strong_etag_from_bare_graph_token(token: &str) -> Result { + if token.is_empty() + || token.starts_with("W/") + || token + .chars() + .any(|character| character.is_whitespace() || matches!(character, '"' | ',' | '*')) + { + bail!( + "graph token must be one non-empty bare opaque token, without whitespace, quotes, \ + wildcard, or comma syntax" + ); + } + Ok(HeaderValue::from_str(&format!("\"{token}\""))?) +} + +fn validate_strong_etag(value: &HeaderValue) -> Result<()> { + let value = value + .to_str() + .map_err(|_| eyre!("stream ingest preflight returned a non-text ETag"))?; + if value.starts_with("W/") + || value.len() < 2 + || !value.starts_with('"') + || !value.ends_with('"') + { + bail!("stream ingest preflight returned a malformed or weak ETag"); + } + strong_etag_from_bare_graph_token(&value[1..value.len() - 1])?; + Ok(()) +} + +async fn stream_request_error(response: reqwest::Response) -> color_eyre::Report { + let status = response.status(); + match response.text().await { + Ok(text) => { + if let Ok(error) = serde_json::from_str::(&text) { + eyre!(error.error) + } else { + eyre!("server returned {status}: {text}") + } + } + Err(error) => eyre!("server returned {status}; failed to read error body: {error}"), + } +} + +fn streaming_request_body(reader: Box) -> reqwest::Body { + let chunks = stream::try_unfold(reader, |mut reader| async move { + let mut chunk = vec![0_u8; STREAM_INPUT_CHUNK_BYTES]; + let read = reader.read(&mut chunk).await?; + if read == 0 { + return Ok::<_, std::io::Error>(None); + } + chunk.truncate(read); + Ok(Some((chunk, reader))) + }); + reqwest::Body::wrap_stream(chunks) +} + +async fn open_stream_input(path: &Path) -> Result> { + if path == Path::new("-") { + Ok(Box::new(tokio::io::stdin())) + } else { + Ok(Box::new(tokio::fs::File::open(path).await?)) + } +} + pub(crate) enum GraphClient { /// Local engine at `uri`. Reads (`resolve()`) leave `actor` empty; /// writes (`resolve_with_policy()`) attribute the resolved actor. @@ -237,6 +309,45 @@ impl GraphClient { capability, crate::scope::ScopeFlags { profile, store, server, cluster: None, graph, uri }, )?; + Self::resolve_with_policy_scope(scope, cli_as).await + } + + /// Resolve the served graph selected by `stream ingest`. Unlike the + /// legacy data-command resolver, this surface has no flat/single-graph + /// server fallback: OmniGraph servers are cluster-only, so the route must + /// include one graph selected explicitly or by configuration. + pub(crate) async fn resolve_stream_ingest( + server: Option<&str>, + graph: Option<&str>, + cli_as: Option<&str>, + profile: Option<&str>, + store: Option<&str>, + ) -> Result { + let scope = crate::scope::resolve_scope( + &crate::operator::load_operator_config()?, + crate::planes::Capability::Served, + crate::scope::ScopeFlags { + profile, + store, + server, + cluster: None, + graph, + uri: None, + }, + )?; + if scope.graph.is_none() { + bail!( + "`stream ingest` requires one selected graph; pass --graph with \ + --server, or configure default_graph on the selected server profile" + ); + } + Self::resolve_with_policy_scope(scope, cli_as).await + } + + async fn resolve_with_policy_scope( + scope: crate::scope::ResolvedScope, + cli_as: Option<&str>, + ) -> Result { require_graph_for_multi_graph_server(&scope).await?; let (server, graph, uri) = ( scope.server.as_deref(), @@ -817,6 +928,94 @@ impl GraphClient { } } + /// `stream ingest` — send one incremental NDJSON request to a served + /// graph and copy its ordered NDJSON outcomes directly into `writer`. + /// + /// With no caller-supplied graph token, the client first makes one empty + /// request and requires the server's 428 strong-ETag challenge. Only then + /// does it open `data`, so a refused preflight cannot consume stdin or + /// touch a file. A supplied token skips the preflight. In either case the + /// input body is opened and invoked at most once: 412 and every other + /// response error are surfaced without replacing the token or replaying + /// an owned body. + pub(crate) async fn stream_ingest( + &self, + data: &Path, + graph_token: Option<&str>, + writer: &mut W, + ) -> Result<()> { + let GraphClient::Remote { + http, + base_url, + token, + } = self + else { + bail!( + "`stream ingest` requires a served graph; address it with --server \ + or a server-backed --profile" + ); + }; + + let url = remote_url(base_url, &["stream", "ingest"], &[])?; + let graph_etag = match graph_token { + Some(token) => strong_etag_from_bare_graph_token(token)?, + None => { + let preflight = + apply_bearer_token(http.request(Method::POST, url.clone()), token.as_deref()) + .header(CONTENT_TYPE, NDJSON_CONTENT_TYPE) + .header(ACCEPT, NDJSON_CONTENT_TYPE) + .send() + .await?; + if preflight.status() != reqwest::StatusCode::PRECONDITION_REQUIRED { + return Err(stream_request_error(preflight).await); + } + let etag = + preflight.headers().get(ETAG).cloned().ok_or_else(|| { + eyre!("stream ingest preflight omitted its ETag challenge") + })?; + validate_strong_etag(&etag)?; + // The ETag is authoritative. Drop rather than buffer an + // untrusted challenge body; the exact request may use a fresh + // connection. + drop(preflight); + etag + } + }; + + // This is deliberately below the successful preflight. The resulting + // Body is single-use and no response branch reconstructs it. + let body = streaming_request_body(open_stream_input(data).await?); + let request = apply_bearer_token(http.request(Method::POST, url), token.as_deref()) + .header(CONTENT_TYPE, NDJSON_CONTENT_TYPE) + .header(ACCEPT, NDJSON_CONTENT_TYPE) + .header(IF_MATCH, graph_etag) + .body(body); + let mut response = request.send().await?; + if response.status() != reqwest::StatusCode::OK { + return Err(stream_request_error(response).await); + } + let response_is_ndjson = response + .headers() + .get(CONTENT_TYPE) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.split(';').next()) + .is_some_and(|media_type| { + media_type + .trim() + .eq_ignore_ascii_case(NDJSON_CONTENT_TYPE) + }); + if !response_is_ndjson { + bail!( + "stream ingest requires an exact 200 application/x-ndjson response from the server" + ); + } + while let Some(chunk) = response.chunk().await? { + writer.write_all(&chunk)?; + } + writer.flush()?; + Ok(()) + } + /// `export` — stream the branch as JSONL into `writer`. The streaming /// shape (a `W: Write`, not a returned DTO) is why this lands in 3c /// rather than 3b. Opens WITHOUT policy (like reads), so it is reached @@ -902,6 +1101,93 @@ impl GraphClient { #[cfg(test)] mod tests { use super::*; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + use tokio::net::{TcpListener, TcpStream}; + + struct CapturedRequest { + head: String, + body: Vec, + } + + async fn ensure_buffered(stream: &mut TcpStream, bytes: &mut Vec, needed: usize) { + while bytes.len() < needed { + let mut chunk = [0_u8; 4096]; + let read = stream.read(&mut chunk).await.unwrap(); + assert_ne!(read, 0, "connection closed before the request completed"); + bytes.extend_from_slice(&chunk[..read]); + } + } + + async fn read_http_request(stream: &mut TcpStream) -> CapturedRequest { + let mut bytes = Vec::new(); + let header_end = loop { + if let Some(pos) = bytes.windows(4).position(|window| window == b"\r\n\r\n") { + break pos + 4; + } + let needed = bytes.len() + 1; + ensure_buffered(stream, &mut bytes, needed).await; + }; + let head = String::from_utf8(bytes[..header_end].to_vec()).unwrap(); + let mut remaining = bytes.split_off(header_end); + let lowercase = head.to_ascii_lowercase(); + let body = if lowercase.contains("transfer-encoding: chunked") { + let mut body = Vec::new(); + loop { + let line_end = loop { + if let Some(pos) = remaining.windows(2).position(|window| window == b"\r\n") { + break pos; + } + let needed = remaining.len() + 1; + ensure_buffered(stream, &mut remaining, needed).await; + }; + let size_text = std::str::from_utf8(&remaining[..line_end]).unwrap(); + let size = + usize::from_str_radix(size_text.split(';').next().unwrap().trim(), 16).unwrap(); + remaining.drain(..line_end + 2); + if size == 0 { + ensure_buffered(stream, &mut remaining, 2).await; + assert_eq!(&remaining[..2], b"\r\n"); + break; + } + ensure_buffered(stream, &mut remaining, size + 2).await; + body.extend_from_slice(&remaining[..size]); + assert_eq!(&remaining[size..size + 2], b"\r\n"); + remaining.drain(..size + 2); + } + body + } else { + let content_length = lowercase + .lines() + .find_map(|line| line.strip_prefix("content-length:")) + .map(|value| value.trim().parse::().unwrap()) + .unwrap_or(0); + ensure_buffered(stream, &mut remaining, content_length).await; + remaining[..content_length].to_vec() + }; + CapturedRequest { head, body } + } + + async fn write_response( + stream: &mut TcpStream, + status: &str, + headers: &[(&str, &str)], + body: &[u8], + ) { + let mut head = format!( + "HTTP/1.1 {status}\r\nContent-Length: {}\r\nConnection: close\r\n", + body.len() + ); + for (name, value) in headers { + head.push_str(name); + head.push_str(": "); + head.push_str(value); + head.push_str("\r\n"); + } + head.push_str("\r\n"); + stream.write_all(head.as_bytes()).await.unwrap(); + stream.write_all(body).await.unwrap(); + stream.shutdown().await.unwrap(); + } #[test] fn resolve_registry_is_sync_and_yields_the_bare_base_url() { @@ -917,4 +1203,222 @@ mod tests { assert_eq!(client.uri(), "http://server.invalid:9"); assert!(client.is_remote()); } + + #[tokio::test] + async fn stream_ingest_preflights_before_opening_input_and_copies_jsonl() { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base_url = format!("http://{}", listener.local_addr().unwrap()); + let temp = tempfile::tempdir().unwrap(); + let data = temp.path().join("created-after-preflight.ndjson"); + assert!(!data.exists()); + let server_data = data.clone(); + let input = b"{\"type\":\"Person\",\"data\":{\"name\":\"Alice\"}}\n".to_vec(); + let expected_input = input.clone(); + let output = + b"{\"ordinal\":0,\"status\":\"durable\"}\n{\"ordinal\":1,\"status\":\"invalid\"}\n" + .to_vec(); + let expected_output = output.clone(); + + let server = tokio::spawn(async move { + let (mut preflight, _) = listener.accept().await.unwrap(); + let request = read_http_request(&mut preflight).await; + assert!(request.head.starts_with("POST /stream/ingest HTTP/1.1")); + assert!(!request.head.to_ascii_lowercase().contains("if-match:")); + assert!(request.body.is_empty()); + tokio::fs::write(&server_data, &input).await.unwrap(); + write_response( + &mut preflight, + "428 Precondition Required", + &[ + ("Content-Type", "application/json"), + ("ETag", "\"sha256:graph-a\""), + ], + br#"{"graph_token":"sha256:graph-a"}"#, + ) + .await; + + let (mut ingest, _) = listener.accept().await.unwrap(); + let request = read_http_request(&mut ingest).await; + assert!(request.head.starts_with("POST /stream/ingest HTTP/1.1")); + let lowercase = request.head.to_ascii_lowercase(); + assert!(lowercase.contains("content-type: application/x-ndjson")); + assert!(lowercase.contains("accept: application/x-ndjson")); + assert!(lowercase.contains("if-match: \"sha256:graph-a\"")); + write_response( + &mut ingest, + "200 OK", + &[("Content-Type", "application/x-ndjson")], + &output, + ) + .await; + request.body + }); + + let client = GraphClient::Remote { + http: reqwest::Client::new(), + base_url, + token: None, + }; + let mut actual_output = Vec::new(); + client + .stream_ingest(&data, None, &mut actual_output) + .await + .unwrap(); + assert_eq!(server.await.unwrap(), expected_input); + assert_eq!(actual_output, expected_output); + } + + #[tokio::test] + async fn supplied_graph_token_is_sent_once_and_never_replaced_on_412() { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base_url = format!("http://{}", listener.local_addr().unwrap()); + let temp = tempfile::tempdir().unwrap(); + let data = temp.path().join("input.ndjson"); + let input = b"{\"type\":\"Person\",\"data\":{\"name\":\"Alice\"}}\n"; + tokio::fs::write(&data, input).await.unwrap(); + + let server = tokio::spawn(async move { + let (mut ingest, _) = listener.accept().await.unwrap(); + let request = read_http_request(&mut ingest).await; + let lowercase = request.head.to_ascii_lowercase(); + assert!(lowercase.contains("if-match: \"sha256:stale\"")); + assert_eq!(request.body, input); + write_response( + &mut ingest, + "412 Precondition Failed", + &[("Content-Type", "application/json")], + br#"{"error":"stale graph token"}"#, + ) + .await; + tokio::time::timeout(std::time::Duration::from_millis(150), listener.accept()) + .await + .is_err() + }); + + let client = GraphClient::Remote { + http: reqwest::Client::new(), + base_url, + token: None, + }; + let error = client + .stream_ingest(&data, Some("sha256:stale"), &mut Vec::new()) + .await + .unwrap_err() + .to_string(); + assert!(error.contains("stale graph token")); + assert!(server.await.unwrap(), "the client retried a consumed body"); + } + + #[tokio::test] + async fn stream_ingest_requires_exact_200_ndjson_response() { + let temp = tempfile::tempdir().unwrap(); + let data = temp.path().join("input.ndjson"); + tokio::fs::write(&data, b"{}\n").await.unwrap(); + + for (status, content_type) in [ + ("204 No Content", "application/x-ndjson"), + ("206 Partial Content", "application/x-ndjson"), + ("200 OK", "application/json"), + ] { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let base_url = format!("http://{}", listener.local_addr().unwrap()); + let server = tokio::spawn(async move { + let (mut ingest, _) = listener.accept().await.unwrap(); + let request = read_http_request(&mut ingest).await; + assert_eq!(request.body, b"{}\n"); + write_response( + &mut ingest, + status, + &[("Content-Type", content_type)], + &[], + ) + .await; + }); + let client = GraphClient::Remote { + http: reqwest::Client::new(), + base_url, + token: None, + }; + let error = client + .stream_ingest(&data, Some("sha256:graph-a"), &mut Vec::new()) + .await + .unwrap_err() + .to_string(); + if status == "200 OK" { + assert!(error.contains("exact 200 application/x-ndjson"), "{error}"); + } else { + assert!(error.contains(status.split_whitespace().next().unwrap()), "{error}"); + } + server.await.unwrap(); + } + } + + #[test] + fn graph_token_is_always_bare_and_strong_quoted() { + assert_eq!( + strong_etag_from_bare_graph_token("sha256:abc") + .unwrap() + .to_str() + .unwrap(), + "\"sha256:abc\"" + ); + for invalid in [ + "", + "*", + "W/sha256:abc", + "\"sha256:abc\"", + "sha256:a,sha256:b", + "sha256:a b", + "a\nb", + ] { + assert!(strong_etag_from_bare_graph_token(invalid).is_err()); + } + } + + #[tokio::test] + async fn invalid_graph_token_is_rejected_before_input_open_or_network() { + let client = GraphClient::Remote { + http: reqwest::Client::new(), + base_url: "http://127.0.0.1:9".to_string(), + token: None, + }; + let error = client + .stream_ingest( + Path::new("/must-not-open.ndjson"), + Some("*"), + &mut Vec::new(), + ) + .await + .unwrap_err() + .to_string(); + assert!(error.contains("graph token must be one"), "{error}"); + } + + #[tokio::test] + async fn stream_ingest_requires_and_resolves_a_selected_graph_without_network_io() { + let error = match GraphClient::resolve_stream_ingest( + Some("http://127.0.0.1:9"), + None, + None, + None, + None, + ) + .await + { + Ok(_) => panic!("stream ingest accepted a server without a selected graph"), + Err(error) => error.to_string(), + }; + assert!(error.contains("requires one selected graph"), "{error}"); + + let client = GraphClient::resolve_stream_ingest( + Some("http://127.0.0.1:9"), + Some("knowledge"), + None, + None, + None, + ) + .await + .unwrap(); + assert_eq!(client.uri(), "http://127.0.0.1:9/graphs/knowledge"); + } } diff --git a/crates/omnigraph-cli/src/main.rs b/crates/omnigraph-cli/src/main.rs index 33d51898..dbaea807 100644 --- a/crates/omnigraph-cli/src/main.rs +++ b/crates/omnigraph-cli/src/main.rs @@ -729,6 +729,24 @@ async fn main() -> Result<()> { .export(&branch, &type_names, &table_keys, &mut stdout) .await?; } + Command::Stream { command } => match command { + StreamCommand::Ingest { data, graph_token } => { + let client = client::GraphClient::resolve_stream_ingest( + cli.server.as_deref(), + cli.graph.as_deref(), + cli.as_actor.as_deref(), + cli.profile.as_deref(), + cli.store.as_deref(), + ) + .await?; + echo_write_target(cli.quiet, "stream ingest", client.uri(), true); + let stdout = io::stdout(); + let mut stdout = stdout.lock(); + client + .stream_ingest(&data, graph_token.as_deref(), &mut stdout) + .await?; + } + }, Command::Query { name, query, diff --git a/crates/omnigraph-cli/src/planes.rs b/crates/omnigraph-cli/src/planes.rs index 3e6f7da6..d4456385 100644 --- a/crates/omnigraph-cli/src/planes.rs +++ b/crates/omnigraph-cli/src/planes.rs @@ -14,7 +14,8 @@ use color_eyre::eyre::bail; use crate::cli::{ Cli, ClusterCommand, ClusterStreamCommand, Command, GraphsCommand, QueriesCommand, - SchemaCommand, StreamBlockCommand, StreamDeadLetterCommand, StreamRetireForRebuildCommand, + SchemaCommand, StreamBlockCommand, StreamCommand, StreamDeadLetterCommand, + StreamRetireForRebuildCommand, }; #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -147,18 +148,19 @@ fn flag_applies(flag: ScopeFlag, capability: Capability, cmd: &Command) -> bool let cluster_ok = accepts_cluster_addressing(cmd); let graph_ok = accepts_graph_selector(cmd); match flag { - // Served-graph addressing; `served` is the registry scope (the bare - // server), which still needs a server to talk to. + // Served addressing always needs a server. `graphs list` uses the bare + // registry scope; `stream ingest` selects one graph on that server. ScopeFlag::Server => matches!(capability, Any | Served), ScopeFlag::Cluster => cluster_ok, - // The one graph selector across scopes: a served graph (`any`), or a - // cluster graph on the verbs that take cluster addressing. `served` - // rejects it — `graphs list` IS the enumeration, and a selected graph - // would corrupt the registry URL. + // The one graph selector across scopes: a served graph (`any` or the + // served-only `stream ingest`), or a cluster graph on verbs that take + // cluster addressing. The other served-only command, `graphs list`, + // rejects it because that command IS the registry enumeration. ScopeFlag::Graph => match capability { Any => true, Direct | Control => graph_ok, - Served | Local => false, + Served => graph_ok, + Local => false, }, // `direct` refines per command: the maintenance verbs (optimize/ // repair/cleanup/schema plan/lint) resolve their target through @@ -207,12 +209,13 @@ fn flag_applies(flag: ScopeFlag, capability: Capability, cmd: &Command) -> bool } /// The capability a subcommand needs, derived from its `Plane` (the exhaustive -/// classifier) plus the one Data→Served refinement: `graphs` is remote-only. +/// classifier) plus the Data→Served refinements: graph-scoped `stream ingest` +/// and registry-scoped `graphs` are both remote-only. /// /// This reflects *current enforced behavior*, so messages stay truthful: /// `queries`/`policy` read a cluster's applied state (`Control`). pub(crate) fn command_capability(cmd: &Command) -> Capability { - if let Command::Graphs { .. } = cmd { + if matches!(cmd, Command::Graphs { .. } | Command::Stream { .. }) { return Capability::Served; } match command_plane(cmd) { @@ -237,7 +240,8 @@ pub(crate) fn command_plane(cmd: &Command) -> Plane { | Command::Snapshot { .. } | Command::Export { .. } | Command::Commit { .. } - | Command::Graphs { .. } => Plane::Data, + | Command::Graphs { .. } + | Command::Stream { .. } => Plane::Data, Command::Schema { command: SchemaCommand::Show { .. } | SchemaCommand::Apply { .. }, } => Plane::Data, @@ -339,6 +343,9 @@ pub(crate) fn command_label(cmd: &Command) -> &'static str { Command::Graphs { command } => match command { GraphsCommand::List { .. } => "graphs list", }, + Command::Stream { command } => match command { + StreamCommand::Ingest { .. } => "stream ingest", + }, } } @@ -374,9 +381,10 @@ fn accepts_graph_selector(cmd: &Command) -> bool { accepts_cluster_addressing(cmd) || matches!( cmd, - Command::Cluster { - command: ClusterCommand::Stream { .. }, - } + Command::Stream { .. } + | Command::Cluster { + command: ClusterCommand::Stream { .. }, + } ) } @@ -470,9 +478,9 @@ mod tests { // The full flag × capability contract in one place. Rows cover every // capability, both cluster_ok refinements of `direct` (optimize vs // init) and of `control` (queries vs cluster), plus the graph-only - // stream-control selectors. `served` is the registry scope: server - // addressing only — --graph/--store/--as are rejected (--graph used - // to corrupt the registry URL to /graphs//graphs). + // stream-control selectors. Served commands split by scope: `graphs` + // is registry-only, while `stream ingest` consumes --graph. Neither + // accepts --store/--as. let parse = |args: &[&str]| Cli::try_parse_from(args).unwrap().command; // (command, [server, cluster, graph, store, as, profile]) let rows = [ @@ -484,6 +492,10 @@ mod tests { parse(&["omnigraph", "graphs", "list"]), [true, false, false, false, false, true], ), + ( + parse(&["omnigraph", "stream", "ingest"]), + [true, false, true, false, false, true], + ), ( parse(&["omnigraph", "optimize", "g.omni"]), [false, true, true, true, false, true], @@ -557,9 +569,10 @@ mod tests { #[test] fn command_capability_classifies_representative_verbs() { let cap = |args: &[&str]| command_capability(&Cli::try_parse_from(args).unwrap().command); - // The one Data→Served refinement — if the `graphs` guard were deleted, - // every other assertion here would still pass. + // Both Data→Served refinements are explicit: one registry-scoped and + // one graph-scoped. assert_eq!(cap(&["omnigraph", "graphs", "list"]), Capability::Served); + assert_eq!(cap(&["omnigraph", "stream", "ingest"]), Capability::Served); assert_eq!(cap(&["omnigraph", "alias", "who"]), Capability::Local); assert_eq!( cap(&["omnigraph", "optimize", "graph.omni"]), @@ -630,6 +643,47 @@ mod tests { ); } + #[test] + fn stream_ingest_is_served_only_before_input_dispatch() { + let cli = Cli::try_parse_from([ + "omnigraph", + "stream", + "ingest", + "--store", + "file:///must-not-open.omni", + "--data", + "/must-not-open.ndjson", + ]) + .unwrap(); + let error = guard_addressing(&cli).unwrap_err().to_string(); + assert!(error.contains("`stream ingest` is a served command")); + assert!(error.contains("--store addresses")); + + let cli = Cli::try_parse_from([ + "omnigraph", + "stream", + "ingest", + "--server", + "http://server.invalid:9", + "--graph", + "knowledge", + ]) + .unwrap(); + guard_addressing(&cli).unwrap(); + match cli.command { + Command::Stream { + command: + StreamCommand::Ingest { + data, graph_token, .. + }, + } => { + assert_eq!(data, std::path::Path::new("-")); + assert_eq!(graph_token, None); + } + other => panic!("expected stream ingest, got {other:?}"), + } + } + #[test] fn policy_explain_accepts_stream_action_wire_names_and_kebab_aliases() { let parse_action = |spelling: &str| { diff --git a/crates/omnigraph-server/src/handlers.rs b/crates/omnigraph-server/src/handlers.rs index 4c70bfa2..291f290e 100644 --- a/crates/omnigraph-server/src/handlers.rs +++ b/crates/omnigraph-server/src/handlers.rs @@ -3,6 +3,7 @@ //! verbatim from lib.rs in the modularization). use super::*; +use futures::StreamExt; /// Liveness probe. /// @@ -680,6 +681,192 @@ pub(crate) async fn server_export( .into_response()) } +enum GraphIfMatch { + Missing, + Strong(String), + Malformed, +} + +fn graph_stream_if_match(headers: &axum::http::HeaderMap) -> GraphIfMatch { + let mut values = headers.get_all(IF_MATCH).iter(); + let Some(value) = values.next() else { + return GraphIfMatch::Missing; + }; + if values.next().is_some() { + return GraphIfMatch::Malformed; + } + let Ok(value) = value.to_str() else { + return GraphIfMatch::Malformed; + }; + let value = value.trim(); + if value.len() < 2 || !value.starts_with('"') || !value.ends_with('"') { + return GraphIfMatch::Malformed; + } + let opaque = &value[1..value.len() - 1]; + if opaque.is_empty() + || !opaque + .bytes() + .all(|byte| byte == b'!' || (b'#'..=b'~').contains(&byte)) + { + return GraphIfMatch::Malformed; + } + GraphIfMatch::Strong(opaque.to_string()) +} + +fn graph_stream_etag(authority_token: &str) -> std::result::Result { + HeaderValue::try_from(format!("\"{authority_token}\"")) + .map_err(|_| ApiError::internal("graph stream authority token is not a valid ETag")) +} + +fn require_graph_stream_content_type( + headers: &axum::http::HeaderMap, +) -> std::result::Result<(), ApiError> { + let accepted = headers + .get(CONTENT_TYPE) + .and_then(|value| value.to_str().ok()) + .and_then(|value| value.split(';').next()) + .is_some_and(|media_type| { + media_type + .trim() + .eq_ignore_ascii_case("application/x-ndjson") + }); + if accepted { + Ok(()) + } else { + Err(ApiError::unsupported_media_type( + "graph stream ingest requires Content-Type application/x-ndjson", + )) + } +} + +#[utoipa::path( + post, + path = "/stream/ingest", + tag = "streaming", + operation_id = "stream_ingest", + params( + ("If-Match" = Option, Header, description = "Strong ETag containing the opaque graph token returned by the 428 challenge."), + ), + request_body( + content = String, + description = "Graph-native node and edge rows, one JSON object per line. Results are emitted in caller order and the request is not atomic.", + content_type = "application/x-ndjson", + ), + responses( + (status = 200, description = "Ordered graph-native ingest results", body = api::StreamIngestLineOutput, content_type = "application/x-ndjson"), + (status = 401, description = "Unauthorized", body = ErrorOutput), + (status = 403, description = "Forbidden", body = ErrorOutput), + (status = 409, description = "Checked streaming runtime or lifecycle conflict", body = ErrorOutput), + (status = 412, description = "Malformed or stale graph token; no replacement token is disclosed", body = ErrorOutput), + (status = 413, description = "Streaming request admission capacity exhausted", body = ErrorOutput), + (status = 415, description = "Request Content-Type is not application/x-ndjson", body = ErrorOutput), + (status = 428, description = "Graph token required; retry with the strong ETag in If-Match", body = StreamIngestChallenge, + headers( + ("ETag" = String, description = "Strong ETag containing the current opaque graph token"), + ("Cache-Control" = String, description = "Always `no-store`; the challenge is authority-bearing"), + )), + (status = 503, description = "Overlapping durable recovery must resolve before ingest", body = ErrorOutput), + ), + security(("bearer_token" = [])), +)] +/// Ingest a graph-native NDJSON firehose through the checked served runtime. +/// +/// A request without `If-Match` is authorized and preflighted without polling +/// its body, then receives a 428 challenge. A malformed or stale precondition +/// receives 412 without replacement authority. An exact strong ETag transfers +/// the Axum body stream directly into the engine and streams one redacted +/// newline-delimited result per input row; neither direction is buffered as a +/// complete request or response. +pub(crate) async fn server_stream_ingest( + Extension(handle): Extension>, + actor: Option>, + headers: axum::http::HeaderMap, + body: Body, +) -> std::result::Result { + require_graph_stream_content_type(&headers)?; + let actor = actor.as_ref().map(|Extension(actor)| actor); + if handle.policy.is_none() { + // With a graph policy installed, the engine's graph-scoped check is + // the sole Cedar decision for this request. Without one the engine + // intentionally has no checker, so preserve the server's open-mode / + // authenticated-default-deny contract here before body ownership. + authorize_request( + actor, + None, + PolicyRequest { + action: PolicyAction::StreamIngest, + branch: None, + target_branch: None, + }, + )?; + } + + let supplied = graph_stream_if_match(&headers); + let precondition_supplied = !matches!(&supplied, GraphIfMatch::Missing); + // A malformed header still enters the checked engine preflight with an + // impossible token. This preserves policy/runtime ordering while keeping + // strong-ETag syntax part of the HTTP contract. + let supplied_token = match &supplied { + GraphIfMatch::Missing => None, + GraphIfMatch::Strong(token) => Some(token.as_str()), + GraphIfMatch::Malformed => Some(""), + }; + let chunks = body.into_data_stream().map(|chunk| { + chunk.map(|bytes| bytes.to_vec()).map_err(|error| { + OmniError::Io(std::io::Error::other(format!( + "graph stream request body failed: {error}" + ))) + }) + }); + let start = handle + .engine + .start_served_graph_stream_ingest_as( + actor.map_or("anonymous", |actor| actor.actor_id.as_ref()), + supplied_token, + Box::pin(chunks), + ) + .await; + let start = start + .map_err(|error| ApiError::from_graph_stream_start(error, precondition_supplied))?; + + match start { + omnigraph::db::GraphStreamIngestStart::TokenRequired { authority_token } => { + let etag = graph_stream_etag(&authority_token)?; + Ok(( + StatusCode::PRECONDITION_REQUIRED, + [ + (ETAG, etag), + (CACHE_CONTROL, HeaderValue::from_static("no-store")), + ], + Json(StreamIngestChallenge { + graph_token: authority_token, + }), + ) + .into_response()) + } + omnigraph::db::GraphStreamIngestStart::Ready(result_handle) => { + if matches!(supplied, GraphIfMatch::Malformed | GraphIfMatch::Missing) { + return Err(ApiError::internal( + "graph stream engine accepted a missing or malformed precondition", + )); + } + let results = futures::stream::try_unfold(result_handle, |mut result_handle| async { + match result_handle.recv().await { + Ok(Some(line)) => Ok(Some((Bytes::from(line), result_handle))), + Ok(None) => Ok(None), + Err(error) => Err(std::io::Error::other(error.to_string())), + } + }); + Ok(( + StatusCode::OK, + [(CONTENT_TYPE, "application/x-ndjson; charset=utf-8")], + Body::from_stream(results), + ) + .into_response()) + } + } +} + /// Shared implementation behind `POST /mutate` (canonical) and /// `POST /change` (deprecated alias). Returns the bare `ChangeOutput`; /// each route handler wraps it (the alias also attaches Deprecation diff --git a/crates/omnigraph-server/src/lib.rs b/crates/omnigraph-server/src/lib.rs index 75b0658e..4f906c23 100644 --- a/crates/omnigraph-server/src/lib.rs +++ b/crates/omnigraph-server/src/lib.rs @@ -30,15 +30,17 @@ use api::{ CommitListQuery, ErrorCode, ErrorOutput, ExportRequest, GraphInfo, GraphListResponse, HealthOutput, IngestOutput, IngestRequest, InvokeStoredQueryRequest, InvokeStoredQueryResponse, QueriesCatalogOutput, QueryRequest, ReadOutput, ReadRequest, SchemaApplyOutput, - SchemaApplyRequest, SchemaOutput, SnapshotQuery, ingest_output, schema_apply_output, - snapshot_payload, + SchemaApplyRequest, SchemaOutput, SnapshotQuery, StreamIngestChallenge, ingest_output, + schema_apply_output, snapshot_payload, }; pub use auth::{AWS_SECRET_ENV, EnvOrFileTokenSource, TokenSource, resolve_token_source}; use axum::body::{Body, Bytes}; use axum::extract::DefaultBodyLimit; use axum::extract::{Extension, OriginalUri, Path, Query, Request, State}; use axum::http::StatusCode; -use axum::http::header::{AUTHORIZATION, CONTENT_TYPE, HeaderName, HeaderValue}; +use axum::http::header::{ + AUTHORIZATION, CACHE_CONTROL, CONTENT_TYPE, ETAG, HeaderName, HeaderValue, IF_MATCH, +}; use axum::middleware::{self, Next}; use axum::response::{IntoResponse, Response}; use axum::routing::{delete, get, post}; @@ -92,6 +94,7 @@ fn hash_bearer_token(token: &str) -> BearerTokenHash { #[allow(deprecated)] handlers::server_read, handlers::server_query, handlers::server_export, + handlers::server_stream_ingest, #[allow(deprecated)] handlers::server_change, handlers::server_mutate, handlers::server_list_queries, @@ -586,8 +589,9 @@ impl AppState { ) -> Self { // Engine-layer policy gate (MR-722). With a per-graph policy // installed, every `_as` writer on `Omnigraph` calls into the - // PolicyChecker. HTTP-layer `authorize_request` is the first - // gate; engine-layer is the redundant-but-correct backstop. + // PolicyChecker. Most handlers retain an HTTP-layer first gate; + // graph firehose deliberately uses the engine's single graph-scoped + // decision so lazy private-lane work cannot create policy drift. let db = if let Some(policy) = policy_engine.as_ref() { let checker = Arc::clone(policy) as Arc; db.with_policy(checker) @@ -785,6 +789,34 @@ impl ApiError { } } + fn precondition_failed(message: impl Into) -> Self { + Self { + status: StatusCode::PRECONDITION_FAILED, + code: Some(ErrorCode::Conflict), + message: message.into(), + merge_conflicts: Vec::new(), + manifest_conflict: None, + read_set_conflict: None, + key_conflict: None, + resource_limit: None, + recovery_required: None, + } + } + + fn unsupported_media_type(message: impl Into) -> Self { + Self { + status: StatusCode::UNSUPPORTED_MEDIA_TYPE, + code: Some(ErrorCode::BadRequest), + message: message.into(), + merge_conflicts: Vec::new(), + manifest_conflict: None, + read_set_conflict: None, + key_conflict: None, + resource_limit: None, + recovery_required: None, + } + } + pub fn internal(message: impl Into) -> Self { Self { status: StatusCode::INTERNAL_SERVER_ERROR, @@ -976,11 +1008,11 @@ impl ApiError { actual, }, ), - // There is no public stream row route yet. Keep the exhaustive - // engine translation conservative until that surface defines - // structured wire fields: fold-required and strict-blocked are - // retryable logical conflicts, while an invoked-but-unconfirmed - // append is unavailable/ambiguous. + // The graph firehose converts invoked row outcomes to its ordered + // NDJSON union inside the engine. Only request-level stream errors + // reach this ordinary JSON translation: fold-required and strict- + // blocked are retryable logical conflicts, while an invoked-but- + // unconfirmed append is unavailable/ambiguous. err @ (OmniError::FoldRequired { .. } | OmniError::StreamDataBlocked { .. }) => { Self::conflict(err.to_string()) } @@ -1021,12 +1053,12 @@ impl ApiError { Self::conflict(format!("retryable storage commit conflict: {message}")) } OmniError::Io(err) => Self::internal(format!("io: {err}")), - // Engine-layer policy enforcement (MR-722). All denials and - // evaluation failures surface here as 403. The HTTP-layer - // `authorize_request` already distinguishes 401 (missing - // bearer) from 403 (policy denial), so by the time the - // engine gate fires, the bearer is valid — any failure from - // the engine is a policy outcome, not an auth one. + // Engine-layer policy enforcement (MR-722). Authentication + // middleware has already distinguished a missing/invalid bearer + // (401); policy denials and evaluation failures surface as 403. + // Most legacy handlers also perform an HTTP-layer policy check, + // while graph firehose intentionally makes the engine's one + // graph-scoped decision authoritative. OmniError::Policy(message) => Self::forbidden(message), // `Omnigraph::init` against an existing graph URI in strict // mode. Not currently HTTP-reachable (POST /graphs was @@ -1036,6 +1068,49 @@ impl ApiError { err @ OmniError::AlreadyInitialized { .. } => Self::conflict(err.to_string()), } } + + /// Translate failures before the graph firehose owns its request body. + /// + /// Generic engine errors legitimately carry table keys, lifecycle IDs, + /// recovery operation IDs, block tokens, and storage diagnostics for + /// trusted control-plane callers. None of that is part of the graph-only + /// row-ingest contract. Preserve the useful HTTP class while removing + /// every structured/private detail at this transport boundary. + fn from_graph_stream_start(err: OmniError, precondition_supplied: bool) -> Self { + if precondition_supplied && matches!(&err, OmniError::StreamingAuthorityMismatch { .. }) { + return Self::precondition_failed("graph stream ingest precondition failed"); + } + + let acknowledgement_unknown = matches!(&err, OmniError::AckUnknown { .. }); + let mut translated = Self::from_omni(err); + if acknowledgement_unknown { + // A startup fold may already have invoked storage. Keep that + // ambiguity retryable/unavailable rather than presenting it as a + // generic server fault, while still suppressing every physical + // acknowledgement coordinate below. + translated.status = StatusCode::SERVICE_UNAVAILABLE; + translated.code = None; + } + translated.message = match translated.status { + StatusCode::BAD_REQUEST => "graph stream ingest preflight was refused", + StatusCode::FORBIDDEN => "graph stream ingest is forbidden", + StatusCode::NOT_FOUND => "graph stream ingest target was not found", + StatusCode::CONFLICT => "graph stream ingest is not ready", + StatusCode::PAYLOAD_TOO_LARGE => "graph stream request capacity is unavailable", + StatusCode::SERVICE_UNAVAILABLE => { + "graph recovery must complete before stream ingest can begin" + } + _ => "graph stream ingest preflight failed", + } + .to_string(); + translated.merge_conflicts.clear(); + translated.manifest_conflict = None; + translated.read_set_conflict = None; + translated.key_conflict = None; + translated.resource_limit = None; + translated.recovery_required = None; + translated + } } fn summarize_merge_conflicts(conflicts: &[api::MergeConflictOutput]) -> String { @@ -1167,6 +1242,114 @@ mod api_error_tests { assert!(error.recovery_required.is_none()); } + #[tokio::test] + async fn graph_stream_start_errors_redact_private_engine_evidence() { + let cases = vec![ + ( + OmniError::FoldRequired { + table_key: "node:PrivatePerson".to_string(), + rows: 7, + bytes: 11, + }, + StatusCode::CONFLICT, + "PrivatePerson", + ), + ( + OmniError::StreamDataBlocked { + block_token: "private-block-token".to_string(), + }, + StatusCode::CONFLICT, + "private-block-token", + ), + ( + OmniError::RecoveryRequired { + operation_id: "private-recovery-operation".to_string(), + reason: "private recovery reason".to_string(), + }, + StatusCode::SERVICE_UNAVAILABLE, + "private-recovery-operation", + ), + ( + OmniError::StreamBindingChanged { + stable_table_id: 41, + table_incarnation_id: 43, + current_stream_incarnation_id: "private-stream-incarnation".to_string(), + }, + StatusCode::CONFLICT, + "private-stream-incarnation", + ), + ( + OmniError::ResourceLimitExceeded { + resource: "private-storage-scan-limit".to_string(), + limit: 1, + actual: 2, + }, + StatusCode::PAYLOAD_TOO_LARGE, + "private-storage-scan-limit", + ), + ( + OmniError::Lance("private object-store URI".to_string()), + StatusCode::INTERNAL_SERVER_ERROR, + "private object-store URI", + ), + ( + OmniError::AckUnknown { + stable_table_id: 41, + table_incarnation_id: 43, + enrollment_id: "private-enrollment".to_string(), + shard_id: "private-shard".to_string(), + writer_epoch: 47, + caller_ordinal_start: 0, + caller_ordinal_end: 1, + admission_attempt_id: Some("private-attempt".to_string()), + logical_write_ids: vec!["private-write".to_string()], + unconfirmed_candidate_token: Some("private-token".to_string()), + reason: "private acknowledgement reason".to_string(), + }, + StatusCode::SERVICE_UNAVAILABLE, + "private-enrollment", + ), + ( + OmniError::StreamingAuthorityMismatch { + reason: "private fold delegation".to_string(), + }, + StatusCode::PRECONDITION_FAILED, + "private fold delegation", + ), + ]; + + for (engine_error, expected_status, private_evidence) in cases { + let response = ApiError::from_graph_stream_start(engine_error, true).into_response(); + assert_eq!(response.status(), expected_status); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let error: ErrorOutput = serde_json::from_slice(&body).unwrap(); + assert!(!error.error.contains(private_evidence), "{}", error.error); + assert!(error.merge_conflicts.is_empty()); + assert!(error.manifest_conflict.is_none()); + assert!(error.read_set_conflict.is_none()); + assert!(error.key_conflict.is_none()); + assert!(error.resource_limit.is_none()); + assert!(error.recovery_required.is_none()); + } + + let response = ApiError::from_graph_stream_start( + OmniError::StreamingAuthorityMismatch { + reason: "private checked-runtime binding".to_string(), + }, + false, + ) + .into_response(); + assert_eq!(response.status(), StatusCode::CONFLICT); + let body = axum::body::to_bytes(response.into_body(), usize::MAX) + .await + .unwrap(); + let error: ErrorOutput = serde_json::from_slice(&body).unwrap(); + assert_eq!(error.error, "graph stream ingest is not ready"); + assert!(!error.error.contains("private checked-runtime binding")); + } + #[tokio::test] async fn stream_management_conflicts_serialize_as_409() { let cases = [ @@ -1278,6 +1461,7 @@ pub fn build_app(state: AppState) -> Router { let per_graph_protected = Router::new() .route("/snapshot", get(server_snapshot)) .route("/export", post(server_export)) + .route("/stream/ingest", post(server_stream_ingest)) // /read and /change are kept indefinitely for back-compat; // their handlers carry #[deprecated] so the OpenAPI operation is // flagged and their responses include RFC 9745 Deprecation + diff --git a/crates/omnigraph-server/tests/auth_policy.rs b/crates/omnigraph-server/tests/auth_policy.rs index 68e89b8d..330ffce8 100644 --- a/crates/omnigraph-server/tests/auth_policy.rs +++ b/crates/omnigraph-server/tests/auth_policy.rs @@ -4,8 +4,9 @@ use std::env; use std::fs; use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; -use axum::body::Body; +use axum::body::{Body, Bytes}; use axum::http::header::AUTHORIZATION; use axum::http::{Method, Request, StatusCode}; use omnigraph::db::{Omnigraph, ReadTarget}; @@ -67,6 +68,28 @@ async fn protected_routes_require_bearer_token() { error.code, Some(omnigraph_server::api::ErrorCode::Unauthorized) ); + + let stream_body_polled = Arc::new(AtomicBool::new(false)); + let body_probe = Arc::clone(&stream_body_polled); + let response = app + .oneshot( + Request::builder() + .uri(g("/stream/ingest")) + .method(Method::POST) + .header("content-type", "application/x-ndjson") + .body(Body::from_stream(futures::stream::once(async move { + body_probe.store(true, Ordering::SeqCst); + Ok::<_, std::io::Error>(Bytes::from_static(b"{}\n")) + }))) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::UNAUTHORIZED); + assert!( + !stream_body_polled.load(Ordering::SeqCst), + "bearer authentication must reject graph ingest before polling input" + ); } #[tokio::test(flavor = "multi_thread")] diff --git a/crates/omnigraph-server/tests/data_routes.rs b/crates/omnigraph-server/tests/data_routes.rs index e1358cc9..20b69973 100644 --- a/crates/omnigraph-server/tests/data_routes.rs +++ b/crates/omnigraph-server/tests/data_routes.rs @@ -3,9 +3,10 @@ use std::fs; use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; use std::time::Duration; -use axum::body::{Body, to_bytes}; +use axum::body::{Body, Bytes, to_bytes}; use axum::http::{Method, Request, StatusCode}; use omnigraph::db::{Omnigraph, ReadTarget}; use omnigraph::loader::LoadMode; @@ -107,6 +108,69 @@ fn export_request(type_names: Vec) -> Request { .unwrap() } +fn body_poll_probe(polled: Arc) -> Body { + Body::from_stream(futures::stream::once(async move { + polled.store(true, Ordering::SeqCst); + Ok::<_, std::io::Error>(Bytes::from_static( + b"{\"type\":\"Person\",\"data\":{\"id\":\"must-not-run\",\"age\":1},\"$stream\":{\"write_id\":\"00000000-0000-4000-8000-000000000001\",\"predecessor_token\":null}}\n", + )) + })) +} + +#[tokio::test(flavor = "multi_thread")] +async fn stream_ingest_media_and_runtime_refusals_do_not_poll_the_body() { + let (_temp, app) = app_for_loaded_graph().await; + + let wrong_media_polled = Arc::new(AtomicBool::new(false)); + let response = app + .clone() + .oneshot( + Request::builder() + .uri(g("/stream/ingest")) + .method(Method::POST) + .header("content-type", "application/json") + .body(body_poll_probe(Arc::clone(&wrong_media_polled))) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::UNSUPPORTED_MEDIA_TYPE); + assert!(!wrong_media_polled.load(Ordering::SeqCst)); + + let missing_media_polled = Arc::new(AtomicBool::new(false)); + let response = app + .clone() + .oneshot( + Request::builder() + .uri(g("/stream/ingest")) + .method(Method::POST) + .body(body_poll_probe(Arc::clone(&missing_media_polled))) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::UNSUPPORTED_MEDIA_TYPE); + assert!(!missing_media_polled.load(Ordering::SeqCst)); + + let no_runtime_polled = Arc::new(AtomicBool::new(false)); + let response = app + .oneshot( + Request::builder() + .uri(g("/stream/ingest")) + .method(Method::POST) + .header("content-type", "application/x-ndjson") + .body(body_poll_probe(Arc::clone(&no_runtime_polled))) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::CONFLICT); + assert!(!no_runtime_polled.load(Ordering::SeqCst)); + let body = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let error: ErrorOutput = serde_json::from_slice(&body).unwrap(); + assert_eq!(error.error, "graph stream ingest is not ready"); +} + #[tokio::test(flavor = "multi_thread")] async fn export_invalid_filter_refuses_before_success_headers() { let (_temp, app) = app_for_loaded_graph().await; diff --git a/crates/omnigraph-server/tests/multi_graph.rs b/crates/omnigraph-server/tests/multi_graph.rs index 67b41eb8..dfe16c77 100644 --- a/crates/omnigraph-server/tests/multi_graph.rs +++ b/crates/omnigraph-server/tests/multi_graph.rs @@ -2,12 +2,20 @@ //! Moved verbatim from tests/server.rs in the modularization. use std::fs; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::time::Duration; -use axum::body::{Body, to_bytes}; +use axum::body::{Body, Bytes, to_bytes}; +use axum::http::header::{CACHE_CONTROL, ETAG, HeaderValue, IF_MATCH}; use axum::http::{Method, Request, StatusCode}; +use futures::StreamExt; use omnigraph::db::Omnigraph; use omnigraph::loader::{LoadMode, load_jsonl}; -use omnigraph_server::api::{ChangeRequest, ErrorOutput, ExportRequest, ReadRequest}; +use omnigraph_server::api::{ + ChangeRequest, ErrorOutput, ExportRequest, QueryRequest, ReadRequest, StreamIngestChallenge, + StreamIngestKindOutput, StreamIngestLineOutput, StreamIngestStatusOutput, +}; use omnigraph_server::{AppState, build_app}; use serde_json::Value; use serial_test::serial; @@ -462,13 +470,50 @@ async fn cluster_boot_serves_applied_state() { assert_eq!(status, StatusCode::OK, "{body}"); } -#[tokio::test] -#[serial] -async fn cluster_boot_installs_enabled_stream_runtime_authority() { +fn stream_body_poll_probe(polled: Arc) -> Body { + Body::from_stream(futures::stream::once(async move { + polled.store(true, Ordering::SeqCst); + Ok::<_, std::io::Error>(Bytes::from_static(b"{}\n")) + })) +} + +fn served_graph_node_line(type_name: &str, id: &str, score: i32, write_id: &str) -> Vec { + let mut line = serde_json::to_vec(&serde_json::json!({ + "type": type_name, + "data": {"id": id, "score": score}, + "$stream": {"write_id": write_id, "predecessor_token": null}, + })) + .unwrap(); + line.push(b'\n'); + line +} + +fn served_graph_edge_line( + type_name: &str, + id: &str, + from: &str, + to: &str, + write_id: &str, +) -> Vec { + let mut line = serde_json::to_vec(&serde_json::json!({ + "edge": type_name, + "from": from, + "to": to, + "data": {"id": id}, + "$stream": {"write_id": write_id, "predecessor_token": null}, + })) + .unwrap(); + line.push(b'\n'); + line +} + +async fn enabled_stream_state( + tokens: Vec<(String, String)>, +) -> (tempfile::TempDir, AppState, Arc) { let temp = tempfile::tempdir().unwrap(); fs::write( temp.path().join("people.pg"), - "\nnode Person {\n name: String @key\n}\n", + "\nnode Person {\n score: I32\n}\nedge Knows: Person -> Person\n", ) .unwrap(); fs::write( @@ -516,13 +561,34 @@ graphs: let state = omnigraph_server::open_multi_graph_state( graphs, - Vec::new(), + tokens, server_policy.as_ref(), config_path, false, ) .await .unwrap(); + let engine = Arc::clone( + &state + .routing() + .registry + .list() + .into_iter() + .next() + .expect("enabled cluster graph must be registered") + .engine, + ); + engine + .start_stream_fold_driver() + .await + .expect("served graph starts its resident fold driver"); + (temp, state, engine) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn cluster_boot_installs_enabled_stream_runtime_authority() { + let (_temp, state, engine) = enabled_stream_state(Vec::new()).await; let app = build_app(state); let (status, body) = json_response( &app, @@ -550,9 +616,9 @@ graphs: ); let request = ChangeRequest { - query: "query insert_person($name: String) { insert Person { name: $name } }".to_string(), + query: "query insert_person($score: I32) { insert Person { score: $score } }".to_string(), name: Some("insert_person".to_string()), - params: Some(serde_json::json!({ "name": "served-runtime" })), + params: Some(serde_json::json!({ "score": 7 })), branch: Some("main".to_string()), }; let (status, body) = json_response( @@ -566,6 +632,322 @@ graphs: ) .await; assert_eq!(status, StatusCode::OK, "{body}"); + + let challenge_polled = Arc::new(AtomicBool::new(false)); + let challenge_response = app + .clone() + .oneshot( + Request::builder() + .method(Method::POST) + .uri("/graphs/knowledge/stream/ingest") + .header("content-type", "application/x-ndjson") + .body(stream_body_poll_probe(Arc::clone(&challenge_polled))) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!( + challenge_response.status(), + StatusCode::PRECONDITION_REQUIRED + ); + assert!( + !challenge_polled.load(Ordering::SeqCst), + "the graph-token challenge must not poll the request body" + ); + assert_eq!( + challenge_response.headers().get(CACHE_CONTROL), + Some(&HeaderValue::from_static("no-store")) + ); + let etag = challenge_response + .headers() + .get(ETAG) + .cloned() + .expect("428 challenge carries a strong ETag"); + let etag_text = etag.to_str().unwrap(); + assert!( + etag_text.starts_with("\"sha256:") && etag_text.ends_with('"'), + "challenge must carry the opaque graph token as a strong ETag: {etag_text}" + ); + let authority_token = etag_text + .strip_prefix('"') + .and_then(|value| value.strip_suffix('"')) + .unwrap() + .to_string(); + let challenge: StreamIngestChallenge = serde_json::from_slice( + &to_bytes(challenge_response.into_body(), usize::MAX) + .await + .unwrap(), + ) + .unwrap(); + assert_eq!(challenge.graph_token, authority_token); + assert!( + engine.stream_status().await.unwrap().tables.is_empty(), + "an effect-free graph-token challenge cannot enroll a table" + ); + + let weak_etag = HeaderValue::from_str(&format!("W/{etag_text}")).unwrap(); + let stale_etag = HeaderValue::from_str(&format!("\"sha256:{}\"", "0".repeat(64))).unwrap(); + for (label, supplied) in [ + ("wildcard", HeaderValue::from_static("*")), + ("weak", weak_etag), + ("stale", stale_etag), + ] { + let refused_polled = Arc::new(AtomicBool::new(false)); + let response = app + .clone() + .oneshot( + Request::builder() + .method(Method::POST) + .uri("/graphs/knowledge/stream/ingest") + .header("content-type", "application/x-ndjson") + .header(IF_MATCH, supplied) + .body(stream_body_poll_probe(Arc::clone(&refused_polled))) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!( + response.status(), + StatusCode::PRECONDITION_FAILED, + "{label} precondition" + ); + assert!( + response.headers().get(ETAG).is_none(), + "{label} refusal must not disclose replacement authority" + ); + assert!( + !refused_polled.load(Ordering::SeqCst), + "{label} refusal must not poll the request body" + ); + let refusal = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let refusal: Value = serde_json::from_slice(&refusal).unwrap(); + assert!(refusal.get("graph_token").is_none(), "{label}: {refusal}"); + assert!( + !refusal.to_string().contains(&authority_token), + "{label} refusal leaked replacement authority: {refusal}" + ); + } + + let multiple_polled = Arc::new(AtomicBool::new(false)); + let mut multiple = Request::builder() + .method(Method::POST) + .uri("/graphs/knowledge/stream/ingest") + .header("content-type", "application/x-ndjson") + .body(stream_body_poll_probe(Arc::clone(&multiple_polled))) + .unwrap(); + multiple.headers_mut().append(IF_MATCH, etag.clone()); + multiple.headers_mut().append(IF_MATCH, etag.clone()); + let response = app.clone().oneshot(multiple).await.unwrap(); + assert_eq!(response.status(), StatusCode::PRECONDITION_FAILED); + assert!(response.headers().get(ETAG).is_none()); + assert!(!multiple_polled.load(Ordering::SeqCst)); + let refusal = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + assert!(!String::from_utf8_lossy(&refusal).contains(&authority_token)); + + let (input_sender, input_receiver) = + futures::channel::mpsc::unbounded::>(); + let mut first_chunk = served_graph_node_line( + "Person", + "served-alice", + 11, + "50505050-5050-4050-8050-505050505050", + ); + first_chunk.extend(served_graph_node_line( + "Person", + "served-bob", + 12, + "51515151-5151-4151-8151-515151515151", + )); + first_chunk.extend(served_graph_edge_line( + "Knows", + "served-alice-knows-bob", + "served-alice", + "served-bob", + "52525252-5252-4252-8252-525252525252", + )); + input_sender + .unbounded_send(Ok(Bytes::from(first_chunk))) + .unwrap(); + + let response = app + .clone() + .oneshot( + Request::builder() + .method(Method::POST) + .uri("/graphs/knowledge/stream/ingest") + .header("content-type", "application/x-ndjson") + .header(IF_MATCH, etag) + .body(Body::from_stream(input_receiver)) + .unwrap(), + ) + .await + .unwrap(); + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response.headers().get("content-type").unwrap(), + "application/x-ndjson; charset=utf-8" + ); + let mut response_stream = response.into_body().into_data_stream(); + let first = tokio::time::timeout(Duration::from_secs(30), response_stream.next()) + .await + .expect("the first result must stream before request EOF") + .expect("the exact request produces a first result") + .expect("the first response chunk is readable"); + let mut result_bytes = first.to_vec(); + drop(input_sender); + while let Some(chunk) = response_stream.next().await { + result_bytes.extend_from_slice(&chunk.unwrap()); + } + let result_lines = result_bytes + .split(|byte| *byte == b'\n') + .filter(|line| !line.is_empty()) + .collect::>(); + assert_eq!( + result_lines.len(), + 3, + "{}", + String::from_utf8_lossy(&result_bytes) + ); + let outcomes = result_lines + .iter() + .map(|line| { + let raw: Value = serde_json::from_slice(line).unwrap(); + const ALLOWED_KEYS: &[&str] = &[ + "actual", + "blocking_ordinal", + "blocking_status", + "current_token", + "id", + "kind", + "limit", + "message", + "ordinal", + "scope", + "status", + "stream_token", + "type", + "unconfirmed_candidate_token", + "write_id", + ]; + for key in raw.as_object().unwrap().keys() { + assert!( + ALLOWED_KEYS.contains(&key.as_str()), + "served graph result leaked physical evidence '{key}': {raw}" + ); + } + assert!( + !raw.to_string().contains("enrollment_id") + && !raw.to_string().contains("shard_id") + && !raw.to_string().contains("writer_epoch") + && !raw.to_string().contains("dataset"), + "served graph result leaked physical evidence: {raw}" + ); + serde_json::from_value::(raw).unwrap() + }) + .collect::>(); + assert_eq!( + outcomes.iter().map(|line| line.ordinal).collect::>(), + [0, 1, 2] + ); + assert!( + outcomes + .iter() + .all(|line| line.status == StreamIngestStatusOutput::Durable), + "all three streamed graph rows must be durable: {}", + String::from_utf8_lossy(&result_bytes) + ); + assert_eq!(outcomes[0].kind, Some(StreamIngestKindOutput::Node)); + assert_eq!(outcomes[1].kind, Some(StreamIngestKindOutput::Node)); + assert_eq!(outcomes[2].kind, Some(StreamIngestKindOutput::Edge)); + assert_eq!(outcomes[0].type_name.as_deref(), Some("Person")); + assert_eq!(outcomes[2].type_name.as_deref(), Some("Knows")); + assert!(outcomes.iter().all(|line| line.stream_token.is_some())); + + let query = QueryRequest { + query: r#" +query streamed_edge() { + match { + $p: Person { score: 11 } + $p knows $f + } + return { $p.score, $f.score } +} +"# + .to_string(), + name: Some("streamed_edge".to_string()), + params: None, + branch: Some("main".to_string()), + snapshot: None, + }; + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + loop { + let (status, body) = json_response( + &app, + Request::builder() + .method(Method::POST) + .uri("/graphs/knowledge/query") + .header("content-type", "application/json") + .body(Body::from(serde_json::to_vec(&query).unwrap())) + .unwrap(), + ) + .await; + assert_eq!(status, StatusCode::OK, "{body}"); + if body["row_count"] == 1 { + assert_eq!(body["rows"][0]["p.score"], 11); + assert_eq!(body["rows"][0]["f.score"], 12); + break; + } + assert!( + tokio::time::Instant::now() < deadline, + "resident fold driver did not make streamed edge queryable: {body}" + ); + tokio::task::yield_now().await; + } + engine + .shutdown_stream_fold_driver() + .await + .expect("served graph fold driver shuts down cleanly"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +#[serial] +async fn authenticated_default_deny_refuses_stream_ingest_before_body_ownership() { + let (_temp, state, engine) = enabled_stream_state(vec![( + "stream-actor".to_string(), + "stream-secret".to_string(), + )]) + .await; + let app = build_app(state); + let body_polled = Arc::new(AtomicBool::new(false)); + let response = app + .oneshot( + Request::builder() + .method(Method::POST) + .uri("/graphs/knowledge/stream/ingest") + .header("authorization", "Bearer stream-secret") + .header("content-type", "application/x-ndjson") + .body(stream_body_poll_probe(Arc::clone(&body_polled))) + .unwrap(), + ) + .await + .unwrap(); + + assert_eq!(response.status(), StatusCode::FORBIDDEN); + assert!( + !body_polled.load(Ordering::SeqCst), + "default-deny authorization must run before graph ingest owns the body" + ); + let body = to_bytes(response.into_body(), usize::MAX).await.unwrap(); + let error: ErrorOutput = serde_json::from_slice(&body).unwrap(); + assert!(error.error.contains("default-deny mode"), "{error:?}"); + assert!( + engine.stream_status().await.unwrap().tables.is_empty(), + "a default-denied request cannot lazily enroll a private lane" + ); + engine + .shutdown_stream_fold_driver() + .await + .expect("served graph fold driver shuts down cleanly"); } #[tokio::test] diff --git a/crates/omnigraph-server/tests/openapi.rs b/crates/omnigraph-server/tests/openapi.rs index 619ca08b..02429216 100644 --- a/crates/omnigraph-server/tests/openapi.rs +++ b/crates/omnigraph-server/tests/openapi.rs @@ -170,6 +170,7 @@ const EXPECTED_PATHS: &[&str] = &[ "/graphs/{graph_id}/read", "/graphs/{graph_id}/query", "/graphs/{graph_id}/export", + "/graphs/{graph_id}/stream/ingest", "/graphs/{graph_id}/change", "/graphs/{graph_id}/mutate", "/graphs/{graph_id}/queries", @@ -235,6 +236,48 @@ fn openapi_export_is_post() { assert!(doc["paths"]["/graphs/{graph_id}/export"]["post"].is_object()); } +#[test] +fn graph_stream_ingest_documents_ndjson_and_token_preconditions() { + let doc = openapi_json(); + let operation = &doc["paths"]["/graphs/{graph_id}/stream/ingest"]["post"]; + assert!(operation.is_object()); + + let request_body = &operation["requestBody"]; + assert!(request_body.is_object()); + assert!(request_body["content"]["application/x-ndjson"].is_object()); + assert!( + request_body["content"].get("application/json").is_none(), + "graph streaming input is NDJSON, not a buffered JSON document" + ); + assert!( + operation["parameters"] + .as_array() + .unwrap() + .iter() + .any(|parameter| parameter["name"] == "If-Match" && parameter["in"] == "header"), + "graph stream ingest must document its strong graph-token precondition" + ); + + let responses = &operation["responses"]; + assert_eq!( + responses["200"]["content"]["application/x-ndjson"]["schema"]["$ref"], + "#/components/schemas/StreamIngestLineOutput" + ); + for status in ["401", "403", "409", "412", "413", "415", "503"] { + assert_eq!( + responses[status]["content"]["application/json"]["schema"]["$ref"], + "#/components/schemas/ErrorOutput", + "stream ingest {status} must use ErrorOutput" + ); + } + assert_eq!( + responses["428"]["content"]["application/json"]["schema"]["$ref"], + "#/components/schemas/StreamIngestChallenge" + ); + assert!(responses["428"]["headers"]["ETag"].is_object()); + assert!(responses["428"]["headers"]["Cache-Control"].is_object()); +} + #[test] fn stream_aware_export_documents_pre_header_failures() { let doc = openapi_json(); @@ -418,6 +461,11 @@ const EXPECTED_SCHEMAS: &[&str] = &[ "SchemaApplyRequest", "SnapshotOutput", "SnapshotTableOutput", + "StreamIngestChallenge", + "StreamIngestKindOutput", + "StreamIngestLineOutput", + "StreamIngestScopeOutput", + "StreamIngestStatusOutput", ]; #[test] @@ -812,6 +860,7 @@ fn protected_endpoints_reference_bearer_token_security() { ("/graphs/{graph_id}/load", "post"), ("/graphs/{graph_id}/ingest", "post"), ("/graphs/{graph_id}/export", "post"), + ("/graphs/{graph_id}/stream/ingest", "post"), ("/graphs/{graph_id}/snapshot", "get"), ("/graphs/{graph_id}/branches", "get"), ("/graphs/{graph_id}/branches", "post"), @@ -999,6 +1048,7 @@ fn recovery_barrier_write_endpoints_document_recovery_required() { ("/graphs/{graph_id}/queries/{name}", "post"), ("/graphs/{graph_id}/load", "post"), ("/graphs/{graph_id}/ingest", "post"), + ("/graphs/{graph_id}/stream/ingest", "post"), ("/graphs/{graph_id}/branches", "post"), ("/graphs/{graph_id}/branches/{branch}", "delete"), ("/graphs/{graph_id}/branches/merge", "post"), @@ -1025,6 +1075,7 @@ fn bounded_keyed_write_endpoints_document_resource_limit() { ("/graphs/{graph_id}/queries/{name}", "post"), ("/graphs/{graph_id}/load", "post"), ("/graphs/{graph_id}/ingest", "post"), + ("/graphs/{graph_id}/stream/ingest", "post"), ("/graphs/{graph_id}/branches/merge", "post"), ] { let response = &doc["paths"][path][method]["responses"]["413"]; @@ -1183,6 +1234,7 @@ async fn auth_mode_spec_has_security_on_protected_operations() { let protected_paths = [ ("/graphs/{graph_id}/read", "post"), ("/graphs/{graph_id}/change", "post"), + ("/graphs/{graph_id}/stream/ingest", "post"), ("/graphs/{graph_id}/snapshot", "get"), ("/graphs/{graph_id}/branches", "get"), ("/graphs/{graph_id}/commits", "get"), @@ -1262,6 +1314,7 @@ const EXPECTED_CLUSTER_PATHS: &[&str] = &[ "/graphs/{graph_id}/snapshot", "/graphs/{graph_id}/read", "/graphs/{graph_id}/export", + "/graphs/{graph_id}/stream/ingest", "/graphs/{graph_id}/change", "/graphs/{graph_id}/schema", "/graphs/{graph_id}/schema/apply", @@ -1336,6 +1389,7 @@ async fn multi_mode_openapi_drops_flat_protected_paths() { "/snapshot", "/read", "/export", + "/stream/ingest", "/change", "/schema", "/schema/apply", @@ -1532,6 +1586,7 @@ async fn served_spec_always_nests_under_cluster_prefix() { "/read", "/query", "/export", + "/stream/ingest", "/change", "/mutate", "/queries", diff --git a/crates/omnigraph/src/db/mod.rs b/crates/omnigraph/src/db/mod.rs index a3d335bb..e7599a45 100644 --- a/crates/omnigraph/src/db/mod.rs +++ b/crates/omnigraph/src/db/mod.rs @@ -24,6 +24,8 @@ pub use omnigraph::{ StreamDeadLetterPayloadPage, StreamExportCut, StreamStatus, StreamTableStatus, StreamingProfileResult, TableCleanupStats, TableOptimizeStats, TableRepairStats, }; +#[doc(hidden)] +pub use omnigraph::{GraphStreamChunkSource, GraphStreamIngestHandle, GraphStreamIngestStart}; pub(crate) use omnigraph::{ DeferredTableFork, StreamAuthorityRetirementExportProvenance, WriteAuthorityToken, WriteTxn, }; diff --git a/crates/omnigraph/src/db/omnigraph.rs b/crates/omnigraph/src/db/omnigraph.rs index 046a17f9..e9eee235 100644 --- a/crates/omnigraph/src/db/omnigraph.rs +++ b/crates/omnigraph/src/db/omnigraph.rs @@ -74,6 +74,8 @@ pub use stream_dead_letter::{ StreamDeadLetterEncodingCostForTest, failpoint_measure_stream_dead_letter_object_for_test, }; #[doc(hidden)] +pub use stream_ndjson::{GraphStreamChunkSource, GraphStreamIngestHandle, GraphStreamIngestStart}; +#[doc(hidden)] pub use stream_profile::{ CheckedClusterApplyAuthority, CheckedClusterBlockAuthority, CheckedClusterDeadLetterAuthority, CheckedClusterMaintenanceAuthority, CheckedClusterRetirementAuthority, diff --git a/crates/omnigraph/src/db/omnigraph/stream_ingest.rs b/crates/omnigraph/src/db/omnigraph/stream_ingest.rs index 17a5e9df..4f587e8f 100644 --- a/crates/omnigraph/src/db/omnigraph/stream_ingest.rs +++ b/crates/omnigraph/src/db/omnigraph/stream_ingest.rs @@ -18,6 +18,7 @@ use futures::TryStreamExt; use lance::dataset::mem_wal::scanner::LsmScanner; use lance::dataset::mem_wal::{DatasetMemWalExt, ShardManifestStore, ShardWriter, WalTailer}; use lance_index::mem_wal::{MemWalIndexDetails, MergedGeneration, ShardId, ShardStatus}; +use sha2::{Digest, Sha256}; #[cfg(feature = "failpoints")] use crate::db::ReadTarget; @@ -162,8 +163,8 @@ pub(super) struct NormalizedStreamJsonRow { /// manifest version and graph head, so neither is part of this witness. The /// accepted graph identity/catalog and active profile delegation are the /// stable authority that must remain exact while a logical row is routed to a -/// private lane. This is an internal-v19 compare witness, not the later public -/// graph-control token. +/// private lane. F7 derives the opaque served-ingest precondition token from +/// this witness; the witness itself remains internal and non-serialized. #[derive(Debug, Clone, PartialEq, Eq)] pub(super) struct StreamGraphIngestWitness { graph_identity_digest: String, @@ -212,6 +213,30 @@ impl StreamGraphIngestWitness { pub(super) fn schema_ir_hash(&self) -> &str { &self.schema_ir_hash } + + /// Derive the opaque served-ingest compare token from this exact witness. + /// + /// This is deliberately a projection of existing manifest authority, not + /// a persisted token or a second source of truth. Every variable-width + /// field is length framed and every integer is big endian, so the v1 + /// domain has one canonical byte representation without relying on JSON + /// object ordering or a serializer version. + pub(super) fn authority_token(&self) -> String { + fn hash_bytes(hasher: &mut Sha256, bytes: &[u8]) { + hasher.update(u64::try_from(bytes.len()).unwrap_or(u64::MAX).to_be_bytes()); + hasher.update(bytes); + } + + let mut hasher = Sha256::new(); + hasher.update(b"omnigraph.graph-stream-ingest-authority.v1\0"); + hash_bytes(&mut hasher, self.graph_identity_digest.as_bytes()); + hash_bytes(&mut hasher, self.schema_ir_hash.as_bytes()); + hasher.update(self.schema_identity_version.to_be_bytes()); + hasher.update(self.profile_revision.to_be_bytes()); + hash_bytes(&mut hasher, self.fold_delegation_id.as_bytes()); + hash_bytes(&mut hasher, self.fold_delegation_digest.as_bytes()); + format!("sha256:{:x}", hasher.finalize()) + } } /// Effect-free disposition for the first row at a physical-run boundary. @@ -8490,6 +8515,26 @@ pub(super) fn exact_merged_generation( mod tests { use super::*; + #[test] + fn graph_ingest_authority_token_pins_canonical_witness_encoding() { + let witness = StreamGraphIngestWitness { + graph_identity_digest: format!("sha256:{}", "1".repeat(64)), + schema_ir_hash: format!("sha256:{}", "2".repeat(64)), + schema_identity_version: 2, + profile_revision: 7, + fold_delegation_id: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa".to_string(), + fold_delegation_digest: format!("sha256:{}", "3".repeat(64)), + }; + + assert_eq!( + witness.authority_token(), + "sha256:e7662e871c1eb31a2cc86098cdf5bb13a767c3aff852fb1db2999c272eadfda1" + ); + let mut changed = witness.clone(); + changed.profile_revision += 1; + assert_ne!(changed.authority_token(), witness.authority_token()); + } + #[test] fn data_block_error_preserves_typed_correction_token() { let error = stream_data_block_error("block-token"); diff --git a/crates/omnigraph/src/db/omnigraph/stream_ndjson.rs b/crates/omnigraph/src/db/omnigraph/stream_ndjson.rs index d1c128d2..f9da17d2 100644 --- a/crates/omnigraph/src/db/omnigraph/stream_ndjson.rs +++ b/crates/omnigraph/src/db/omnigraph/stream_ndjson.rs @@ -1,13 +1,15 @@ -#![allow(dead_code)] // Hidden F4 engine seam; production transport activation is intentionally deferred. +#![allow(dead_code)] // Table-scoped F4 seams remain private; F7 exposes only the graph bridge. -//! Hidden RFC-026 F4 NDJSON request driver. +//! RFC-026 F4 NDJSON request driver and F7 served graph bridge. //! -//! This is deliberately transport-neutral and crate-private. It composes the +//! The table/lane driver remains crate-private. F7 exposes only a doc-hidden, +//! transport-neutral graph request seam; HTTP, CLI, SDK, and OpenAPI contracts +//! are defined by their respective transport crates. Both layers compose the //! bounded framing/result primitives with the existing B2 authority and -//! durability corridor; it does not define an HTTP, CLI, SDK, or OpenAPI -//! surface. +//! durability corridor. use std::collections::{BTreeMap, BTreeSet}; +use std::fmt; use std::pin::Pin; use std::sync::{Arc, Mutex}; use std::time::Duration; @@ -54,7 +56,41 @@ const STREAM_SIZING_ATTEMPT_ID: &str = "00000000-0000-4000-8000-000000000001"; const GRAPH_STREAM_SIZING_UUID: &str = "00000000-0000-4000-8000-000000000002"; const STREAM_TAIL_SIZING_RETRY_DELAY: Duration = Duration::from_millis(1); -pub(super) type StreamChunkSource = Pin>> + Send + 'static>>; +/// Publicly observable maximum coalescing dwell for one non-full graph +/// request run. Storage, authority, and backpressure can add acknowledgement +/// latency after submission; this bound only prevents a complete row from +/// waiting indefinitely for another body chunk, a lane switch, or EOF. +const GRAPH_STREAM_RUN_MAX_COALESCE_DELAY: Duration = Duration::from_millis(50); + +fn served_graph_authority_mismatch() -> OmniError { + OmniError::StreamingAuthorityMismatch { + reason: "supplied graph stream ingest authority token is malformed or stale".to_string(), + } +} + +fn normalize_token_bound_start_error(error: OmniError, token_bound: bool) -> OmniError { + if token_bound + && matches!( + &error, + OmniError::StreamingRequiresClusterRuntime { .. } + | OmniError::StreamingAuthorityMismatch { .. } + | OmniError::StreamAuthorityRetired { .. } + ) + { + served_graph_authority_mismatch() + } else { + error + } +} + +/// Transport-neutral chunk source accepted by the served graph firehose. +/// +/// Each item is one bounded transport chunk, not necessarily one NDJSON line. +/// The engine's existing framer retains the 32-MiB chunk/line limits. +#[doc(hidden)] +pub type GraphStreamChunkSource = Pin>> + Send + 'static>>; + +pub(super) type StreamChunkSource = GraphStreamChunkSource; struct TransportOwnership { _permit: StreamRequestPermit, @@ -109,16 +145,41 @@ struct GraphStreamLineOutcome { outcome: StreamLineOutcome, } -/// Hidden graph-native request handle. The existing line outcome retains all -/// private evidence needed for stop-tail safety, while this wrapper attaches -/// only caller-logical row identity for the redacted graph projection. -struct GraphStreamRequestHandle { +/// Move-only served graph-ingest request. +/// +/// The existing line outcome retains all private evidence needed for stop-tail +/// safety, while this wrapper attaches only caller-logical row identity for the +/// redacted graph projection. +/// +/// Dropping the handle closes the bounded result receiver. The existing +/// request task observes that closure before polling more body input while any +/// already-invoked durable call remains owned until it settles. +#[doc(hidden)] +pub struct GraphStreamIngestHandle { inner: StreamRequestHandle, routes: Arc>>, } -impl GraphStreamRequestHandle { - async fn recv(&mut self) -> Result> { +impl GraphStreamIngestHandle { + /// Receive one newline-terminated, graph-logical NDJSON result. + /// + /// The projection deliberately omits table, dataset, lane, binding, + /// writer, shard, epoch, generation, and recovery-sidecar identities. + pub async fn recv(&mut self) -> Result>> { + let Some(outcome) = self.recv_inner().await? else { + return Ok(None); + }; + let mut encoded = + serde_json::to_vec(&graph_stream_line_outcome_json(&outcome)).map_err(|error| { + OmniError::manifest_internal(format!( + "graph stream result serialization failed: {error}" + )) + })?; + encoded.push(b'\n'); + Ok(Some(encoded)) + } + + async fn recv_inner(&mut self) -> Result> { let Some(outcome) = self.inner.recv().await? else { return Ok(None); }; @@ -131,6 +192,28 @@ impl GraphStreamRequestHandle { } } +/// Effect-free served-ingest start disposition. +#[doc(hidden)] +pub enum GraphStreamIngestStart { + /// The caller must retry the request with this exact graph authority token. + /// The supplied body was never polled. + TokenRequired { authority_token: String }, + /// The supplied token matched current checked-runtime graph authority and + /// the bounded request task now owns body polling. + Ready(GraphStreamIngestHandle), +} + +impl fmt::Debug for GraphStreamIngestStart { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::TokenRequired { .. } => { + formatter.write_str("TokenRequired { authority_token: }") + } + Self::Ready(_) => formatter.write_str("Ready()"), + } + } +} + struct GraphLaneContext { table_key: String, stream_incarnation_id: String, @@ -335,7 +418,34 @@ impl Omnigraph { self: &Arc, actor_id: &str, body: StreamChunkSource, - ) -> Result { + ) -> Result { + self.enforce( + omnigraph_policy::PolicyAction::StreamIngest, + &omnigraph_policy::ResourceScope::Graph, + Some(actor_id), + )?; + let contributor_id = TrustedContributorId::new(actor_id.to_string()) + .map_err(|error| OmniError::manifest(error.to_string()))?; + + self.start_graph_stream_ingest_after_token(actor_id, contributor_id, None, body) + .await + } + + /// Start one checked-runtime graph-native served firehose request. + /// + /// Authorization, runtime validation, and the exact graph-authority + /// challenge complete before `body` is polled. A missing token returns the + /// current opaque token. A malformed or stale token refuses without + /// returning replacement authority and before lazy enrollment, recovery, + /// folding, or MemWAL invocation. An exact token transfers the body into + /// the existing bounded F4 request owner. + #[doc(hidden)] + pub async fn start_served_graph_stream_ingest_as( + self: &Arc, + actor_id: &str, + supplied_authority_token: Option<&str>, + body: GraphStreamChunkSource, + ) -> Result { self.enforce( omnigraph_policy::PolicyAction::StreamIngest, &omnigraph_policy::ResourceScope::Graph, @@ -343,25 +453,70 @@ impl Omnigraph { )?; let contributor_id = TrustedContributorId::new(actor_id.to_string()) .map_err(|error| OmniError::manifest(error.to_string()))?; + + let expected_graph_witness = { + let profile_guard = self.write_queue().acquire_stream_profile_shared().await; + self.ensure_streaming_ingest_runtime_authorized().await?; + let txn = self.open_write_txn(None).await?; + let witness = StreamGraphIngestWitness::capture(&txn)?; + drop(profile_guard); + witness + }; + let authority_token = expected_graph_witness.authority_token(); + let Some(supplied) = supplied_authority_token else { + return Ok(GraphStreamIngestStart::TokenRequired { authority_token }); + }; + if supplied != authority_token { + return Err(served_graph_authority_mismatch()); + } + + let handle = self + .start_graph_stream_ingest_after_token( + actor_id, + contributor_id, + Some(&expected_graph_witness), + body, + ) + .await?; + Ok(GraphStreamIngestStart::Ready(handle)) + } + + async fn start_graph_stream_ingest_after_token( + self: &Arc, + actor_id: &str, + contributor_id: TrustedContributorId, + expected_graph_witness: Option<&StreamGraphIngestWitness>, + body: GraphStreamChunkSource, + ) -> Result { + let token_bound = expected_graph_witness.is_some(); let permit = self.stream_requests.try_acquire(actor_id)?; { let initial_profile_guard = self.write_queue().acquire_stream_profile_shared().await; - self.ensure_streaming_ingest_runtime_authorized().await?; + self.ensure_streaming_ingest_runtime_authorized() + .await + .map_err(|error| normalize_token_bound_start_error(error, token_bound))?; drop(initial_profile_guard); } // Settle the existing coordinator's cold startup cut before this // request can make an edge lane resident. Otherwise the sole resident // slot could make a previously acknowledged endpoint node lose the // node-before-edge round and turn a valid edge into a dead letter. - self.settle_initial_stream_lanes_for_graph_ingest().await?; + self.settle_initial_stream_lanes_for_graph_ingest() + .await + .map_err(|error| normalize_token_bound_start_error(error, token_bound))?; // The root→profile lock order above deliberately opens a transition // window. Reacquire profile authority and revalidate it before the // request pins its graph witness/catalog or polls the body. let profile_guard = self.write_queue().acquire_stream_profile_shared().await; - self.ensure_streaming_ingest_runtime_authorized().await?; + self.ensure_streaming_ingest_runtime_authorized() + .await + .map_err(|error| normalize_token_bound_start_error(error, token_bound))?; let txn = self.open_write_txn(None).await?; - let expected_graph_witness = StreamGraphIngestWitness::capture(&txn)?; + let current_graph_witness = StreamGraphIngestWitness::capture(&txn)?; + if expected_graph_witness.is_some_and(|expected| expected != ¤t_graph_witness) { + return Err(served_graph_authority_mismatch()); + } let catalog = Arc::new(super::public_catalog_view(&txn.catalog)?); drop(profile_guard); @@ -379,14 +534,14 @@ impl Omnigraph { actor_id, contributor_id, catalog, - expected_graph_witness, + current_graph_witness, body, sender, task_routes, ) .await; }); - Ok(GraphStreamRequestHandle { + Ok(GraphStreamIngestHandle { inner: StreamRequestHandle { receiver, task: Some(task), @@ -409,8 +564,10 @@ impl Omnigraph { let body = Box::pin(futures::stream::iter(chunks.into_iter().map(Ok))); let mut handle = self.stream_ingest_graph_ndjson_as(actor_id, body).await?; let mut results = Vec::new(); - while let Some(outcome) = handle.recv().await? { - results.push(graph_stream_line_outcome_json(&outcome).to_string()); + while let Some(line) = handle.recv().await? { + let line = std::str::from_utf8(&line) + .map_err(|error| OmniError::manifest_internal(error.to_string()))?; + results.push(line.trim_end_matches('\n').to_string()); } Ok(results) } @@ -474,11 +631,26 @@ async fn drive_graph_request( let mut blocker: Option = None; let mut results = OrderedResultBuffer::new(0); let mut next_input_ordinal = 0_u64; + let mut run_flush_deadline = None; loop { tokio::select! { biased; _ = sender.closed() => return, + _ = wait_for_graph_run_flush(run_flush_deadline.map(|(_, deadline)| deadline)) => { + if !flush_graph_run( + &db, + &contributor_id, + lane.as_ref(), + &mut run, + &mut blocker, + &mut results, + &sender, + ).await { + return; + } + sync_graph_run_flush_deadline(&mut run_flush_deadline, run.as_ref()); + } next = body.next() => { match next { Some(Ok(chunk)) => { @@ -512,7 +684,7 @@ async fn drive_graph_request( return; } next_input_ordinal = frame.ordinal().saturating_add(1); - if !process_graph_frame( + let keep_driving = process_graph_frame( &db, &actor_id, &contributor_id, @@ -525,7 +697,12 @@ async fn drive_graph_request( &mut results, &sender, &routes, - ).await { + ).await; + sync_graph_run_flush_deadline( + &mut run_flush_deadline, + run.as_ref(), + ); + if !keep_driving { return; } } @@ -596,6 +773,35 @@ async fn drive_graph_request( .await; } +async fn wait_for_graph_run_flush(deadline: Option) { + match deadline { + Some(deadline) => tokio::time::sleep_until(deadline).await, + None => std::future::pending().await, + } +} + +fn sync_graph_run_flush_deadline( + deadline: &mut Option<(u64, tokio::time::Instant)>, + run: Option<&AccumulatingRun>, +) { + let first_ordinal = run.and_then(|run| run.rows.first()).map(|row| row.ordinal); + sync_graph_run_flush_deadline_at(deadline, first_ordinal, tokio::time::Instant::now()); +} + +fn sync_graph_run_flush_deadline_at( + deadline: &mut Option<(u64, tokio::time::Instant)>, + first_ordinal: Option, + now: tokio::time::Instant, +) { + match (first_ordinal, deadline.as_ref()) { + (Some(first_ordinal), Some((current_ordinal, _))) if first_ordinal == *current_ordinal => {} + (Some(first_ordinal), _) => { + *deadline = Some((first_ordinal, now + GRAPH_STREAM_RUN_MAX_COALESCE_DELAY)); + } + (None, _) => *deadline = None, + } +} + #[allow(clippy::too_many_arguments)] async fn process_graph_frame( db: &Arc, @@ -2315,7 +2521,6 @@ async fn emit_ordered( true } -#[cfg(feature = "failpoints")] fn graph_stream_line_outcome_json(outcome: &GraphStreamLineOutcome) -> serde_json::Value { let line = &outcome.outcome; let status = graph_status_name(line.status()); @@ -2374,7 +2579,6 @@ fn graph_stream_line_outcome_json(outcome: &GraphStreamLineOutcome) -> serde_jso value } -#[cfg(feature = "failpoints")] fn graph_status_name(status: StreamLineStatus) -> &'static str { match status { StreamLineStatus::StreamBindingChanged @@ -2385,7 +2589,6 @@ fn graph_status_name(status: StreamLineStatus) -> &'static str { } } -#[cfg(feature = "failpoints")] fn graph_status_scope(status: StreamLineStatus) -> &'static str { match status { StreamLineStatus::AckUnknown @@ -2408,7 +2611,6 @@ fn graph_status_scope(status: StreamLineStatus) -> &'static str { } } -#[cfg(feature = "failpoints")] fn graph_safe_message(status: StreamLineStatus) -> Option<&'static str> { match status { StreamLineStatus::Durable @@ -2526,6 +2728,38 @@ const _: () = assert!(STREAM_REQUEST_MAX_RESULT_STATUSES >= STREAM_RESULT_CHANNE mod tests { use super::*; + #[test] + fn graph_run_coalescing_deadline_is_non_resetting() { + let start = tokio::time::Instant::now(); + let mut deadline = None; + + sync_graph_run_flush_deadline_at(&mut deadline, Some(7), start); + let original = deadline.expect("a first row starts one deadline"); + assert_eq!(original.0, 7); + assert_eq!(original.1, start + GRAPH_STREAM_RUN_MAX_COALESCE_DELAY); + + sync_graph_run_flush_deadline_at( + &mut deadline, + Some(7), + start + Duration::from_millis(40), + ); + assert_eq!( + deadline, + Some(original), + "more rows in the same run cannot extend its first-row deadline" + ); + + let next_start = start + Duration::from_millis(60); + sync_graph_run_flush_deadline_at(&mut deadline, Some(8), next_start); + assert_eq!( + deadline, + Some((8, next_start + GRAPH_STREAM_RUN_MAX_COALESCE_DELAY)) + ); + + sync_graph_run_flush_deadline_at(&mut deadline, None, next_start); + assert!(deadline.is_none(), "an empty run owns no timer"); + } + fn test_request_handle( receiver: tokio::sync::mpsc::Receiver, task: tokio::task::JoinHandle<()>, diff --git a/crates/omnigraph/src/db/omnigraph/stream_request.rs b/crates/omnigraph/src/db/omnigraph/stream_request.rs index e0953b6a..3d5ea185 100644 --- a/crates/omnigraph/src/db/omnigraph/stream_request.rs +++ b/crates/omnigraph/src/db/omnigraph/stream_request.rs @@ -1,11 +1,12 @@ -#![allow(dead_code)] // Hidden F4 request vocabulary; production transport activation is intentionally deferred. +#![allow(dead_code)] // Shared private vocabulary includes controls not activated by F7a. //! Private, transport-neutral request mechanics for RFC-026 F4. //! //! This module deliberately owns no HTTP, CLI, or public DTO shape. It provides -//! only the bounded pieces shared by a future transport adapter: incremental +//! only the bounded pieces shared by the graph transport adapter: incremental //! NDJSON framing, per-line result classification, request admission, and -//! caller-order result buffering. +//! caller-order result buffering. F7a's graph-native served adapter reuses +//! these mechanics; table/lane controls and physical evidence remain private. use std::collections::{BTreeMap, HashMap}; use std::sync::{Arc, Mutex, OnceLock, Weak}; diff --git a/crates/omnigraph/tests/forbidden_apis.rs b/crates/omnigraph/tests/forbidden_apis.rs index bfbedbf0..e299e884 100644 --- a/crates/omnigraph/tests/forbidden_apis.rs +++ b/crates/omnigraph/tests/forbidden_apis.rs @@ -226,6 +226,9 @@ const STREAM_LIFECYCLE_V14_V15_V18: WriteProtocol = WriteProtocol::Composed("private firehose lifecycle recovery v14 + resume v15 + rebind v18"); const STREAM_FOLD_DRIVER_V14: WriteProtocol = WriteProtocol::Composed("hidden resident stream fold driver + recovery v14"); +const SERVED_GRAPH_STREAM_INGEST_V14_V21: WriteProtocol = WriteProtocol::Composed( + "served graph ingress over enrollment/fold recovery v14 + dead-letter recovery v21", +); /// RFC-026 checked profile authority: ENABLED/DISABLED terminal transitions /// pair one immutable token-ledger receipt with one profile/lineage publish /// under recovery-v13. Disable additionally persists the admission-cutoff @@ -296,6 +299,9 @@ write_surfaces! { "failpoint_stream_ingest_ndjson_as_for_test", "failpoint_stream_ingest_ndjson_cancel_for_test", ], + "db/omnigraph/stream_ndjson.rs" => SERVED_GRAPH_STREAM_INGEST_V14_V21 => [ + "start_served_graph_stream_ingest_as", + ], "db/omnigraph.rs" => OPTIMIZE_V9 => ["optimize"], "db/omnigraph.rs" => WriteProtocol::ManifestAdoption => ["repair"], "db/omnigraph/stream_profile.rs" => WriteProtocol::Composed("control-plane-required refusal") => ["set_streaming_enabled_as"], diff --git a/crates/omnigraph/tests/memwal_stream.rs b/crates/omnigraph/tests/memwal_stream.rs index 569a6a9b..9c549f35 100644 --- a/crates/omnigraph/tests/memwal_stream.rs +++ b/crates/omnigraph/tests/memwal_stream.rs @@ -39,10 +39,10 @@ use object_store::{ UploadPart, }; use omnigraph::db::{ - CleanupPolicyOptions, Omnigraph, ReadTarget, StreamAuthorityRetirementPlan, - StreamDataCorrectionAction, StreamDataCorrectionRequest, StreamOldestUncoveredAgeStatus, - StreamPendingGenerationStatus, StreamRebuildBlockReason, StreamTablePhysicalOperationalStatus, - StreamTableStatus, StreamTokenIndexCoverageStatus, + CleanupPolicyOptions, GraphStreamChunkSource, GraphStreamIngestStart, Omnigraph, ReadTarget, + StreamAuthorityRetirementPlan, StreamDataCorrectionAction, StreamDataCorrectionRequest, + StreamOldestUncoveredAgeStatus, StreamPendingGenerationStatus, StreamRebuildBlockReason, + StreamTablePhysicalOperationalStatus, StreamTableStatus, StreamTokenIndexCoverageStatus, }; use omnigraph::error::OmniError; use omnigraph::failpoints::{ScopedFailPoint, names}; @@ -3094,6 +3094,13 @@ fn parse_ndjson_outcomes(outcomes: Vec) -> Vec { .collect() } +fn graph_body_poll_probe(polled: Arc) -> GraphStreamChunkSource { + Box::pin(futures::stream::poll_fn(move |_| { + polled.store(true, Ordering::SeqCst); + std::task::Poll::Ready(None::, OmniError>>) + })) +} + async fn prepare_stream_ingest( db: &Omnigraph, enrollment_request_id: &str, @@ -9649,6 +9656,20 @@ rules: .await .expect_err("Cedar must reject an actor without stream_ingest"); assert!(matches!(denied, OmniError::Policy(_)), "{denied:?}"); + let denied_graph_polled = Arc::new(AtomicBool::new(false)); + let denied_graph = db + .start_served_graph_stream_ingest_as( + "act-denied", + None, + graph_body_poll_probe(Arc::clone(&denied_graph_polled)), + ) + .await + .expect_err("served graph ingress must authorize before challenging or polling"); + assert!( + matches!(denied_graph, OmniError::Policy(_)), + "{denied_graph:?}" + ); + assert!(!denied_graph_polled.load(Ordering::SeqCst)); let no_runtime_prepare = prepare_stream_ingest( &db, @@ -9676,6 +9697,23 @@ rules: ), "{no_runtime:?}" ); + let no_runtime_graph_polled = Arc::new(AtomicBool::new(false)); + let no_runtime_graph = db + .start_served_graph_stream_ingest_as( + "act-allowed", + None, + graph_body_poll_probe(Arc::clone(&no_runtime_graph_polled)), + ) + .await + .expect_err("ambient graph ingress cannot mint a served challenge"); + assert!( + matches!( + no_runtime_graph, + OmniError::StreamingRequiresClusterRuntime { ref mode } if mode == "ENABLED" + ), + "{no_runtime_graph:?}" + ); + assert!(!no_runtime_graph_polled.load(Ordering::SeqCst)); } assert_eq!( db.snapshot_of(ReadTarget::branch("main")) @@ -9688,6 +9726,194 @@ rules: assert_no_recovery_sidecars(&dir); } +#[tokio::test] +#[serial] +async fn served_graph_ndjson_challenges_before_body_and_streams_only_redacted_lines() { + let _scenario = FailScenario::setup(); + let (_dir, db) = init_unenrolled_served_with_schema(TWO_TABLE_STREAM_SCHEMA).await; + let manifest_before = db + .snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(); + + let challenge_polled = Arc::new(AtomicBool::new(false)); + let authority_token = match db + .start_served_graph_stream_ingest_as( + "agent:served-graph", + None, + graph_body_poll_probe(Arc::clone(&challenge_polled)), + ) + .await + .expect("checked served ingest returns an authority challenge") + { + GraphStreamIngestStart::TokenRequired { authority_token } => authority_token, + GraphStreamIngestStart::Ready(_) => panic!("a missing token cannot start body polling"), + }; + assert!(!challenge_polled.load(Ordering::SeqCst)); + assert_eq!(authority_token.len(), "sha256:".len() + 64); + assert!( + authority_token + .strip_prefix("sha256:") + .is_some_and(|digest| digest + .bytes() + .all(|byte| byte.is_ascii_hexdigit() && !byte.is_ascii_uppercase())) + ); + assert_eq!( + db.snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(), + manifest_before, + "the challenge must not publish graph or enrollment state" + ); + assert!(db.stream_status().await.unwrap().tables.is_empty()); + + let repeated_polled = Arc::new(AtomicBool::new(false)); + let repeated = match db + .start_served_graph_stream_ingest_as( + "agent:served-graph", + None, + graph_body_poll_probe(Arc::clone(&repeated_polled)), + ) + .await + .expect("unchanged authority returns the same challenge") + { + GraphStreamIngestStart::TokenRequired { authority_token } => authority_token, + GraphStreamIngestStart::Ready(_) => panic!("a repeated challenge cannot start a request"), + }; + assert_eq!(repeated, authority_token); + assert!(!repeated_polled.load(Ordering::SeqCst)); + + for supplied in [ + "not-a-token".to_string(), + format!("sha256:{}", "0".repeat(64)), + ] { + let refused_polled = Arc::new(AtomicBool::new(false)); + let error = db + .start_served_graph_stream_ingest_as( + "agent:served-graph", + Some(&supplied), + graph_body_poll_probe(Arc::clone(&refused_polled)), + ) + .await + .expect_err("malformed and stale tokens must refuse effect-free"); + assert!( + matches!(error, OmniError::StreamingAuthorityMismatch { .. }), + "{error:?}" + ); + assert!(!refused_polled.load(Ordering::SeqCst)); + assert!( + !error.to_string().contains(&authority_token), + "a stale-token refusal must not disclose replacement authority" + ); + } + assert_eq!( + db.snapshot_of(ReadTarget::branch("main")) + .await + .unwrap() + .version(), + manifest_before + ); + assert!(db.stream_status().await.unwrap().tables.is_empty()); + + let body: GraphStreamChunkSource = Box::pin(futures::stream::iter([Ok(graph_node_line( + "Person", + "served-person", + 17, + "50505050-5050-4050-8050-505050505050", + None, + ))])); + let mut handle = match db + .start_served_graph_stream_ingest_as("agent:served-graph", Some(&authority_token), body) + .await + .expect("an exact graph token starts the existing request owner") + { + GraphStreamIngestStart::Ready(handle) => handle, + GraphStreamIngestStart::TokenRequired { .. } => { + panic!("the exact token cannot be rechallenged") + } + }; + let line = handle + .recv() + .await + .expect("served result receive succeeds") + .expect("one input produces one result"); + assert_eq!(line.last(), Some(&b'\n')); + assert_eq!(line.iter().filter(|byte| **byte == b'\n').count(), 1); + let outcome: serde_json::Value = + serde_json::from_slice(&line[..line.len() - 1]).expect("result line is valid JSON"); + assert_eq!(outcome["status"], "durable"); + assert_eq!(outcome["scope"], "row"); + assert_eq!(outcome["kind"], "node"); + assert_eq!(outcome["type"], "Person"); + assert_eq!(outcome["id"], "served-person"); + assert_graph_stream_result_redacted(&outcome); + assert!(handle.recv().await.unwrap().is_none()); +} + +#[tokio::test] +#[serial] +async fn served_graph_ndjson_flushes_one_complete_run_before_body_eof() { + let _scenario = FailScenario::setup(); + let (_dir, db) = init_unenrolled_served_with_schema(TWO_TABLE_STREAM_SCHEMA).await; + let challenge_body: GraphStreamChunkSource = Box::pin(futures::stream::pending()); + let authority_token = match db + .start_served_graph_stream_ingest_as("agent:low-rate", None, challenge_body) + .await + .expect("checked served ingest returns an authority challenge") + { + GraphStreamIngestStart::TokenRequired { authority_token } => authority_token, + GraphStreamIngestStart::Ready(_) => panic!("a missing token cannot start body polling"), + }; + + let (body_sender, body_receiver) = + futures::channel::mpsc::unbounded::, OmniError>>(); + body_sender + .unbounded_send(Ok(graph_node_line( + "Person", + "low-rate-person", + 19, + "53535353-5353-4353-8353-535353535353", + None, + ))) + .unwrap(); + let body: GraphStreamChunkSource = Box::pin(body_receiver); + let mut handle = match db + .start_served_graph_stream_ingest_as("agent:low-rate", Some(&authority_token), body) + .await + .expect("the exact graph token starts the request") + { + GraphStreamIngestStart::Ready(handle) => handle, + GraphStreamIngestStart::TokenRequired { .. } => { + panic!("the exact token cannot be rechallenged") + } + }; + + // Keep `body_sender` alive: after delivering the complete line the source + // is Pending, not EOF. The fixed coalescing boundary must submit the run. + let line = tokio::time::timeout(Duration::from_secs(5), handle.recv()) + .await + .expect("one low-rate graph row must acknowledge while the body remains pending") + .expect("served result receive succeeds") + .expect("one input produces one result"); + assert_eq!(line.last(), Some(&b'\n')); + let outcome: serde_json::Value = + serde_json::from_slice(&line[..line.len() - 1]).expect("result line is valid JSON"); + assert_eq!(outcome["status"], "durable"); + assert_eq!(outcome["id"], "low-rate-person"); + assert_graph_stream_result_redacted(&outcome); + + drop(body_sender); + assert!( + tokio::time::timeout(Duration::from_secs(5), handle.recv()) + .await + .expect("request reaches EOF after the pending body is closed") + .unwrap() + .is_none() + ); +} + #[tokio::test] #[serial] async fn graph_ndjson_authorizes_one_graph_not_each_lazy_lane() { diff --git a/docs/dev/canon.md b/docs/dev/canon.md index 805b9463..3bd3994c 100644 --- a/docs/dev/canon.md +++ b/docs/dev/canon.md @@ -1199,8 +1199,8 @@ receipt and complete physical-output envelope remain documented limits on any future bounded-storage claim, not blockers for this profile. B2a itself activated no schema or product surface. -**Implemented private B2 token/fold, bounded profile authority, and hidden -lifecycle core; public row and lifecycle controls remain inactive:** +**Implemented B2 token/fold, bounded profile authority, graph-native served +row ingress, and hidden lifecycle core; public lifecycle controls remain inactive:** Internal schema v9/config-v3/state-v2 adds canonical payload and token digests, trusted hidden row metadata, exact compare-and-chain/idempotency classification, same-generation token overlays, and the manifest-selected graph-global @@ -1223,10 +1223,12 @@ as `RETIRED`, and retired export carries the selected root receipt plus a recomputable selected-branch-member witness and ordered cut-membership proof. V18/recovery-v20 adds the separate stopped/offline, cluster-only exact `DataBlock` show/correct exit. Checked offline disable is the sole supported -production quiescence owner. Explicit public enrollment, general lifecycle -control, `AuthorityBlock` repair, public row admission, operational-status -transport, and SDK/HTTP/OpenAPI row/lifecycle/maintenance parity remain -inactive. F6b6's internal checked status core observes one read-only authority +production quiescence owner. F7a exposes one graph-native mixed node/edge +NDJSON route plus remote client/CLI/OpenAPI parity over the existing lazy +private-lane prepare and resident driver; it adds no coordinator or format. +Explicit public lane enrollment, general lifecycle control, `AuthorityBlock` +repair, operational-status transport, and SDK/HTTP/OpenAPI +lifecycle/maintenance parity remain inactive. F6b6's internal checked status core observes one read-only authority cut and retains honest unavailable states for cold replay, flushed LWW projection, exact canonical-main recovery-owned physical movement, and oldest uncovered age; all pending sidecars inside the accepted envelope remain visible and rebuild- @@ -1254,7 +1256,7 @@ heads format **(RFC-024, research-blocked after Gate A rejected the first physical access shape)**; checkpoint-pinned retention **(RFC-025, research-blocked after Gate 0 rejected the current compacted registry-access shape)**; -public MemWAL row admission and later lifecycle/read phases **(RFC-026, draft; +remaining MemWAL lifecycle/read phases **(RFC-026, draft; private v9 compare-and-chain/token-fold core, unbounded retain-all, bounded v11/profile-v2 checked authority, hidden v12/lifecycle-v3 quiescence, v13/recovery-v15 private resume/guarded drain-abort, narrow @@ -1267,11 +1269,12 @@ plus three-disposition retirement implemented; hidden F4 caller-shaped ingress/prepare and format-neutral F5a/F5b0 scheduling are also implemented; managed reclamation is optional later work; no active producer creates reserved `AuthorityBlock`, so its repair remains -fail-closed rather than blocking public enrollment, quiesce, row admission, +fail-closed rather than blocking later lane management, quiesce, operational-status transport, and transport-parity evidence. F5b's bounded deterministic NDJSON object, current `DEAD_LETTERED` authority, fresh ordinary successor correction, selected-token inspection/export, and extended -same-format retirement are active behind the hidden row path. F6b3 closes the +same-format retirement are active behind the engine row path, now reached only +through F7a's graph-level served bridge. F6b3 closes the exact-selected uncovered-tail current-token hit/miss and terminal-page instrument. F6b7 preserves that baseline and adds the paired failpoints-only selected-index decision evidence for current-token and receipt-key lookup work; @@ -1302,7 +1305,7 @@ resource budgets. | **R4: Manifest authority access grows with commit count.** Current-state resolution folds history; a selective index does not by itself bound the complete physical read. | Medium | `optimize` compacts internal tables (keeps periodically-optimized shipped paths flat where separately cost-gated). RFC-024 Gate A rejected durable heads because representative RustFS latest-manifest reads/bytes grow despite flat exact-BTREE row/range work. RFC-025 Gate 0 independently rejected checkpoint-registry activation: at local 10→1,000 on RC.1, uncompacted reconciled work and the eight-fragment tail stay flat, but compacted list/cleanup scan bytes grow 17,012→38,000 cold and 12,336→15,064 warm; exact-show bytes and operation counts also grow. Both RFCs are research-blocked; v8 retains the journal fold and internal-table *cleanup* remains deferred behind the resurrection watermark. | | **R5: Schema identity corruption or alias/identity drift.** Internal schema v5 introduced stable IDs/incarnation as durable authority; v6, v7, and v8 preserve them. | Medium | Open/init validate the SchemaIR domain and exact bidirectional IR↔manifest identity/path/alias contract; every active recovery envelope carries the identity pair; zero, duplicate, missing, or mismatched identity fails closed. | | **R6: Merge cost at divergence** — full-width classification and history-growing manifest folds. | Medium | Coherent coordinator scans plus retained probe handles reduced the pre-slice measured depth-5/depth-80 baseline from 59/651 manifest reads to 40/410 and cap the common fast-forward route at three internal opens and three scans, but the uncompacted-history slope remains. `merge_cost.rs` keeps both facts visible; O(delta) merge is blocked on a real deletion-delta source **(RFC-027)**; fragment adoption is **(draft RFC-0001)**. | -| **R7: No public streaming row path** — production writes are still capped by the `graph_head` CAS rate; high-frequency small writes remain wasteful outside the private evidence seam. | Medium | MemWAL is the strategic substrate. The bounded worker provides watcher-plus-post-fence acknowledgement and conservative replay; its legal near-cap shape closes under logical dense-slice accounting. Schema v9–v18 establish compare-and-chain tokens, checked profile/lifecycle authority, maintenance/rebind, retirement, and exact DataBlock correction. Current v19/recovery-v21 adds the hidden bounded terminal fold, current `DEAD_LETTERED` authority, ordinary successors, cluster/offline inspection/export, and three-disposition retirement. F6b5 activates only exact-terminal served export on the existing HTTP/remote-CLI/OpenAPI route; F6b6 implements the checked read-only operational-status core internally; F6b7 adds paired failpoints-only selected token-index decision evidence; its bounded NO-GO applies only to the uncompacted profile-cycle fixture, schedules no standalone production reconciler, and reopens at greater depth, after a Lance/index-grammar change, or before considering graph-manifest-compacted or checked-Optimize-coupled maintenance. Exact evidence remains in [RFC-026](../rfcs/0026-memwal-streaming-ingest.md) and the [testing map](testing.md). Public row activation remains closed on production enrollment/lifecycle/rebind, operational-status transport parity, and F6's remaining measurement/guardrail acceptance. Managed reclamation and a whole-root history budget are optional later work; a public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | +| **R7: The public graph firehose has a deliberately narrow operating envelope** — F7a serves absent/`OPEN` lanes in the single-writer checked runtime, while lifecycle/status/maintenance transport and overlapping-process ownership remain incomplete. | Medium | MemWAL is the strategic substrate. The bounded worker provides watcher-plus-post-fence acknowledgement and conservative replay; its legal near-cap shape closes under logical dense-slice accounting. Schema v9–v18 establish compare-and-chain tokens, checked profile/lifecycle authority, maintenance/rebind, retirement, and exact DataBlock correction. Current v19/recovery-v21 adds bounded terminal folds, current `DEAD_LETTERED` authority, ordinary successors, cluster/offline inspection/export, and three-disposition retirement. F7a activates graph-native served row ingress by reusing that core and the resident driver; it adds no new coordinator or format. F6b5 separately activates exact-terminal served export; F6b6 implements checked operational status internally; F6b7 adds paired failpoints-only selected token-index evidence and records the uncompacted-profile-cycle bounded NO-GO. Exact evidence remains in [RFC-026](../rfcs/0026-memwal-streaming-ingest.md) and the [testing map](testing.md). Public resume/rebind, operational-status and maintenance transport, and the remaining guardrail acceptance stay open. Managed reclamation and a whole-root history budget are optional later work; a public exact enrollment receipt plus reversible admission seal gates broader overlapping-process topology. | | **R8: Some operations lack enforced memory/time budgets.** | Medium | Known gap, narrowed and accepted for RFC-023. Its direct-substrate instrument rejected the first whole-delta fenced adopt (~447 MB peak at 100K × 256 versus ~74 MB Append), and the first corrected production 10K series failed at 30.0× / 108,625,920 bytes overhead; both negative results remain evidence. Mutation/Load now refuses a keyed table above 8,192 rows / 32 MiB before arm, while BranchMerge uses a recovery-enrolled chain with the same per-chunk bounds and a 1,024-transaction ceiling. The inductive certificate route removes the general diff, temporary delta, target preflight, and target join without weakening that chain. Final five-pair production medians passed at 31/8 ms (3.875×) for 10K and 136/35 ms (~3.886×) for 100K; maximum signed paired RSS overheads were 24,297,472 and 32,604,160 bytes. Inclusive row/transaction ceilings, byte refusal (including materialized blobs), operation-wide validation retention, exact source/target incarnation revalidation, second-generation certificate composition, and both between-chunk recovery directions are pinned; other operations still need explicit bounds. | | **R9: Local-FS conditional-write emulation** (`write_text_if_match` check-then-act gap). | Low | All current callers sit behind the cluster lock protocol; S3 uses true conditional puts; close before admitting any lock-free caller. | | **R10: Doc/spec drift as the system grows** — this document included. | Low | Maintenance contract (same-PR doc updates, `check-agents-md.sh` link CI, "don't lie" stale markers); this canon defers to area docs by construction. | @@ -1398,7 +1401,7 @@ The plan of record is the RFC-022…028 family (all under | [0023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | Substrate-native keyed-write fencing via Lance's unenforced-PK filter; fleet/format activation barrier | **Implemented** (2026-07-15) | | [0024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | Materialized head-row research; the first exact-BTREE candidate bounded scan work but failed the full latest-manifest/object-byte cost gate | **Research blocked** | | [0025 — Checkpoint-pinned retention](../rfcs/0025-checkpoint-retention.md) | Named checkpoints as authoritative retention roots, materialized as Lance tags; current in-manifest registry lookup rejected by Gate 0 | **Research-blocked** | -| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: bounded watcher-plus-post-fence acknowledgement, compare-and-chain token authority, unbounded retain-all, checked profile/lifecycle control, recovery-v15–v20 maintenance/rebind/retirement/correction, and v19/recovery-v21 bounded terminal dead-letter folds plus three-disposition retirement. Hidden lifecycle/rebind/dead-letter, narrow cluster/offline inspection/correction/rebuild exits, F6b5 exact-terminal served export, F6b6 internal checked operational status, and the F6b7 failpoints-only token-index decision instrument (uncompacted-profile-cycle bounded NO-GO; no standalone production reconciler) are implemented; public enrollment/lifecycle and row/status transports remain inactive | **Draft; hidden firehose protocol through F5b plus F6a–F6b7 evidence subsets implemented; F6 remainder and F7 row/control/status activation remain** | +| [0026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | Durability-first streaming writes: bounded watcher-plus-post-fence acknowledgement, compare-and-chain token authority, unbounded retain-all, checked profile/lifecycle control, recovery-v15–v20 maintenance/rebind/retirement/correction, and v19/recovery-v21 bounded terminal dead-letter folds plus three-disposition retirement. Hidden lifecycle/rebind/dead-letter, narrow cluster/offline inspection/correction/rebuild exits, F6b5 exact-terminal served export, F6b6 internal checked operational status, and the F6b7 failpoints-only token-index decision instrument (uncompacted-profile-cycle bounded NO-GO; no standalone production reconciler) are implemented; F7a graph-native served row ingress is active, while public lane enrollment/lifecycle and status transports remain inactive | **Draft; hidden protocol through F5b, F6 evidence, and F7a graph row activation implemented; remaining control/status activation stays staged** | | [0027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | O(delta) merge classification from row-version lineage | Research-blocked | Deliberately split, not one mega-format: identity, key fencing, head rows, diff --git a/docs/dev/firehose-path-specs.md b/docs/dev/firehose-path-specs.md index 6805ef7f..ed280a83 100644 --- a/docs/dev/firehose-path-specs.md +++ b/docs/dev/firehose-path-specs.md @@ -13,7 +13,11 @@ same-binding `SEALED` EnsureIndices plus recovery-v17 same-binding `SEALED` Optimize, recovery-v18 private exact-`SEALED` physical rebind, recovery-v19 lineage-neutral root-wide authority retirement, recovery-v20 exact DataBlock correction, and recovery-v21 mixed/all-diverted dead-letter folds plus -three-disposition retirement. Public ingress, operator lifecycle/rebind verbs, and +three-disposition retirement. F7a activates one graph-native served ingress +surface (`POST /graphs/{graph_id}/stream/ingest`) plus its remote client/CLI +path. It exposes logical node/edge rows only: declaration routing, lazy private +lane preparation, MemWAL ownership, and folding remain behind the graph +boundary. Operator lifecycle/rebind verbs, checked operational status, and every maintenance transport surface remain inactive; retirement and DataBlock inspection/correction are exposed only by narrow offline cluster controls. The format-neutral F4, F5a, and F5b0 hidden-path slices are also implemented: @@ -50,8 +54,9 @@ short cut repeats only mutable witnesses. Only an exact canonical-main recovery participant outcome can explain physical movement as an unavailable projection rather than a movement error. Cold-replay and flushed-LWW pending accounting plus exact oldest-uncovered-token age remain explicit unavailable values; the public -`stream_status` stays nonblocking and manifest-only, and CLI/HTTP/OpenAPI/SDK -transport remains F7. F6b7 supplies covered/reconciled decision evidence; its +`stream_status` stays nonblocking and manifest-only, and its +CLI/HTTP/OpenAPI/SDK transport remains in the F7 remainder. F6b7 supplies +covered/reconciled decision evidence; its configured-RustFS result is a bounded NO-GO only for the uncompacted profile- cycle fixture, so no standalone production reconciler is scheduled. F6b8 closes the resume-to-driver handoff without changing format or recovery: resume @@ -61,7 +66,9 @@ empty-owner housekeeping prepass before the unchanged node-before-edge round so the sole root slot is released promptly. Driver-first, resume-first/caller- cancelled, cross-lane reuse, and clean-shutdown cells are green. The remaining guardrail matrix and the broader post-claim install/retirement-failure matrix -remain in F6; F7 remains forbidden until all of F6 passes. +remain in F6 before the remaining management surfaces activate. F7a does not +weaken those gates because it exposes only the already-proved graph-ingest and +resident-driver composition. **Design authority:** [RFC-026](../rfcs/0026-memwal-streaming-ingest.md) — this file never overrides it. Where they disagree, the RFC wins and this file is wrong. §4.7 records the selected experimental profile; §4.3/§4.6 record the @@ -274,8 +281,9 @@ until a reachable producer and finalized evidence grammar exist. Stopped/offline authority; it is not an import or replay surface. A typed failpoints-only snapshot now exposes process-local driver scheduling evidence to tests; it is explicitly advisory, and its pending triggers are not a durable backlog. -Public durable `StreamStatus` remains manifest-only; public driver status and -every served row/lifecycle/maintenance surface remain inactive. F6b6 adds a +Public durable `StreamStatus` remains manifest-only. F7a activates one +graph-native served row surface; public driver status and lifecycle/maintenance +surfaces remain inactive. F6b6 adds a separate engine-internal checked operational cut over physical, token, recovery, advisory-driver, and rebuild evidence. It has no public transport; `DISABLING` uses explicit checked cluster-apply status authority. Every sidecar @@ -325,7 +333,8 @@ byte/capacity/timing and isolated peak-RSS acceptance described below. | ~~F6b7 token-index decision evidence~~ | Paired current-token and profile-receipt hit/miss work before/after one content-identical exact-selected index refresh, with maintenance I/O and semantic-equivalence proof | no format or recovery change | bounded NO-GO only for the uncompacted profile-cycle fixture; failpoints-only and no standalone production reconciler | | ~~F6b8 resume/driver handoff~~ | Compile-enforced root-producer-permit transfer into detached resume installation, urgent trigger-before-release, exact empty-owner housekeeping before the unchanged node-before-edge round, cancellation-safe shutdown, and cross-lane root-slot reuse | no format or recovery change | implemented behind existing hidden lifecycle/driver seams; broader retirement-failure matrix remains in F6 | | **F6b remainder** | Remaining guardrail matrix, including F6b8's post-claim install/retirement-failure cells; token-index evidence reopens at greater depth, after a Lance/index-grammar change, or before considering graph-manifest-compacted / checked-Optimize-coupled maintenance | — | later | -| **F7** | Remaining served row ingress, lifecycle, maintenance, operational-status transport, and their SDK/HTTP/remote-CLI/OpenAPI parity | — | only after all F6 cells pass; export is already the F6b5 exception | +| ~~F7a served graph ingress~~ | One graph-native mixed node/edge NDJSON route, strong graph-authority precondition, remote client/CLI, OpenAPI, and direct-mode refusals over the existing checked runtime and resident driver | no format or recovery change | implemented; no public table/lane selector or management surface | +| **F7 remainder** | Graph-level lifecycle, maintenance, operational-status transport, and their SDK/HTTP/remote-CLI/OpenAPI parity | — | only after their remaining F6 cells pass; export is already the F6b5 exception | These are dependency milestones, not mandates for giant PRs. Keep each PR reviewable behind the hidden seam: the next lifecycle tranche may land receipts, @@ -2289,7 +2298,8 @@ selected-token inspection/export, and `WITHDRAWN | DEAD_LETTERED` retirement active; the genuine v18↔v19 adjacent-binary and frozen-receipt cell closes the format-boundary evidence. The broader retirement race/failpoint/freeze/export matrix below remains required before -public ingress activates; `AuthorityBlock` repair remains separate. +the remaining management surfaces activate; F7a's narrower ingress evidence +is owned separately, and `AuthorityBlock` repair remains separate. - Failpoints through the hidden candidate-runtime path at acknowledgement, claim, lifecycle, maintenance, both fold participants, canonical dead-letter @@ -2479,32 +2489,40 @@ surfaces and must pass before that activation PR merges. --- -## 8. F7 — atomic public activation - -The remaining server-owned row/control runtime, shared wire DTOs, HTTP/OpenAPI, -remote `GraphClient`, and remote CLI arms land together. F6b5's export arm is -already active, and F6b6's checked status core is ready internally; F7 defines -and transports its unavailable-value/error shape rather than weakening that -cut. The raw physical operations -never become ambient `Omnigraph` writers in this cluster-only profile. By the -time F7 executes, F2 will already have landed the profile adapter and +## 8. F7 — staged public activation + +F7a activates only the server-owned graph-ingest runtime, shared challenge DTO, +HTTP/OpenAPI route, remote `GraphClient`, and remote CLI arm. The public seam is +graph-first: callers submit mixed logical node and edge rows and never select a +Lance dataset, table incarnation, lane, writer, shard, epoch, or generation. +The engine resolves each declaration and reuses the existing lazy private-lane +prepare, checked runtime, bounded request owner, and resident fold driver. It +adds no coordinator, recovery owner, persisted authority, or format strand. + +F6b5's export arm is already active, and F6b6's checked status core remains +internal. Lifecycle, status, and maintenance transport stay in the F7 +remainder and must not be exposed by weakening that cut. Raw physical +operations never become ambient `Omnigraph` writers in this cluster-only +profile. F2 already landed the profile adapter and `cluster apply --confirm-stream-offline`; F7 does not restage that control. | Capability | Owned cluster runtime | HTTP | Remote client / CLI | |---|---|---|---| -| ingest preparation + rows | capability-bound prepare then hidden core | `POST /graphs/{graph_id}/streams/{type_name}/prepare` (JSON), then `POST .../ingest` (NDJSON in/out) | `stream ingest` performs prepare automatically | -| status | F6b6 checked exclusive-cut status | `GET /graphs/{graph_id}/streams[/{type_name}]` | `stream status` | -| fold | explicit operator fold + internal driver fold | `POST .../streams/{type_name}/fold` | `stream fold` | -| quiesce | capability-bound quiesce | `POST .../streams/{type_name}/quiesce` | `stream quiesce` | -| resume / abort | capability-bound resume | `POST .../streams/{type_name}/resume` | `stream resume [--abort-drain]` | +| graph ingest | checked graph authority then the existing hidden core | `POST /graphs/{graph_id}/stream/ingest` (NDJSON in/out); missing `If-Match` returns a bodyless `428` challenge | `stream ingest --data [--graph-token ]` performs the challenge before opening input | +| status | F6b6 checked exclusive-cut status | not exposed in F7a | later graph-level `stream status` | +| fold | internal resident driver; explicit operator fold remains private | not exposed in F7a | later graph-level management surface | +| quiesce | capability-bound quiesce remains private | not exposed in F7a | later graph-level management surface | +| resume / abort | capability-bound resume remains private | not exposed in F7a | later graph-level management surface | | graph export / rebuild artifact | runtime-pinned exact sealed cut | existing `POST /graphs/{graph_id}/export` with stream-aware guards | existing `export --server`; direct `--store` refuses an enrolled graph | -| same-binding maintenance | lifecycle-aware Optimize / EnsureIndices | `POST /graphs/{graph_id}/maintenance/optimize`, `POST /graphs/{graph_id}/maintenance/ensure-indices` | `optimize --server`, `maintenance ensure-indices --server` | +| same-binding maintenance | lifecycle-aware Optimize / EnsureIndices remain private | not exposed in F7a | later graph-level maintenance surface | | physical rebind | no serving runtime; exact terminal `DISABLED` revision + `CheckedClusterMaintenanceAuthority`; accepted schema unchanged | none | disable to `DISABLED`, then `cluster apply --confirm-stream-offline`; later enable/restart/resume | | schema change | no in-place EXP writer; freeze one checked sealed/retired export cut | none | initialize a fresh graph with the desired schema and load the artifact; never load over the enrolled source | -The table contains the five primary stream workflows plus existing export and -maintenance integration. Bodyless prepare is an automatic ingest handshake, -not a sixth workflow. Reachable terminal states retain narrow cluster/offline +The table separates the activated graph row workflow from future graph-level +management plus existing export and private maintenance integration. The strong-ETag +challenge is an authority precondition, not a user-visible lane prepare verb. +After the exact token is accepted, lazy lane preparation remains internal. +Reachable terminal states retain narrow cluster/offline exits without served HTTP/OpenAPI parity: | Cluster/offline support | Command shape | @@ -2514,9 +2532,9 @@ exits without served HTTP/OpenAPI parity: | exact AuthorityBlock repair (**future**) | `cluster stream block repair-authority --confirm-stream-offline` | | authority retirement / rebuild exit | `cluster stream retire-for-rebuild plan|confirm --confirm-stream-offline` | -All request/response types live in `omnigraph-api-types`; pagination, canonical -token/digest parsing, and tagged per-line dispositions are shared rather than -reimplemented in handlers. Every single-lane mutating management call requires +The challenge and HTTP error types live in `omnigraph-api-types`; the engine +owns the canonical newline-terminated, redacted per-line result projection so +handlers do not rederive private evidence. Every future single-lane mutating management call requires its operation ID and expected `lifecycle_revision`, with receipt-first replay; profile apply instead binds the expected profile revision. Root-wide authority retirement binds `(graph identity, AUTHORITY_RETIREMENT, @@ -2529,8 +2547,9 @@ any armed plan before a retry replans against current graph/catalog/lifecycle authority. A no-work retry is therefore a true no-op rather than a replayed terminal receipt. EnsureIndices already implements that engine boundary; Optimize must match it before F7 exposes either route. -Prepare and ingest use `stream_ingest`; lifecycle and fold use -`stream_manage`. Cluster-only DataBlock correction, future AuthorityBlock repair, and +Graph ingest uses one graph-scoped `stream_ingest` decision before body +ownership; lazy private-lane preparation does not add per-table policy checks. +Lifecycle and fold use `stream_manage`. Cluster-only DataBlock correction, future AuthorityBlock repair, and retirement require their exact offline checked authority plus `stream_manage`. Read-only status uses operational-metadata authorization. Dead-letter payload export additionally requires the existing `export` action. @@ -2691,8 +2710,9 @@ advisory driver diagnostics; F6b1–F6b5 close export/process/cost ownership and F6b6 adds the checked read-only operational cut. F6b7 adds the failpoints-only paired token-index decision instrument and records a bounded NO-GO for the uncompacted profile-cycle fixture, so the F6b remainder owns only the remaining -guardrails. F7 alone activates the -remaining row/control/status surfaces. +guardrails. F7a activates the proved graph-ingest composition without exposing +its private lanes; the F7 remainder later owns graph-level control/status +surfaces. This ordering makes every intermediate merge safe: @@ -2711,9 +2731,11 @@ This ordering makes every intermediate merge safe: - after F6b7, tests can compare one exact selected uncovered cut with its content-identical reconciled successor, but no production maintenance owner or recovery protocol exists; the fixture-scoped bounded NO-GO schedules neither; -- after F6, all gates are proved but no compatibility surface is committed; -- F7 exposes the served SDK, HTTP, remote CLI, and OpenAPI together while - direct mutation remains a typed refusal. +- after the ingest-owned F6 evidence, F7a exposes graph-native served SDK, + HTTP, remote CLI, and OpenAPI ingress together while direct mutation remains + a typed refusal; +- the F7 remainder exposes only separately proved graph-level management and + status workflows; it does not turn private lanes into user resources. A performance spike may still invoke the hidden F4/F5a/F5b0 seam after F3. It never lands a production writer or claims an SLO before F6. diff --git a/docs/dev/invariants.md b/docs/dev/invariants.md index 449e86b2..acf3f42d 100644 --- a/docs/dev/invariants.md +++ b/docs/dev/invariants.md @@ -251,7 +251,7 @@ shared side and remain concurrent with one another. | Constructive mutations | In-memory `MutationStaging`, one end-of-query table commit per touched table, then one manifest publish | [writes.md](writes.md), [execution.md](execution.md) | | Keyed writes | Every current-format node/edge table declares exact non-null physical `id` as Lance's unenforced PK from creation, using the v6-introduced and later-preserved exact-`id` fence. General production strict insert and upsert use the sealed exact-`id`, forced-v2 MergeInsert adapter; strict insert exact-probes its pinned parent before minting `omnigraph.insert_absence=v1`, while an all-new upsert may mint it only when completed effect statistics prove one attempt inserted every source row with zero updates, deletes, or skipped duplicates. Certificate admission is optional: BranchMerge uses the shortcut only when every transaction in the complete contiguous source interval carries v1 and its persisted operation is the full pure-insert `Update` shape (exact parent, no removed or updated fragments, nonempty new fragments with `physical_rows`, no field or generation rewrites, `RewriteRows`, exact-`id` filter, full nested schema preorder, and matching physical-row total). It then rechecks both source and target native ref incarnations and passes owned batches through an opaque capability whose one production mint site is structurally guarded. The proven publisher stages immutable fragments with `InsertBuilder`, replaces the uncommitted Append operation with that filter-bearing `Update`, and performs zero target preflights, target merge joins, or committed Appends. Missing, cleaned, unknown, or malformed proof uses the general ordered diff. Mutation/Load remains one keyed transaction per table and rejects more than 8,192 rows or 32 MiB before arm; BranchMerge keeps its bounded v4 chain and exact-recovery limits. Raw Lance graph writers are unsupported, and the certificate is an internal non-cryptographic capability rather than an authenticity mechanism. The final production cost gate passed at 10K (3.875× median; 24,297,472-byte max paired RSS overhead) and 100K (~3.886×; 32,604,160 bytes) | [RFC-023](../rfcs/0023-key-conflict-fencing.md), [writes.md](writes.md), [execution.md](execution.md) | | Deletes | Staged like inserts/updates (`stage_delete` via Lance 7.0 `DeleteBuilder::execute_uncommitted`, MR-A) — no inline HEAD advance; mixed insert/update/delete in one query rejected by D2 as a deliberate boundary (constructive XOR destructive per query; compose via separate mutations or a branch) | [query-language.md](../user/queries/index.md), [writes.md](writes.md) | -| Streaming ingest | RFC-026 adopts Lance MemWAL and remains Draft; public row ingress is inactive. V7–v9 established enrollment, the bounded worker, compare-and-chain token authority, and exact base/token fold; v10–v18 added checked profile/lifecycle authority plus recovery-owned profile, claim, drain, resume, maintenance, rebind, retirement, and DataBlock correction. Current v19/token-schema-v3/recovery-v21 adds deterministic mixed/all-diverted folds, one bounded dead-letter object, current `DEAD_LETTERED` sequencing, ordinary-successor correction, selected-token stopped/offline inspection/export, and three-disposition retirement. F4/F5 provide the hidden caller path, resident folding, deterministic disable, and terminal diversion. F6a–F6b4 provide composition, process/lifecycle, token-cost, and production dead-letter-envelope evidence. F6b1's checked exact-version move-only export cut blocks current terminal authority and owns one nonwaiting root slot; F6b5 now connects it to the existing served HTTP/remote-CLI/OpenAPI export route with pre-header validation, incremental exact-version scans using approximate Lance targets, strict 64-KiB chunks, a complete queue-envelope/process queue reservation under a deadline, backpressure, and disconnect-safe release. The queue reservation is not a whole-response or RSS bound. F6b6 adds an engine-internal checked read-only operational cut over physical lane, token, recovery, advisory-driver, and rebuild evidence. Expensive immutable token/base parity, the bounded sample, index coverage, and selected lifecycle-ledger proofs run once without writer gates; a short second phase validates the exact selected manifest/recovery cut and repeats only mutable witnesses under root/profile/lane fences. It binds the exact runtime owner for `ENABLED`, served-export owner for terminal `DISABLED | RETIRED`, or explicit checked cluster-apply status owner for `DISABLING`. Within the hard status envelope, every pending recovery sidecar is reported and rebuild-blocking; exceeding any discovery bound refuses the whole status; only exact canonical-main recovery ownership makes physical movement explicit unavailable evidence, while overlap or unexplained movement is `StreamStatusChanged`. Cold replay, flushed LWW projection accounting, and exact oldest-uncovered age are unavailable rather than inferred. The public nonblocking `stream_status` remains manifest-only and no CLI/HTTP/OpenAPI/SDK operational-status transport exists. Embedded/direct export of an enrolled graph still refuses; the served graph must be exact terminal `DISABLED | RETIRED`, and `RETIRED` export carries verified provenance. Checked offline disable remains the supported production quiescence owner; same-schema rebind requires terminal `DISABLED` and exact `SEALED` authority, while enrolled schema evolution uses checked export into a fresh graph. There is still no public row ingress/enrollment, general lifecycle/rebind, operational-status transport, fresh read, or generation GC. F6b7 adds a failpoints-only paired selected-index decision instrument for current-token and receipt-key work. Its configured-RustFS result is a bounded NO-GO only for the uncompacted profile-cycle fixture, so no standalone production token-index reconciler is scheduled; the remaining guardrail matrix stays open. No active path produces reserved `AuthorityBlock`, so its repair stays fail-closed. | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | +| Streaming ingest | RFC-026 adopts Lance MemWAL and remains Draft; F7a graph-native served row ingress is active while public management remains inactive. V7–v9 established enrollment, the bounded worker, compare-and-chain token authority, and exact base/token fold; v10–v18 added checked profile/lifecycle authority plus recovery-owned profile, claim, drain, resume, maintenance, rebind, retirement, and DataBlock correction. Current v19/token-schema-v3/recovery-v21 adds deterministic mixed/all-diverted folds, one bounded dead-letter object, current `DEAD_LETTERED` sequencing, ordinary-successor correction, selected-token stopped/offline inspection/export, and three-disposition retirement. F4/F5 provide the hidden caller path, resident folding, deterministic disable, and terminal diversion. F6a–F6b4 provide composition, process/lifecycle, token-cost, and production dead-letter-envelope evidence. F6b1's checked exact-version move-only export cut blocks current terminal authority and owns one nonwaiting root slot; F6b5 now connects it to the existing served HTTP/remote-CLI/OpenAPI export route with pre-header validation, incremental exact-version scans using approximate Lance targets, strict 64-KiB chunks, a complete queue-envelope/process queue reservation under a deadline, backpressure, and disconnect-safe release. The queue reservation is not a whole-response or RSS bound. F6b6 adds an engine-internal checked read-only operational cut over physical lane, token, recovery, advisory-driver, and rebuild evidence. Expensive immutable token/base parity, the bounded sample, index coverage, and selected lifecycle-ledger proofs run once without writer gates; a short second phase validates the exact selected manifest/recovery cut and repeats only mutable witnesses under root/profile/lane fences. It binds the exact runtime owner for `ENABLED`, served-export owner for terminal `DISABLED | RETIRED`, or explicit checked cluster-apply status owner for `DISABLING`. Within the hard status envelope, every pending recovery sidecar is reported and rebuild-blocking; exceeding any discovery bound refuses the whole status; only exact canonical-main recovery ownership makes physical movement explicit unavailable evidence, while overlap or unexplained movement is `StreamStatusChanged`. Cold replay, flushed LWW projection accounting, and exact oldest-uncovered age are unavailable rather than inferred. The public nonblocking `stream_status` remains manifest-only and no CLI/HTTP/OpenAPI/SDK operational-status transport exists. Embedded/direct export of an enrolled graph still refuses; the served graph must be exact terminal `DISABLED | RETIRED`, and `RETIRED` export carries verified provenance. Checked offline disable remains the supported production quiescence owner; same-schema rebind requires terminal `DISABLED` and exact `SEALED` authority, while enrolled schema evolution uses checked export into a fresh graph. F7a exposes one graph-only mixed node/edge NDJSON route and remote command behind a strong derived graph-authority ETag, reusing lazy private-lane preparation and the resident driver without a new coordinator or format. Public lane enrollment, general lifecycle/rebind, operational-status transport, fresh read, and generation GC remain inactive. F6b7 adds a failpoints-only paired selected-index decision instrument for current-token and receipt-key work. Its configured-RustFS result is a bounded NO-GO only for the uncompacted profile-cycle fixture, so no standalone production token-index reconciler is scheduled; the remaining guardrail matrix stays open. No active path produces reserved `AuthorityBlock`, so its repair stays fail-closed. | [RFC-026](../rfcs/0026-memwal-streaming-ingest.md), [writes.md](writes.md) | | Branch create/delete | `__manifest` `BranchContents` is the single logical authority. Lance create is physically two-phase, so OmniGraph prevalidates names, enforces path-prefix-disjoint live graph names, reclaims an absent-ref clone-only tree, and uses a bounded completion classifier; delete removes authority before tree cleanup, so an absent ref is success and derived tree reclaim may converge later. Neither control emits graph lineage. Under schema/source-target/all-table gates, each control uses one operation-local post-gate manifest/namespace capture rather than refreshing the handle-local coordinator around table-gate acquisition; successful ref movement explicitly invalidates derived read caches. Per-table forks are derived state, reclaimed best-effort with `cleanup` as backstop. A target-scoped unresolved sidecar may be made unreachable by deletion and is then audit-discarded by recovery; graph-global SchemaApply still blocks. Public create/create-from/delete also acquire the stream-export root slot nonwaitingly, preventing native named-branch delete/recreate ABA while an immutable cut is live | [branches-commits.md](../user/branching/index.md), [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Cleanup retention | Explicit cleanup derives exact `keep` cutoffs from Lance's available version list, caps each main-table GC cutoff at the oldest exact main version inherited by any live lazy graph branch, and refuses uncovered main HEAD drift. Lance protects native per-table branch refs itself. The graph-wide live-reference preflight fails closed before the first table GC, after which individual table failures remain fault-isolated | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md) | | Optimize visibility | One operation-local accepted catalog and fresh main snapshot are planned under schema → main → all-table gates. Every productive table shares one identity-bearing v9 recovery envelope; bounded-parallel physical effects become visible through at most one monotonic manifest/lineage CAS. Complete crash residuals roll forward together and partial residuals compensate before visibility. Optimize's payload remains a bounded maintenance adapter rather than an exact caller-minted Lance transaction proof, so it is supported within the single-writer-process recovery boundary. Exact provenance is deferred until Lance exposes a stable public caller-controlled maintenance transaction API and OmniGraph has distributed recovery fencing | [maintenance.md](../user/operations/maintenance.md), [writes.md](writes.md), [RFC-022](../rfcs/0022-unified-write-path.md) | @@ -288,7 +288,7 @@ Do not hide these behind invariant wording. Either move them forward or keep them explicit. - **Private B2 token/fold core and B2a retain-all profile are implemented; - public row streaming is not active:** RC.1's public + F7a graph-native served row streaming is active:** RC.1's public initializer commits the singleton MemWAL index internally and shard claim is a separate durable effect. RFC-026 no longer treats an upstream release as the only calendar path: a production-neutral harness proved exact @@ -306,8 +306,9 @@ them explicit. delete ambiguous artifacts. Phase A then activated historical internal schema v7 with a recovery-v10 enrollment intent, identity-keyed lifecycle CAS, process-local exclusion, compatible-open validation, partial-format refusal, - and strict v6↔v7 rebuild. Its private enrollment adapter can create only an - exact empty enrollment; no production caller can put or acknowledge a row. + and strict v6↔v7 rebuild. At that checkpoint its private enrollment adapter + could create only an exact empty enrollment; no production caller could put + or acknowledge a row. RC.1's writer-wide durability watermark is unsafe with batch positions that reset after MemTable rollover, `put_no_wait` may mutate before returning an error, replay @@ -342,8 +343,10 @@ them explicit. effect-free exact eligibility witness, and only an exact echo may reuse the existing recovery-v14 enrollment path. Durable same-request retries return the actor-bound receipt and concurrent request IDs converge on one validated - lane. The witness is never persisted and introduces no format shape. Neither - seam is an incremental NDJSON, transport, or product surface. Blob-bearing + lane. The witness is never persisted and introduces no format shape. Those + low-level seams remain private; F7a composes them behind the incremental + graph-native NDJSON route and exposes no eligibility witness or lane identity. + Blob-bearing tables are refused before any MemWAL put: Lance's LSM fold scanner returns Blob descriptors rather than foldable logical Blob values, so exact-generation Blob materialization across warm and cold recovery remains a separate diff --git a/docs/dev/lance-memwal-pr.md b/docs/dev/lance-memwal-pr.md index 16b5de3d..b5bbb1e0 100644 --- a/docs/dev/lance-memwal-pr.md +++ b/docs/dev/lance-memwal-pr.md @@ -83,11 +83,12 @@ are implemented and evidence-green at a deliberately narrow boundary: and v9 lifecycle-v2/recovery-v12 state cross that boundary only through export/init/load. -The core is intentionally reachable only through a feature-gated, doc-hidden -engine seam. Checked profile control exists, including restart-stable -`DISABLING`, but there is no `@stream` schema intent, public enrollment, SDK -ingest method, HTTP route, CLI ingest command, OpenAPI surface, or enrolled-lane -operator workflow. The hidden core implements recovery-covered empty/non-empty +The lane core remains private and feature-gated tests own its fault matrix. +F7a reaches it through one doc-hidden graph bridge plus the checked HTTP, +remote-CLI, and OpenAPI row surface. Checked profile control exists, including +restart-stable `DISABLING`, but there is no `@stream` schema intent, public lane +enrollment, embedded/direct SDK ingest method, or enrolled-lane operator +workflow. The hidden core implements recovery-covered empty/non-empty `OPEN → DRAINING → SEALED` plus crate-private `SEALED → OPEN` resume and guarded `DRAINING → OPEN` abort, checked `SEALED` maintenance, and checked offline physical rebind. Public/production rebind and control/transport @@ -106,7 +107,7 @@ retention. The measured one-exclusive-fold RSS delta was 286,441,472 bytes (about 273 MiB); CI's 384-MiB threshold is a remeasurement tripwire, not a runtime allocator limit. -### Public streaming is still specified but inactive +### Graph row streaming is active; controls remain staged RFC-026 Phase B2 specifies the remaining public contract: @@ -135,10 +136,12 @@ cluster-only authority-retirement and root-receipt plus closed selected-branch-member export exit; v18/recovery-v20 adds exact DataBlock correction; and v19/token-schema-v3/recovery-v21 adds terminal dead-letter folds, selected-token inspection/export, and three-disposition retirement. -Checked offline disable is the supported quiescence owner. Public enrollment, -general resume/abort, physical rebind, SDK/HTTP/OpenAPI row streaming, public -operational-status transport, and every other streaming product surface remain -inactive. F6b6's checked read-only operational cut is implemented internally. +Checked offline disable is the supported quiescence owner. F7a activates +graph-native HTTP/remote-CLI/OpenAPI row streaming over absent or `OPEN` lanes. +Public lane enrollment, general resume/abort, physical rebind, +embedded/direct SDK row ingress, operational-status transport, and every other +streaming management surface remain inactive. F6b6's checked read-only +operational cut is implemented internally. ## The missing Lance capabilities diff --git a/docs/dev/rfc-022-027-architecture-review.md b/docs/dev/rfc-022-027-architecture-review.md index 26dfcacf..dfdcd049 100644 --- a/docs/dev/rfc-022-027-architecture-review.md +++ b/docs/dev/rfc-022-027-architecture-review.md @@ -121,7 +121,8 @@ the full MemWAL table and system-index specifications, including the durability/fencing behavior carried forward from beta.17, are reflected in the RFC's survey and acceptance guards. Gate E0 passed at the explicit bounded process/topology profile and Phase A consumed it as described above; RFC-026 -remains draft and public row streaming is inactive. Gate R0's historical +remains draft, and F7a now activates only graph-native served row streaming. +Public lane management remains inactive. Gate R0's historical finite-retention no-go, the selected unbounded B2a disposition, the optional B2b route, and the common contract inventory are recorded in RFC-026 §0.2, §4.1–§4.6, and BLOCKER-14 below. @@ -561,8 +562,9 @@ owner; v17/recovery-v19 adds terminal stopped/offline authority retirement and receipt-bearing export; v18/recovery-v20 adds stopped/offline exact DataBlock correction; current v19/token-schema-v3/recovery-v21 adds deterministic mixed/all-diverted terminal folding and three-disposition retirement while -leaving public/production rebind inactive. RFC-026 -remains draft and public row streaming remains inactive. +leaving public/production rebind inactive. RFC-026 remains draft; F7a exposes +one graph-native served row route, while lane enrollment/control, rebind, and +operational-status transport remain inactive. Enrollment creates persistent MemWAL metadata and `stream_state` changes the correctness preconditions for schema, branch, maintenance, and data operations. diff --git a/docs/dev/testing.md b/docs/dev/testing.md index d23a9887..b827b19a 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -19,6 +19,17 @@ export route using that exact-terminal binding and doc-hidden move-only engine cut; it does not activate row ingress or lifecycle/maintenance/status surfaces. The dedicated F6b5 ownership section below lists the current tests. +**F7a amendment:** graph-native row ingress is now the narrow public exception. +`memwal_stream.rs` owns the checked-runtime bridge, bodyless token challenge, +redacted ordered results, and existing driver composition. Server +`auth_policy.rs`, `data_routes.rs`, and `multi_graph.rs` own pre-body auth/media/ +runtime refusal plus the enabled-runtime 428/412/200 route composition; +`openapi.rs` pins the generated contract. CLI in-source `client::tests` owns +preflight-before-file-open, single-use streaming request/response bytes, and +no replay after 412, while `planes::tests` pins served graph addressing and +direct-mode refusal before dispatch. No test exposes a table/lane selector or +adds a lifecycle, status, or maintenance route. + The engine's `tests/` is the principal coverage surface; most graph-shaped behavior is exercised there. ## CI control-plane tests @@ -53,10 +64,10 @@ it is not inferred from a local syntax check. See [ci.md](ci.md). | `merge_fast_forward.rs` | Branch-adopt cost + correctness under RFC-023. The one-batch and 8,193-row fixtures prove that a complete v1 insertion-absence history chain publishes one/two bounded exact-`id` filtered `Update` transactions with zero target strict-insert preflights, target MergeInsert joins, committed Appends, ordered-cursor scans, or whole-delta staged combines. `pure_insert_fast_forward_retains_value_constraint_validation` proves the certificate skips only redundant key work, not logical row constraints; its all-new Upsert source is certified from completed effect statistics. `proven_fast_forward_certificate_composes_across_merge_generation` proves the publisher re-mints v1 and a second merge consumes that output as the next proof-chain link. A missing intermediate transaction proves cleaned history is an optimization miss: the merge enters the general ordered diff, preserves exact rows, and leaves no recovery residue. `lazy_target_ref_only_fast_forward_uses_pin_after_main_advances` distinguishes a valid old lazy graph pin from drift when the inherited main ref advances. A nested `main → feature → experiment` cell prevents a deeper valid `BranchIdentifier` from becoming a false read-set conflict. Every general-route base/source/target `OrderedTableCursor` scan applies both Lance `batch_size(8,192)` and `batch_size_bytes(32 MiB)`. Validation streams projected `id`/`src`/`dst`/scalar batches, charges exact Arrow memory before retention, and shares one 32 MiB operation-wide budget across candidate tables; `branch_merge_validation_delta_is_aggregate_bounded_pre_arm` crosses it with two individually valid ~18 MiB deltas while proving zero HEAD/manifest/lineage/sidecar movement. Deletes use exact escaped-filter chunks with the same row/byte and retained-plan bounds. Production-helper unit cells pin chain/delete/recovery limits. The subprocess scenario owns the final production latency/RSS evidence; these integration tests own route semantics, not timings | | `writes.rs` | Direct-publish writes: cancellation, RFC-022 non-strict full-attempt reprepare from fresh branch authority, strict stale-write conflicts, multi-statement atomicity, MR-794 staged-write rewire (D₂ rejection, insert+update coalesce, multi-append coalesce, partial-failure recovery, load RI/cardinality recovery); RFC-023 pins the inclusive 8,192-row keyed input ceiling, the same exact/+1 boundary on streamed mutation-update matches, no-effect state for both refusals, and oversized stored-Blob rejection before payload read. Crate-internal pending-scan cells pin inclusive/+1 32 MiB accounting plus pending-key shadow-before-charge. The lance#7444 row-id-overlap regression (`filtered_read_after_merge_update_and_delete_keeps_row_ids_consistent` — merge-load → same-key merge-load → delete → keyed point lookup, green only under the vendored lance-table patch — plus its append-only control) | | `src/table_store/staged_tests.rs` | Crate-internal staged primitives. RFC-023 pins one exact target preflight for general StrictInsert, durable v1 mint/commit/reopen/history persistence, exact-`id` filter emission, typed `KeyConflict`, and missing/wrong PK refusal. `all_new_upsert_certifies_insert_absence_and_persists_it_in_history` proves an all-new completed Upsert receives the optional certificate, a mixed/update Upsert does not, unrelated transaction properties survive, and UUID rebinding does not erase it. Proven-insert cells show the opaque path performs zero strict preflights; stages with `InsertBuilder` but commits the full pure-insert `Update` shape (exact parent and `id` filter, `RewriteRows`, no updates/removals, full nested schema preorder, physical rows); persists/re-admits its own output for proof composition; leaves new fragments outside old index coverage; and fails same-key races loudly in proven/proven and proven/general orders. The in-source `exec/merge.rs` certificate unit table rejects missing/unknown properties, wrong parent/filter/full-preorder/mode/offsets, rewrite/removal shapes, missing `physical_rows`, and Append. Source-interval cells pin exact selection, lazy retained-parent splitting, coalescing, and pinned Lance's approximate raw-emission boundary while every normalized/writer chunk remains hard-capped. Generic `stage_append`/`stage_merge_insert` remain primitive tests only. The file also owns index staging and `commit_staged{,_exact}` | -| `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. At the RFC-026 Phase-A checkpoint the guard registered only the exact v10 enrollment gateway and feature-gated test seam, counted its sidecar/index/shard durability primitives, and kept every row-put/ack/fold surface absent; the Phase-B1 owner below exact-counts only the approved crate-private put/fold durable-call sites. V11 adds only the bounded checked offline profile/runtime authority factories and recovery-v13 profile gateway; the guard must keep ambient constructors and public row, enrollment, drain, claim, lifecycle, SDK, HTTP, CLI, and OpenAPI side doors absent. F5a classifies the doc-hidden start/shutdown bridge as composed recovery-v14 fold orchestration while adding no durable primitive allowance for the supervisor module itself. F6b1 structurally pins the doc-hidden export cut as private-field, move-only, and non-forgeable while adding no public export seam. The embedded SDK's read-only `stream_status` is classified separately and must remain free of durable calls. B2a inventories the only production `_mem_wal` literal owners, forbids MemWAL reclamation/adoption symbols and destructive primitives in the adapter, keeps generic maintenance unaware of MemWAL, and keeps raw inventory/classifier helpers private. This remains defense in depth rather than macro expansion, alias, or data-flow analysis; the visibility boundary and behavior tests are still primary. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | +| `forbidden_apis.rs` | Defense-in-depth syntax-tree/source guard over the whole engine. The primary boundary is Rust visibility: raw storage/coordinator/handle-cache modules are crate-private; public `Snapshot::open` returns `SnapshotTable`; and `SnapshotScanner` executes reads without exposing Lance's raw scanner or physical plan. The guard pins those visibility/return-type boundaries, classifies public async inherent `Omnigraph` methods plus loader conveniences, classifies every crate-visible async method on `GraphCoordinator` / `ManifestCoordinator`, and exact-counts registered method/UFCS durable-call shapes including recovery. RFC-023 rejects production graph call sites of generic `stage_append{,_stream}` and `proven_insert_capability_has_one_production_mint_site` pins `ProvenInsertChunk::from_verified_history` to the complete-history classifier in `exec/merge.rs`, preventing the no-preflight capability from becoming a reusable bypass. At the RFC-026 Phase-A checkpoint the guard registered only the exact v10 enrollment gateway and feature-gated test seam, counted its sidecar/index/shard durability primitives, and kept every row-put/ack/fold surface absent; the Phase-B1 owner below exact-counts only the approved crate-private put/fold durable-call sites. V11 adds only the bounded checked offline profile/runtime authority factories and recovery-v13 profile gateway; the guard must keep ambient constructors and public row, enrollment, drain, claim, lifecycle, SDK, HTTP, CLI, and OpenAPI side doors absent. F5a classifies the doc-hidden start/shutdown bridge as composed recovery-v14 fold orchestration while adding no durable primitive allowance for the supervisor module itself. F6b1 structurally pins the doc-hidden export cut as private-field, move-only, and non-forgeable while adding no public export seam. F7a registers exactly one doc-hidden public graph-ingest bridge as composed recovery-v14/v21 ownership while retaining the ban on raw table/lane writers and public management side doors. The embedded SDK's read-only `stream_status` is classified separately and must remain free of durable calls. B2a inventories the only production `_mem_wal` literal owners, forbids MemWAL reclamation/adoption symbols and destructive primitives in the adapter, keeps generic maintenance unaware of MemWAL, and keeps raw inventory/classifier helpers private. This remains defense in depth rather than macro expansion, alias, or data-flow analysis; the visibility boundary and behavior tests are still primary. It also counts selected raw `SnapshotHandle` / Dataset shapes, rejects renamed-owner/macro/include/path-lookalike forms, skips structurally test-only code, and pins retired escape hatches absent. `// forbidden-api-allow: ` exempts reviewed inline-Lance lines only | | `lance_surface_guards.rs` | Pins the Lance API surfaces omnigraph depends on (named runtime + compile-only guards; see [lance.md](lance.md)) — the first smoke check on any Lance version bump. `cached_and_zero_cache_sessions_share_store_registry_not_metadata_cache` proves a cached data Session and zero-cache control Session reuse one live `ObjectStoreRegistry` client while their metadata caches remain isolated. `_compile_uncommitted_full_table_vector_index_shape` pins the public `IndexMetadata` shape suitable for `Operation::CreateIndex`; `compact_files_succeeds_on_blob_columns` pins blob-v2 compaction; Guard 9 pins clone-only branch reclaim semantics. RFC-023's `unenforced_pk_filter_shape_is_route_dependent` explicitly forces v2 versus indexed routes and pins the `Some(populated)` / `Some(empty)` / `None` key-filter shapes; `unenforced_pk_conflict_matrix_is_directional` pins the directional filtered/unfiltered and filtered/Append matrix. RFC-024's compile guard pins the public `BranchIdentifier` + current table version + current `Transaction.uuid` + `ManifestLocation.e_tag` current-HEAD witness; the local/shared-`Session` guard proves unchanged-reopen stability, ordinary-commit movement, and same-version ABA, while RustFS covers object-store ABA. RFC-025 adds exact main/named-branch tag-target, sparse cleanup pin/unpin, and branch-tree-deletion guards. RFC-026 pins doc-hidden `has_successor_version`, initializer/readback/shard-writer/durability/fencing, flush/drain, replay watermark, scanner, and merged-generation shapes; runtime Gate E0 classification belongs to `memwal_enrollment_gate.rs`, while v7 Phase-A and v8 B1 publication/recovery belong to the manifest/failpoint suites. B2b adds `cleanup_old_versions_does_not_reclaim_mem_wal_objects` and `mem_wal_deleted_fence_slot_allows_stale_writer_success_on_pinned_lance`: the first proves generic cleanup leaves the present MemWAL fixture unchanged and the second proves deleting the successor's empty fence sentinel is unsafe. The pinned source audit, not those two tests alone, establishes that stock RC.1 exposes no owned MemWAL reclamation API. The RC.1 compiler guard pins the five surveyed public Lance virtual system-column constants to early `.pg` rejection. These guards prove substrate shapes/tokens and negative ownership boundaries; they do not by themselves prove heads/checkpoint activation, the current publisher, or a safe reclamation implementation | | `memwal_enrollment_gate.rs` | RFC-026's green production-neutral Gate E0 harness, isolated from the production manifest and graph writer. Fourteen substantive local cells plus one explicit unconfigured-S3 skip cover exact no-effect / `N + 1` index / pre-minted empty-shard classification, buried-effect refusal, marker survival, strict inventory/error handling, and the broad fail-closed matrix. The rejected first instrument used `checkout_latest` plus `IOTracker`, which missed local `read_dir`. The accepted exact-version classifier pins doc-hidden `has_successor_version`; its `AttemptTracker` records failed/`NotFound` attempts before forwarding and proves the identical complete six-attempt shape at baseline versions 8/80: four successful manifest HEADs, one `NotFound` manifest HEAD, one successful manifest GET, zero lists. A Unix execute-only `_versions` tripwire proves exact probing works when latest enumeration fails and an unreadable exact HEAD errors. The configured RustFS exact cell passes non-vacuously with the same zero-list shape and owns the positive lost-result/index/empty-shard/reopen sequence plus foreign shard, malformed/loose root, durable WAL, persisted cursor, and corrupt-manifest negatives. S3 ABA remains in `lance_surface_guards.rs`; CI rejects skipped E0/ABA cells. This file never mutates production manifest/schema state or deletes ambiguous artifacts; Phase A consumes its classifier through the private adapter | -| `memwal_stream.rs` | Feature-gated RFC-026 private B1 mechanics, B2 compare-and-chain behavior, B2a provider-failure evidence, and the hidden lifecycle-v3 integration. B1 owns bounded put/ack/replay, authority, cancellation, and manifest-only visibility, including row-local value rejection before any WAL or manifest effect. B2 owns idempotency conflicts, same-generation overlays, stale-authority recapture, durable attribution, and one watcher/fence result over a distinct-key contiguous multi-row physical prefix. The hidden F4 request proof additionally pins graph-scoped `stream_ingest` policy and exact checked-runtime authority before body work; separate root-wide and per-actor transport admission before polling; incremental NDJSON framing across accepted chunks, CRLF, EOF, and over-limit-line boundaries without whole-request retention; strict `$stream` parsing; duplicate, unknown, and reserved-field refusal; explicit canonical IDs; dense schema-ordered node/edge conversion; scalar/list/enum/vector and value-constraint validation before recovery entry; and effect-free Blob-table refusal through a deliberately stale pre-schema-apply handle before either the request or lower B1/B2 seam can invoke MemWAL. Run-splitting cells cover invalid lines, repeated keys, token dispositions, and row/byte ceilings; bounded result/reorder ownership preserves caller order and stop-tail `blocking_ordinal` precedence, while disconnect stops new body polling/admission and transfers the invoked tail to root-owned settlement. The graph-native hidden slice pins strict mixed node/edge rows with no caller-visible table key, one graph-scoped policy decision, catalog-resolved lazy enrollment, move-only bounded normalization, declaration handoff through the existing finite node-before-edge driver round, graph-wide ambiguous-result blocking, and a scalar logical result whitelist with no physical evidence. The bodyless prepare cells pin effect-free witness challenge, checked-runtime/policy ordering, Blob refusal before enrollment, actor-bound durable-receipt replay, and two concurrent request IDs converging on one OPEN lane before ingest/fold composition. F5a extends this same owner with checked-runtime start refusal, coalesced timer/cap folding, trigger-during-fold preservation, cold-reopen discovery without an in-memory pending bit, deterministic finite node-before-edge rounds, root singleflight, retry/backoff visibility, and bounded supervisor shutdown; every effect is still proved by the existing recovery-v14 fold/crash cells rather than a duplicate replay suite. F6a adds one hidden in-process candidate-runtime composition: prepare, ordered NDJSON, an automatic mixed visible/dead-letter fold, stopped/offline selected-token list/export, an ordinary corrected successor, driver restart, clean shutdown ownership, and checked offline disable. F6b1 adds ambient-enrolled pre-byte refusal; managed and unmanaged-terminal checked success; checked `WITHDRAWN | DEAD_LETTERED` pre-byte refusal; one immutable exact-version cut across a later writer; sole nonwaiting root-slot exclusion and release; named-branch delete/recreate exclusion; and preservation of a post-start storage error with slot release. It remains inaccessible to production callers and exposes no SDK/HTTP/CLI/API/OpenAPI ingress, public driver status, or public rebind surface. Lifecycle-v3 owns recovery-covered cold/fold claims, exact full-generation projection after flush/reopen, recovery-v14 ordinary/drain folds, empty and non-empty `OPEN → DRAINING → SEALED`, an empty successor after an ordinary published fold, durable/reopen-stable typed `DataBlock` publication with no base or graph commit (including a fresh-source minimum-cardinality violation whose streamed edge supplies correction identity), idempotent same-request restart, conflicting request/stale-revision refusal, and the claim-before-seal plus seal-before-fold crash boundaries. Recovery-v15 adds receipt-first idempotent `SEALED → OPEN` resume, guarded `DRAINING → OPEN` abort, higher-epoch claim, terminal receipt publication, named-branch refusal, and current-binding-chain ancestry checks. F6b8 adds compile-enforced root-producer ownership transfer through detached resume installation, urgent trigger-before-release, exact empty-owner housekeeping before the unchanged node-before-edge round, prompt retirement, driver-first and resume-first/caller-cancelled races, cross-lane slot reuse, and shutdown waiting for the detached owner. The strict-block path streams `DRAINING` validation directly into the bounded evidence collector; unit owners below pin the detailed cap, empty-evidence refusal, and non-materialized overflow digest. F3d's checked-offline cell releases the served runtime, reaches terminal `DISABLED`, crashes after arming v18 but before a physical effect, and proves retry selects one fresh `SEALED` scope while retaining the old MemWAL inventory; repeating the same occurrence is physically effect-free. B2a injects a recording/failing store at the real Lance table-store boundary and covers post-invocation ambiguity and inert orphan residue. No test here implies a transport-backed/public ingress or a supported lifecycle API. | +| `memwal_stream.rs` | Feature-gated RFC-026 private B1 mechanics, B2 compare-and-chain behavior, B2a provider-failure evidence, and the hidden lifecycle-v3 integration. B1 owns bounded put/ack/replay, authority, cancellation, and manifest-only visibility, including row-local value rejection before any WAL or manifest effect. B2 owns idempotency conflicts, same-generation overlays, stale-authority recapture, durable attribution, and one watcher/fence result over a distinct-key contiguous multi-row physical prefix. The hidden F4 request proof additionally pins graph-scoped `stream_ingest` policy and exact checked-runtime authority before body work; separate root-wide and per-actor transport admission before polling; incremental NDJSON framing across accepted chunks, CRLF, EOF, and over-limit-line boundaries without whole-request retention; strict `$stream` parsing; duplicate, unknown, and reserved-field refusal; explicit canonical IDs; dense schema-ordered node/edge conversion; scalar/list/enum/vector and value-constraint validation before recovery entry; and effect-free Blob-table refusal through a deliberately stale pre-schema-apply handle before either the request or lower B1/B2 seam can invoke MemWAL. Run-splitting cells cover invalid lines, repeated keys, token dispositions, and row/byte ceilings; bounded result/reorder ownership preserves caller order and stop-tail `blocking_ordinal` precedence, while disconnect stops new body polling/admission and transfers the invoked tail to root-owned settlement. The graph-native hidden slice pins strict mixed node/edge rows with no caller-visible table key, one graph-scoped policy decision, catalog-resolved lazy enrollment, move-only bounded normalization, declaration handoff through the existing finite node-before-edge driver round, graph-wide ambiguous-result blocking, and a scalar logical result whitelist with no physical evidence. The bodyless prepare cells pin effect-free witness challenge, checked-runtime/policy ordering, Blob refusal before enrollment, actor-bound durable-receipt replay, and two concurrent request IDs converging on one OPEN lane before ingest/fold composition. F5a extends this same owner with checked-runtime start refusal, coalesced timer/cap folding, trigger-during-fold preservation, cold-reopen discovery without an in-memory pending bit, deterministic finite node-before-edge rounds, root singleflight, retry/backoff visibility, and bounded supervisor shutdown; every effect is still proved by the existing recovery-v14 fold/crash cells rather than a duplicate replay suite. F6a adds one hidden in-process candidate-runtime composition: prepare, ordered NDJSON, an automatic mixed visible/dead-letter fold, stopped/offline selected-token list/export, an ordinary corrected successor, driver restart, clean shutdown ownership, and checked offline disable. F6b1 adds ambient-enrolled pre-byte refusal; managed and unmanaged-terminal checked success; checked `WITHDRAWN | DEAD_LETTERED` pre-byte refusal; one immutable exact-version cut across a later writer; sole nonwaiting root-slot exclusion and release; named-branch delete/recreate exclusion; and preservation of a post-start storage error with slot release. It remains inaccessible to production callers and exposes no SDK/HTTP/CLI/API/OpenAPI ingress, public driver status, or public rebind surface. Lifecycle-v3 owns recovery-covered cold/fold claims, exact full-generation projection after flush/reopen, recovery-v14 ordinary/drain folds, empty and non-empty `OPEN → DRAINING → SEALED`, an empty successor after an ordinary published fold, durable/reopen-stable typed `DataBlock` publication with no base or graph commit (including a fresh-source minimum-cardinality violation whose streamed edge supplies correction identity), idempotent same-request restart, conflicting request/stale-revision refusal, and the claim-before-seal plus seal-before-fold crash boundaries. Recovery-v15 adds receipt-first idempotent `SEALED → OPEN` resume, guarded `DRAINING → OPEN` abort, higher-epoch claim, terminal receipt publication, named-branch refusal, and current-binding-chain ancestry checks. F6b8 adds compile-enforced root-producer ownership transfer through detached resume installation, urgent trigger-before-release, exact empty-owner housekeeping before the unchanged node-before-edge round, prompt retirement, driver-first and resume-first/caller-cancelled races, cross-lane slot reuse, and shutdown waiting for the detached owner. The strict-block path streams `DRAINING` validation directly into the bounded evidence collector; unit owners below pin the detailed cap, empty-evidence refusal, and non-materialized overflow digest. F3d's checked-offline cell releases the served runtime, reaches terminal `DISABLED`, crashes after arming v18 but before a physical effect, and proves retry selects one fresh `SEALED` scope while retaining the old MemWAL inventory; repeating the same occurrence is physically effect-free. B2a injects a recording/failing store at the real Lance table-store boundary and covers post-invocation ambiguity and inert orphan residue. F7a extends this owner with the production checked-runtime graph token bridge, redacted newline result contract, and a non-resetting 50-ms same-declaration coalescing boundary that acknowledges a complete row while its body stream remains pending; server and CLI owners cover transport. No test here implies a supported public lifecycle API. | | `memwal_stream_cost.rs` | Feature-gated RFC-026 B1, Gate-R0, and B2a decision instrument. It separately measures warm already-claimed durability acknowledgement, cold replay, selected-generation fold scanning, visibility, retained merged metadata, the uncompacted graph-manifest term, legal no-roll estimates, and paired peak RSS. Gate R0 adds a revision-pinned source-audit tripwire, strict current-object classification/reference census, listed path/class/size retain-all comparisons at one/four/eight folds, referenced-cut retry reuse, and deterministic high-entropy near-cap local/configured-RustFS cells. The near-cap cell proves the exact B2-attributed boundary through the real adapter: 3,742 payload bytes per row admits 8,192 rows at 33,550,336 logical bytes, while 3,743 is rejected effect-free at 33,558,528 bytes. The legal generation acknowledges without graph visibility, then folds and publishes exactly once after logical-slice charging plus dense per-scanner-batch take. The reference-environment paired fold peak-RSS lift measured 286,441,472 bytes (about 273 MiB), below a one-sided 384-MiB remeasurement tripwire; common initialization may censor that lifetime high-water lift to zero or a negative value on another runner, and the tripwire is not a runtime allocator limit. B2a adds 1/8/32/128 local and configured-RustFS retained-history sweeps whose terms remain separate: warm ack, cold reopen/replay, fold, visibility, MemWAL/base-table/token-authority/other table-store work, graph-manifest/adapter work, advisory current-object bytes, and whole-process peak RSS. Older retained roots must receive zero reads, writes, or deletes. The only allowed delete shape is Lance's losing manifest-CAS `.binpb.tmp.` staging; canonical durable MemWAL delete requests remain zero. LIST totals, wall times, and RSS are advisory—not a quota, SLO, isolated WAL slope, or provider billing. A green test proves private closure/retention behavior; it does not activate a public API. | | `memwal_stream_cost.rs` (F6b3) | Exact-selected uncovered-tail current-token cost owner. The normal local 1/8-cycle cell and ignored local/configured-RustFS 1/8/32/128 sweeps hold current-token/page cardinality fixed while growing immutable token-ledger receipt history; graph-manifest history also advances during setup. Per sample they report selected version, lookup-index coverage, serialized page bytes, and cumulative advisory whole-process RSS. Fresh-handle hit/miss plus the first terminal page, then warm hit/miss and repeat terminal pages, report token-read counts, total table-store read bytes, manifest reads/bytes, adapter operations, and the applicable per-sample warm/repeat p50 plus max-of-eight. This historical F6b3 fixture did not claim a cold provider cache, receipt-key lookup, or a covered/reconciled curve; F6b7 adds the latter two terms. | | `memwal_stream_cost.rs` (F6b7) | Paired selected-token lookup-index decision owner. A failpoints-only test writer creates and selects one content-identical fully covered `CreateIndex` cut, then the same fixture measures current-token and profile-receipt hit/miss on both cuts. The 2026-08-03 configured-RustFS 6/20/68/260-fragment uncompacted profile-cycle sweep records a bounded NO-GO for that physical shape: all four recurring terms retain a 3× token-table read-request ratio and the deepest byte term remains 2.084×, but total maintenance request-cost amortization grows to 1,697 calls at 260 fragments, above the predeclared 1,000-call ceiling. No standalone production reconciler is scheduled; remeasure beyond 260 exact uncovered fragments, after a Lance-pin/index-grammar change, or before coupling reconciliation to graph-manifest compaction/checked Optimize. | @@ -205,8 +216,9 @@ and member digests, selected index, sibling digest, and source schema hash. This evidence activates only the stopped/offline retirement/export exit. The later F3f suite separately owns the production `WITHDRAWN` correction path; -`memwal_stream.rs` owns hidden F4/F5a evidence. Public ingress and lifecycle -transports remain outside both slices. +`memwal_stream.rs` owns hidden F4/F5a evidence and the engine side of the F7a +graph bridge. Server and CLI suites own the public F7a ingress transport; +general lifecycle transports remain outside these slices. ## Fixtures @@ -660,18 +672,17 @@ the stopped/offline `cluster stream block show|correct` DataBlock surface. No test in the v11 profile-authority slice activated ingress, enrollment, claim, or ordinary lifecycle mutation. The current v19 hidden lifecycle owners are `memwal_stream.rs` plus the v14/v15/v16/v17/v18/v19/v20/v21 in-source -recovery/lifecycle suites described above; those tests still do not activate a -production row or ordinary lifecycle transport surface. - -The remaining B1/B2a and private B2-common work adds no parser/server/ingest-CLI -tests because it has no public row surface. The -common product contract inventory specifies the missing contracts without -activating them. Any public implementation -must extend the existing compiler, server/OpenAPI, CLI parity, Cedar, shutdown, -audit, no-raw-GC, provider-failure, and genuine rebuild suites together with explicit -enrollment, persistent revisioned status/quiesce/resume/abort-drain, bounded -terminal management receipts, compare-and-chain tokens, attribution, and strict -correction. The selected retain-all profile has no byte/object/file/history +recovery/lifecycle suites described above. F7a extends the existing graph-row +owner and adds server/CLI/OpenAPI coverage; those tests still do not activate +an ordinary lifecycle transport surface. + +The historical B1/B2a and private B2-common slices added no +parser/server/ingest-CLI tests because they had no public row surface. F7a now +extends the existing server/OpenAPI, CLI, Cedar, shutdown, no-raw-GC, and +provider-failure ownership for graph ingress without activating lane controls. +Future public management must additionally cover persistent revisioned +status/quiesce/resume/abort-drain, bounded terminal management receipts, +attribution, and strict correction. The selected retain-all profile has no byte/object/file/history quota; its tests must instead prove that provider exhaustion is loud and cannot drop an acknowledgement or bypass recovery/manifest visibility. Storage watermarks and graph-history admission controls belong only to a future B2b diff --git a/docs/dev/versioning.md b/docs/dev/versioning.md index cdf3a184..99618342 100644 --- a/docs/dev/versioning.md +++ b/docs/dev/versioning.md @@ -225,17 +225,24 @@ recovery and does not reinterpret recovery-v14's frozen `StreamTokenLedgerIndexMaintenance` placeholder. Any later production reconciler therefore needs its own exact recovery grammar/strand. +F7a also changes no storage, recovery, or persisted wire version. Its strong +graph-ingest token is derived from existing graph identity, accepted schema, +profile revision, and fold delegation; it is an HTTP authority precondition, +not persisted state. The graph route reuses recovery-v14 lazy enrollment and +recovery-v21 terminal outcomes without reinterpreting either grammar. + Recovery-v13 remains exactly the v11 profile-change protocol. Historical recovery-v10 enrollment and recovery-v12 lifecycle-v2 folds retain their old wire meanings and are refused under lifecycle-v3 rather than synthesized. -There is still no public firehose ingress, public production enrollment, -quiesce, resume/abort, physical rebind, or streaming or maintenance -transport surface. The narrow offline operator surfaces are +F7a activates graph-native served firehose ingress plus remote CLI/OpenAPI +parity. It exposes no table/lane selector or physical evidence. Public lane +enrollment, quiesce, resume/abort, physical rebind, operational status, and +maintenance transport remain inactive. The narrow offline operator surfaces are `cluster stream retire-for-rebuild plan|confirm` and exact `DataBlock` inspection/correction through `cluster stream block show|correct`, plus manifest-selected current dead-letter inspection through `cluster stream dead-letter list|export`. Their recovery owners remain -crate-private; none activates public row ingress or ordinary lifecycle control. +crate-private; none widens the graph row route or activates ordinary lifecycle control. F6b3 closes the exact-selected uncovered-tail current-token hit/miss and terminal-page instrument; F6b7 closes the paired uncovered/reconciled current-token and receipt-key decision evidence. F6b4 already closes isolated diff --git a/docs/dev/wal-options.md b/docs/dev/wal-options.md index d2261955..227b47e3 100644 --- a/docs/dev/wal-options.md +++ b/docs/dev/wal-options.md @@ -54,7 +54,9 @@ Option 4 remains useful precedent for the acknowledgement boundary; OmniGraph's selected profile keeps stronger fencing, recovery, sequencing, and terminal- disposition requirements than the surveyed LanceDB path. -No option in this document authorizes public streaming today. +This options analysis selected the substrate and acknowledgement boundary; F7a +later authorizes the narrow graph-native served row path. It does not authorize +public lane controls, embedded/direct SDK ingest, or overlapping-process writers. ## The contracts that must not be conflated diff --git a/docs/dev/wal-thinking.md b/docs/dev/wal-thinking.md index 01eb9590..c9704fee 100644 --- a/docs/dev/wal-thinking.md +++ b/docs/dev/wal-thinking.md @@ -1,6 +1,6 @@ # WAL Thinking -Working notes, updated 2026-08-01. Plain-language grounding for the WAL/streaming +Working notes, updated 2026-08-04. Plain-language grounding for the WAL/streaming discussion ([RFC-018](../rfcs/0018-ingest-wal.md) → [RFC-026](../rfcs/0026-memwal-streaming-ingest.md)). Three parts: the contract difference between an interactive graph commit and durable stream @@ -11,7 +11,9 @@ Current boundary: RFC-026 Phase A, the Phase-B1 private core, private B2a retain-all, the common-B2 compare-and-chain core, bounded profile authority, hidden lifecycle-v3 quiescence, private resume/maintenance, checked offline physical rebind, terminal authority retirement/export, and stopped/offline -DataBlock correction are built, but public streaming is not. Gate R0 found +DataBlock correction are built. F7a now exposes the graph-native served row +path over absent or `OPEN` lanes; public lane enrollment, resume/status/ +maintenance transport, and embedded/direct SDK ingress are not active. Gate R0 found and the follow-up fixed the one known all-shape closure failure: a legal high-entropy near-cap generation is durably acknowledged, materialized, folded, and published without lowering the @@ -32,19 +34,21 @@ batch, acknowledge only after its Lance durability watcher and the same writer's post-durability epoch check both succeed, route replay or one flushed-unmerged generation fold-only, and publish one exact fold at the `__manifest` CAS. It prevents MemTable rollover and retires the writer before a -successor generation can put. This is implementation/evidence machinery, not a -product surface. There is still no `@stream`, public enrollment or row -admission, SDK/HTTP/CLI/OpenAPI route, operator drain/resume workflow, or fresh -read. V9 adds config-v3/state-v2, the grammar-impossible trusted attribution +successor generation can put. That seam remains implementation/evidence +machinery; F7a reaches it only through the graph bridge. There is still no +`@stream`, public lane enrollment, embedded/direct SDK row admission, operator +drain/resume workflow, operational-status transport, or fresh read. V9 adds +config-v3/state-v2, the grammar-impossible trusted attribution field, manifest-selected token authority, compare-and-chain admission, and recovery-v12's exact base-plus-token publication. V10 added graph profile enablement; v11 replaces its boolean with checked profile protocol v2 and adds exact recovery-v13 `StreamProfileChange`. V12 replaces inline lifecycle history with fixed-size ledger heads and recovery-v14 hidden enrollment, writer claims, ordinary/drain folds, and terminal management receipts. Its private seam can -quiesce empty and non-empty lanes `OPEN → DRAINING → SEALED`. Public -ingress/enrollment/quiesce, resume/abort, correction, retirement, and every -product surface are still inactive. RFC-026 remains Draft. +quiesce empty and non-empty lanes `OPEN → DRAINING → SEALED`. At that v12 +slice, every row/control product surface was inactive. F7a later activates only +the served graph row path; quiesce/resume/abort and ordinary management remain +private. RFC-026 remains Draft. RFC-026 selects **unbounded retain-all on stock Lance** as the first storage profile, and its private B2a gate is implemented. OmniGraph deletes no canonical @@ -63,8 +67,8 @@ monotonic storage and loud provider-capacity exhaustion. Managed reclamation and the Lance patch are deferred optimizations. The compare-and-chain token and trusted attribution core, checked profile authority, and hidden lifecycle-v3 claim/fold/quiesce path are implemented. The stopped/offline retirement/export -exit is active; production lifecycle control, correction, and product-parity -contracts remain inactive. +exit is active; F7a graph row transport is active, while production lifecycle +control and management/status parity remain inactive. --- diff --git a/docs/dev/writes.md b/docs/dev/writes.md index e42f4c4c..f27b215f 100644 --- a/docs/dev/writes.md +++ b/docs/dev/writes.md @@ -902,9 +902,10 @@ deletable. Measurements validate those bounds; they do not create them. Generic Lance cleanup does not reclaim `_mem_wal`, and OmniGraph must never delete its raw paths. Internal schema v9/config-v3/state-v2/recovery-v12 is active for the private row/fold slice; v11/profile-v2/recovery-v13 adds only checked profile -authority and its exact receipt. Every public wire surface remains inactive -until cancellation, enrolled-lane lifecycle/correction/status, compatibility, -and parity evidence passes in a later strict strand. The selected +authority and its exact receipt. F7a exposes only graph-native served row +ingress over absent or `OPEN` lanes and reuses that private core. Public lane +enrollment, resume/correction/status/maintenance transport, and embedded/direct +SDK row ingress remain inactive pending their own evidence. The selected unbounded retain-all profile has no physical watermark and no graph-history quota. If a future B2b bounded/managed profile is pursued, its physical watermark is per binding and does not bound base/token or shared manifest @@ -1471,8 +1472,8 @@ recovery-v16 private SEALED EnsureIndices; v15 activates the distinct are not supported general lifecycle APIs. Historical v10 enrollment, v12 fold, and v14 resume/maintenance/correction/retirement sidecars are refused, not reinterpreted. Checked offline disable is the sole - active quiescence owner; public ingress/enrollment and general - resume/abort/rebind integration remain inactive. + active quiescence owner. Graph-native served ingress is active; explicit + enrollment and general resume/abort/rebind integration remain inactive. The stamp history (v1 PK-less, v2 unenforced-PK, v3 `__run__*` sweep, v4 lineage in `__manifest` with the commit-graph tables retired, v5 stable table identity, diff --git a/docs/dev/writing-path-state-of-affairs.md b/docs/dev/writing-path-state-of-affairs.md index 3fe12bc7..4814328c 100644 --- a/docs/dev/writing-path-state-of-affairs.md +++ b/docs/dev/writing-path-state-of-affairs.md @@ -114,20 +114,22 @@ cluster state, profile/declaration revision, actor, and runtime lifetime. `DISABLING` persists an exact restart/resume plan with drain-only continuation authority. Checked offline apply is the one active quiescence owner: it freezes admission, drains a finite manifest-selected lane cut, and resumes that stored -plan after restart. Public ingress/enrollment, general lifecycle -resume/abort/rebind, and maintenance/status transports remain inactive; +plan after restart. F7a activates one graph-native served-ingress route and +remote CLI/OpenAPI contract; declaration routing and lazy lane enrollment stay +private. Public lane enrollment, general lifecycle resume/abort/rebind, and +maintenance/status transports remain inactive; retirement, `stream block show|correct`, and `stream dead-letter list|export` are the narrow cluster-only CLI exceptions. -The RFC remains Draft. Row admission remains reachable only through a -feature-gated private engine seam, while the checked cluster/offline controls -above are active. The graph-scoped Cedar vocabulary, public manifest-only -status, checked profile-control boundary, and engine-internal checked -operational-status cut exist, but there is still no `@stream` -syntax, production/public enrollment or put/ack/fold API, -enrolled-lane drain/resume/correction surface, HTTP/CLI/OpenAPI streaming -contract, or fresh-read mode. +The RFC remains Draft. Row admission is reachable through the checked served +graph bridge at `POST /graphs/{graph_id}/stream/ingest`; its engine mechanics +remain private and feature-gated tests own their fault matrix. The graph-scoped +Cedar vocabulary, public manifest-only status, checked profile-control +boundary, and engine-internal checked operational-status cut exist, but there +is still no `@stream` syntax, public lane-enrollment/fold API, enrolled-lane +drain/resume/correction surface, operational-status transport, or fresh-read +mode. This page answers four practical questions: @@ -311,7 +313,7 @@ These operations do not create a side door around the protocol: | Recovery | Same-process handles share ordered queues for the same canonical local root or identical normalized opaque remote/custom URI. Stream-admission domains are acquired outside profile-shared → schema → branch → stream-token → table gates where applicable. Mutation/load, SchemaApply, BranchMerge, EnsureIndices, Optimize, active v14–v21 stream operations, StreamProfileChange, and `refresh` heal roll-forward-only; Repair and Cleanup refuse pending recovery, while branch controls use specialized barriers. Read-write open performs the quiesced full sweep, including v14 enrollment/claim/fold/terminal-receipt, v15 resume/abort, v16 SEALED EnsureIndices, v17 SEALED Optimize, v18 SEALED physical rebind, historical v19 two-disposition retirement, v20 exact `DataBlock` correction, v21 terminal fold/three-disposition retirement, and v13 profile-receipt completion; read-only open never repairs and refuses unresolved stream recovery. Historical recovery-v10 enrollment, v11 B1 fold, v12 lifecycle-v2 fold, and v14 resume/maintenance/retirement/rebind are refused. A resolved graph-lineage intent is audited internally with the original actor when present; retirement instead uses its selected immutable receipt/profile chain and emits no graph lineage or `RecoveryAudit`. No-effect cleanup need not create graph lineage. | | RFC-026 Phase A (v7 foundation) | Internal schema v7 introduced identity-keyed lifecycle rows and exact empty main/unsharded enrollment. At that Phase-A-only boundary, any lifecycle row—including `SEALED`—rejected base-table, schema, maintenance, repair-adoption, and recovery effects under a process-local admission lease because no drain/fold witness-update adapter existed. Native branch create/delete alone could proceed at `SEALED` because it did not move table HEAD; `OPEN`/`DRAINING` refused it. Enrollment and open-time validation rejected named-branch overlap and any uncovered lifecycle/MemWAL mismatch. Current v19 preserves these foundation guarantees while adding the guarded lifecycle-v3 claim/fold/quiesce/resume transitions and explicit SEALED EnsureIndices/Optimize/rebind bridges. | | RFC-026 Phase B1 (historical private core) | Internal schema v8/config-v2 added one root-scoped, cross-handle serialized worker and a hard-bounded 8,192-row/32-MiB logical dense-slice Arrow no-roll generation. Watcher success proves durability; a clean `DurableBatchAck` additionally requires the same `ShardWriter::check_fenced()` to succeed immediately afterward. Fence loss, epoch-read failure, owner-task failure, or deadline ambiguity is post-invocation `AckUnknown` plus worker retirement. Reopen/replay is conservative and exact drain proof precedes quiesced abort. Gate R0's deterministic legal high-entropy near-cap cell exposed sparse scanner arrays; logical-slice charging plus dense copies repaired it. Physical RSS is evidence only. V9 preserves those worker/closure mechanics, but recovery-v11 itself is historical only. | -| RFC-026 private B2 token/fold core + profile/lifecycle authority | Internal schema v9/config-v3/state-v2 adds canonical payload/token digests, hidden trusted row metadata, exact idempotency and compare-and-chain classification, same-generation token overlays, and post-admission authority recapture. `_stream_tokens.lance` is graph-global durable sequencing state selected only by its manifest witness. V11/profile-v2 adds checked stopped/offline and runtime ownership, resumable `DISABLING`, fail-closed `RETIRED`, and exact recovery-v13 profile receipts. V12/lifecycle-v3 replaces inline histories with fixed-size ledger heads; recovery-v14 covers enrollment, writer claims, ordinary/drain folds, and terminal management receipts. V13–v20 add private resume, SEALED maintenance/rebind, authority retirement, and DataBlock correction. Current v19/token-schema-v3/recovery-v21 adds deterministic terminal diversion and extends retirement to `WITHDRAWN | DEAD_LETTERED`. The hidden one-lane core can cycle `OPEN → DRAINING → SEALED → OPEN`; checked offline apply is the supported `OPEN/DRAINING → SEALED` owner. Public enrollment, row admission, general lifecycle/rebind control, status transport, and maintenance transport remain inactive. F6b5 is the narrow served HTTP/remote-CLI/OpenAPI exception: exact terminal checked export streams a pinned cut with bounded transport ownership. F6b6 implements an engine-internal checked read-only operational cut with explicit checked `DISABLING` authority and complete pending-sidecar inventory inside a hard advisory envelope: 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, 32 MiB per sidecar body, and 32 MiB of cumulative bodies. Exceeding any bound is a typed refusal, never partial status. The cut retains honest recovery/unavailable physical evidence and unavailable cold-replay/flushed/oldest-age fields. Retirement, DataBlock inspection/correction, and selected current dead-letter list/export remain narrow cluster-only CLI controls. F6 owns remaining measurements and guardrail acceptance. | +| RFC-026 B2 token/fold core + profile/lifecycle authority + F7a graph ingress | Internal schema v9/config-v3/state-v2 adds canonical payload/token digests, hidden trusted row metadata, exact idempotency and compare-and-chain classification, same-generation token overlays, and post-admission authority recapture. `_stream_tokens.lance` is graph-global durable sequencing state selected only by its manifest witness. V11/profile-v2 adds checked stopped/offline and runtime ownership, resumable `DISABLING`, fail-closed `RETIRED`, and exact recovery-v13 profile receipts. V12/lifecycle-v3 replaces inline histories with fixed-size ledger heads; recovery-v14 covers enrollment, writer claims, ordinary/drain folds, and terminal management receipts. V13–v20 add private resume, SEALED maintenance/rebind, authority retirement, and DataBlock correction. Current v19/token-schema-v3/recovery-v21 adds deterministic terminal diversion and extends retirement to `WITHDRAWN | DEAD_LETTERED`. The private one-lane core can cycle `OPEN → DRAINING → SEALED → OPEN`; checked offline apply is the supported `OPEN/DRAINING → SEALED` owner. F7a exposes graph-native served row admission over absent or `OPEN` lanes while keeping lane enrollment private. Public lane enrollment, resume/rebind control, status transport, and maintenance transport remain inactive. F6b5 separately activates exact-terminal served HTTP/remote-CLI/OpenAPI export with bounded transport ownership. F6b6 implements an engine-internal checked read-only operational cut with explicit checked `DISABLING` authority and complete pending-sidecar inventory inside a hard advisory envelope: 256 matching direct `.json` sidecars, 256 irrelevant direct-or-nested objects encountered below the prefix, 4 MiB of cumulative input-anchored URI bytes across all encountered objects, 32 MiB per sidecar body, and 32 MiB of cumulative bodies. Exceeding any bound is a typed refusal, never partial status. The cut retains honest recovery/unavailable physical evidence and unavailable cold-replay/flushed/oldest-age fields. Retirement, DataBlock inspection/correction, and selected current dead-letter list/export remain narrow cluster-only CLI controls. F6 owns remaining measurements and guardrail acceptance. | | Lance access | One process-wide `ObjectStoreRegistry` reuses clients. Each `Omnigraph` handle owns its cached data-table `Session`; one process-wide zero-cache control `Session` opens mutable tips. Only the object-store registry is shared between the data and control sessions. This is “cache the past, never the present,” not one global cached session. | | Maintenance | EnsureIndices stages exact missing-index transactions. Its private checked-runtime recovery-v16 path can refresh an enrolled table only at exact `SEALED`; ambient calls remain refused. Optimize coordinates graph-wide compaction/index work under one bounded recovery envelope; its private recovery-v17 path applies the same checked-runtime, exact-`SEALED` pointer/proof refresh while ambient calls remain refused for enrolled tables. Recovery-v18 separately rebinds an exact `SEALED` lane without opening it. Periodic optimize compacts `__manifest`, but unmaintained history-dependent paths are not globally flat. | @@ -336,7 +338,7 @@ retry rules, recovery classification, and the full owned limits. | [023 — Key-conflict fencing](../rfcs/0023-key-conflict-fencing.md) | **Implemented** | Internal schema v6 introduced exact-`id` PK metadata, closed keyed routing, typed conflicts, bounded replay, rebuild/refusal, and accepted performance evidence; v19 preserves that contract. | | [024 — Durable table heads](../rfcs/0024-durable-table-heads.md) | **Research-blocked** | The first in-manifest BTREE candidate has a specified logical contract and flat indexed row/range work, but fails the complete physical-I/O gate. No head rows or heads format are active. | | [025 — Checkpoint retention](../rfcs/0025-checkpoint-retention.md) | **Research-blocked** | Lance tag/pin semantics pass, but the proposed in-manifest registry access shape is not history-flat after compaction. No checkpoint rows, `ogcp_` production tags, API, or cleanup integration are active. | -| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; hidden ingest/fold/lifecycle core, narrow cluster/offline controls, exact-terminal served export, and internal checked status implemented; no public row path** | Schema v9 introduced config-v3/state-v2 token authority plus recovery-v12 exact base+token fold. V11–v21 add checked profile/lifecycle ownership, recovery-covered claims/quiesce/resume/maintenance/rebind/correction/retirement, deterministic terminal folds, exact retry/ordinary successor, and selected-token inspection. Historical recovery payloads retain their exact grammar. F6b3/F6b7 provide uncovered and failpoints-only paired reconciled token-index evidence; F6b7 records a bounded standalone-reconciler NO-GO for the uncompacted profile-cycle fixture, so recovery-owned maintenance remains open only to new evidence (including graph-manifest-compacted/checked-Optimize coupling). F6b4 closes the dead-letter envelope; F6b5 connects the move-only exact-terminal cut to existing HTTP/remote-CLI/OpenAPI export with strict chunk/queue ownership; F6b6 implements the checked read-only operational core internally. Public status transport and remaining guardrails stay open. Checked offline disable is the supported production quiescence owner; production enrollment, general resume/abort/rebind, public row admission, and all served row/lifecycle/maintenance/status surfaces remain inactive. B2b managed reclamation remains optional future work. | +| [026 — MemWAL streaming ingest](../rfcs/0026-memwal-streaming-ingest.md) | **Draft; graph-native served row ingress, hidden lane/fold/lifecycle core, narrow cluster/offline controls, exact-terminal served export, and internal checked status implemented** | Schema v9 introduced config-v3/state-v2 token authority plus recovery-v12 exact base+token fold. V11–v21 add checked profile/lifecycle ownership, recovery-covered claims/quiesce/resume/maintenance/rebind/correction/retirement, deterministic terminal folds, exact retry/ordinary successor, and selected-token inspection. Historical recovery payloads retain their exact grammar. F6b3/F6b7 provide uncovered and failpoints-only paired reconciled token-index evidence; F6b7 records a bounded standalone-reconciler NO-GO for the uncompacted profile-cycle fixture, so recovery-owned maintenance remains open only to new evidence (including graph-manifest-compacted/checked-Optimize coupling). F6b4 closes the dead-letter envelope; F6b5 connects the move-only exact-terminal cut to existing HTTP/remote-CLI/OpenAPI export with strict chunk/queue ownership; F6b6 implements the checked read-only operational core internally. Public status transport and remaining guardrails stay open. Checked offline disable is the supported production quiescence owner. F7a activates one graph-level mixed node/edge NDJSON row route and remote CLI/OpenAPI contract over the existing lazy private-lane prepare and resident driver. Public lane enrollment, general resume/abort/rebind, and served lifecycle/maintenance/status surfaces remain inactive. B2b managed reclamation remains optional future work. | | [027 — Lineage merge deltas](../rfcs/0027-lineage-merge-deltas.md) | **Research-blocked** | The desired O(delta) classifier and fallback contract are specified. Selective live-row and deletion-delta discovery are not yet bounded, so `OrderedTableCursor` remains the correctness path. | | [028 — Stable schema identity](../rfcs/0028-stable-schema-identity.md) | **Implemented** | Rename-stable IDs, table incarnation, identity-derived paths, schema/recovery integration, and strict rebuild activation were introduced in v5 and remain active in v19. | diff --git a/docs/releases/v0.10.0.md b/docs/releases/v0.10.0.md index 41c257b8..1ab72a87 100644 --- a/docs/releases/v0.10.0.md +++ b/docs/releases/v0.10.0.md @@ -1,5 +1,27 @@ # OmniGraph v0.10.0 (unreleased) +## Experimental graph-native firehose ingress + +Cluster-served graphs with `streaming: true` now accept mixed node/edge NDJSON +at `POST /graphs/{graph_id}/stream/ingest` while each selected declaration lane +is absent (and prepared lazily) or `OPEN`. F7a does not expose resume, so a +`SEALED` lane after disable/re-enable remains closed. The URL, Cedar resource, remote +client, and `omnigraph stream ingest --data ` command are graph-level; +Lance datasets, table incarnations, and MemWAL lanes remain private runtime +details. The engine reuses the existing lazy enrollment, recovery, bounded +request owner, and resident fold driver. This activation adds no coordinator, +recovery version, or storage-format strand. + +Before a body is owned, the endpoint requires a strong graph-authority ETag. +Missing `If-Match` returns an effect-free `428` challenge; malformed or stale +authority returns `412` without a replacement token. The CLI completes the +missing-token challenge before opening its file or stdin and never +automatically replays an owned body after `412`. A successful response is +ordered NDJSON containing only logical row identity, disposition, and safe +sequencing evidence. A `durable` result means Lance MemWAL durability; the +resident driver publishes visibility later through the atomic graph commit +path. Embedded/direct `--store` firehose ingestion remains unsupported. + ## Experimental streaming-profile ownership This release moves the experimental graph-wide streaming profile behind the @@ -16,13 +38,13 @@ The confirmation is an operator attestation of the stop/apply/restart handoff, not a distributed lease. Run the apply offline and restart the server after it completes. -`streaming: true` is **not additive in this release**. While the durable -profile is enabled, embedded SDK and direct `--store` Mutation/Load/delete +`streaming: true` is **not additive to the existing direct-writer surfaces**. +While the durable profile is enabled, embedded SDK and direct `--store` Mutation/Load/delete calls are refused before body/file reads, staging, recovery arming, or Lance effects. Existing served mutations are authorized only by the exact checked -runtime attached during cluster server boot. There is no public firehose -ingress endpoint yet. Branch merge is refused while the profile is `ENABLED` -or `DISABLING`, even through that served runtime. +runtime attached during cluster server boot; graph-native firehose clients use +the endpoint above. Branch merge is refused while the profile is `ENABLED` or +`DISABLING`, even through that served runtime. Profile apply evaluates `stream_manage` under both the currently applied graph policy and the desired graph policy. A simultaneous policy change must allow @@ -127,7 +149,8 @@ without prefix listing or a second inventory. Payload export is inspection, not replay or import. Recovery-v21 also extends irreversible retirement to an exact `WITHDRAWN | DEAD_LETTERED` cut. Recovery-v19 and recovery-v20 retain their historical meanings. No public HTTP, SDK, remote-CLI, or OpenAPI row -surface is activated. F6b3 closes the exact-selected uncovered-tail current- +surface was activated by that F5b slice; F7a above later activates the narrow +graph-native served row path. F6b3 closes the exact-selected uncovered-tail current- token hit/miss and terminal-page instrument. F6b7 preserves that baseline and adds a paired failpoints-only exact-selected lookup-index decision instrument: current-token and profile-receipt hit/miss work is measured before and after a diff --git a/docs/rfcs/0026-memwal-streaming-ingest.md b/docs/rfcs/0026-memwal-streaming-ingest.md index cd8871b3..280fd3f7 100644 --- a/docs/rfcs/0026-memwal-streaming-ingest.md +++ b/docs/rfcs/0026-memwal-streaming-ingest.md @@ -58,7 +58,8 @@ unavailable rather than falsely inconsistent. Cold-replay pending accounting, flushed LWW projection accounting, and exact oldest-uncovered-token age are reported as unavailable rather than inferred. The already-public `Omnigraph::stream_status` remains the nonblocking manifest-only projection; -CLI/HTTP/OpenAPI/SDK transport for the operational shape remains F7. +CLI/HTTP/OpenAPI/SDK transport for the operational shape remains in the F7 +remainder. F6b8 closes the format-neutral resume-to-driver ownership handoff: resume transfers its non-clone root producer permit into detached writer installation and arms an urgent trigger before that transfer can release. Under the @@ -66,14 +67,17 @@ exclusive root fence, the driver snapshots and retires only exact empty owners in a housekeeping prepass before its unchanged node-before-edge round. Driver- first, caller-cancelled resume-first, cross-lane reuse, and clean-shutdown cells are green. The broader post-claim install/retirement-failure matrix remains F6 -work. -Public row streaming, public enrollment, general lifecycle/rebind verbs, -`AuthorityBlock` repair, +work. F7a activates graph-native mixed node/edge row streaming through one +checked-runtime HTTP/remote-client/CLI/OpenAPI surface and strong graph- +authority precondition. It reuses hidden lazy enrollment and the resident +driver, exposes no lane selector or physical evidence, and changes no format +or recovery grammar. +Public lane enrollment, general lifecycle/rebind verbs, `AuthorityBlock` repair, the standalone production token-index reconciler deferred by F6b7's uncompacted- profile-cycle bounded NO-GO, the F6b-remainder -guardrail acceptance, and SDK/HTTP/OpenAPI row-ingress, -lifecycle, operational-status, or maintenance surfaces remain inactive. -Stream-aware served export is the narrow activated transport exception. +guardrail acceptance, and SDK/HTTP/OpenAPI lifecycle, operational-status, or +maintenance surfaces remain inactive. +Stream-aware served export remains the separate terminal-cut transport. Retirement, DataBlock show/correct, and current dead-letter list/export are exposed only by narrow stopped/offline cluster controls. **Date:** 2026-07-10 @@ -968,24 +972,21 @@ branch-scoping question is proven by a surface guard and end-to-end test. Later overlapping-process enrollment/failover still requires the upstream receipt / admission lifecycle or a separately accepted distributed fence. -Neither public profile hides enrollment inside the first ingest row. The full -non-experimental B2 contract leaves an `@stream` table `UNENROLLED` until its -standalone explicit enrollment request. The selected experimental profile -amends that surface with §4.7 P2's automatic **bodyless prepare** handshake: -the client obtains a complete `StreamEligibilityWitness`, sends a -caller-minted non-nil `enrollment_request_id`, and receives the engine-minted -logical `stream_incarnation_id` before any row body is owned. It reuses the -physical mechanics above but arms only recovery-v14 `StreamEnrollmentV2`, -whose fixed actor/witness intent and result are retained in actor-bound -`EnrollmentReceiptV2`; historical `protocol_v10` is not reinterpreted. -Same ID/actor/intent after a lost result returns that receipt; another actor or -intent conflicts; concurrent prepare losers resolve through the winner's -complete receipt and return bounded `already_enrolled`. A successful prepare -with no subsequent body intentionally leaves an empty `OPEN` lane. Actual -ingest still requires the exact returned incarnation on every row, and an -absent lane returns request-level `StreamPrepareRequired` before body -ownership. Thus row-body ingest never creates enrollment and there is no -first-row exception. +The full non-experimental B2 contract leaves an `@stream` table `UNENROLLED` +until its standalone explicit enrollment request. The selected experimental +profile instead keeps enrollment private behind §4.7 P2's graph adapter. The +public bodyless step is the graph-authority ETag challenge, not a table prepare +or incarnation exchange. After an exact graph token transfers body ownership, +the first logical row for an absent declaration may invoke the existing +recovery-v14 `StreamEnrollmentV2` prepare before that row's admission. Its +fixed actor/witness intent and engine-minted result remain retained in +actor-bound `EnrollmentReceiptV2`; historical `protocol_v10` is not +reinterpreted. Same ID/actor/intent after an internal lost result returns that +receipt, another actor or intent conflicts, and concurrent prepare losers +converge through the winner's complete receipt. A disconnect after prepare may +leave an empty `OPEN` lane, which the existing lifecycle owner handles. The +adapter injects the private stream incarnation into the row call; the client +neither supplies nor observes it. ## 4. B2 contract and future public activation @@ -2744,44 +2745,59 @@ or an advisory counter. ### 4.6 Public surface after the gates close The shipped `POST /graphs/{id}/ingest` path remains the deprecated, compatible -alias of `/load`. Streaming receives a new, non-conflicting surface: +alias of `/load`. F7a gives streaming one graph-native, non-conflicting row +surface: ```text -POST /graphs/{graph_id}/streams/{type_name}/enroll -POST /graphs/{graph_id}/streams/{type_name}/ingest?branch=main -GET /graphs/{graph_id}/streams -GET /graphs/{graph_id}/streams/{type_name} -GET /graphs/{graph_id}/streams/{type_name}/blocks/{block_token} -POST /graphs/{graph_id}/streams/{type_name}/fold -POST /graphs/{graph_id}/streams/{type_name}/quiesce -POST /graphs/{graph_id}/streams/{type_name}/resume -POST /graphs/{graph_id}/streams/{type_name}/correct -POST /graphs/{graph_id}/streams/{type_name}/rebuild-preflight +POST /graphs/{graph_id}/stream/ingest ``` +The URL and policy resource stop at the graph. A caller may mix logical node +and edge declarations in one ordered request; it never selects a physical +dataset, table incarnation, MemWAL lane, writer, shard, epoch, or generation. +Declaration resolution, lazy private-lane preparation, and automatic folding +remain engine/runtime responsibilities. F7a adds no durable coordinator or +format state. Graph-level lifecycle, status, and maintenance routes require +their own later design and evidence; the former type/lane-specific route sketch +is not a public contract. + The ingest request and response use `Content-Type: application/x-ndjson` and `Accept: application/x-ndjson`. -Each input line is one row payload plus the compare-and-chain envelope. The -contributor is never accepted from the body: +Before body ownership, the route requires one strong `If-Match` value derived +from existing graph identity, accepted schema/catalog authority, streaming +profile revision, and live fold delegation. Missing `If-Match` is an +effect-free HTTP `428` challenge carrying the current strong `ETag`; it does +not poll the body or create lane state. A malformed or stale value returns +HTTP `412` without replacement authority and without polling the body. Clients +may perform the missing-token challenge before opening input, but must never +automatically replay an already-owned body after `412`. The token is derived +authority, not persisted coordination state. + +Each input line is one graph row payload plus the compare-and-chain envelope. +The contributor is never accepted from the body: ```json -{"$stream":{"stream_incarnation_id":"d288f7a0-38b4-4e63-a841-60f323df0dd8","write_id":"8a880f0a-3f41-4a42-9b0e-f34af0a9a4df","predecessor_token":null},"id":"n-17","name":"Ada"} +{"type":"Person","data":{"id":"n-17","name":"Ada"},"$stream":{"write_id":"8a880f0a-3f41-4a42-9b0e-f34af0a9a4df","predecessor_token":null}} ``` +An edge line uses `edge`, `from`, `to`, `data`, and the same `$stream` +envelope. The engine resolves the private stream incarnation from the accepted +graph catalog; clients cannot supply or observe it. + Each output line corresponds to the same input ordinal: ```json -{"ordinal":17,"status":"durable","stream_incarnation_id":"d288f7a0-38b4-4e63-a841-60f323df0dd8","write_id":"8a880f0a-3f41-4a42-9b0e-f34af0a9a4df","stream_token":"sha256:...","origin":{"kind":"admission","admission_attempt_id":"...","caller_ordinal":17},"enrollment_id":"...","shard_id":"...","writer_epoch":8} +{"ordinal":17,"status":"durable","scope":"row","kind":"node","type":"Person","id":"n-17","write_id":"8a880f0a-3f41-4a42-9b0e-f34af0a9a4df","stream_token":"sha256:..."} ``` -The response union is tagged rather than pretending every outcome created a -new admission attempt. Exact JSON `status` values are `durable`, +The response union is tagged and graph-logical rather than pretending every +outcome created a new admission attempt or exposing private lane evidence. +Exact JSON `status` values are `durable`, `ack_unknown`, `already_durable`, `withdrawn`, `dead_lettered`, `invalid`, -`stream_input_too_large`, `stream_binding_changed`, -`stream_lifecycle_changed`, `stream_authority_changed`, +`stream_input_too_large`, `stream_authority_changed`, `stream_sequence_conflict`, `stream_idempotency_conflict`, -`stream_resume_required`, `stream_fold_required`, `stream_backpressure`, +`stream_fold_required`, `stream_backpressure`, `recovery_required`, and `stream_retry_required`; CamelCase names below denote the corresponding engine error/disposition types. An unresolved recovery found by the request-level barrier before any body line @@ -2790,45 +2806,31 @@ the adapter has accepted an ordinal or emitted any line, the same condition is represented only by per-line `recovery_required` plus the stop-tail rule below; partial success never changes into an HTTP error. -For variants that reach the engine's authoritative binding capture, response -`enrollment_id`, `shard_id`, and `writer_epoch` describe that freshly -revalidated current physical binding. They are not copied from a token row's -immutable `origin_enrollment_id`, so a later Phase-D rebind does not make an -`already_durable` response advertise a stale writer. An adapter-level `invalid` -before binding capture omits those fields. A tail `stream_retry_required` -identifies the blocking attempt's captured binding under explicitly named -`blocking_binding`; it does not pretend to have revalidated a new per-line -binding. OpenAPI makes presence variant-specific rather than nullable-by-habit. - -- `durable` returns the confirmed token and its new `Admission` origin; -- `ack_unknown` / `AckUnknown` returns that attempt plus an explicitly - `candidate_stream_token_unconfirmed`; it never labels the token current; -- `already_durable` returns the persisted token and its persisted - `Admission | Correction` origin, with no new attempt; -- `withdrawn` returns the current terminal token and its original persisted - `Admission | Correction` origin plus the separate withdrawal correction - actor, operation, and receipt, with no new attempt; -- `dead_lettered` (activated only by §4.7 F5's new format) returns the current - terminal token, persisted tagged origin, violation/object candidate - reference, and fold operation with no WAL attempt. The object contains only - that key's canonical final LWW candidate; superseded occurrences are outside - the §4.2 audit contract and are never response identities; +Every line includes `ordinal`, `status`, and `scope` (`row | graph`). When +known from the submitted logical row it also includes `kind`, `type`, `id`, +and `write_id`. Token fields are intentionally semantic: `stream_token` is +confirmed authority, `current_token` names a confirmed conflicting/current +occurrence, and `unconfirmed_candidate_token` is never represented as current. +Safe messages, exact single-row `limit`/`actual`, and +`blocking_ordinal`/`blocking_status` are included only when applicable. +Responses never expose stream incarnation, enrollment, binding, table, +dataset, shard, writer, epoch, generation, recovery sidecar, object URI, digest, +or native HEAD evidence. + +- `durable` and `already_durable` return the confirmed `stream_token`; +- `ack_unknown` / `AckUnknown` returns + `unconfirmed_candidate_token`; it never labels that token current; +- `withdrawn` and `dead_lettered` return the current terminal token without + exposing correction, dead-letter-object, or fold internals; - `invalid` is the effect-free per-line parse/schema/normalization error and creates no attempt; - `stream_input_too_large` means the exact normalized single row cannot fit an otherwise empty legal generation. It is terminal for that line, creates no attempt, and does not ask the caller to fold and retry an impossible row; -- `stream_binding_changed`, `stream_lifecycle_changed`, - `stream_authority_changed`, `stream_sequence_conflict`, and - `stream_idempotency_conflict` are effect-free and return only the - authoritative binding/lifecycle/current-token evidence safe for the - corresponding typed error, with no fabricated attempt. Lifecycle change - covers `OPEN -> DRAINING/SEALED` between request runs; authority change covers - an exhausted pre-invocation reprepare after schema/main/token movement; -- `stream_resume_required` is the effect-free result for an existing `SEALED` - lane under an `ENABLED` profile. It carries the current lifecycle revision - needed by the separate authorized resume and never resumes under the ingest - actor; +- `stream_authority_changed` graph-redacts binding, lifecycle, profile, + schema, and resume-required movement into one effect-free retry boundary; +- `stream_sequence_conflict` and `stream_idempotency_conflict` are effect-free + row outcomes and may return only safe current-token evidence; - `stream_fold_required` is an effect-free admission refusal with no attempt. It means the next legal row/run fits an empty generation but not the bounded resident generation and must be folded before retry. B2a defines no retained- @@ -2839,11 +2841,10 @@ binding. OpenAPI makes presence variant-specific rather than nullable-by-habit. attempt; - `recovery_required` means a pre-invocation recovery/retirement operation remains authoritative and could not finish within the request deadline. It - carries that recovery operation ID, no new admission attempt, and no token - claim; and + exposes no recovery identity, new admission attempt, or token claim; and - `stream_retry_required` means the line was not invoked because an earlier - physical run became `AckUnknown`; it carries that blocking attempt ID but no - attempt or token claim for this line. + physical run became `AckUnknown`; it carries the blocking ordinal/status but + no private attempt or token claim for this line. RC.1 exposes a durability completion, not an exact per-put WAL receipt. `BatchDurableWatcher::wait()` returns only `Result<()>`; @@ -2935,27 +2936,25 @@ CLI commands mirror the new namespace rather than overloading deprecated `omnigraph ingest`: ```text -omnigraph stream enroll --enrollment-request-id ... -omnigraph stream ingest --data ... -omnigraph stream status [] ... -omnigraph stream block show --block-token ... -omnigraph stream fold [] --operation-id --expected-lifecycle-revision ... -omnigraph stream quiesce [] --drain-id --expected-lifecycle-revision ... -omnigraph stream resume [] --resume-id --expected-lifecycle-revision ... -omnigraph stream resume [] --abort-drain --resume-id --expected-lifecycle-revision ... -omnigraph stream correct --block-token --correction-id --expected-lifecycle-revision --plan ... -omnigraph stream rebuild-preflight [] ... +omnigraph stream ingest --data [--graph-token ] ... ``` -HTTP fold, quiesce, resume/abort-drain, and correction bodies likewise require +It is served-only and graph-addressed. Without `--graph-token`, the client +completes the bodyless `428` challenge before opening the path or stdin, then +sends the body exactly once. Supplying a token skips that preflight. A `412` +never triggers token replacement or replay. Direct `--store`, embedded use, +and client-supplied `--as` refuse before input is opened. Future lifecycle, +status, correction, and maintenance commands must be graph-level even if their +implementation delegates to private lanes; F7a does not reserve their CLI +grammar. + +Future HTTP fold, quiesce, resume/abort-drain, and correction bodies likewise require their operation ID plus expected lifecycle revision; status and block-view responses expose the revision to use as the compare token. Exact occurrence plus intent is retry-safe after a lost response, while stale revision refuses without retargeting. `enroll` requires caller-minted -`enrollment_request_id` and returns tagged `enrolled | already_enrolled` with -the durable stream incarnation/current binding; ingest on a declared but -unenrolled table returns request-level typed `StreamNotEnrolled` before reading -or acknowledging body rows. `correct` requires `block_token`, +`enrollment_request_id` internally and returns tagged `enrolled | already_enrolled` to +the graph adapter; those lane identities are not public. `correct` requires `block_token`, caller-minted `correction_id`, and explicit ordered `REPLACE | WITHDRAW` actions. The engine derives the canonical plan digest from §4.4; an optional client digest is only an equality assertion. It returns the immutable @@ -3503,67 +3502,38 @@ no operator, no resident fold driver, and no lifecycle owner. With the profile exactly `ENABLED`, every graph table is stream-eligible; no per-table opt-in exists. The experimental profile amends §3's standalone -enrollment surface, but not §4.1's exact wire incarnation. A table with no -lifecycle uses an automatic bodyless prepare handshake before its first row: - -1. Status returns no stream incarnation for the absent lane, but returns a - canonical `StreamEligibilityWitness` over graph identity, stable table and - table-incarnation IDs, accepted-catalog digest, profile revision, and live - fold-delegation ID, plus the exact canonical-main table/ref - `CurrentHeadWitness` and lifecycle-slot-absent compare evidence. An - ingest-only actor need not hold status permission. Prepare resolves retained - receipt/current lifecycle authority first: under exact `ENABLED`, an - existing `OPEN | DRAINING | SEALED` lane returns bounded - `already_enrolled` with current stream incarnation, binding digest, - lifecycle, and revision even when no witness was supplied. Only an absent - eligible lane with no/stale witness returns effect-free - `witness_required` plus the current bounded witness under `stream_ingest`; - neither result retains a new operation occurrence. -2. `POST /graphs/{graph_id}/streams/{type_name}/prepare` carries a - caller-minted non-nil `enrollment_request_id`; the witness is optional only - for an effect-free challenge and mandatory before arming. It requires - `stream_ingest` and checked server-runtime authority. The remote - `GraphClient`/CLI uses a cached/status witness when available or performs - witness challenge → prepare automatically. One `stream ingest` call follows - at most one fresh challenge within its bounded deadline; another movement - returns typed `StreamAuthorityChanged`/retry guidance before body ownership - instead of polling. There is no manual `stream enroll` command or per-table - policy decision. Under the enrollment - ReadSet it byte-revalidates the whole witness. Concurrent lifecycle creation - restarts at the current-authority branch and returns `already_enrolled`. - `DISABLED`, `DISABLING`, a changed graph/table incarnation, or a no-longer- - eligible table returns a typed profile/eligibility refusal with no witness. - Only a still-`ENABLED`, still-absent lane whose HEAD/ref/catalog/profile - witness moved returns effect-free `witness_required` with a fresh witness - and does not arm. -3. For an absent lane, prepare feeds the existing §3 recovery adapter. Before - its first effect, recovery fixes the request/witness intent, authenticated - actor, and engine-minted stream incarnation/binding. The durable - `EnrollmentReceiptV2` retains that actor. The same request ID may be reused - after an effect-free witness challenge. Once a participant effect makes the - receipt durable, the same ID, actor, and intent after a lost response - returns the receipt; another actor or intent conflicts. Durable intent - covers the graph/table lifetime, accepted schema, original table HEAD, and - fixed stream configuration; profile revision and fold delegation remain - pre-arm freshness evidence because the receipt does not persist them. - Concurrent IDs resolve through the one-winner lifecycle - CAS, and a loser returns `already_enrolled` only after revalidating the - winner's complete receipt and current authority. A successful prepare - followed by no body leaves an empty enrolled `OPEN` lane; the F2 empty-lane - drain path owns its quiesce/disable. If recovery proves that an armed - sidecar had zero participant effects, it may retire that sidecar and re-arm - the same request with new engine-minted result IDs; no receipt or - acknowledgement existed at that boundary. -4. Only a later ingest request whose every `$stream` envelope carries that - exact incarnation may own or read NDJSON. Ingest against an absent lane - returns request-level `StreamPrepareRequired` before body admission. There - is no omitted-incarnation first-row exception. A supported client never - overwrites an explicitly supplied stale incarnation or automatically - reprepares/replays that body after `StreamBindingChanged`; crossing a strict - rebuild/re-enrollment requires the caller to choose a new occurrence and - predecessor. - -Before the table lease, prepare—an enrollment control rather than a +enrollment surface, but not §4.1's exact internal wire incarnation. A table +with no lifecycle is prepared lazily behind the graph adapter: + +1. Before body ownership, the checked served entry point captures one + `StreamGraphIngestWitness` over graph identity, accepted schema identity and + catalog hash, profile revision, and live fold delegation. It derives the + opaque graph token described in §4.6. This is the only public prepare + handshake; an ingest-only actor needs no status permission and the graph is + the sole Cedar resource. +2. Missing token returns the bodyless `428` challenge. Malformed/stale token or + authority movement during final recapture returns `412` before polling the + body and without replacement authority. After the exact token transfers + body ownership, later schema/profile/delegation or private-lane movement is + a redacted per-line `stream_authority_changed` stop-tail boundary. There is + no manual `stream enroll`, public table prepare, or per-table policy + decision. +3. For each logical node/edge row, the adapter resolves the declaration against + the captured catalog. An existing `OPEN` lane is reused. For an absent lane, + the adapter captures the table-scoped eligibility/HEAD evidence and feeds + the existing §3 recovery-owned prepare using engine-minted enrollment and + stream-incarnation identities. Concurrent lifecycle creation converges + through the same one-winner CAS/receipt rules. A successful prepare may + leave an empty enrolled `OPEN` lane if the request disconnects; the existing + empty-lane drain path owns its eventual quiesce/disable. +4. The adapter injects the exact private stream incarnation into the internal + row call. Clients supply only logical declaration, row identity, + `write_id`, and predecessor token; they neither choose nor observe the lane + incarnation or binding. Crossing a rebuild/re-enrollment still requires a + new graph-authority preflight and the caller's deliberate sequencing choice; + the client never silently reprepares or replays an already-owned body. + +Before the table lease, private prepare—an enrollment control rather than a resident-producing row put—acquires the graph-profile gate shared. Under the same exclusive table admission lease and existing schema/main/token/table gates, prepare reruns the recovery barrier and rereads canonical-main @@ -3571,8 +3541,8 @@ gates, prepare reruns the recovery barrier and rereads canonical-main `FoldDelegation`, and checked runtime must all match. `DISABLED`, `DISABLING`, a changed graph/table identity, ineligibility, or a delegation mismatch refuses before sidecar or Lance effect; a still-eligible -absent lane with moved HEAD/ref/catalog/profile evidence returns the bounded -witness challenge above. Resident-producing ordinary admission retains bounded +absent lane with moved HEAD/ref/catalog/profile evidence refuses the graph +request without publishing new authority. Resident-producing ordinary admission retains bounded preprocessing/inflight ownership, then takes root MemWAL opportunity shared, graph-profile shared, and table admission before performing the same final profile/delegation/runtime match and handing off a run. Those permits remain @@ -3583,8 +3553,9 @@ gate exclusively only for the first profile CAS and releases it before per-table drains. The supported production race is closed by the server-exit/apply-start handoff, not a cross-process lock. -An existing `OPEN` lane may admit; an existing `SEALED` lane returns typed -`StreamResumeRequired` and never auto-resumes under an ingest actor. The +An existing `OPEN` lane may admit; an existing `SEALED` lane returns internal +typed `StreamResumeRequired`, graph-redacted to `stream_authority_changed`, and +never auto-resumes under an ingest actor. The prepare exchange is binding negotiation hidden by supported clients, not an operator opt-in: the graph remains one connected model without requiring external producers to choose which tables participate. @@ -3691,10 +3662,11 @@ The selected dead-letter protocol is deliberately small: Strict blocks retain §4.4's exact exits: same-cut retry after fixing a transient cause, reason-gated data correction, or proof-bound authority repair. Cluster/offline block inspection, correction, repair, payload export, and -retirement are supported before public ingress activates, but EXP gives them -no served HTTP/OpenAPI parity. The five primary workflows are ingest, status, -fold, quiesce, and resume; bodyless prepare remains an automatic ingest -handshake. +retirement remain narrow emergency exits after F7a and receive no served +HTTP/OpenAPI parity. Ingest is the sole activated graph workflow; status, +fold, quiesce, and resume remain later graph-level workflows. The bodyless +graph-authority challenge is part of ingest, while private lane prepare is an +engine detail. The selected profile adds no new attribution history merely because an object exists. It reuses the authenticated contributor/payload identity needed for @@ -3951,9 +3923,9 @@ control, never load over the enrolled source. #### Surface retained, trimmed, and non-trimmable -Retained from §4.6: the automatic bodyless prepare handshake plus NDJSON -`ingest` with its full per-line response union and ordering/cancellation rules -(plus P4's `dead_lettered` terminal result); full +Retained from §4.6: the graph-authority ETag handshake plus graph-native NDJSON +`ingest` with its redacted per-line response union and ordering/cancellation +rules (plus P4's `dead_lettered` terminal result); full status needed by lifecycle and operations; operator `fold`; persistent revision-fenced `quiesce` and `resume`; post-`SEALED` rebuild preflight; stream-aware export and same-binding maintenance; and the `stream_ingest` / @@ -3968,9 +3940,10 @@ policy. Non-trimmable regardless of experimental status: Cedar enforcement, typed bounded failures, shutdown ownership, durable attribution, terminal dead-letter sequencing, safe export, OpenAPI/parity/failpoint/genuine-rebuild evidence, and the no-raw-GC boundary. Because this profile is cluster-only, the -server-owned runtime, HTTP/OpenAPI, remote `GraphClient`, and remote CLI -activate together only after the hidden ingress/driver path and acceptance -evidence pass. Ambient embedded SDK and direct `--store` mutation remain a +server-owned runtime, HTTP/OpenAPI, remote `GraphClient`, and remote CLI for +each workflow activate together only after its hidden path and acceptance +evidence pass. F7a does so for graph ingress without activating management. +Ambient embedded SDK and direct `--store` mutation remain a typed `StreamingRequiresClusterRuntime` refusal before body/effect; embedded manifest-only status remains. The experimental designation is also an explicit Hyrum boundary: acknowledgement and P4/P5 terminal semantics are committed, @@ -3979,19 +3952,19 @@ declared unstable. By the time F7 executes, F2 will already have landed `cluster apply --confirm-stream-offline` and its profile adapter, and F6b5 will -already own exact-terminal served export. F7 co-lands the remaining served row, -lifecycle, maintenance, and status routes plus their remote commands, -DTO/authorization/audit tests, and the existing stream surface. F6 proves the -hidden candidate runtime first; export is its narrow HTTP/remote exception. +already own exact-terminal served export. F7a co-lands the graph row route, +remote command, challenge/error DTOs, authorization tests, and OpenAPI contract +over the existing candidate runtime. It deliberately leaves lifecycle, +maintenance, and checked-status transport to later F7 slices. Export remains +the earlier narrow HTTP/remote exception. -That activation PR extends F2's already-public cluster-ownership, +The F7a activation PR extends F2's already-public cluster-ownership, direct-mutation-refusal, and v10→v11 rebuild baseline with the activated stream -operating contract. CLI reference, server, policy, and error docs add -prepare/ingest, status/lifecycle, maintenance/export, authorization, tagged -results, and the stream/export-specific extension of the +operating contract. CLI reference, server, policy, and error docs add the +graph-token/ingest handshake, authorization, tagged redacted results, and the stream/export-specific extension of the served-only/direct-refusal boundary. Cluster docs separately cover the narrow offline dead-letter, correction, authority-repair, and retirement exits. -Maintenance docs show exact +Later maintenance docs show exact `quiesce -> served Optimize/EnsureIndices -> resume`; cluster and upgrade docs show the distinct `graceful stop -> offline disable to terminal DISABLED -> cluster-state-locked physical rebind -> separate enable -> restart -> explicit @@ -5755,11 +5728,11 @@ diagnostics only and enforce no product threshold. This B2a result itself added no schema or product surface. The subsequent private B2-common row/fold slice activated schema v9; F3f later added the narrow -stopped/offline DataBlock exit. Production row callers, general lifecycle -control, `AuthorityBlock` repair, operational-status transport, and transport -parity remain §12.6 work. F6b6 implements the checked operational core behind -an internal seam; the authorization/manifest-status slice shipped earlier -under §4.7. +stopped/offline DataBlock exit, and F7a later activated the graph-native served +row caller plus HTTP/remote-CLI/OpenAPI parity. General lifecycle control, +`AuthorityBlock` repair, and operational-status transport remain §12.6 work. +F6b6 implements the checked operational core behind an internal seam; the +authorization/manifest-status slice shipped earlier under §4.7. ### 12.6 Private B2-common implementation and remaining public/B2b gates @@ -5772,13 +5745,13 @@ genuine v8↔v9 refusal/rebuild. F3e later activated the cluster/offline retirement/export escape for a verified current-`WITHDRAWN` cut; F3f adds exact stopped/offline `DataBlock` show/correct with recovery-v20, and F5b adds current `DEAD_LETTERED`, selected-token inspection/export, ordinary successors, -and three-disposition retirement through recovery-v21. Public -row activation still waits for explicit -production enrollment in the full profile or §4.7 P2's selected automatic -prepare handshake, general lifecycle controls, `AuthorityBlock` repair, -public operational-status transport, cancellation/shutdown, API compatibility, -and transport parity. F6b6 implements the checked operational-status core -internally. Cold-replay and flushed-LWW accounting plus exact oldest-uncovered +and three-disposition retirement through recovery-v21. F7a activates the +selected profile's graph-native served row bridge, lazy private prepare, +cancellation ownership, and HTTP/remote-CLI/OpenAPI parity. Explicit lane +enrollment, general lifecycle controls, `AuthorityBlock` repair, public +operational-status transport, and embedded/direct SDK row ingress remain +inactive. F6b6 implements the checked operational-status core internally. +Cold-replay and flushed-LWW accounting plus exact oldest-uncovered age are explicitly unavailable. `DISABLING` uses explicit checked cluster- apply status authority; all within-envelope sidecars are reported and rebuild- blocking, while an over-bound discovery refuses the whole status. Likewise, @@ -5792,12 +5765,11 @@ The design does not waive the persistent escape requirement: a user must never be left with a table that ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. -- **Inactive row/control product surface:** `@stream(mode="upsert", - on_reject="strict")`, production first use, SDK row/control methods, HTTP, - CLI, and OpenAPI must all route through the same private core. The Cedar - actions are registered, but `stream_ingest` has no production caller and - `stream_manage` currently reaches only profile enablement and the separate - stopped/offline retirement handshake. Existing +- **Graph row surface activated in F7a; controls remain inactive:** + `@stream(mode="upsert", on_reject="strict")` production first use, the + served client, HTTP, remote CLI, and OpenAPI all route through the same + private core. `stream_ingest` has one production graph caller; + `stream_manage` still reaches only existing cluster/offline controls. Existing `/ingest` behavior must remain compatible. The full surface requires embedded/remote command parity; the selected §4.7 profile instead tests served/remote success against the exact @@ -5805,45 +5777,40 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. Accepted-schema and runtime guards must refuse `@stream` on a type requiring `@embed` or any external/provider-derived field; caller-supplied physical vectors must round-trip without provider invocation. -- **Private prepare proof; inactive enrollment product surface:** the full +- **Private prepare proof; no enrollment product surface:** the full non-experimental profile retains §3/§4.6's explicit `/enroll`, `stream_manage`, and request-level `StreamNotEnrolled` contract. The selected experimental profile instead - requires P2's `stream_ingest`-authorized bodyless prepare: complete - `StreamEligibilityWitness` including exact current HEAD/ref and lifecycle - absence; effect-free witness challenge; actor-bound request ID/intent; + requires P2's `stream_ingest`-authorized graph challenge plus lazy private + prepare: complete graph witness before body ownership and exact table + eligibility/current-HEAD/lifecycle-absence evidence before enrollment; + effect-free missing/stale graph-token handling; actor-bound internal request ID/intent; same/different-intent and lost-result receipt replay; bounded `already_enrolled` for an existing lane; concurrent first-prepare CAS; - returned stream incarnation before body ownership; stale explicit - incarnation refusal; and request-level `StreamPrepareRequired` when body - ingest finds no lane. Row-body ingest itself never creates physical - enrollment. Tests cover every selected-profile + engine-owned stream incarnation injection; and graph-redacted refusal on + stale private authority. Tests cover every selected-profile bootstrap/shard/lifecycle crash boundary plus Cedar/served/remote success and embedded/direct refusal. B2b additionally covers every genesis body/pointer/new-details crash boundary. The feature-gated engine proof now - covers the bodyless challenge and recovery-v14 enrollment subset; public - enrollment, transport, and row-body activation remain inactive. -- **Inactive public acknowledgement adapter:** the response must be status-only - and caller-ordered. It must map the private durable batch result back to each - caller ordinal and report the - stream incarnation, write ID, and current binding, not a WAL position, - generation, or Lance `batch_positions` value. Only durable/current outcomes - may report a confirmed token and persisted tagged origin; `ack_unknown` must label - its candidate unconfirmed, while invalid/conflict/not-invoked outcomes mint - neither. Every exact response variant in §4.6, including `invalid`, - `stream_input_too_large`, lifecycle/authority/recovery blockers, and - `stream_retry_required`, must have a schema. The public adapter must own contiguous - physical-run boundaries around invalid lines and token dispositions plus its - bounded reorder buffer. Tests must cross a partially full generation, row/logical- - memory and queue/deadline limits, - an intrinsically oversized row on an empty generation, authority/lifecycle - movement or `RecoveryRequired` between runs, and a pre-invocation queue - deadline after earlier lines are already durable; the - blocking and remaining otherwise-admissible lines must receive their exact effect- - free capacity/backpressure status while later parse/schema/normalization - failures remain `invalid` and intrinsically oversized rows retain their own - status. The same precedence must be pinned after `AckUnknown`. No handler may convert - partial NDJSON success into an HTTP error. + covers the graph challenge and recovery-v14 enrollment subset; F7a exposes + row transport, while public lane enrollment remains intentionally absent. +- **Public graph acknowledgement adapter activated in F7a:** the response is + status-only and caller-ordered. It maps each private durable batch result back + to its caller ordinal and may report only graph-logical kind/type/id, + `write_id`, safe confirmed/current/unconfirmed token evidence, and bounded + blocker/limit fields—not stream incarnation, binding, WAL position, + generation, Lance `batch_positions`, or recovery/object identity. Only + durable/current outcomes may report a confirmed token; `ack_unknown` labels + its candidate unconfirmed, while invalid and uninvoked outcomes mint neither. + Every exact response variant in §4.6 has a schema. The existing private F4 + adapter continues to own contiguous physical-run boundaries around invalid + lines and token dispositions plus its bounded reorder buffer; F7a adds the + graph redaction and incremental transport. Existing engine tests cover + partially full generations, row/logical-memory and queue/deadline limits, + intrinsically oversized rows, authority/lifecycle/recovery movement, and + stopped-tail and cancellation precedence. Served tests pin incremental mixed + graph rows, redaction, and the bodyless precondition refusals. No handler + converts partial NDJSON success into an HTTP error. - **Implemented privately:** schema v9/config-v3/state-v2 provisions the hidden row metadata and manifest-selected token dataset. Canonical payload/token digests bind accepted schema, table/key identity, stream incarnation, @@ -5859,11 +5826,11 @@ ordinary writers refuse but cannot be corrected, quiesced, or rebuilt. exact base-only recovery may complete only the planned token effect. V11 is historical and refused under v9. The graph-global token gate and release-all-gates/restart rule cover every manifest writer. -- **Still inactive at the product boundary:** OpenAPI/SDK/CLI round-trips must - pin the exact `sha256:` plus 64 lowercase-hex wire form for public tokens and - protocol digests; uppercase, base64, whitespace, bad prefix, and wrong length - must refuse pre-effect. Public caller ordering, mixed request shaping, - cancellation/shutdown, and transport parity retain their gates above. +- **Still inactive at the product boundary:** embedded/direct SDK row ingress, + public lifecycle/status/maintenance transports, and public lane enrollment. + Any later SDK or control surface must preserve the graph-only boundary and + the token/redaction rules above; it cannot promote the private lane adapter + into a user-visible API. - **Operator controls:** two-step same-format `retire-for-rebuild` plan/confirm is active only as a narrow cluster/offline support surface; it has no served HTTP/OpenAPI equivalent. Current dead-letter list/payload export is likewise @@ -5924,7 +5891,8 @@ and three-disposition retirement evidence. The genuine v18↔v19 adjacent binary cell pins both refusals, ordinary rebuild fidelity, and import of frozen final- v18 retirement receipt-v1 bytes without authority transfer. The broader race/ failpoint/freeze/export -matrix below remains required before public ingress activates, and none of +matrix below remains required before the remaining control surfaces activate; +F7a's narrower served-ingress evidence is called out separately, and none of this activates `AuthorityBlock` repair. - V17 authority-retirement planning begins with at least one current @@ -6177,7 +6145,7 @@ remain concurrent with one another. | B2a | selected unbounded retain-all/no-GC profile on stock Lance | **Private gate implemented 2026-07-21 (§12.5):** no OmniGraph byte/object/file/history quota; zero canonical `_mem_wal` deletion; complete/partial provider residue remains retained, unreferenced, and untouched below its root through retry/reopen; provider failures are loud; local/configured-RustFS history sweeps are advisory. This gate itself activated no schema or product surface; the later private B2-common slice activates v9 | | B2b | candidate managed-reclamation retention profile | Inactive. Requires the Lance-owned durable inspect/plan/execute + receipt, post-success fencing, bounded checkpoint/inventory/accounting, local/RustFS enforced-bound validation, and the profile-specific crash matrix (§4.5.2/§12.6). Passing it alone activates no product surface | | B2-common | schema v9/config-v3/state-v2, compare-and-chain token/attribution, graph-global token authority, recovery-v12 base+token fold; then explicit enrollment, revision-fenced lifecycle/correction/full status, SDK row/control methods, HTTP, CLI, and OpenAPI | **Private row/fold subset implemented 2026-07-22 (§11/§12.6):** canonical digests, hidden attribution, stale-authority revalidation after shared admission, same-generation chains, exact two-participant recovery/publication, durable fold attribution, retain-all, and genuine v8↔v9 refusal/rebuild are green. Explicit production enrollment, general lifecycle mutation, public row admission, cancellation/shutdown, API compatibility, and transport parity remain inactive. The later F6b6 slice implements the checked read-only operational-status core internally; its public transport remains inactive. The Cedar vocabulary, embedded manifest-only status, and narrow stopped/offline F3f DataBlock correction shipped in later EXP slices. `GraphHistoryBudget` belongs only to a future bounded/managed profile | -| EXP | experimental cluster-only activation of the §4.7 profile: offline capability-bound enablement, lazy enrollment, caller-supplied vectors, terminal per-key dead letter plus correction, irreversible authority retirement for fresh-root rebuild, SEALED maintenance/rebind, starvation-free serial folding, and upsert-only hidden ingress | **Selected 2026-07-27 and amended 2026-07-29 (§4.7); F3a–F3f, hidden F4, F5a/F5b0/F5b, and F6a–F6b7 evidence subsets are implemented.** Current v19/token-schema-v3/recovery-v21 publishes deterministic mixed/all-diverted folds, current `DEAD_LETTERED` authority, exact retry/ordinary successor, stopped/offline inspection, and three-disposition retirement. F6b1 freezes an exact-terminal move-only cut; F6b5 connects it to existing served HTTP/remote-client/CLI/OpenAPI export with incremental exact-version scans using approximate Lance targets, strict 64-KiB chunks, complete queue-envelope reservation, pre-header typed refusal, backpressure, and disconnect-safe body-plus-producer ownership. F6b6 adds the engine-internal checked operational cut with explicit checked `DISABLING` cluster-apply status authority. F6b7 adds a paired failpoints-only exact-selected token-index decision instrument without recovery or production maintenance. Within the hard status envelope it reports every sidecar as rebuild-blocking, while an over-bound discovery refuses the whole status; it makes only exact sidecar-owned base-HEAD movement physically unavailable, and reports cold-replay/flushed-LWW accounting plus exact oldest-uncovered age as unavailable; the public manifest-only status is unchanged. Public ingress/enrollment/lifecycle/rebind control, every served row/lifecycle/maintenance/status transport, and unreachable `AuthorityBlock` repair remain inactive. F6b7's bounded NO-GO applies only to the uncompacted profile-cycle fixture and schedules no standalone production token-index reconciler; remeasurement begins beyond 260 uncovered fragments, after a Lance/index-grammar change, or before considering graph-manifest-compacted or checked-Optimize-coupled maintenance, while the remaining guardrail matrix stays open; full row/control activation still requires the remaining F6 evidence before F7. | +| EXP | experimental cluster-only activation of the §4.7 profile: offline capability-bound enablement, lazy enrollment, caller-supplied vectors, terminal per-key dead letter plus correction, irreversible authority retirement for fresh-root rebuild, SEALED maintenance/rebind, starvation-free serial folding, and graph-native served ingress | **Selected 2026-07-27 and amended through F7a (§4.7); F3a–F3f, hidden F4, F5a/F5b0/F5b, F6a–F6b8 evidence subsets, and F7a graph ingress are implemented.** Current v19/token-schema-v3/recovery-v21 publishes deterministic mixed/all-diverted folds, current `DEAD_LETTERED` authority, exact retry/ordinary successor, stopped/offline inspection, and three-disposition retirement. F6b1 freezes an exact-terminal move-only cut; F6b5 connects it to existing served HTTP/remote-client/CLI/OpenAPI export with incremental exact-version scans using approximate Lance targets, strict 64-KiB chunks, complete queue-envelope reservation, pre-header typed refusal, backpressure, and disconnect-safe body-plus-producer ownership. F6b6 adds the engine-internal checked operational cut with explicit checked `DISABLING` cluster-apply status authority. F6b7 adds a paired failpoints-only exact-selected token-index decision instrument without recovery or production maintenance. Within the hard status envelope it reports every sidecar as rebuild-blocking, while an over-bound discovery refuses the whole status; it makes only exact sidecar-owned base-HEAD movement physically unavailable, and reports cold-replay/flushed-LWW accounting plus exact oldest-uncovered age as unavailable; the public manifest-only status is unchanged. F7a exposes one graph-only mixed node/edge NDJSON route and remote command over the existing checked runtime, hidden lazy enrollment, and resident fold driver. A strong graph-authority ETag gates body ownership; results are graph-logical and redact all table/lane/binding evidence. Public lane enrollment, lifecycle/rebind control, served maintenance/status transport, and unreachable `AuthorityBlock` repair remain inactive. F6b7's bounded NO-GO applies only to the uncompacted profile-cycle fixture and schedules no standalone production token-index reconciler; remeasurement begins beyond 260 uncovered fragments, after a Lance/index-grammar change, or before considering graph-manifest-compacted or checked-Optimize-coupled maintenance. | | C | restart-stable reject-row identity, atomic dead letter, richer status, and evidence-backed configurable bounds | reject crash matrix; reject-retention proof; backpressure and RSS/latency evidence. The §4.7 profile pulls a bounded object-form dead-letter subset forward using the §4.1 token as reject identity | | D | automatic operation drain, broader schema/branch/upgrade integration, and orchestrated rematerialization rebind beyond P7's explicit bridge | two-coordinator race, old/new physical-binding crash matrix, and format-transition suite | | E | fresh cuts and maintained-index reads; cross-process `Fresh` ships only if the substrate generation-retention guard exists (§9), otherwise same-process only | cut consistency; merged-generation exclusion | diff --git a/docs/rfcs/0028-stable-schema-identity.md b/docs/rfcs/0028-stable-schema-identity.md index 65a53155..363e97c0 100644 --- a/docs/rfcs/0028-stable-schema-identity.md +++ b/docs/rfcs/0028-stable-schema-identity.md @@ -485,10 +485,10 @@ V16/recovery-v18 adds the separate private physical-rebind owner for an exact `SEALED` lane. V17/recovery-v19 adds the narrow stopped/offline, cluster-only authority-retirement and receipt-bearing export exit. Current v18/recovery-v20 adds exact stopped/offline `DataBlock` correction through the -cluster-only `stream block show|correct` commands. Public ingress, enrollment, -ordinary lifecycle control, authority repair, rebind, and public maintenance -surfaces remain inactive; retirement and DataBlock control are the two narrow -CLI exceptions. +cluster-only `stream block show|correct` commands. Graph-native served ingress +is active; explicit enrollment, ordinary lifecycle control, authority repair, +rebind, and public maintenance surfaces remain inactive. Retirement and +DataBlock control remain the two narrow cluster-control CLI exceptions. None of the later formats reinterprets or backfills v5 in place. A v5 graph was never served with a diff --git a/docs/user/cli/reference.md b/docs/user/cli/reference.md index 06b1e700..dea1efa8 100644 --- a/docs/user/cli/reference.md +++ b/docs/user/cli/reference.md @@ -11,6 +11,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po | `init` | `--schema ` → initialize a graph (start cluster configs from the [cluster.md](../clusters/index.md) quick-start) | | `load` | bulk load a branch, local or remote (`--mode overwrite\|append\|merge` is **required** — overwrite is destructive, so there is no default). Without `--from` the target branch must exist; `--from ` forks a missing `--branch` from `` first | | `ingest` | deprecated alias of `load --from ` (defaults: `--from main --mode merge`); prints a one-line warning to stderr | +| `stream ingest` | experimental graph-native firehose; served-only. `stream ingest [--data ] [--graph-token ]` streams mixed node/edge NDJSON to `/graphs/{graph_id}/stream/ingest`, performs at most one missing-token preflight before opening the body, and emits ordered NDJSON acknowledgement lines. `--data` defaults to `-` (stdin). It never targets a table or direct `--store` | | `query ` (alias: `read`) | run a read query. **Catalog lane** (default): `` is a stored query invoked **by name** from the served catalog (served-only — address with `--server`/`--profile`; the verb asserts the query is a read). **Ad-hoc lane**: with `--query ` or `-e`/`--query-string `, runs that source (the positional `` then selects which query in it). No positional graph URI — address via `--store`/`--server`/`--profile`. `read` is the deprecated previous name (one-line stderr warning) | | `mutate ` (alias: `change`) | run a mutation query; same catalog (by-name, served-only, verb asserts mutation) / ad-hoc (`--query`/`-e`) lanes as `query`. `change` is the deprecated previous name (one-line stderr warning) | | `alias [args]` | invoke an operator alias — a read-only personal binding (under `aliases:` in `~/.omnigraph/config.yaml`) to a stored query on a named server (replaces the removed `--alias` flag; stored mutations are rejected before execution) | @@ -36,7 +37,7 @@ Top-level command families and subcommands. Graph-targeting commands accept a po Every command declares the **capability** it needs — what it requires to reach a graph — which determines the addressing flags that apply: - **`any`** — `query`, `mutate`, `load`, `ingest`, `branch *`, `snapshot`, `export`, `commit *`, `schema show`, `schema apply`. Run against a graph **served (via a server) or embedded (direct against a store)**: accept a positional `file://`/`s3://` URI, `--server ` (+ `--graph ` for multi-graph servers), `--store `, or `--profile `. A remote server is addressed with `--server` — a positional `http(s)://` URI does **not** dispatch to one. -- **`served`** — `graphs list`. Requires a server, and addresses the server's graph *registry* (the bare server URL), not a graph within it: only `--server` / `--profile` apply, and `--graph`, `--store`, and `--as` are rejected loudly. +- **`served`** — `graphs list` and `stream ingest`. Both require a server. `graphs list` addresses the graph *registry* (the bare server URL), while `stream ingest` requires one selected graph through `--server` / `--profile` plus `--graph` or a configured default. Direct `--store` and client-supplied `--as` are rejected for firehose ingest; the server resolves the authenticated actor. - **`direct`** — `init`, `optimize`, `repair`, `cleanup`, `schema plan`, `lint`. Need **direct storage access** (`file://` / `s3://`), never through a server. They accept a positional `URI`, but **not** `--server`, and a remote (`http(s)://`) URI is rejected. `optimize` / `repair` / `cleanup` additionally accept **`--cluster --graph `** (`--cluster` is a cluster directory or storage-root URI, named via `clusters:` in `~/.omnigraph/config.yaml` or a literal root), which resolves the graph's storage URI from the served cluster state (so you needn't know the `/graphs/.omni` layout). `--graph` is the one graph selector across all scopes — on these three verbs it picks the cluster graph; on the other `direct` verbs it does not apply. `--as` does not apply to any `direct` verb — maintenance records no actor. - **`control`** — `cluster *` via `--config `; `policy *` and `queries *` via `--cluster ` or a cluster profile. - **`local`** — `alias`, `embed`, `login`, `logout`, `profile`, `version`. Address no explicit graph scope. diff --git a/docs/user/clusters/config.md b/docs/user/clusters/config.md index b3da422d..da4b1294 100644 --- a/docs/user/clusters/config.md +++ b/docs/user/clusters/config.md @@ -167,8 +167,11 @@ profile is enabled, Mutation/Load/delete through an embedded SDK handle or direct `--store` CLI handle fail before input-file reads, staging, recovery arming, or Lance effects. Existing served Mutation/Load operations continue only through the one cluster-booted server carrying the exact checked runtime -authority. There is not yet a public firehose-ingest endpoint in this release. -Do not set `streaming: true` expecting an additional input route. +authority. The graph-native served firehose is available through +`POST /graphs/{graph_id}/stream/ingest` and remote `stream ingest`, but only an +absent lane (prepared lazily) or an existing `OPEN` lane can admit rows in F7a. +An existing `SEALED` lane after disable/re-enable needs the later public resume +surface; repeated ingest requests do not reopen it. Branch merge is stricter: it is refused while the profile is `ENABLED` or `DISABLING`, even through that checked served runtime, because this release has no token-aware merge transition. diff --git a/docs/user/clusters/index.md b/docs/user/clusters/index.md index 5dec2751..1e28ad99 100644 --- a/docs/user/clusters/index.md +++ b/docs/user/clusters/index.md @@ -194,11 +194,14 @@ after the graph effect landed but the state CAS did not, use `cluster refresh` to reconcile the ledger from manifest truth before replanning under another actor; a different actor cannot adopt the original receipt directly. -In this release, `streaming: true` is not additive: it makes embedded SDK and -direct `--store` Mutation/Load/delete fail before input reads or durable +In this release, `streaming: true` is not additive to the existing direct +writer surfaces: it makes embedded SDK and direct `--store` +Mutation/Load/delete fail before input reads or durable effects. Existing served mutations work only through the restarted -cluster-booted server's checked runtime authority. There is no public firehose -ingress yet. Branch merge remains refused while the profile is `ENABLED` or +cluster-booted server's checked runtime authority. Graph-native producers use +the served [`/stream/ingest` firehose](../operations/server.md#streaming), which +keeps physical datasets and lanes private. F7a admits absent or `OPEN` lanes; +it does not resume a `SEALED` lane after disable/re-enable. Branch merge remains refused while the profile is `ENABLED` or `DISABLING`, including through the checked server runtime. A later explicit `streaming: false` offline apply publishes `DISABLING`, derives one finite manifest lane cut, and serially drains `OPEN`, goal-`SEALED`, and adopted @@ -257,9 +260,13 @@ omnigraph --graph knowledge --as andrew \ Follow `next_cursor` with `--cursor` until it is absent. Both commands pin the manifest-selected token version; export verifies the recovery-owned object descriptor and does not prefix-list storage. Payload export is an inspection -artifact, not replay or import. The hidden row path can restore `PRESENT` with -a fresh ordinary stream occurrence naming the terminal token as predecessor; -there is no public row-ingress command or HTTP/SDK surface yet. +artifact, not replay or import. A fresh ordinary stream occurrence can restore +`PRESENT` by naming the terminal token as predecessor. +The graph-native `stream ingest` command and +`POST /graphs/{graph_id}/stream/ingest` route can submit that occurrence while +the enabled lane is absent or `OPEN`; payload export itself does not replay it +automatically. Once the lane is terminally disabled/`SEALED`, retirement and +rebuild remain the public exit until resume control ships. ### Terminal authority retirement: plan → confirm → rebuild diff --git a/docs/user/operations/errors.md b/docs/user/operations/errors.md index 1d76cd80..624e6ad9 100644 --- a/docs/user/operations/errors.md +++ b/docs/user/operations/errors.md @@ -44,11 +44,18 @@ - `RecoveryRequired { operation_id, reason }` — an overlapping durable recovery intent remains unresolved. Its physical effects may already have landed, or it may still be armed before the first effect. HTTP returns **503** with `recovery_required.operation_id`. Resolve the sidecar through a read-write reopen/server restart before retrying; this is intentionally not an ordinary OCC retry. - `StreamExportBlocked { withdrawn_token_count, dead_lettered_token_count }` — ordinary export found current `WITHDRAWN` or `DEAD_LETTERED` sequencing - authority that a row-only artifact cannot preserve. The hidden ingest path - can install a fresh ordinary `PRESENT` successor, but no public row-ingress - surface exposes it yet; the current operator exit is the stopped/offline + authority that a row-only artifact cannot preserve. Graph-native firehose + ingest can install a fresh ordinary `PRESENT` successor only while the + enabled declaration lane is absent or `OPEN`. A terminally disabled/ + `SEALED` lane has no public resume in F7a; its operator exit is the stopped/offline `cluster stream retire-for-rebuild` handshake. The served export route returns this as HTTP **409** before `200`. +- Graph-ingest HTTP preconditions are transport errors rather than new durable + engine states. Missing `If-Match` returns **428** plus the current opaque + graph-ingest ETag without polling the body. Malformed or stale `If-Match` + returns **412** without a replacement token, also before body polling. A + non-`application/x-ndjson` request returns **415**. None of these outcomes + enrolls a lane or invokes MemWAL. - `StreamRetirementPlanChanged` — graph/profile/lifecycle/token authority moved between retirement plan and confirm. No retirement effect is accepted; rerun the plan and review the new digest. diff --git a/docs/user/operations/policy.md b/docs/user/operations/policy.md index c790d51a..1df635cc 100644 --- a/docs/user/operations/policy.md +++ b/docs/user/operations/policy.md @@ -14,7 +14,7 @@ Per-graph actions (bind to `Omnigraph::Graph::""`): 6. `branch_delete` 7. `branch_merge` 8. `admin` — reserved for policy-management surfaces (hot reload, audit log, approvals). No call site today. -9. `stream_ingest` — **experimental (RFC-026)**; gates admitting rows into a table's streaming lane. Graph-scoped: the experimental streaming profile is main-only, so a rule that sets `branch_scope` or `target_branch_scope` on it is rejected at validation. Deliberately separate from `change`: a stream append acknowledges *durability* without graph visibility, so an operator can grant high-rate ingestion without granting direct-lane writes, or the reverse. No production ingest surface exists yet. +9. `stream_ingest` — **experimental (RFC-026)**; gates the graph-native firehose at `POST /graphs/{graph_id}/stream/ingest`. The graph is the authorization resource: logical node/edge declarations are row data, while the private Lance datasets and MemWAL lanes are never policy resources. The profile is main-only, so a rule that sets `branch_scope` or `target_branch_scope` is rejected at validation. Deliberately separate from `change`: a stream append acknowledges *durability* without graph visibility, so an operator can grant high-rate ingestion without granting direct-lane writes, or the reverse. 10. `stream_manage` — **experimental (RFC-026)**; gates streaming *lifecycle management*: enabling or disabling the graph's streaming flag today, plus fold / quiesce / resume / abort-drain as they ship. Graph-scoped for the same main-only reason. Split from `stream_ingest` because the blast radii differ in kind — ingestion adds rows, while management can seal a lane, drain acknowledged data, or reopen it at a new epoch. Read-only stream *status* is authorized like other graph operational metadata rather than by this action, so an operator can always see whether a lane is stuck even without the rights to act on it. 11. `invoke_query` — gates invoking a server-side stored query (the `queries:` registry). Graph-scoped (like `admin`) — per-branch access is enforced by the inner `read` / `change` gate, so a rule that sets `branch_scope` on `invoke_query` is rejected. Coarse in this release: an `invoke_query` allow rule permits any stored query on the graph; a future, additive refinement adds an optional per-query-name scope without changing rules written against the coarse action. Enforced at `POST /queries/{name}` (see [server](server.md)). A stored *mutation* is double-gated: `invoke_query` to reach the tool, plus `change` for the write itself (the engine `_as` writers still enforce per the query body). diff --git a/docs/user/operations/server.md b/docs/user/operations/server.md index 7b1288c7..93cc3bcf 100644 --- a/docs/user/operations/server.md +++ b/docs/user/operations/server.md @@ -158,8 +158,59 @@ the request body and response handling alone. ## Streaming -Only `/export` streams (`application/x-ndjson`); everything else is buffered -JSON. Export authorization, relevant checked recovery settlement/validation, +The experimental graph-native firehose is +`POST /graphs/{graph_id}/stream/ingest`. Its request and successful response +are both `application/x-ndjson`; logical node and edge declarations may be +mixed in one ordered request. The request is a firehose, not one graph +transaction: earlier durable rows are not rolled back by a later failure, and +an edge must follow the endpoint nodes it needs because forward references are +not buffered. The URL never names a table, dataset, lane, or stream +incarnation. Each response line reports only caller-logical identity, the +acknowledgement disposition, and safe sequencing evidence. + +```json +{"type":"Person","data":{"id":"p1","name":"Ada"},"$stream":{"write_id":"8a880f0a-3f41-4a42-9b0e-f34af0a9a4df","predecessor_token":null}} +{"edge":"Knows","from":"p1","to":"p2","data":{},"$stream":{"write_id":"83ed1d92-87bb-489f-a64a-202e8369b46f","predecessor_token":null}} +``` + +Ingest uses an opaque graph-authority ETag to prevent a producer from writing +an owned body into a rebuilt or reconfigured graph at the same URL. A request +without `If-Match` is authorized and preflighted without polling its body, then +returns HTTP **428** with the current strong `ETag`, `Cache-Control: no-store`, +and a small JSON challenge. Retry once with that exact value in `If-Match`. A malformed or stale value +returns HTTP **412** without a replacement token; the caller must re-check its +target and must not automatically replay an already-owned body. The CLI +performs the missing-token handshake before opening its input. With bearer +authentication but no applied policy bundle, default-deny mode returns **403** +before polling the body; an explicit graph-level `stream_ingest` grant is +required. + +```http +ETag: "sha256:…" +Cache-Control: no-store + +{"graph_token":"sha256:…"} +``` + +After the exact precondition passes, input and results stream incrementally +through bounded queues. Acknowledgement means the row is durable in Lance +MemWAL, not yet graph-visible. The resident server driver later validates and +publishes folds through the ordinary atomic graph publication path. A row-local +shape/value error does not invalidate earlier acknowledgements. An ambiguous +acknowledgement or graph-authority/backpressure failure stops later physical +invocation. For `ack_unknown`, retry the exact same logical row with the same +`write_id`, predecessor, and payload; the unconfirmed candidate token is not +valid as a successor predecessor. Uninvoked rows may be retried only after the +graph blocker is resolved. Disconnect +stops future body polling while already-invoked work remains owned until its +durability result is classified. + +F7a admits an absent declaration lane (prepared lazily) or an existing `OPEN` +lane. It does not expose resume: a `SEALED` lane after disable/re-enable returns +`stream_authority_changed` until a later graph-level resume surface exists. + +The existing `/export` route also streams `application/x-ndjson`; other routes +remain buffered JSON. Export authorization, relevant checked recovery settlement/validation, branch/filter validation, and stream-authority validation all finish before the server sends `200`. A pristine graph may export normally. An enrolled `DISABLED` graph must be cluster-served at that exact terminal cut; ordinary @@ -224,7 +275,9 @@ structured field is additive and rolling-safe. Do not blindly resubmit the write: let a read-write open or the recovery sweep resolve that operation first, then retry from a fresh snapshot. -HTTP status codes used: 200, 400, 401, 403, 404, 405, 409, 429, 500, 503. +HTTP status codes used include 200, 400, 401, 403, 404, 405, 409, 412, 413, +415, 428, 429, 500, and 503. The firehose-specific 412/415/428 meanings are +defined in [Streaming](#streaming). ## Per-actor admission control @@ -261,6 +314,12 @@ and `/schema/apply`. Read-only endpoints (`/snapshot`, `/query`, `/read`, `/export`, `/branches` GET, `/commits`, `/schema` GET) are not admission-gated. +Graph firehose uses the engine's separate bounded per-actor and per-graph +stream-request registry because its body is incremental rather than a buffered +request with an up-front byte estimate. That permit is acquired before body +polling and remains owned by the request task through invoked-tail settlement; +saturation returns structured HTTP **413** without consuming the body. + ## Body limits - Default: 1 MB diff --git a/docs/user/operations/upgrade.md b/docs/user/operations/upgrade.md index ae5eab15..dc2bb4dd 100644 --- a/docs/user/operations/upgrade.md +++ b/docs/user/operations/upgrade.md @@ -202,11 +202,13 @@ omnigraph --graph --as \ ``` Pass the returned `--cursor` to request the next page. Payload export is an -inspection artifact, not a replay or import protocol. In the hidden row path, -a corrected value is a fresh ordinary stream admission naming the current -terminal token as its predecessor; while it remains current, exact retry -returns the same terminal result. No public HTTP, SDK, remote-CLI, or OpenAPI -row-ingress surface exposes that successor yet. +inspection artifact, not a replay or import protocol. A corrected value is a +fresh ordinary stream admission naming the current terminal token as its +predecessor; while it remains current, exact retry returns the same terminal +result. The graph-native HTTP/remote-CLI firehose can submit that successor only +while the enabled declaration lane is absent or `OPEN`. A terminally disabled/ +`SEALED` lane cannot be reopened by F7a ingress; until public resume ships, +retirement/rebuild remains the public exit there. If terminal authority must intentionally be discarded for a fresh-root rebuild, use the same v19 binary's irreversible diff --git a/docs/user/reference/constants.md b/docs/user/reference/constants.md index 6efd4f58..522de1ef 100644 --- a/docs/user/reference/constants.md +++ b/docs/user/reference/constants.md @@ -30,6 +30,9 @@ | RFC-026 stopped/offline dead-letter page ceiling | `256 entries`, `268,435,456` complete serialized page bytes (256 MiB); decoded cursor `4,096` bytes | bounded selected-current-token `cluster stream dead-letter list|export`; payload export verifies the recovery-owned object descriptor and is not an import/replay surface | | Served export scan targets and chunk ceiling | initial `8,192`-row estimate and approximate `33,554,432` decoded-Arrow-byte target; emitted chunks: hard `65,536`-byte maximum | `POST /graphs/{id}/export` incrementally scans exact pinned Lance versions without whole-table collection. Lance's byte target overrides the row setting; neither setting is a hard limit, and Lance may emit a larger batch. Blob descriptor batches are explicitly sliced to one logical row before its complete Blob-property set is materialized. One row's Blob values and encoded JSON remain indivisible scratch before the row is split into strictly bounded independently owned transport chunks | | Served export transport budget | `2` queued chunks; `262,144` bytes reserved per response queue envelope; `2,097,152` bytes process-wide; `250 ms` reservation deadline; `1` nonwaiting immutable cut per graph root | each queue reservation accounts for two queued chunks, one producer chunk awaiting admission, and one consumer-current chunk; it is not the complete response or process RSS. At most eight reservations coexist process-wide. Saturation or an occupied graph cut returns typed HTTP 413 before success headers; the body and producer jointly retain the queue permit, and the cut remains in the producer or terminal frame until completion or disconnect unwinds both owners | +| Served graph-ingest authority precondition | strong opaque `ETag` / `If-Match`; one missing-token challenge, no automatic stale-token replacement | `POST /graphs/{id}/stream/ingest` derives the token from existing graph identity, accepted schema, streaming-profile revision, and fold delegation. Missing/stale checks finish before request-body polling and add no persisted authority | +| Served graph-ingest run coalescing | `GRAPH_STREAM_RUN_MAX_COALESCE_DELAY = 50 ms` | public observable, fixed maximum dwell while collecting more same-declaration rows after one non-full request run contains a complete row; new arrivals do not reset the deadline. Capacity, declaration change, and EOF still flush immediately. This is not an end-to-end acknowledgement SLO: storage, authority, and backpressure begin after submission and may add latency | +| Served graph-ingest request envelope | `1` request per actor; `2` per graph root; `134,217,728` reserved envelope bytes per request (`268,435,456` root-wide); `8,192` retained result statuses; `33,554,432` bytes per input chunk and raw line | acquired only after an exact graph token and before body polling. The permit follows the detached request owner through any invoked tail; saturation is HTTP 413. These are process-memory/ownership admission bounds, not retained-storage quota or a promise that one request allocates the full reservation | | Private RFC-026 B2 token-projection ceiling | `B2_MAX_TOKEN_PROJECTION_ARROW_BYTES = 33,554,432` (32 MiB) | exact winning token projection and cumulative exact-authority lookup retention | | Private RFC-026 B2 recovery-JSON ceiling | `B2_MAX_TOKEN_RECOVERY_JSON_BYTES = 33,554,432` (32 MiB) | bounded token rows embedded in one stream-fold recovery sidecar; lifecycle-v3 recovery-v14 preserves the same logical token-plan bound | | Maintenance concurrency | `OMNIGRAPH_MAINTENANCE_CONCURRENCY=8` | optimize/cleanup | diff --git a/openapi.json b/openapi.json index 73bc3850..b9d67d4d 100644 --- a/openapi.json +++ b/openapi.json @@ -1773,6 +1773,161 @@ ] } }, + "/graphs/{graph_id}/stream/ingest": { + "post": { + "tags": [ + "streaming" + ], + "summary": "Ingest a graph-native NDJSON firehose through the checked served runtime.", + "description": "A request without `If-Match` is authorized and preflighted without polling\nits body, then receives a 428 challenge. A malformed or stale precondition\nreceives 412 without replacement authority. An exact strong ETag transfers\nthe Axum body stream directly into the engine and streams one redacted\nnewline-delimited result per input row; neither direction is buffered as a\ncomplete request or response.", + "operationId": "cluster_stream_ingest", + "parameters": [ + { + "name": "graph_id", + "in": "path", + "description": "Graph id to route the request to.", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "If-Match", + "in": "header", + "description": "Strong ETag containing the opaque graph token returned by the 428 challenge.", + "required": false, + "schema": { + "type": [ + "string", + "null" + ] + } + } + ], + "requestBody": { + "description": "Graph-native node and edge rows, one JSON object per line. Results are emitted in caller order and the request is not atomic.", + "content": { + "application/x-ndjson": { + "schema": { + "type": "string" + } + } + }, + "required": true + }, + "responses": { + "200": { + "description": "Ordered graph-native ingest results", + "content": { + "application/x-ndjson": { + "schema": { + "$ref": "#/components/schemas/StreamIngestLineOutput" + } + } + } + }, + "401": { + "description": "Unauthorized", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "403": { + "description": "Forbidden", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "409": { + "description": "Checked streaming runtime or lifecycle conflict", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "412": { + "description": "Malformed or stale graph token; no replacement token is disclosed", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "413": { + "description": "Streaming request admission capacity exhausted", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "415": { + "description": "Request Content-Type is not application/x-ndjson", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + }, + "428": { + "description": "Graph token required; retry with the strong ETag in If-Match", + "headers": { + "Cache-Control": { + "schema": { + "type": "string" + }, + "description": "Always `no-store`; the challenge is authority-bearing" + }, + "ETag": { + "schema": { + "type": "string" + }, + "description": "Strong ETag containing the current opaque graph token" + } + }, + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/StreamIngestChallenge" + } + } + } + }, + "503": { + "description": "Overlapping durable recovery must resolve before ingest", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/ErrorOutput" + } + } + } + } + }, + "security": [ + { + "bearer_token": [] + } + ] + } + }, "/healthz": { "get": { "tags": [ @@ -2895,6 +3050,165 @@ "minimum": 0 } } + }, + "StreamIngestChallenge": { + "type": "object", + "description": "Effect-free precondition challenge for graph-native streaming ingest.\n\nThe same value is returned as a strong `ETag` response header. Clients\nretry the request with that tag in `If-Match`; this convenience copy keeps\nthe graph authority token distinct from the per-row sequencing\n[`StreamIngestLineOutput::stream_token`].", + "required": [ + "graph_token" + ], + "properties": { + "graph_token": { + "type": "string" + } + } + }, + "StreamIngestKindOutput": { + "type": "string", + "description": "Logical declaration kind selected by one graph-native stream row.", + "enum": [ + "node", + "edge" + ] + }, + "StreamIngestLineOutput": { + "type": "object", + "description": "One ordered, newline-delimited result from graph-native streaming ingest.\n\nEvery field is caller-logical or directly actionable retry evidence.\nPhysical table and MemWAL identities are deliberately absent.", + "required": [ + "ordinal", + "status", + "scope" + ], + "properties": { + "actual": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "blocking_ordinal": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "blocking_status": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/StreamIngestStatusOutput" + } + ] + }, + "current_token": { + "type": [ + "string", + "null" + ], + "description": "Current per-row token returned with a sequencing or terminal outcome." + }, + "id": { + "type": [ + "string", + "null" + ] + }, + "kind": { + "oneOf": [ + { + "type": "null" + }, + { + "$ref": "#/components/schemas/StreamIngestKindOutput" + } + ] + }, + "limit": { + "type": [ + "integer", + "null" + ], + "format": "int64", + "minimum": 0 + }, + "message": { + "type": [ + "string", + "null" + ] + }, + "ordinal": { + "type": "integer", + "format": "int64", + "minimum": 0 + }, + "scope": { + "$ref": "#/components/schemas/StreamIngestScopeOutput" + }, + "status": { + "$ref": "#/components/schemas/StreamIngestStatusOutput" + }, + "stream_token": { + "type": [ + "string", + "null" + ], + "description": "Confirmed per-row sequencing token. This is not the graph-level ETag." + }, + "type": { + "type": [ + "string", + "null" + ] + }, + "unconfirmed_candidate_token": { + "type": [ + "string", + "null" + ], + "description": "Candidate token whose durability acknowledgement is unknown." + }, + "write_id": { + "type": [ + "string", + "null" + ] + } + } + }, + "StreamIngestScopeOutput": { + "type": "string", + "description": "Whether one stream result applies to a single row or blocks the graph-wide\nremainder of the request.", + "enum": [ + "row", + "graph" + ] + }, + "StreamIngestStatusOutput": { + "type": "string", + "description": "Stable, graph-logical status vocabulary for one streaming-ingest line.\n\nSeveral private physical-authority transitions intentionally collapse to\n`stream_authority_changed`; the transport never exposes lane, binding,\nshard, epoch, generation, dataset, or recovery-sidecar identity.", + "enum": [ + "durable", + "ack_unknown", + "already_durable", + "withdrawn", + "dead_lettered", + "invalid", + "stream_input_too_large", + "stream_authority_changed", + "stream_sequence_conflict", + "stream_idempotency_conflict", + "stream_fold_required", + "stream_backpressure", + "recovery_required", + "stream_retry_required" + ] } }, "securitySchemes": { From 963837f6775f844ede4df28c332f2bd454288036 Mon Sep 17 00:00:00 2001 From: aaltshuler Date: Tue, 4 Aug 2026 15:59:42 +0300 Subject: [PATCH 2/4] fix(streaming): bound enrollment recovery stack --- crates/omnigraph/src/db/manifest/recovery.rs | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/crates/omnigraph/src/db/manifest/recovery.rs b/crates/omnigraph/src/db/manifest/recovery.rs index ce68aaa0..b86ed694 100644 --- a/crates/omnigraph/src/db/manifest/recovery.rs +++ b/crates/omnigraph/src/db/manifest/recovery.rs @@ -10811,13 +10811,16 @@ pub(crate) async fn complete_stream_enrollment_sidecar_v14( snapshot: &Snapshot, sidecar: &RecoverySidecar, ) -> Result<()> { - process_stream_enrollment_sidecar_v14( + // Keep the deep recovery-and-manifest-publish future on the heap. Lazy + // graph enrollment runs on a Tokio worker, and the nested debug Lance + // planning stack otherwise overflows the default 2 MiB worker stack. + Box::pin(process_stream_enrollment_sidecar_v14( root_uri, &storage, snapshot, sidecar, StreamEnrollmentCleanup::BestEffortAfterVisible, - ) + )) .await .map(|_| ()) } From c87dd676f12d3e8bb972c9585a1167a148ebd6ef Mon Sep 17 00:00:00 2001 From: aaltshuler Date: Tue, 4 Aug 2026 17:56:25 +0300 Subject: [PATCH 3/4] fix(ci): align AWS test stack floor --- .github/workflows/ci.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 14eb5053..486a12fc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -360,6 +360,7 @@ jobs: contents: read env: CARGO_TERM_COLOR: always + RUST_MIN_STACK: 16777216 steps: - name: Skip for documentation-only changes if: needs.classify_changes.outputs.run_full_ci != 'true' From faeda4be2245bbe19dbe1758b82ca86d9edc35b7 Mon Sep 17 00:00:00 2001 From: aaltshuler Date: Tue, 4 Aug 2026 18:21:08 +0300 Subject: [PATCH 4/4] fix(ci): scope AWS stack floor to tests --- .github/workflows/ci.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 486a12fc..88428382 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -360,7 +360,6 @@ jobs: contents: read env: CARGO_TERM_COLOR: always - RUST_MIN_STACK: 16777216 steps: - name: Skip for documentation-only changes if: needs.classify_changes.outputs.run_full_ci != 'true' @@ -396,6 +395,8 @@ jobs: - name: Test omnigraph-server with aws feature if: needs.classify_changes.outputs.run_full_ci == 'true' + env: + RUST_MIN_STACK: 16777216 run: cargo test --locked -p omnigraph-server --features aws rustfs_integration: