diff --git a/docs/dev/firehose-path-specs.md b/docs/dev/firehose-path-specs.md index 32ec0775..d9adccd2 100644 --- a/docs/dev/firehose-path-specs.md +++ b/docs/dev/firehose-path-specs.md @@ -279,7 +279,7 @@ v16→v17/recovery-v19 authority retirement, and v17→v18/recovery-v20 exact grammar takes another strand when its final shape differs from a dormant scaffold. Dormant discriminator names never authorize reinterpretation of their frozen payload. The exact pre-release strand count is recorded as shapes -settle and freezes at the 0.10.0 release gate; an honest extra strand is cheaper +settle and freezes at the 0.9.0 release gate; an honest extra strand is cheaper than pre-registering a guessed on-disk contract. The lifecycle tranche plus F3 are the operator lifecycle and maintenance @@ -2229,7 +2229,7 @@ root. V17 owns `WITHDRAWN` retirement; F5 extends it for `DEAD_LETTERED`. is either exactly the frozen scaffold already registered or takes a new honest pre-release strand. There is no guessed numeric ceiling: every added strand is recorded with predecessor refusal/rebuild evidence, and the complete count -freezes at the 0.10.0 release gate. No discriminator acquires a different +freezes at the 0.9.0 release gate. No discriminator acquires a different payload meaning in place merely to save a rebuild. **What the experimental designation does and does not buy.** It licenses diff --git a/docs/dev/testing.md b/docs/dev/testing.md index 125f8f9b..4914b37c 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -940,8 +940,9 @@ OMNIGRAPH_V8_BIN=/path/to/final-v8/omnigraph \ RFC-026 §4.7 P1 (the v10 stream-profile format) added the historical `OMNIGRAPH_V9_BIN` seam. It mints a genuine internal-v9 graph with the pinned -final-v9 binary, proves CURRENT refuses it naming the published `0.9.x` line in -both message slots (`created by omnigraph 0.9.x` and `with an omnigraph 0.9.x +final-v9 binary, proves CURRENT refuses it naming the `0.9.x` line in +both message slots (no 0.9.x binary was ultimately published — the label +predates that outcome and awaits the release-map relabel) (`created by omnigraph 0.9.x` and `with an omnigraph 0.9.x binary` — the exact strings are also pinned in-source by `migrations.rs::release_names_the_writing_line_for_each_stamp`), exports with v9, rebuilds a distinct current-format root, proves row/vector fidelity plus @@ -955,7 +956,7 @@ OMNIGRAPH_V9_BIN=/path/to/final-v9/omnigraph \ ``` The historical `OMNIGRAPH_V10_BIN` seam mints a genuine final-v10 graph with -the matching 0.10.0-dev source build, proves CURRENT refuses it with +the matching 0.9.0-dev source build, proves CURRENT refuses it with source-build/export guidance, exports with v10, rebuilds a distinct current-format root, and proves row/vector/blob fidelity plus exact-`id` PK metadata. The old v10 binary must refuse the current root. It remains available @@ -997,7 +998,7 @@ OMNIGRAPH_V12_BIN=/path/to/final-v12/omnigraph \ ``` The historical `OMNIGRAPH_V13_BIN` seam mints a -genuine final-v13 graph with the matching 0.10.0-dev source build, proves CURRENT +genuine final-v13 graph with the matching 0.9.0-dev source build, proves CURRENT refuses it with source-build/export guidance, exports with v13, rebuilds a distinct current-format root, and proves row/vector/blob fidelity plus exact-`id` PK metadata. The old v13 binary must refuse the current root. The fixture is clean, diff --git a/docs/dev/versioning.md b/docs/dev/versioning.md index a3515dd9..6fa13325 100644 --- a/docs/dev/versioning.md +++ b/docs/dev/versioning.md @@ -46,8 +46,12 @@ declares exactly non-null physical `id` as Lance's unenforced primary key from creation, and production strict insert/upsert routes use the exact-`id` filter-bearing adapter. -Internal schema v9 maps to OmniGraph **0.9.x** — the first published release -line to serve any of these formats. +Internal schema v9 was also a 0.9.0-dev format: the 0.9.0 release was +prepared at this stamp (workspace bump plus release notes) but never tagged or +published, so — like v5–v8 — only source builds off `main` serve it. The +in-code release map still labels stamp 9 `0.9.x`, and the refusal grammar +tests pin that string; relabeling it to source-build guidance is release-prep +work, annotated next to the arm-split note in `migrations.rs`. It preserves v8's private data-bearing MemWAL core, then activates RFC-026's common B2 storage/recovery contract: stream-config v3, lifecycle state v2, the grammar-impossible trusted base-row field @@ -72,7 +76,7 @@ writer. The physical field's trailing `$` is outside the `.pg` identifier grammar, so a genuine v8 user property named `__omnigraph_stream_v1` remains ordinary user data and round-trips unchanged. -Internal schema v10 was the first 0.10.0-dev streaming-profile format. It +Internal schema v10 was the first 0.9.0-dev streaming-profile format. It preserved the complete v9 contract and added RFC-026 §4.7 P1's enablement authority: one required graph-global `stream_profile` singleton row, present from genesis (disabled, revision 1), flipped through the shared publisher's @@ -109,7 +113,7 @@ accepted v13 discriminator. It owns the exact token-ledger selects its achieved token witness and the next profile together makes either authoritative. -Internal schema v12 was an unreleased 0.10.0-dev format. It replaces lifecycle +Internal schema v12 was an unreleased 0.9.0-dev format. It replaces lifecycle state-v2's inline receipt histories with lifecycle-v3 fixed-size ledger-chain/current pointers and an authenticated WAL-tail commitment. The recovery-sidecar ceiling is v14. Its active hidden discriminators are @@ -126,7 +130,7 @@ historical. It did not encode the complete prior lifecycle/profile/topology, the physical claim attempt, or the two terminal receipt families required to recover resume safely, so v13 does not reinterpret it. -Internal schema v13 was an unreleased 0.10.0-dev format. It preserves lifecycle-v3 +Internal schema v13 was an unreleased 0.9.0-dev format. It preserves lifecycle-v3 and raises the recovery-sidecar ceiling to v15. Recovery-v15 has one active hidden discriminator, `StreamResume`, which owns the complete revision-fenced `SEALED → OPEN` resume or guarded `DRAINING → OPEN` abort: exact request and @@ -136,7 +140,7 @@ Receipt lookup precedes revision refusal for idempotent retry. The v14 resume/correction/retirement/ledger-maintenance/sealed-maintenance/rebind scaffolds retain their old bytes and continue to fail closed. -Internal schema v14 was an unreleased 0.10.0-dev format. It raised the sidecar +Internal schema v14 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v16 for one active hidden discriminator, `StreamSealedEnsureIndices`. V16 reuses the frozen recovery-v8 exact CreateIndex plan and layers the enabled profile, selected @@ -149,7 +153,7 @@ retry idempotency. Ambient EnsureIndices remains refused for enrolled tables. Recovery-v14's sealed-maintenance scaffold keeps its original bytes and is not reinterpreted. -Internal schema v15 was an unreleased 0.10.0-dev format. It raised the sidecar +Internal schema v15 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v17 for the distinct hidden `StreamSealedOptimize` discriminator. V17 owns Optimize's bounded, internally committing maintenance plan, exact confirmed outputs, and complete @@ -158,7 +162,7 @@ refreshes publish atomically; a true no-work invocation is effect-free. It writes no token row, advances no receipt chain, and accepts no caller operation ID. Ambient Optimize remains refused for enrolled tables. -Internal schema v16 was an unreleased 0.10.0-dev format. It raised the sidecar +Internal schema v16 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v18 for the distinct hidden `StreamRebind` discriminator. V18 binds the complete prior `SEALED` authority, exact fresh physical enrollment and empty shard, immutable binding @@ -168,7 +172,7 @@ recovery-v15 resume must claim a higher epoch within the fresh binding scope. The v14 rebind scaffold and recovery-v17 Optimize envelope retain their exact historical meanings and are never reinterpreted. -Internal schema v17 was an unreleased 0.10.0-dev format. It raised the sidecar ceiling to recovery-v19 for +Internal schema v17 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v19 for one active `StreamAuthorityRetirement` discriminator. Under checked stopped/offline cluster authority, a read-only plan proves an exact `DISABLED` profile, settled recovery, every enrolled lane `SEALED`, base/token parity, and @@ -188,7 +192,7 @@ retired source cut; it need not equal the fresh target graph identity, whose schema compatibility remains ordinary loader validation. This slice added no production path that creates `WITHDRAWN`. -Internal schema v18 was an unreleased 0.10.0-dev format. It raised the sidecar +Internal schema v18 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v20 for exact `DataBlock` correction while leaving the frozen recovery-v14 correction scaffold unchanged. Recovery-v20 binds the exact blocked generation cut, one @@ -198,7 +202,7 @@ management-receipt effect, fixed graph lineage, and the complete next block; receipt lookup precedes stale block/revision refusal on exact retries. Internal schema **v19 is the currently served format** (unreleased, current -0.10.0-dev source builds). It upgrades `_stream_tokens.lance` to schema v3, +0.9.0-dev source builds). It upgrades `_stream_tokens.lance` to schema v3, adds terminal `DEAD_LETTERED` evidence and versioned fold attribution, and raises the sidecar ceiling to recovery-v21. `DeadLetterFold` deterministically partitions one bounded generation, preserves valid winners, and writes all diff --git a/docs/releases/v0.10.0.md b/docs/releases/v0.10.0.md deleted file mode 100644 index 3dcc04c9..00000000 --- a/docs/releases/v0.10.0.md +++ /dev/null @@ -1,169 +0,0 @@ -# OmniGraph v0.10.0 (unreleased) - -## Experimental streaming-profile ownership - -This release moves the experimental graph-wide streaming profile behind the -cluster control plane. A profile change requires all writer-capable processes -for the graph to be stopped, the cluster state lock to be enabled, an -authenticated apply actor, and: - -```bash -omnigraph cluster apply --config \ - --as --confirm-stream-offline -``` - -The confirmation is an operator attestation of the stop/apply/restart handoff, -not a distributed lease. Run the apply offline and restart the server after it -completes. - -`streaming: true` is **not additive in this release**. While the durable -profile is enabled, embedded SDK and direct `--store` Mutation/Load/delete -calls are refused before body/file reads, staging, recovery arming, or Lance -effects. Existing served mutations are authorized only by the exact checked -runtime attached during cluster server boot. There is no public firehose -ingress endpoint yet. Branch merge is refused while the profile is `ENABLED` -or `DISABLING`, even through that served runtime. - -Profile apply evaluates `stream_manage` under both the currently applied graph -policy and the desired graph policy. A simultaneous policy change must allow -the actor on both sides of the cluster-state CAS; if only one side binds a -policy, that policy governs. When one side would deny, use two applies: grant -then change the profile, or change the profile then revoke. If the profile -change is blocked, apply also blocks every current- or desired-bound policy -change for that graph and keeps the currently applied policy selected. In -particular, a simultaneous revoke cannot land and strand the next profile -retry. - -Receipt replay is actor-bound. Retry a lost profile-apply result with the same -`--as` actor. If that identity is unavailable after the graph effect but -before the state CAS, run `cluster refresh` to reconcile the ledger from the -manifest, then re-plan under the replacement actor. - -An explicit offline `streaming: false` apply restores the direct content-write -lane only for an unenrolled graph. Internal schema v19 includes the hidden, -recovery-covered lifecycle core that can quiesce an enrolled lane through -`OPEN → DRAINING → SEALED`, including never-written and non-empty lanes. The -checked offline disable owner invokes that core after publishing `DISABLING`, -visits one finite manifest-derived lane cut in deterministic order, and adopts -an existing `OPEN_AFTER_FOLD` drain without minting a second occurrence. A -selected `DataBlock` leaves the apply durably pending until stopped/offline -correction and an apply retry. There is still no standalone public lifecycle -verb. The crate-private evidence path can also recovery-cover an explicit -`SEALED → OPEN` resume or a guarded `DRAINING → OPEN` abort; that is not a -supported operator surface. Existing `SEALED` enrollments remain fenced after -disable, so only the no-lane case restores direct content writes. An enrolled -source must satisfy ordinary export's clean preflight or use the exact -same-format retirement/export authority-discard exit; neither logical rebuild -path transfers live sequencing authority. Disabling itself never discards that -authority. - -The private F3b evidence seam can run content-preserving EnsureIndices on -canonical main through the exact checked serving runtime and a `stream_manage` -actor, but only when every enrolled productive table is exactly `SEALED`. -Recovery-v16 publishes index pointers with refreshed lifecycle HEAD witnesses, -proof digests, and revisions atomically. This naturally idempotent maintenance -call has no caller operation ID or token-ledger ManagementReceipt. Ambient -EnsureIndices remains refused for enrolled tables. - -The private F3c seam adds the corresponding checked `SEALED` Optimize path. -Recovery-v17 owns Optimize's internally committing compaction/index-maintenance -effects, records their achieved physical HEADs, and publishes the table -pointers with refreshed lifecycle proofs in one manifest CAS. A true no-work -Optimize stays effect-free. Ambient Optimize remains refused for enrolled -tables; all CLI/HTTP/OpenAPI maintenance surfaces remain unavailable. - -The private F3d seam adds recovery-covered physical rebind for an exact -`SEALED` lane. It retains the old binding and claim history, creates a fresh -enrollment and empty shard namespace, records the new binding and fence-only -claim receipts, and publishes a new exact proof while remaining `SEALED`. -Rebind never admits a writer or put; a separate resume is required. No public -rebind command or transport surface is added in this release. - -The F3e slice adds the one supported terminal exit for a v17 graph whose -current `WITHDRAWN` sequencing authority blocks ordinary export. The -cluster-only `stream retire-for-rebuild plan|confirm` handshake requires the -state lock, an authenticated actor, explicit stopped/offline confirmation, an -exact `DISABLED` profile, every enrolled lane `SEALED`, settled recovery, and -verified base/token parity. Planning is read-only. Confirmation records one -immutable actor- and plan-bound receipt and irreversibly publishes `RETIRED` -without moving graph or branch lineage. The source is thereafter permanently -read/query/status/export-only; retired export re-proves the frozen cut and emits -the selected root receipt plus a `branch_member` witness whose -`table_witness_digest` makes its `branch_member_digest` recomputable. The -`source_schema_ir_hash`, exact `ordered_branch_member_digests`, and -`selected_member_index` prove that member belongs to the receipt's -`export_cut_digest`. The source schema hash commits the retired source cut; it -need not equal the fresh target graph identity, whose compatibility remains -ordinary loader validation. At the F3e boundary, every production path that -could create `WITHDRAWN` remained inactive. - -The F3f slice adds the stopped/offline cluster-only `stream block show|correct` -surface for one exact `DataBlock` while the lane remains `DRAINING`. Show -reconstructs bounded, cursor-stable correction evidence from the retained WAL -cut. Correct requires an actor, block token, lifecycle revision, correction -UUID, and an ordered REPLACE/WITHDRAW plan. Recovery-v20 owns one pre-minted -base transaction and one combined token-successor plus immutable correction/ -management-receipt transaction. It may recreate only the latter after proving -the exact base effect; token-only, foreign, buried, or mixed outcomes fail -closed. Unchanged winners retain their current token rows, replacements chain -from the blocked token, and withdrawals retain terminal sequencing authority. -The sole manifest publication advances both pointers, records lineage and any -PRESENT attribution, releases that exact block, and leaves the drain in -`DRAINING`. This is correction, not resume: the ordinary drain owner must still -finish the lane to `SEALED`. - -The F5b slice advances the current development format to internal schema v19, -upgrades `_stream_tokens.lance` to schema v3, and adds recovery-v21. One -deterministic bounded fold publishes valid winners and records every losing -terminal candidate in one canonical NDJSON object; each losing key becomes -current `DEAD_LETTERED` authority. An all-diverted generation still advances -the base through a marker-only transaction. Exact retry returns the same -terminal result while its token remains current, and a fresh ordinary -admission naming that predecessor can restore `PRESENT`. - -Stopped/offline `cluster stream dead-letter list|export` pages through the -manifest-selected current terminal authority and verifies payload descriptors -without prefix listing or a second inventory. Payload export is inspection, -not replay or import. Recovery-v21 also extends irreversible retirement to an -exact `WITHDRAWN | DEAD_LETTERED` cut. Recovery-v19 and recovery-v20 retain -their historical meanings. No public HTTP, SDK, remote-CLI, or OpenAPI row -surface is activated; F6 still owns representative object/RSS/lookup -measurements and the full guardrail matrix. - -## Storage-format change - -The current development strand advances from the interim v18 format to v19, -upgrades current-token authority to schema v3, and adds recovery-sidecar schema -v21 `DeadLetterFold` plus `StreamAuthorityRetirementV2`; a later pre-release -strand may supersede v19 before v0.10.0 ships. Recovery-v20 retains its exact -DataBlock-correction meaning, recovery-v19 retains its exact historical -two-disposition authority-retirement meaning, recovery-v18 retains its exact -rebind meaning, -recovery-v17 retains its exact Optimize meaning, recovery-v16 retains its exact -EnsureIndices meaning, recovery-v15 retains its exact resume meaning, and -recovery-v14's incomplete correction/retirement/rebind scaffolds remain frozen. -These development binaries remain strict-single-version: v18 graphs must be -rebuilt into a different v19 root, and older binaries refuse v19. - -Before exporting a v18 graph, stop its writers. The profile must be explicitly -`DISABLED`; every enrolled lane must be `SEALED`, recovery settled, base/token -parity exact, and current terminal authority empty. Such a graph uses ordinary -export. A graph with current `WITHDRAWN` authority instead requires v18's exact -same-format retirement handshake; if it cannot satisfy that protocol, preserve -the old root because the v19 binary cannot open or upgrade it. Export the -eligible source with the matching final v18 source build, then initialize and -load a fresh v19 root with this binary. -Logical rows, vectors, blobs, and ordinary user -properties are preserved; private stream lifecycle, WAL, token, receipt, -maintenance, correction, rebind, and dead-letter authority is not transferred -by ordinary export. The recorded v17↔v18 binary result remains historical; a -genuine v18↔v19 adjacent-binary cell now proves both refusals, ordinary rebuild -fidelity, and compatibility with final-v18 receipt-v1 retirement exports. - -The preceding F3d development strand's v15→v16 transition remains historical: -its final v15 source build is -`84f3af758947970d16040a987cb1d6ea0f0931e8`, and its genuine v15↔v16 refusal/ -rebuild cell remains on demand. - -See [the upgrade guide](../user/operations/upgrade.md) for the complete -export/init/load procedure. diff --git a/docs/releases/v0.9.0.md b/docs/releases/v0.9.0.md index 5782f5f3..4c631247 100644 --- a/docs/releases/v0.9.0.md +++ b/docs/releases/v0.9.0.md @@ -1,12 +1,17 @@ -# Omnigraph v0.9.0 +# Omnigraph v0.9.0 (unreleased — rolling draft) -A substrate and durability release. The storage substrate reaches **Lance -9.0.0 stable**, which restores crates.io publication; the write path gains -stable schema identity across renames, substrate-native key-conflict fencing, -and crash recovery for every writer. +The next release: everything since v0.8.1. A substrate and durability +release — the storage substrate reaches **Lance 9.0.0 stable**, which restores +crates.io publication; the write path gains stable schema identity across +renames, substrate-native key-conflict fencing, and crash recovery for every +writer — plus the experimental streaming-profile line described in the +sections further below. This file is the rolling draft for the unreleased +0.9.0 line; every development build since v0.8.1 identifies as 0.9.0. **This release changes the on-disk format.** Internal schema advances from v4 -(0.8.x) to v9, so **v0.8.x graphs must be rebuilt via export/import** — see +(0.8.x) to the current development stamp — v19 at the time of writing; the +release freezes whatever stamp is current at tag time — so **v0.8.x graphs +must be rebuilt via export/import** — see [Upgrade notes](#upgrade-notes) below and [docs/user/operations/upgrade.md](../user/operations/upgrade.md). @@ -110,17 +115,22 @@ channels. - **Storage-format change — rebuild required.** Opening a v0.8.x graph is refused with a message naming the release line that wrote it and the exact commands. The recipe: export with a 0.8.x binary, `init` a **different** root - with 0.9.x, then `load --mode overwrite`. Data, vectors, and blobs are + with the 0.9.0 release binary, then `load --mode overwrite`. Data, vectors, and blobs are preserved; commit history and branches are not. Keep the old root unchanged through the rollback window, and never force-init the old root in place. - **Server deployments:** take the graph out of the serving set, rebuild it offline with the CLI, then repoint the cluster with `cluster apply`. -- **Internal schema stamps v5 through v8 were never published.** The format - advanced five times inside the single 0.8.1 → 0.9.0 development window, so - only source builds off `main` can carry them; the refusal message names them - `0.9.0-dev`. A published binary only ever wrote v4 (0.8.x) or v9 (0.9.x). -- Downgrade to 0.8.x is not possible for a v9 graph — a 0.8.x binary refuses it, - by design. +- **No internal schema stamp between v4 and the release stamp was ever + published.** The format advanced repeatedly inside the single + 0.8.1 → 0.9.0 development window (v5 through v18 at the time of writing), + so only source builds off `main` can carry those stamps. A published binary + only ever wrote v4 (0.8.x) or earlier. The refusal message names stamps + v5–v8 `0.9.0-dev`; stamp v9's refusal label still reads `0.9.x` from when + this release was expected to ship at that stamp — treat it as "final v9 + source build" (the release-map relabel is release-prep work, annotated in + `migrations.rs`). +- Downgrade to 0.8.x is not possible for a current-format graph — a 0.8.x + binary refuses it, by design. ## Developer-facing @@ -133,3 +143,180 @@ channels. [docs/dev/writes.md](../dev/writes.md). - Cross-version rebuild tests run against genuine older binaries rather than simulated stamps, including the immediately preceding format. + +--- + +# The experimental streaming line + +*(The sections below were previously drafted under a "v0.10.0" heading on the +assumption that 0.9.0 had already shipped. It had not: v0.8.1 remains the +latest release, this line identifies as 0.9.0, and these notes belong to the +same unreleased 0.9.0 release above.)* + +## Experimental streaming-profile ownership + +This release moves the experimental graph-wide streaming profile behind the +cluster control plane. A profile change requires all writer-capable processes +for the graph to be stopped, the cluster state lock to be enabled, an +authenticated apply actor, and: + +```bash +omnigraph cluster apply --config \ + --as --confirm-stream-offline +``` + +The confirmation is an operator attestation of the stop/apply/restart handoff, +not a distributed lease. Run the apply offline and restart the server after it +completes. + +`streaming: true` is **not additive in this release**. While the durable +profile is enabled, embedded SDK and direct `--store` Mutation/Load/delete +calls are refused before body/file reads, staging, recovery arming, or Lance +effects. Existing served mutations are authorized only by the exact checked +runtime attached during cluster server boot. There is no public firehose +ingress endpoint yet. Branch merge is refused while the profile is `ENABLED` +or `DISABLING`, even through that served runtime. + +Profile apply evaluates `stream_manage` under both the currently applied graph +policy and the desired graph policy. A simultaneous policy change must allow +the actor on both sides of the cluster-state CAS; if only one side binds a +policy, that policy governs. When one side would deny, use two applies: grant +then change the profile, or change the profile then revoke. If the profile +change is blocked, apply also blocks every current- or desired-bound policy +change for that graph and keeps the currently applied policy selected. In +particular, a simultaneous revoke cannot land and strand the next profile +retry. + +Receipt replay is actor-bound. Retry a lost profile-apply result with the same +`--as` actor. If that identity is unavailable after the graph effect but +before the state CAS, run `cluster refresh` to reconcile the ledger from the +manifest, then re-plan under the replacement actor. + +An explicit offline `streaming: false` apply restores the direct content-write +lane only for an unenrolled graph. Internal schema v19 includes the hidden, +recovery-covered lifecycle core that can quiesce an enrolled lane through +`OPEN → DRAINING → SEALED`, including never-written and non-empty lanes. The +checked offline disable owner invokes that core after publishing `DISABLING`, +visits one finite manifest-derived lane cut in deterministic order, and adopts +an existing `OPEN_AFTER_FOLD` drain without minting a second occurrence. A +selected `DataBlock` leaves the apply durably pending until stopped/offline +correction and an apply retry. There is still no standalone public lifecycle +verb. The crate-private evidence path can also recovery-cover an explicit +`SEALED → OPEN` resume or a guarded `DRAINING → OPEN` abort; that is not a +supported operator surface. Existing `SEALED` enrollments remain fenced after +disable, so only the no-lane case restores direct content writes. An enrolled +source must satisfy ordinary export's clean preflight or use the exact +same-format retirement/export authority-discard exit; neither logical rebuild +path transfers live sequencing authority. Disabling itself never discards that +authority. + +The private F3b evidence seam can run content-preserving EnsureIndices on +canonical main through the exact checked serving runtime and a `stream_manage` +actor, but only when every enrolled productive table is exactly `SEALED`. +Recovery-v16 publishes index pointers with refreshed lifecycle HEAD witnesses, +proof digests, and revisions atomically. This naturally idempotent maintenance +call has no caller operation ID or token-ledger ManagementReceipt. Ambient +EnsureIndices remains refused for enrolled tables. + +The private F3c seam adds the corresponding checked `SEALED` Optimize path. +Recovery-v17 owns Optimize's internally committing compaction/index-maintenance +effects, records their achieved physical HEADs, and publishes the table +pointers with refreshed lifecycle proofs in one manifest CAS. A true no-work +Optimize stays effect-free. Ambient Optimize remains refused for enrolled +tables; all CLI/HTTP/OpenAPI maintenance surfaces remain unavailable. + +The private F3d seam adds recovery-covered physical rebind for an exact +`SEALED` lane. It retains the old binding and claim history, creates a fresh +enrollment and empty shard namespace, records the new binding and fence-only +claim receipts, and publishes a new exact proof while remaining `SEALED`. +Rebind never admits a writer or put; a separate resume is required. No public +rebind command or transport surface is added in this release. + +The F3e slice adds the one supported terminal exit for a v17 graph whose +current `WITHDRAWN` sequencing authority blocks ordinary export. The +cluster-only `stream retire-for-rebuild plan|confirm` handshake requires the +state lock, an authenticated actor, explicit stopped/offline confirmation, an +exact `DISABLED` profile, every enrolled lane `SEALED`, settled recovery, and +verified base/token parity. Planning is read-only. Confirmation records one +immutable actor- and plan-bound receipt and irreversibly publishes `RETIRED` +without moving graph or branch lineage. The source is thereafter permanently +read/query/status/export-only; retired export re-proves the frozen cut and emits +the selected root receipt plus a `branch_member` witness whose +`table_witness_digest` makes its `branch_member_digest` recomputable. The +`source_schema_ir_hash`, exact `ordered_branch_member_digests`, and +`selected_member_index` prove that member belongs to the receipt's +`export_cut_digest`. The source schema hash commits the retired source cut; it +need not equal the fresh target graph identity, whose compatibility remains +ordinary loader validation. At the F3e boundary, every production path that +could create `WITHDRAWN` remained inactive. + +The F3f slice adds the stopped/offline cluster-only `stream block show|correct` +surface for one exact `DataBlock` while the lane remains `DRAINING`. Show +reconstructs bounded, cursor-stable correction evidence from the retained WAL +cut. Correct requires an actor, block token, lifecycle revision, correction +UUID, and an ordered REPLACE/WITHDRAW plan. Recovery-v20 owns one pre-minted +base transaction and one combined token-successor plus immutable correction/ +management-receipt transaction. It may recreate only the latter after proving +the exact base effect; token-only, foreign, buried, or mixed outcomes fail +closed. Unchanged winners retain their current token rows, replacements chain +from the blocked token, and withdrawals retain terminal sequencing authority. +The sole manifest publication advances both pointers, records lineage and any +PRESENT attribution, releases that exact block, and leaves the drain in +`DRAINING`. This is correction, not resume: the ordinary drain owner must still +finish the lane to `SEALED`. + +The F5b slice advances the current development format to internal schema v19, +upgrades `_stream_tokens.lance` to schema v3, and adds recovery-v21. One +deterministic bounded fold publishes valid winners and records every losing +terminal candidate in one canonical NDJSON object; each losing key becomes +current `DEAD_LETTERED` authority. An all-diverted generation still advances +the base through a marker-only transaction. Exact retry returns the same +terminal result while its token remains current, and a fresh ordinary +admission naming that predecessor can restore `PRESENT`. + +Stopped/offline `cluster stream dead-letter list|export` pages through the +manifest-selected current terminal authority and verifies payload descriptors +without prefix listing or a second inventory. Payload export is inspection, +not replay or import. Recovery-v21 also extends irreversible retirement to an +exact `WITHDRAWN | DEAD_LETTERED` cut. Recovery-v19 and recovery-v20 retain +their historical meanings. No public HTTP, SDK, remote-CLI, or OpenAPI row +surface is activated; F6 still owns representative object/RSS/lookup +measurements and the full guardrail matrix. + +## Storage-format change + +The current development strand advances from the interim v18 format to v19, +upgrades current-token authority to schema v3, and adds recovery-sidecar schema +v21 `DeadLetterFold` plus `StreamAuthorityRetirementV2`; a later pre-release +strand may supersede v19 before v0.9.0 ships. Recovery-v20 retains its exact +DataBlock-correction meaning, recovery-v19 retains its exact historical +two-disposition authority-retirement meaning, recovery-v18 retains its exact +rebind meaning, +recovery-v17 retains its exact Optimize meaning, recovery-v16 retains its exact +EnsureIndices meaning, recovery-v15 retains its exact resume meaning, and +recovery-v14's incomplete correction/retirement/rebind scaffolds remain frozen. +These development binaries remain strict-single-version: v18 graphs must be +rebuilt into a different v19 root, and older binaries refuse v19. + +Before exporting a v18 graph, stop its writers. The profile must be explicitly +`DISABLED`; every enrolled lane must be `SEALED`, recovery settled, base/token +parity exact, and current terminal authority empty. Such a graph uses ordinary +export. A graph with current `WITHDRAWN` authority instead requires v18's exact +same-format retirement handshake; if it cannot satisfy that protocol, preserve +the old root because the v19 binary cannot open or upgrade it. Export the +eligible source with the matching final v18 source build, then initialize and +load a fresh v19 root with this binary. +Logical rows, vectors, blobs, and ordinary user +properties are preserved; private stream lifecycle, WAL, token, receipt, +maintenance, correction, rebind, and dead-letter authority is not transferred +by ordinary export. The recorded v17↔v18 binary result remains historical; a +genuine v18↔v19 adjacent-binary cell now proves both refusals, ordinary rebuild +fidelity, and compatibility with final-v18 receipt-v1 retirement exports. + +The preceding F3d development strand's v15→v16 transition remains historical: +its final v15 source build is +`84f3af758947970d16040a987cb1d6ea0f0931e8`, and its genuine v15↔v16 refusal/ +rebuild cell remains on demand. + +See [the upgrade guide](../user/operations/upgrade.md) for the complete +export/init/load procedure. diff --git a/docs/user/operations/upgrade.md b/docs/user/operations/upgrade.md index ecfebfc7..618130bb 100644 --- a/docs/user/operations/upgrade.md +++ b/docs/user/operations/upgrade.md @@ -37,26 +37,28 @@ from that line (the latest is safest): | internal schema v3 | omnigraph 0.6.2–0.7.2 | the latest 0.7.x (e.g. 0.7.2) | | internal schema v4 | omnigraph 0.8.x | the latest 0.8.x (e.g. 0.8.1) | | internal schema v5–v8 | no published release (see below) | a source build at the matching commit | -| internal schema v9 | omnigraph 0.9.x | the latest 0.9.x | -| internal schema v10 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v11 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v12 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v13 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v14 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v15 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v16 | unreleased (earlier 0.10.0-dev source builds) | final v16 source build at merge `ac59c4f6d1d83acc8118c410c39de2bed91f9c15` | -| internal schema v17 | unreleased (earlier 0.10.0-dev source builds) | final v17 source build at merge `41a5990d53238d63d17e139859c66613f9c25867` | -| internal schema v18 | unreleased (earlier 0.10.0-dev source builds) | final v18 source build at merge `c7c81b186bed37989fe5ce591baf0965b5102648` | -| internal schema v19 | unreleased (current 0.10.0-dev source builds) | — current development format; a later pre-release strand may supersede it | +| internal schema v9 | no published release (0.9.0-dev source builds; the refusal label still reads `0.9.x`) | a source build at the matching commit | +| internal schema v10 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v11 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v12 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v13 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v14 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v15 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v16 | unreleased (earlier 0.9.0-dev source builds) | final v16 source build at merge `ac59c4f6d1d83acc8118c410c39de2bed91f9c15` | +| internal schema v17 | unreleased (earlier 0.9.0-dev source builds) | final v17 source build at merge `41a5990d53238d63d17e139859c66613f9c25867` | +| internal schema v18 | unreleased (earlier 0.9.0-dev source builds) | final v18 source build at merge `c7c81b186bed37989fe5ce591baf0965b5102648` | +| internal schema v19 | unreleased (current 0.9.0-dev source builds) | — current development format; a later pre-release strand may supersede it | **Stamps v5–v8 never shipped.** The storage format advanced five times inside the single 0.8.1 → 0.9.0 development window, so the only graphs carrying those stamps came from source builds off `main`; no published binary reads them and the refusal message names them `0.9.0-dev`. If you have one, export it with a build of the commit that created it, then load into a fresh current-format -graph. A released binary only ever wrote v4 (0.8.x) or v9 (0.9.x); v10–v19 are -pre-release formats written by matching 0.10.0-dev source builds. The final -0.10.0 format may use a later stamp. +graph. A released binary only ever wrote v4 (0.8.x) or earlier stamps: the +0.9.0 release was prepared at v9 but never published, so v9–v19 are all +pre-release formats written by matching 0.9.0-dev source builds (v9's refusal +label still reads `0.9.x` from that expectation). The final 0.9.0 format may +use a later stamp. You can also check versions before you hit a refusal: @@ -487,8 +489,9 @@ retirement records provenance and freezes the source before row-only export to a fresh graph. None of that later F5 behavior is active in v10. A v9 graph must use the standard rebuild recipe above: quiesce writers, export -with the latest 0.9.x binary, initialize a **different** root with the v10 -(0.10.0-dev source build) binary, load the export, and verify the v10 stamp +with a final-v9 source build (no 0.9.x binary was ever published), initialize +a **different** root with the v10 +(0.9.0-dev source build) binary, load the export, and verify the v10 stamp plus row/vector/blob fidelity before cutover. Keep the v9 root unchanged through the rollback window. A v10 binary refuses v9, and a v9 binary refuses v10. @@ -510,7 +513,8 @@ correctness foundations; v9 does not by itself expose a public streaming API. A v8 graph must use the standard rebuild recipe above. Because v8 never shipped in a release, this affects only source builds off `main` during 0.9.0 development: quiesce every v8 writer, export with a build of the commit that -created the graph, initialize a **different** root with the 0.9.x binary, load +created the graph, initialize a **different** root with a final-v9 source +build, load the export, and verify the v9 stamp plus row/vector/blob fidelity before cutover. Keep the v8 root unchanged through the rollback window. A v9 binary refuses v8, and a v8 binary refuses v9.