From 5509dcb80718cbf630595f6b6549cc0cd3867505 Mon Sep 17 00:00:00 2001 From: aaltshuler Date: Sun, 2 Aug 2026 15:17:18 +0300 Subject: [PATCH] docs: the development line is 0.9.0-dev, not 0.10.0-dev MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit v0.9.0 was prepared (workspace bump, release notes) but never tagged or published — v0.8.1 remains the latest release. The docs nevertheless treated 0.9.0 as a completed boundary: they claimed v9 maps to 'the first published release line', told operators to export v9 graphs 'with the latest 0.9.x binary' (a binary that does not exist), and labeled every stamp since v10 as a '0.10.0-dev' line whose binaries actually identify as 0.9.0. Relabel throughout: v9 joins v5-v8 as an unpublished 0.9.0-dev format with source-build export guidance, v10-v19 are 0.9.0-dev, and the release gate is 0.9.0. Merge the rolling 'v0.10.0' release notes into docs/releases/v0.9.0.md as one draft for the actual upcoming release (spanning everything since 0.8.1) and delete v0.10.0.md. Deliberately out of scope (docs describe it honestly instead): the in-code release map still labels stamp 9 as '0.9.x' inside refusal messages, and the grammar tests plus the env-gated v9 crossversion seam pin those exact strings. That relabel changes pinned test strings and is annotated as release-prep work next to the existing arm-split note in migrations.rs. --- docs/dev/firehose-path-specs.md | 4 +- docs/dev/testing.md | 9 +- docs/dev/versioning.md | 26 ++-- docs/releases/v0.10.0.md | 169 ------------------------- docs/releases/v0.9.0.md | 213 ++++++++++++++++++++++++++++++-- docs/user/operations/upgrade.md | 38 +++--- 6 files changed, 243 insertions(+), 216 deletions(-) delete mode 100644 docs/releases/v0.10.0.md diff --git a/docs/dev/firehose-path-specs.md b/docs/dev/firehose-path-specs.md index 32ec0775..d9adccd2 100644 --- a/docs/dev/firehose-path-specs.md +++ b/docs/dev/firehose-path-specs.md @@ -279,7 +279,7 @@ v16→v17/recovery-v19 authority retirement, and v17→v18/recovery-v20 exact grammar takes another strand when its final shape differs from a dormant scaffold. Dormant discriminator names never authorize reinterpretation of their frozen payload. The exact pre-release strand count is recorded as shapes -settle and freezes at the 0.10.0 release gate; an honest extra strand is cheaper +settle and freezes at the 0.9.0 release gate; an honest extra strand is cheaper than pre-registering a guessed on-disk contract. The lifecycle tranche plus F3 are the operator lifecycle and maintenance @@ -2229,7 +2229,7 @@ root. V17 owns `WITHDRAWN` retirement; F5 extends it for `DEAD_LETTERED`. is either exactly the frozen scaffold already registered or takes a new honest pre-release strand. There is no guessed numeric ceiling: every added strand is recorded with predecessor refusal/rebuild evidence, and the complete count -freezes at the 0.10.0 release gate. No discriminator acquires a different +freezes at the 0.9.0 release gate. No discriminator acquires a different payload meaning in place merely to save a rebuild. **What the experimental designation does and does not buy.** It licenses diff --git a/docs/dev/testing.md b/docs/dev/testing.md index 125f8f9b..4914b37c 100644 --- a/docs/dev/testing.md +++ b/docs/dev/testing.md @@ -940,8 +940,9 @@ OMNIGRAPH_V8_BIN=/path/to/final-v8/omnigraph \ RFC-026 §4.7 P1 (the v10 stream-profile format) added the historical `OMNIGRAPH_V9_BIN` seam. It mints a genuine internal-v9 graph with the pinned -final-v9 binary, proves CURRENT refuses it naming the published `0.9.x` line in -both message slots (`created by omnigraph 0.9.x` and `with an omnigraph 0.9.x +final-v9 binary, proves CURRENT refuses it naming the `0.9.x` line in +both message slots (no 0.9.x binary was ultimately published — the label +predates that outcome and awaits the release-map relabel) (`created by omnigraph 0.9.x` and `with an omnigraph 0.9.x binary` — the exact strings are also pinned in-source by `migrations.rs::release_names_the_writing_line_for_each_stamp`), exports with v9, rebuilds a distinct current-format root, proves row/vector fidelity plus @@ -955,7 +956,7 @@ OMNIGRAPH_V9_BIN=/path/to/final-v9/omnigraph \ ``` The historical `OMNIGRAPH_V10_BIN` seam mints a genuine final-v10 graph with -the matching 0.10.0-dev source build, proves CURRENT refuses it with +the matching 0.9.0-dev source build, proves CURRENT refuses it with source-build/export guidance, exports with v10, rebuilds a distinct current-format root, and proves row/vector/blob fidelity plus exact-`id` PK metadata. The old v10 binary must refuse the current root. It remains available @@ -997,7 +998,7 @@ OMNIGRAPH_V12_BIN=/path/to/final-v12/omnigraph \ ``` The historical `OMNIGRAPH_V13_BIN` seam mints a -genuine final-v13 graph with the matching 0.10.0-dev source build, proves CURRENT +genuine final-v13 graph with the matching 0.9.0-dev source build, proves CURRENT refuses it with source-build/export guidance, exports with v13, rebuilds a distinct current-format root, and proves row/vector/blob fidelity plus exact-`id` PK metadata. The old v13 binary must refuse the current root. The fixture is clean, diff --git a/docs/dev/versioning.md b/docs/dev/versioning.md index a3515dd9..6fa13325 100644 --- a/docs/dev/versioning.md +++ b/docs/dev/versioning.md @@ -46,8 +46,12 @@ declares exactly non-null physical `id` as Lance's unenforced primary key from creation, and production strict insert/upsert routes use the exact-`id` filter-bearing adapter. -Internal schema v9 maps to OmniGraph **0.9.x** — the first published release -line to serve any of these formats. +Internal schema v9 was also a 0.9.0-dev format: the 0.9.0 release was +prepared at this stamp (workspace bump plus release notes) but never tagged or +published, so — like v5–v8 — only source builds off `main` serve it. The +in-code release map still labels stamp 9 `0.9.x`, and the refusal grammar +tests pin that string; relabeling it to source-build guidance is release-prep +work, annotated next to the arm-split note in `migrations.rs`. It preserves v8's private data-bearing MemWAL core, then activates RFC-026's common B2 storage/recovery contract: stream-config v3, lifecycle state v2, the grammar-impossible trusted base-row field @@ -72,7 +76,7 @@ writer. The physical field's trailing `$` is outside the `.pg` identifier grammar, so a genuine v8 user property named `__omnigraph_stream_v1` remains ordinary user data and round-trips unchanged. -Internal schema v10 was the first 0.10.0-dev streaming-profile format. It +Internal schema v10 was the first 0.9.0-dev streaming-profile format. It preserved the complete v9 contract and added RFC-026 §4.7 P1's enablement authority: one required graph-global `stream_profile` singleton row, present from genesis (disabled, revision 1), flipped through the shared publisher's @@ -109,7 +113,7 @@ accepted v13 discriminator. It owns the exact token-ledger selects its achieved token witness and the next profile together makes either authoritative. -Internal schema v12 was an unreleased 0.10.0-dev format. It replaces lifecycle +Internal schema v12 was an unreleased 0.9.0-dev format. It replaces lifecycle state-v2's inline receipt histories with lifecycle-v3 fixed-size ledger-chain/current pointers and an authenticated WAL-tail commitment. The recovery-sidecar ceiling is v14. Its active hidden discriminators are @@ -126,7 +130,7 @@ historical. It did not encode the complete prior lifecycle/profile/topology, the physical claim attempt, or the two terminal receipt families required to recover resume safely, so v13 does not reinterpret it. -Internal schema v13 was an unreleased 0.10.0-dev format. It preserves lifecycle-v3 +Internal schema v13 was an unreleased 0.9.0-dev format. It preserves lifecycle-v3 and raises the recovery-sidecar ceiling to v15. Recovery-v15 has one active hidden discriminator, `StreamResume`, which owns the complete revision-fenced `SEALED → OPEN` resume or guarded `DRAINING → OPEN` abort: exact request and @@ -136,7 +140,7 @@ Receipt lookup precedes revision refusal for idempotent retry. The v14 resume/correction/retirement/ledger-maintenance/sealed-maintenance/rebind scaffolds retain their old bytes and continue to fail closed. -Internal schema v14 was an unreleased 0.10.0-dev format. It raised the sidecar +Internal schema v14 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v16 for one active hidden discriminator, `StreamSealedEnsureIndices`. V16 reuses the frozen recovery-v8 exact CreateIndex plan and layers the enabled profile, selected @@ -149,7 +153,7 @@ retry idempotency. Ambient EnsureIndices remains refused for enrolled tables. Recovery-v14's sealed-maintenance scaffold keeps its original bytes and is not reinterpreted. -Internal schema v15 was an unreleased 0.10.0-dev format. It raised the sidecar +Internal schema v15 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v17 for the distinct hidden `StreamSealedOptimize` discriminator. V17 owns Optimize's bounded, internally committing maintenance plan, exact confirmed outputs, and complete @@ -158,7 +162,7 @@ refreshes publish atomically; a true no-work invocation is effect-free. It writes no token row, advances no receipt chain, and accepts no caller operation ID. Ambient Optimize remains refused for enrolled tables. -Internal schema v16 was an unreleased 0.10.0-dev format. It raised the sidecar +Internal schema v16 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v18 for the distinct hidden `StreamRebind` discriminator. V18 binds the complete prior `SEALED` authority, exact fresh physical enrollment and empty shard, immutable binding @@ -168,7 +172,7 @@ recovery-v15 resume must claim a higher epoch within the fresh binding scope. The v14 rebind scaffold and recovery-v17 Optimize envelope retain their exact historical meanings and are never reinterpreted. -Internal schema v17 was an unreleased 0.10.0-dev format. It raised the sidecar ceiling to recovery-v19 for +Internal schema v17 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v19 for one active `StreamAuthorityRetirement` discriminator. Under checked stopped/offline cluster authority, a read-only plan proves an exact `DISABLED` profile, settled recovery, every enrolled lane `SEALED`, base/token parity, and @@ -188,7 +192,7 @@ retired source cut; it need not equal the fresh target graph identity, whose schema compatibility remains ordinary loader validation. This slice added no production path that creates `WITHDRAWN`. -Internal schema v18 was an unreleased 0.10.0-dev format. It raised the sidecar +Internal schema v18 was an unreleased 0.9.0-dev format. It raised the sidecar ceiling to recovery-v20 for exact `DataBlock` correction while leaving the frozen recovery-v14 correction scaffold unchanged. Recovery-v20 binds the exact blocked generation cut, one @@ -198,7 +202,7 @@ management-receipt effect, fixed graph lineage, and the complete next block; receipt lookup precedes stale block/revision refusal on exact retries. Internal schema **v19 is the currently served format** (unreleased, current -0.10.0-dev source builds). It upgrades `_stream_tokens.lance` to schema v3, +0.9.0-dev source builds). It upgrades `_stream_tokens.lance` to schema v3, adds terminal `DEAD_LETTERED` evidence and versioned fold attribution, and raises the sidecar ceiling to recovery-v21. `DeadLetterFold` deterministically partitions one bounded generation, preserves valid winners, and writes all diff --git a/docs/releases/v0.10.0.md b/docs/releases/v0.10.0.md deleted file mode 100644 index 3dcc04c9..00000000 --- a/docs/releases/v0.10.0.md +++ /dev/null @@ -1,169 +0,0 @@ -# OmniGraph v0.10.0 (unreleased) - -## Experimental streaming-profile ownership - -This release moves the experimental graph-wide streaming profile behind the -cluster control plane. A profile change requires all writer-capable processes -for the graph to be stopped, the cluster state lock to be enabled, an -authenticated apply actor, and: - -```bash -omnigraph cluster apply --config \ - --as --confirm-stream-offline -``` - -The confirmation is an operator attestation of the stop/apply/restart handoff, -not a distributed lease. Run the apply offline and restart the server after it -completes. - -`streaming: true` is **not additive in this release**. While the durable -profile is enabled, embedded SDK and direct `--store` Mutation/Load/delete -calls are refused before body/file reads, staging, recovery arming, or Lance -effects. Existing served mutations are authorized only by the exact checked -runtime attached during cluster server boot. There is no public firehose -ingress endpoint yet. Branch merge is refused while the profile is `ENABLED` -or `DISABLING`, even through that served runtime. - -Profile apply evaluates `stream_manage` under both the currently applied graph -policy and the desired graph policy. A simultaneous policy change must allow -the actor on both sides of the cluster-state CAS; if only one side binds a -policy, that policy governs. When one side would deny, use two applies: grant -then change the profile, or change the profile then revoke. If the profile -change is blocked, apply also blocks every current- or desired-bound policy -change for that graph and keeps the currently applied policy selected. In -particular, a simultaneous revoke cannot land and strand the next profile -retry. - -Receipt replay is actor-bound. Retry a lost profile-apply result with the same -`--as` actor. If that identity is unavailable after the graph effect but -before the state CAS, run `cluster refresh` to reconcile the ledger from the -manifest, then re-plan under the replacement actor. - -An explicit offline `streaming: false` apply restores the direct content-write -lane only for an unenrolled graph. Internal schema v19 includes the hidden, -recovery-covered lifecycle core that can quiesce an enrolled lane through -`OPEN → DRAINING → SEALED`, including never-written and non-empty lanes. The -checked offline disable owner invokes that core after publishing `DISABLING`, -visits one finite manifest-derived lane cut in deterministic order, and adopts -an existing `OPEN_AFTER_FOLD` drain without minting a second occurrence. A -selected `DataBlock` leaves the apply durably pending until stopped/offline -correction and an apply retry. There is still no standalone public lifecycle -verb. The crate-private evidence path can also recovery-cover an explicit -`SEALED → OPEN` resume or a guarded `DRAINING → OPEN` abort; that is not a -supported operator surface. Existing `SEALED` enrollments remain fenced after -disable, so only the no-lane case restores direct content writes. An enrolled -source must satisfy ordinary export's clean preflight or use the exact -same-format retirement/export authority-discard exit; neither logical rebuild -path transfers live sequencing authority. Disabling itself never discards that -authority. - -The private F3b evidence seam can run content-preserving EnsureIndices on -canonical main through the exact checked serving runtime and a `stream_manage` -actor, but only when every enrolled productive table is exactly `SEALED`. -Recovery-v16 publishes index pointers with refreshed lifecycle HEAD witnesses, -proof digests, and revisions atomically. This naturally idempotent maintenance -call has no caller operation ID or token-ledger ManagementReceipt. Ambient -EnsureIndices remains refused for enrolled tables. - -The private F3c seam adds the corresponding checked `SEALED` Optimize path. -Recovery-v17 owns Optimize's internally committing compaction/index-maintenance -effects, records their achieved physical HEADs, and publishes the table -pointers with refreshed lifecycle proofs in one manifest CAS. A true no-work -Optimize stays effect-free. Ambient Optimize remains refused for enrolled -tables; all CLI/HTTP/OpenAPI maintenance surfaces remain unavailable. - -The private F3d seam adds recovery-covered physical rebind for an exact -`SEALED` lane. It retains the old binding and claim history, creates a fresh -enrollment and empty shard namespace, records the new binding and fence-only -claim receipts, and publishes a new exact proof while remaining `SEALED`. -Rebind never admits a writer or put; a separate resume is required. No public -rebind command or transport surface is added in this release. - -The F3e slice adds the one supported terminal exit for a v17 graph whose -current `WITHDRAWN` sequencing authority blocks ordinary export. The -cluster-only `stream retire-for-rebuild plan|confirm` handshake requires the -state lock, an authenticated actor, explicit stopped/offline confirmation, an -exact `DISABLED` profile, every enrolled lane `SEALED`, settled recovery, and -verified base/token parity. Planning is read-only. Confirmation records one -immutable actor- and plan-bound receipt and irreversibly publishes `RETIRED` -without moving graph or branch lineage. The source is thereafter permanently -read/query/status/export-only; retired export re-proves the frozen cut and emits -the selected root receipt plus a `branch_member` witness whose -`table_witness_digest` makes its `branch_member_digest` recomputable. The -`source_schema_ir_hash`, exact `ordered_branch_member_digests`, and -`selected_member_index` prove that member belongs to the receipt's -`export_cut_digest`. The source schema hash commits the retired source cut; it -need not equal the fresh target graph identity, whose compatibility remains -ordinary loader validation. At the F3e boundary, every production path that -could create `WITHDRAWN` remained inactive. - -The F3f slice adds the stopped/offline cluster-only `stream block show|correct` -surface for one exact `DataBlock` while the lane remains `DRAINING`. Show -reconstructs bounded, cursor-stable correction evidence from the retained WAL -cut. Correct requires an actor, block token, lifecycle revision, correction -UUID, and an ordered REPLACE/WITHDRAW plan. Recovery-v20 owns one pre-minted -base transaction and one combined token-successor plus immutable correction/ -management-receipt transaction. It may recreate only the latter after proving -the exact base effect; token-only, foreign, buried, or mixed outcomes fail -closed. Unchanged winners retain their current token rows, replacements chain -from the blocked token, and withdrawals retain terminal sequencing authority. -The sole manifest publication advances both pointers, records lineage and any -PRESENT attribution, releases that exact block, and leaves the drain in -`DRAINING`. This is correction, not resume: the ordinary drain owner must still -finish the lane to `SEALED`. - -The F5b slice advances the current development format to internal schema v19, -upgrades `_stream_tokens.lance` to schema v3, and adds recovery-v21. One -deterministic bounded fold publishes valid winners and records every losing -terminal candidate in one canonical NDJSON object; each losing key becomes -current `DEAD_LETTERED` authority. An all-diverted generation still advances -the base through a marker-only transaction. Exact retry returns the same -terminal result while its token remains current, and a fresh ordinary -admission naming that predecessor can restore `PRESENT`. - -Stopped/offline `cluster stream dead-letter list|export` pages through the -manifest-selected current terminal authority and verifies payload descriptors -without prefix listing or a second inventory. Payload export is inspection, -not replay or import. Recovery-v21 also extends irreversible retirement to an -exact `WITHDRAWN | DEAD_LETTERED` cut. Recovery-v19 and recovery-v20 retain -their historical meanings. No public HTTP, SDK, remote-CLI, or OpenAPI row -surface is activated; F6 still owns representative object/RSS/lookup -measurements and the full guardrail matrix. - -## Storage-format change - -The current development strand advances from the interim v18 format to v19, -upgrades current-token authority to schema v3, and adds recovery-sidecar schema -v21 `DeadLetterFold` plus `StreamAuthorityRetirementV2`; a later pre-release -strand may supersede v19 before v0.10.0 ships. Recovery-v20 retains its exact -DataBlock-correction meaning, recovery-v19 retains its exact historical -two-disposition authority-retirement meaning, recovery-v18 retains its exact -rebind meaning, -recovery-v17 retains its exact Optimize meaning, recovery-v16 retains its exact -EnsureIndices meaning, recovery-v15 retains its exact resume meaning, and -recovery-v14's incomplete correction/retirement/rebind scaffolds remain frozen. -These development binaries remain strict-single-version: v18 graphs must be -rebuilt into a different v19 root, and older binaries refuse v19. - -Before exporting a v18 graph, stop its writers. The profile must be explicitly -`DISABLED`; every enrolled lane must be `SEALED`, recovery settled, base/token -parity exact, and current terminal authority empty. Such a graph uses ordinary -export. A graph with current `WITHDRAWN` authority instead requires v18's exact -same-format retirement handshake; if it cannot satisfy that protocol, preserve -the old root because the v19 binary cannot open or upgrade it. Export the -eligible source with the matching final v18 source build, then initialize and -load a fresh v19 root with this binary. -Logical rows, vectors, blobs, and ordinary user -properties are preserved; private stream lifecycle, WAL, token, receipt, -maintenance, correction, rebind, and dead-letter authority is not transferred -by ordinary export. The recorded v17↔v18 binary result remains historical; a -genuine v18↔v19 adjacent-binary cell now proves both refusals, ordinary rebuild -fidelity, and compatibility with final-v18 receipt-v1 retirement exports. - -The preceding F3d development strand's v15→v16 transition remains historical: -its final v15 source build is -`84f3af758947970d16040a987cb1d6ea0f0931e8`, and its genuine v15↔v16 refusal/ -rebuild cell remains on demand. - -See [the upgrade guide](../user/operations/upgrade.md) for the complete -export/init/load procedure. diff --git a/docs/releases/v0.9.0.md b/docs/releases/v0.9.0.md index 5782f5f3..4c631247 100644 --- a/docs/releases/v0.9.0.md +++ b/docs/releases/v0.9.0.md @@ -1,12 +1,17 @@ -# Omnigraph v0.9.0 +# Omnigraph v0.9.0 (unreleased — rolling draft) -A substrate and durability release. The storage substrate reaches **Lance -9.0.0 stable**, which restores crates.io publication; the write path gains -stable schema identity across renames, substrate-native key-conflict fencing, -and crash recovery for every writer. +The next release: everything since v0.8.1. A substrate and durability +release — the storage substrate reaches **Lance 9.0.0 stable**, which restores +crates.io publication; the write path gains stable schema identity across +renames, substrate-native key-conflict fencing, and crash recovery for every +writer — plus the experimental streaming-profile line described in the +sections further below. This file is the rolling draft for the unreleased +0.9.0 line; every development build since v0.8.1 identifies as 0.9.0. **This release changes the on-disk format.** Internal schema advances from v4 -(0.8.x) to v9, so **v0.8.x graphs must be rebuilt via export/import** — see +(0.8.x) to the current development stamp — v19 at the time of writing; the +release freezes whatever stamp is current at tag time — so **v0.8.x graphs +must be rebuilt via export/import** — see [Upgrade notes](#upgrade-notes) below and [docs/user/operations/upgrade.md](../user/operations/upgrade.md). @@ -110,17 +115,22 @@ channels. - **Storage-format change — rebuild required.** Opening a v0.8.x graph is refused with a message naming the release line that wrote it and the exact commands. The recipe: export with a 0.8.x binary, `init` a **different** root - with 0.9.x, then `load --mode overwrite`. Data, vectors, and blobs are + with the 0.9.0 release binary, then `load --mode overwrite`. Data, vectors, and blobs are preserved; commit history and branches are not. Keep the old root unchanged through the rollback window, and never force-init the old root in place. - **Server deployments:** take the graph out of the serving set, rebuild it offline with the CLI, then repoint the cluster with `cluster apply`. -- **Internal schema stamps v5 through v8 were never published.** The format - advanced five times inside the single 0.8.1 → 0.9.0 development window, so - only source builds off `main` can carry them; the refusal message names them - `0.9.0-dev`. A published binary only ever wrote v4 (0.8.x) or v9 (0.9.x). -- Downgrade to 0.8.x is not possible for a v9 graph — a 0.8.x binary refuses it, - by design. +- **No internal schema stamp between v4 and the release stamp was ever + published.** The format advanced repeatedly inside the single + 0.8.1 → 0.9.0 development window (v5 through v18 at the time of writing), + so only source builds off `main` can carry those stamps. A published binary + only ever wrote v4 (0.8.x) or earlier. The refusal message names stamps + v5–v8 `0.9.0-dev`; stamp v9's refusal label still reads `0.9.x` from when + this release was expected to ship at that stamp — treat it as "final v9 + source build" (the release-map relabel is release-prep work, annotated in + `migrations.rs`). +- Downgrade to 0.8.x is not possible for a current-format graph — a 0.8.x + binary refuses it, by design. ## Developer-facing @@ -133,3 +143,180 @@ channels. [docs/dev/writes.md](../dev/writes.md). - Cross-version rebuild tests run against genuine older binaries rather than simulated stamps, including the immediately preceding format. + +--- + +# The experimental streaming line + +*(The sections below were previously drafted under a "v0.10.0" heading on the +assumption that 0.9.0 had already shipped. It had not: v0.8.1 remains the +latest release, this line identifies as 0.9.0, and these notes belong to the +same unreleased 0.9.0 release above.)* + +## Experimental streaming-profile ownership + +This release moves the experimental graph-wide streaming profile behind the +cluster control plane. A profile change requires all writer-capable processes +for the graph to be stopped, the cluster state lock to be enabled, an +authenticated apply actor, and: + +```bash +omnigraph cluster apply --config \ + --as --confirm-stream-offline +``` + +The confirmation is an operator attestation of the stop/apply/restart handoff, +not a distributed lease. Run the apply offline and restart the server after it +completes. + +`streaming: true` is **not additive in this release**. While the durable +profile is enabled, embedded SDK and direct `--store` Mutation/Load/delete +calls are refused before body/file reads, staging, recovery arming, or Lance +effects. Existing served mutations are authorized only by the exact checked +runtime attached during cluster server boot. There is no public firehose +ingress endpoint yet. Branch merge is refused while the profile is `ENABLED` +or `DISABLING`, even through that served runtime. + +Profile apply evaluates `stream_manage` under both the currently applied graph +policy and the desired graph policy. A simultaneous policy change must allow +the actor on both sides of the cluster-state CAS; if only one side binds a +policy, that policy governs. When one side would deny, use two applies: grant +then change the profile, or change the profile then revoke. If the profile +change is blocked, apply also blocks every current- or desired-bound policy +change for that graph and keeps the currently applied policy selected. In +particular, a simultaneous revoke cannot land and strand the next profile +retry. + +Receipt replay is actor-bound. Retry a lost profile-apply result with the same +`--as` actor. If that identity is unavailable after the graph effect but +before the state CAS, run `cluster refresh` to reconcile the ledger from the +manifest, then re-plan under the replacement actor. + +An explicit offline `streaming: false` apply restores the direct content-write +lane only for an unenrolled graph. Internal schema v19 includes the hidden, +recovery-covered lifecycle core that can quiesce an enrolled lane through +`OPEN → DRAINING → SEALED`, including never-written and non-empty lanes. The +checked offline disable owner invokes that core after publishing `DISABLING`, +visits one finite manifest-derived lane cut in deterministic order, and adopts +an existing `OPEN_AFTER_FOLD` drain without minting a second occurrence. A +selected `DataBlock` leaves the apply durably pending until stopped/offline +correction and an apply retry. There is still no standalone public lifecycle +verb. The crate-private evidence path can also recovery-cover an explicit +`SEALED → OPEN` resume or a guarded `DRAINING → OPEN` abort; that is not a +supported operator surface. Existing `SEALED` enrollments remain fenced after +disable, so only the no-lane case restores direct content writes. An enrolled +source must satisfy ordinary export's clean preflight or use the exact +same-format retirement/export authority-discard exit; neither logical rebuild +path transfers live sequencing authority. Disabling itself never discards that +authority. + +The private F3b evidence seam can run content-preserving EnsureIndices on +canonical main through the exact checked serving runtime and a `stream_manage` +actor, but only when every enrolled productive table is exactly `SEALED`. +Recovery-v16 publishes index pointers with refreshed lifecycle HEAD witnesses, +proof digests, and revisions atomically. This naturally idempotent maintenance +call has no caller operation ID or token-ledger ManagementReceipt. Ambient +EnsureIndices remains refused for enrolled tables. + +The private F3c seam adds the corresponding checked `SEALED` Optimize path. +Recovery-v17 owns Optimize's internally committing compaction/index-maintenance +effects, records their achieved physical HEADs, and publishes the table +pointers with refreshed lifecycle proofs in one manifest CAS. A true no-work +Optimize stays effect-free. Ambient Optimize remains refused for enrolled +tables; all CLI/HTTP/OpenAPI maintenance surfaces remain unavailable. + +The private F3d seam adds recovery-covered physical rebind for an exact +`SEALED` lane. It retains the old binding and claim history, creates a fresh +enrollment and empty shard namespace, records the new binding and fence-only +claim receipts, and publishes a new exact proof while remaining `SEALED`. +Rebind never admits a writer or put; a separate resume is required. No public +rebind command or transport surface is added in this release. + +The F3e slice adds the one supported terminal exit for a v17 graph whose +current `WITHDRAWN` sequencing authority blocks ordinary export. The +cluster-only `stream retire-for-rebuild plan|confirm` handshake requires the +state lock, an authenticated actor, explicit stopped/offline confirmation, an +exact `DISABLED` profile, every enrolled lane `SEALED`, settled recovery, and +verified base/token parity. Planning is read-only. Confirmation records one +immutable actor- and plan-bound receipt and irreversibly publishes `RETIRED` +without moving graph or branch lineage. The source is thereafter permanently +read/query/status/export-only; retired export re-proves the frozen cut and emits +the selected root receipt plus a `branch_member` witness whose +`table_witness_digest` makes its `branch_member_digest` recomputable. The +`source_schema_ir_hash`, exact `ordered_branch_member_digests`, and +`selected_member_index` prove that member belongs to the receipt's +`export_cut_digest`. The source schema hash commits the retired source cut; it +need not equal the fresh target graph identity, whose compatibility remains +ordinary loader validation. At the F3e boundary, every production path that +could create `WITHDRAWN` remained inactive. + +The F3f slice adds the stopped/offline cluster-only `stream block show|correct` +surface for one exact `DataBlock` while the lane remains `DRAINING`. Show +reconstructs bounded, cursor-stable correction evidence from the retained WAL +cut. Correct requires an actor, block token, lifecycle revision, correction +UUID, and an ordered REPLACE/WITHDRAW plan. Recovery-v20 owns one pre-minted +base transaction and one combined token-successor plus immutable correction/ +management-receipt transaction. It may recreate only the latter after proving +the exact base effect; token-only, foreign, buried, or mixed outcomes fail +closed. Unchanged winners retain their current token rows, replacements chain +from the blocked token, and withdrawals retain terminal sequencing authority. +The sole manifest publication advances both pointers, records lineage and any +PRESENT attribution, releases that exact block, and leaves the drain in +`DRAINING`. This is correction, not resume: the ordinary drain owner must still +finish the lane to `SEALED`. + +The F5b slice advances the current development format to internal schema v19, +upgrades `_stream_tokens.lance` to schema v3, and adds recovery-v21. One +deterministic bounded fold publishes valid winners and records every losing +terminal candidate in one canonical NDJSON object; each losing key becomes +current `DEAD_LETTERED` authority. An all-diverted generation still advances +the base through a marker-only transaction. Exact retry returns the same +terminal result while its token remains current, and a fresh ordinary +admission naming that predecessor can restore `PRESENT`. + +Stopped/offline `cluster stream dead-letter list|export` pages through the +manifest-selected current terminal authority and verifies payload descriptors +without prefix listing or a second inventory. Payload export is inspection, +not replay or import. Recovery-v21 also extends irreversible retirement to an +exact `WITHDRAWN | DEAD_LETTERED` cut. Recovery-v19 and recovery-v20 retain +their historical meanings. No public HTTP, SDK, remote-CLI, or OpenAPI row +surface is activated; F6 still owns representative object/RSS/lookup +measurements and the full guardrail matrix. + +## Storage-format change + +The current development strand advances from the interim v18 format to v19, +upgrades current-token authority to schema v3, and adds recovery-sidecar schema +v21 `DeadLetterFold` plus `StreamAuthorityRetirementV2`; a later pre-release +strand may supersede v19 before v0.9.0 ships. Recovery-v20 retains its exact +DataBlock-correction meaning, recovery-v19 retains its exact historical +two-disposition authority-retirement meaning, recovery-v18 retains its exact +rebind meaning, +recovery-v17 retains its exact Optimize meaning, recovery-v16 retains its exact +EnsureIndices meaning, recovery-v15 retains its exact resume meaning, and +recovery-v14's incomplete correction/retirement/rebind scaffolds remain frozen. +These development binaries remain strict-single-version: v18 graphs must be +rebuilt into a different v19 root, and older binaries refuse v19. + +Before exporting a v18 graph, stop its writers. The profile must be explicitly +`DISABLED`; every enrolled lane must be `SEALED`, recovery settled, base/token +parity exact, and current terminal authority empty. Such a graph uses ordinary +export. A graph with current `WITHDRAWN` authority instead requires v18's exact +same-format retirement handshake; if it cannot satisfy that protocol, preserve +the old root because the v19 binary cannot open or upgrade it. Export the +eligible source with the matching final v18 source build, then initialize and +load a fresh v19 root with this binary. +Logical rows, vectors, blobs, and ordinary user +properties are preserved; private stream lifecycle, WAL, token, receipt, +maintenance, correction, rebind, and dead-letter authority is not transferred +by ordinary export. The recorded v17↔v18 binary result remains historical; a +genuine v18↔v19 adjacent-binary cell now proves both refusals, ordinary rebuild +fidelity, and compatibility with final-v18 receipt-v1 retirement exports. + +The preceding F3d development strand's v15→v16 transition remains historical: +its final v15 source build is +`84f3af758947970d16040a987cb1d6ea0f0931e8`, and its genuine v15↔v16 refusal/ +rebuild cell remains on demand. + +See [the upgrade guide](../user/operations/upgrade.md) for the complete +export/init/load procedure. diff --git a/docs/user/operations/upgrade.md b/docs/user/operations/upgrade.md index ecfebfc7..618130bb 100644 --- a/docs/user/operations/upgrade.md +++ b/docs/user/operations/upgrade.md @@ -37,26 +37,28 @@ from that line (the latest is safest): | internal schema v3 | omnigraph 0.6.2–0.7.2 | the latest 0.7.x (e.g. 0.7.2) | | internal schema v4 | omnigraph 0.8.x | the latest 0.8.x (e.g. 0.8.1) | | internal schema v5–v8 | no published release (see below) | a source build at the matching commit | -| internal schema v9 | omnigraph 0.9.x | the latest 0.9.x | -| internal schema v10 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v11 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v12 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v13 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v14 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v15 | unreleased (earlier 0.10.0-dev source builds) | a source build at the matching commit | -| internal schema v16 | unreleased (earlier 0.10.0-dev source builds) | final v16 source build at merge `ac59c4f6d1d83acc8118c410c39de2bed91f9c15` | -| internal schema v17 | unreleased (earlier 0.10.0-dev source builds) | final v17 source build at merge `41a5990d53238d63d17e139859c66613f9c25867` | -| internal schema v18 | unreleased (earlier 0.10.0-dev source builds) | final v18 source build at merge `c7c81b186bed37989fe5ce591baf0965b5102648` | -| internal schema v19 | unreleased (current 0.10.0-dev source builds) | — current development format; a later pre-release strand may supersede it | +| internal schema v9 | no published release (0.9.0-dev source builds; the refusal label still reads `0.9.x`) | a source build at the matching commit | +| internal schema v10 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v11 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v12 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v13 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v14 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v15 | unreleased (earlier 0.9.0-dev source builds) | a source build at the matching commit | +| internal schema v16 | unreleased (earlier 0.9.0-dev source builds) | final v16 source build at merge `ac59c4f6d1d83acc8118c410c39de2bed91f9c15` | +| internal schema v17 | unreleased (earlier 0.9.0-dev source builds) | final v17 source build at merge `41a5990d53238d63d17e139859c66613f9c25867` | +| internal schema v18 | unreleased (earlier 0.9.0-dev source builds) | final v18 source build at merge `c7c81b186bed37989fe5ce591baf0965b5102648` | +| internal schema v19 | unreleased (current 0.9.0-dev source builds) | — current development format; a later pre-release strand may supersede it | **Stamps v5–v8 never shipped.** The storage format advanced five times inside the single 0.8.1 → 0.9.0 development window, so the only graphs carrying those stamps came from source builds off `main`; no published binary reads them and the refusal message names them `0.9.0-dev`. If you have one, export it with a build of the commit that created it, then load into a fresh current-format -graph. A released binary only ever wrote v4 (0.8.x) or v9 (0.9.x); v10–v19 are -pre-release formats written by matching 0.10.0-dev source builds. The final -0.10.0 format may use a later stamp. +graph. A released binary only ever wrote v4 (0.8.x) or earlier stamps: the +0.9.0 release was prepared at v9 but never published, so v9–v19 are all +pre-release formats written by matching 0.9.0-dev source builds (v9's refusal +label still reads `0.9.x` from that expectation). The final 0.9.0 format may +use a later stamp. You can also check versions before you hit a refusal: @@ -487,8 +489,9 @@ retirement records provenance and freezes the source before row-only export to a fresh graph. None of that later F5 behavior is active in v10. A v9 graph must use the standard rebuild recipe above: quiesce writers, export -with the latest 0.9.x binary, initialize a **different** root with the v10 -(0.10.0-dev source build) binary, load the export, and verify the v10 stamp +with a final-v9 source build (no 0.9.x binary was ever published), initialize +a **different** root with the v10 +(0.9.0-dev source build) binary, load the export, and verify the v10 stamp plus row/vector/blob fidelity before cutover. Keep the v9 root unchanged through the rollback window. A v10 binary refuses v9, and a v9 binary refuses v10. @@ -510,7 +513,8 @@ correctness foundations; v9 does not by itself expose a public streaming API. A v8 graph must use the standard rebuild recipe above. Because v8 never shipped in a release, this affects only source builds off `main` during 0.9.0 development: quiesce every v8 writer, export with a build of the commit that -created the graph, initialize a **different** root with the 0.9.x binary, load +created the graph, initialize a **different** root with a final-v9 source +build, load the export, and verify the v9 stamp plus row/vector/blob fidelity before cutover. Keep the v8 root unchanged through the rollback window. A v9 binary refuses v8, and a v8 binary refuses v9.