From e60262f5f6b48322c6254878e483cf4f0b35c190 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 4 Oct 2026 01:11:30 -0700 Subject: [PATCH 1/5] fleet: check native source reader successor policy --- crates/cellule-host/docs/README.md | 1 + crates/cellule-host/docs/lifecycle.md | 20 +- crates/cellule-host/docs/source-readers.md | 84 ++++ crates/cellule-host/minion/README.md | 13 + .../reader_tests/evacuation/fixture.rs | 30 +- .../scenario/reader_tests/evacuation/mod.rs | 1 + .../reader_tests/evacuation/source/fixture.rs | 175 ++++++++ .../reader_tests/evacuation/source/mod.rs | 102 +++++ .../evacuation/source/observation.rs | 163 ++++++++ .../reader_tests/evacuation/source/races.rs | 240 +++++++++++ .../reader_tests/evacuation/source/tests.rs | 325 +++++++++++++++ .../reader_tests/evacuation/transport.rs | 25 +- .../writers/successors/collection.rs | 67 +-- .../src/fleet/maintenance_policies/mod.rs | 44 +- crates/cellule-host/src/fleet/mod.rs | 4 +- .../cellule-host/src/fleet/native_writer.rs | 76 ++++ .../src/fleet/reader_evacuation/current.rs | 179 +++++++++ .../src/fleet/reader_evacuation/mod.rs | 6 + .../reader_evacuation/source/collection.rs | 380 ++++++++++++++++++ .../fleet/reader_evacuation/source/digest.rs | 89 ++++ .../src/fleet/reader_evacuation/source/mod.rs | 151 +++++++ .../fleet/reader_evacuation/verification.rs | 163 ++------ .../reconciler/observation/evacuations.rs | 2 +- .../observation/maintenance_policies.rs | 4 +- .../src/fleet/reconciler/observation/mod.rs | 9 +- .../observation/original_writers.rs | 76 ++-- .../reconciler/observation/source_readers.rs | 103 +++++ docs/fleet-operations-plan.md | 9 +- docs/fleet-operations-progress.md | 73 ++++ 29 files changed, 2358 insertions(+), 256 deletions(-) create mode 100644 crates/cellule-host/docs/source-readers.md create mode 100644 crates/cellule-host/minion/scenario/reader_tests/evacuation/source/fixture.rs create mode 100644 crates/cellule-host/minion/scenario/reader_tests/evacuation/source/mod.rs create mode 100644 crates/cellule-host/minion/scenario/reader_tests/evacuation/source/observation.rs create mode 100644 crates/cellule-host/minion/scenario/reader_tests/evacuation/source/races.rs create mode 100644 crates/cellule-host/minion/scenario/reader_tests/evacuation/source/tests.rs create mode 100644 crates/cellule-host/src/fleet/native_writer.rs create mode 100644 crates/cellule-host/src/fleet/reader_evacuation/current.rs create mode 100644 crates/cellule-host/src/fleet/reader_evacuation/source/collection.rs create mode 100644 crates/cellule-host/src/fleet/reader_evacuation/source/digest.rs create mode 100644 crates/cellule-host/src/fleet/reader_evacuation/source/mod.rs create mode 100644 crates/cellule-host/src/fleet/reconciler/observation/source_readers.rs diff --git a/crates/cellule-host/docs/README.md b/crates/cellule-host/docs/README.md index e04ec68f..141a9e9a 100644 --- a/crates/cellule-host/docs/README.md +++ b/crates/cellule-host/docs/README.md @@ -4,6 +4,7 @@ | --- | --- | | [Lifecycle](lifecycle.md) | Start and stop one node safely. | | [Read replicas](read-replicas.md) | Install and supervise immutable views. | +| [Source reader succession](source-readers.md) | Check exact native retirement against the current writer and reader policy. | | [Original failed-boot writers](original-writers.md) | Retain complete original ownership metadata before dependent effects. | | [Fleet journal example](../minion/README.md) | Run the durable local journal foundation and inspect its integration limits. | | [Crate entry](../README.md) | Ownership and test command. | diff --git a/crates/cellule-host/docs/lifecycle.md b/crates/cellule-host/docs/lifecycle.md index ebbc7c00..d03fdfea 100644 --- a/crates/cellule-host/docs/lifecycle.md +++ b/crates/cellule-host/docs/lifecycle.md @@ -968,7 +968,7 @@ ensembles. Signed replacement boots must match the existing producer-specific identity; a changed writer row or follower epoch/ensemble invalidates the input. Attachment order with role coverage, writer successors or failed-boot closure does not change these checks. The reconciler repeats the roster comparison. -Planner digest v11 binds collection presence, canonical record order, full +Planner digest v12 binds collection presence, canonical record order, full barriers, fresh intervals, current authority, reader prefixes and retained native follower inventories. Persisted history, enrollment and transport codecs are unchanged. @@ -993,7 +993,7 @@ fresh complete roster, and reconfirms the full journal snapshot. Retiring rows, extending a deadline or adopting a successor boot preserves the first set. `FleetObservation::with_maintenance_enrollments` retains this evidence inside the outer interval and compares the exact head, registry and roster with other checks, -in either attachment order. Planner digest v11 binds its presence and full digest. +in either attachment order. Planner digest v12 binds its presence and full digest. A missing manifest or page remains unknown. A committed empty manifest proves only that the original unresolved role set was empty at the phase transaction. @@ -1046,6 +1046,15 @@ Pending/Established work, source succession and unknown nonexecution stay in the full matcher even when they are ineligible for donor lookup. This read-only path starts no role effects or tasks and grants no settlement/finalization rights. +### Confirm source reader succession + +Use the [source reader collector](source-readers.md) after actual writer handoff +and exact native reader joining. It consumes the same complete original/current +request set and checks current writer origin, policy and native ready readers. +Retain `with_source_reader_policies` before full maintenance policy matching. +Missing native/provider evidence stays unknown; complete accepted-work/process +coverage and finalization remain separate. + ### Confirm original nonexecution A terminal request without establishment history remains unknown. An independent @@ -1095,7 +1104,7 @@ async fn attach_nonexecution( The observation's outer interval must include both collections. Attach all native policy and nonexecution checks before matching; later input changes refuse. -Checked nonexecution is reported separately and binds planner digest v11. It +Checked nonexecution is reported separately and binds planner digest v12. It cannot upgrade incomplete node observation, prove installed-role redundancy, join other accepted work, or grant SettleRoles/Finalize rights. @@ -1127,7 +1136,8 @@ supply complete request coverage. | `Reader` / `Follower` | Exact current role and fresh native replacement policy checked. | | `Pending` / `Established` | Accepted native outcome or installed role still unresolved. | | `MissingPolicy` | Retired donor lacks current replacement policy evidence. | -| `SourceSuccessor` | Source-side or failed-owner policy still needs separate evidence. | +| `SourceReader` | Exact original native source reader joined, final root verified against an actual managed successor, current ready-reader policy checked. | +| `SourceSuccessor` | Remaining source-side or failed-owner policy still needs separate evidence. | | `Nonexecution` | Independently confirmed original work joining and exclusion; no role effect committed. | | `UnprovenNonexecution` | Closed unknown acceptance lacks checked nonexecution or policy evidence. | @@ -1137,7 +1147,7 @@ empty original manifest and no currently unresolved related requests. `FleetReconcileReport::maintenance_policy` supplies bounded advisory counts with their own head revision, registry and original interval. `None` means unknown. Later allocations cannot restamp these counts to the report's newer snapshot. -Planner digest v11 binds the coverage digest and source inputs; changing the +Planner digest v12 binds the coverage digest and source inputs; changing the attached policy collections after matching is refused. `is_complete()` describes only enumerated request-policy coverage. It does not diff --git a/crates/cellule-host/docs/source-readers.md b/crates/cellule-host/docs/source-readers.md new file mode 100644 index 00000000..a1e41bea --- /dev/null +++ b/crates/cellule-host/docs/source-readers.md @@ -0,0 +1,84 @@ +# Source reader succession + +A reader can live on another node while its original writer enters maintenance. +The host checks that exact reader's native retirement against the actual writer +successor and current reader policy before marking its policy obligation checked. +This uses the existing reader manager, actor FIFO, origin verifier and peer path. + +| Input | Required evidence | +| --- | --- | +| Original request set | Every immutable original page and the full current roster. | +| Reader retirement | The retained opaque `ReaderEnrollmentRetirement` from exact native removal. | +| Final prefix | The last installed root after accepted reader work joined, including later refreshes. | +| Writer successor | An actual managed native writer on an Active physical node outside maintenance, at a strictly newer epoch. | +| Origin | Canonical derivation of the exact final root and availability of all current dependencies. | +| Reader policy | Exact current revision, incarnation and desired count; absent differs from explicit zero. | +| Replacements | Canonically selected Active boots, exact Established requests and actual ready native prefixes. | +| Capture | One full journal/roster barrier, absolute deadline and monotonic interval of at most 30 seconds. | + +## Retain and collect + +The embedding application authenticates both endpoints and canonical backend +mappings. Its existing accepted-work owner retains each native retirement and +returns the same `Arc` on repeated lookup. `None` means +unknown. A terminal enrollment row cannot reconstruct a lost native join witness. + +After writer handoff, join the original reader with +`ReadReplicaManager::remove_enrolled`. Ordinary recruitment installs current +replacements. The same still-Active receiver may hold a new request under the +successor; that request cannot replace the original retirement witness. + +```rust +use cellule_host::fleet::{ + FleetJournal, FleetMaintenanceEnrollments, FleetReaderEvacuationVerifier, + FleetRoster, FleetSourceReaderPolicies, FleetSourceReaderSuccessors, +}; +use cellule_runtime::Result; +use tokio::time::Instant; + +async fn source_policies( + verifier: &FleetReaderEvacuationVerifier, + journal: &dyn FleetJournal, + original: &FleetMaintenanceEnrollments, + roster: &FleetRoster, + successors: &dyn FleetSourceReaderSuccessors, + deadline: Instant, + clock: impl FnMut() -> Result, +) -> Result { + verifier + .collect_source_readers(journal, original, roster, successors, deadline, clock) + .await +} +``` + +The collector visits the complete original/current request set. It verifies +current native writers around origin and reader probes, then globally rechecks +policy, selected boots, native prefixes, provider presence/identity and the full +roster. Changed inputs refuse the capture. Native origin I/O uses the runtime's +existing reservations; applications account bounded metadata copies. + +Build the outer `FleetObservation` after all native collection finishes. Retain +`with_source_reader_policies` and `with_maintenance_enrollments` before +`check_maintenance_policies`. Both attachment orders validate the same original +capture, full head, registry, roster and interval. Signed successor/replacement +boots and any captured writer row must match the checked native evidence. +Planner identity v12 binds these inputs. Missing evidence stays `SourceSuccessor`; +a checked exact source reader is `SourceReader`. + +## Scope and runnable evidence + +The canonical minion suite exercises managed writer handoff, refresh past the +original opening, exact closure, a new reader request on the same receiver, +independent SQLite journal clients and public reconciliation. It also exercises +missing/changed provider evidence, foreign origins, clock/deadline/cancellation +and policy, journal or writer changes behind actual signed native replies. + +```sh +cargo test -p cellule-host --example fleet_operations --all-features --locked \ + scenario::reader_tests::evacuation::source:: -- --test-threads=2 +``` + +These local checks do not upgrade an incomplete observation or authorize +SettleRoles/Finalize. Durable process/provider retention, original writer and +failed-owner proof, follower succession, native/external accepted work and full +maintenance remain required by the [fleet plan](../../../docs/fleet-operations-plan.md). diff --git a/crates/cellule-host/minion/README.md b/crates/cellule-host/minion/README.md index 9cb6fa79..3b738d9b 100644 --- a/crates/cellule-host/minion/README.md +++ b/crates/cellule-host/minion/README.md @@ -578,6 +578,19 @@ Complete authenticated lookup, source/failed-owner successor policy, checked nonexecution, accepted-work joining and SettleRoles/Finalize remain delivery requirements. See the [host matching recipe](../docs/lifecycle.md#match-every-required-maintenance-policy). +## Native source reader succession + +The [source reader recipe](../docs/source-readers.md) composes exact retained +native retirement with an actual managed writer successor and current ready +reader policy. It consumes the complete original/current request set and is +retained through the public observer and matcher. Missing evidence remains an +explicit source obligation. A checked local request does not finish maintenance. + +```sh +cargo test -p cellule-host --example fleet_operations --all-features --locked \ + scenario::reader_tests::evacuation::source:: -- --test-threads=2 +``` + ## Original nonexecution confirmation The actual reader/follower suites consume `FleetMaintenanceNonexecution` through diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/fixture.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/fixture.rs index 0dcbf759..55005262 100644 --- a/crates/cellule-host/minion/scenario/reader_tests/evacuation/fixture.rs +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/fixture.rs @@ -2,6 +2,10 @@ use super::*; impl Fixture { pub(super) async fn new() -> Self { + Self::for_maintenance(1).await + } + + pub(super) async fn for_maintenance(maintenance_node: usize) -> Self { let root = tempfile::tempdir().unwrap(); let journal = Arc::new( SqliteJournal::open( @@ -82,7 +86,11 @@ impl Fixture { .unwrap() .with_native_memory_limit(128 << 20) .unwrap(), - 16 << 20, + if maintenance_node == 0 { + 64 << 20 + } else { + 16 << 20 + }, ) .with_replica_host( Host::default().with_local_disk_budget(DiskBudget::new(8 << 30)), @@ -242,8 +250,8 @@ impl Fixture { MaintenanceOperation::new( OperationId::from_bytes([80; 16]).unwrap(), Digest::from_bytes([81; 32]), - node_id(1), - session(1), + node_id(maintenance_node), + session(maintenance_node), 2, now, now + 60_000, @@ -264,15 +272,23 @@ impl Fixture { .unwrap(); } let operation = snapshot.head().maintenance().unwrap().clone(); - boots[1] - .refresh_capacity(1, journal.as_ref(), Instant::now() + Duration::from_secs(3)) + boots[maintenance_node] + .refresh_capacity( + maintenance_node, + journal.as_ref(), + Instant::now() + Duration::from_secs(3), + ) .await .unwrap(); assert_eq!( - nodes[1].runtime().node_admission().mode().unwrap(), + nodes[maintenance_node] + .runtime() + .node_admission() + .mode() + .unwrap(), cellule_runtime::node::NodeMode::Draining ); - assert!(nodes[1].is_management_ready()); + assert!(nodes[maintenance_node].is_management_ready()); Self { layout, root, diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/mod.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/mod.rs index 4446461a..9baf6b47 100644 --- a/crates/cellule-host/minion/scenario/reader_tests/evacuation/mod.rs +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/mod.rs @@ -12,6 +12,7 @@ use ed25519_dalek::SigningKey; mod fixture; mod inventory_tests; mod persisted; +mod source; mod tests; mod transport; diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/fixture.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/fixture.rs new file mode 100644 index 00000000..00677241 --- /dev/null +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/fixture.rs @@ -0,0 +1,175 @@ +use super::*; + +impl SourceFixture { + pub(super) async fn new() -> Self { + let mut fixture = Fixture::for_maintenance(0).await; + assert_eq!(fixture.original.spec().source.unwrap().node, node_id(0)); + let opened = fixture.reader.lifecycle_observation().await.root(); + let now = clock().unwrap(); + fixture + .handle + .execute( + MutationIdentity { + request_id: RequestId::from_bytes([222; 16]), + issued_at_ms: now, + expires_at_ms: now + 30_000, + }, + Digest::from_bytes([223; 32]), + now, + 64, + 64, + |tx| { + tx.execute("UPDATE counter SET value = 17", [])?; + Ok(HandlerOutcome::Success(Vec::new())) + }, + ) + .await + .unwrap(); + fixture + .reader + .refresh(&fixture.root.path().join("source-final-reader.sqlite")) + .await + .unwrap(); + assert_ne!(fixture.reader.lifecycle_observation().await.root(), opened); + // Join the exact busy writer through its canonical drain, then restore + // its published root on another fully managed native boot. + fixture.handle.drain().await.unwrap(); + let authority = CellAuthority::new(fixture.layout.clone()); + let idle = authority + .load(fixture.target.cell_id()) + .await + .unwrap() + .unwrap(); + let catalog = CellCatalog::new(fixture.layout.clone(), fixture.target.tenant()) + .lookup(fixture.target.cell_id()) + .await + .unwrap() + .unwrap(); + let replica = CellReplica::new( + fixture.layout.clone(), + *fixture.target.cell_id().as_bytes(), + *fixture.description.incarnation.as_bytes(), + Limits { + max_database_bytes: 64 << 20, + max_capture_bytes: 16 << 20, + ..Limits::default() + }, + ) + .unwrap(); + let successor = fixture.nodes[2] + .runtime() + .acquire_idle_restored( + catalog.clone(), + replica.clone(), + authority.clone(), + idle, + fixture.root.path().join("source-successor.sqlite"), + owner(2), + ) + .await + .unwrap(); + fixture.boots[2] + .refresh_capacity(2, fixture.journal.as_ref(), deadline()) + .await + .unwrap(); + let retirement = Arc::new( + fixture.managers[1] + .remove_enrolled(&fixture.original, deadline()) + .await + .unwrap(), + ); + assert_eq!( + retirement.root(), + fixture.reader.lifecycle_observation().await.root() + ); + assert!( + fixture + .reader + .lifecycle_observation() + .await + .locally_joined() + ); + // Activation is authorized by the current writer, not the departed + // source. Pin that actual successor boot's signing key in the same + // authenticated dispatch path; existing donor fixtures retain origin 0. + fixture.transport = Arc::new(transport::NativePeers::for_origin( + &fixture.nodes, + &fixture.managers, + &fixture.layout, + fixture.directory.clone(), + 2, + )); + fixture.peer = ReplicaPeerClient::new( + fixture.nodes[2].application().registry(), + Arc::new(PeerSigner::new( + session(2), + fixture.nodes[2].application().registry().release_digest(), + SigningKey::from_bytes(&[3; 32]), + )), + PeerPrincipal { + issuer: "managed-owner".into(), + subject: "live-owner".into(), + actions: vec!["replica-maintenance".into()], + }, + fixture.transport.clone(), + ); + // The same Active receiver can install a new request under the new + // writer. Its new acceptance must never substitute the original join. + fixture.boots[1] + .refresh_capacity(1, fixture.journal.as_ref(), deadline()) + .await + .unwrap(); + let ad = fixture + .directory + .load(session(1), clock().unwrap()) + .await + .unwrap() + .unwrap(); + fixture + .peer + .activate( + &fixture.target, + &fixture.directory, + ad.advertisement().clone(), + fixture.description, + ) + .await + .unwrap(); + let new_reader = fixture.managers[1] + .resolve(fixture.target.clone()) + .await + .unwrap(); + assert_eq!(new_reader.receipt().await, retirement.receipt()); + assert_ne!( + fixture.managers[1] + .enrollment_completion(fixture.target.cell_id()) + .await + .unwrap() + .unwrap() + .spec, + *retirement.original().spec() + ); + assert_eq!( + new_reader + .query::(Some(retirement.receipt()), 0) + .await + .unwrap() + .output, + 17 + ); + drop(new_reader); + let inputs = Arc::new(FleetSourceReaderInputs { + retirement, + node: node_id(2), + host: fixture.nodes[2].clone(), + catalog, + authority, + replica, + }); + Self { + fixture, + inputs, + successor, + } + } +} diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/mod.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/mod.rs new file mode 100644 index 00000000..65fbeb3d --- /dev/null +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/mod.rs @@ -0,0 +1,102 @@ +//! Actual managed writer handoff, exact native joining and current source policy. +use super::*; +use cellule_host::fleet::{ + FleetAdapterFuture, FleetJournalSnapshot, FleetMaintenanceEnrollments, FleetObservation, + FleetReaderEvacuationVerifier, FleetRoster, FleetSourceReaderInputs, FleetSourceReaderPolicies, + FleetSourceReaderSuccessors, +}; +use cellule_runtime::{fleet::operations::EnrollmentRole, read_policy::ReadPolicyStore}; +use std::sync::atomic::{AtomicUsize, Ordering}; + +mod fixture; +mod observation; +mod races; +mod tests; + +fn deadline() -> Instant { + Instant::now() + Duration::from_secs(5) +} +struct SourceFixture { + fixture: Fixture, + inputs: Arc, + successor: CellHandle, +} +struct Provider { + inputs: Option>, + later: Option>>, + calls: AtomicUsize, +} +impl Provider { + fn stable(inputs: Arc) -> Self { + Self { + inputs: Some(inputs), + later: None, + calls: AtomicUsize::new(0), + } + } +} +impl FleetSourceReaderSuccessors for Provider { + fn successor<'a>( + &'a self, + row: &'a EnrollmentRecord, + expected: &'a FleetJournalSnapshot, + ) -> FleetAdapterFuture<'a, Option>> { + Box::pin(async move { + assert_eq!(expected.head().scope(), scope()); + assert_eq!(row.spec().source.unwrap().node, node_id(0)); + let call = self.calls.fetch_add(1, Ordering::SeqCst); + Ok(if call > 0 { + self.later.clone().unwrap_or_else(|| self.inputs.clone()) + } else { + self.inputs.clone() + }) + }) + } +} +impl SourceFixture { + fn verifier(&self) -> FleetReaderEvacuationVerifier { + FleetReaderEvacuationVerifier::new( + self.fixture.directory.clone(), + CellAuthority::new(self.fixture.layout.clone()), + ReadPolicyStore::new(self.fixture.layout.clone()), + self.fixture.peer.clone(), + ) + } + async fn basis(&self) -> (FleetRoster, FleetMaintenanceEnrollments) { + let snapshot = self.fixture.journal.load_snapshot(scope()).await.unwrap(); + let roster = FleetRoster::collect(self.fixture.journal.as_ref(), &snapshot, deadline()) + .await + .unwrap(); + let original = FleetMaintenanceEnrollments::collect( + self.fixture.journal.as_ref(), + &roster, + deadline(), + clock, + ) + .await + .unwrap(); + assert_eq!(original.entries().count(), 1); + (roster, original) + } + async fn collect( + &self, + provider: &Provider, + ) -> cellule_runtime::Result { + let (roster, original) = self.basis().await; + self.verifier() + .collect_source_readers( + self.fixture.journal.as_ref(), + &original, + &roster, + provider, + deadline(), + clock, + ) + .await + } + async fn finish(self) { + drop(self.inputs); + drop(self.successor); + self.fixture.finish().await; + } +} diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/observation.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/observation.rs new file mode 100644 index 00000000..bf815f38 --- /dev/null +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/observation.rs @@ -0,0 +1,163 @@ +//! Public reconciliation retains the checked source request without completing roles. +use super::*; +use cellule_host::fleet::{FleetActionCompletion, FleetObserver, FleetReconciler, FleetTransport}; +use cellule_runtime::fleet::operations::{ + FleetAction, FleetInspectionObservation, FleetInspectionRequest, +}; +struct Observer { + journal: Arc, + verifier: FleetReaderEvacuationVerifier, + provider: Provider, + directory: NodeDirectory, + effects: AtomicUsize, +} +impl FleetObserver for Observer { + fn observe<'a>( + &'a self, + roster: &'a FleetRoster, + _: i64, + end: Instant, + ) -> FleetAdapterFuture<'a, FleetObservation> { + Box::pin(async move { + let started = clock()?; + let original = + FleetMaintenanceEnrollments::collect(self.journal.as_ref(), roster, end, clock) + .await?; + let policies = self + .verifier + .collect_source_readers( + self.journal.as_ref(), + &original, + roster, + &self.provider, + end, + clock, + ) + .await?; + let mut ads = Vec::new(); + for index in 0..3 { + ads.push( + self.directory + .load(session(index), clock()?) + .await? + .ok_or(Error::Fenced)? + .advertisement() + .clone(), + ); + } + Ok(FleetObservation::new( + scope(), + roster.snapshot().registry(), + roster.snapshot().registry().revision(), + started, + clock()?, + false, + ads, + Vec::new(), + )? + .with_source_reader_policies(policies)? + .with_maintenance_enrollments(original)? + .check_maintenance_policies(roster, clock()?)?) + }) + } +} +impl FleetTransport for Observer { + fn dispatch<'a>( + &'a self, + _: &'a FleetAction, + _: Instant, + ) -> FleetAdapterFuture<'a, Arc> { + self.effects.fetch_add(1, Ordering::SeqCst); + Box::pin(async { Err(invalid("unexpected source-reader action")) }) + } + fn inspect<'a>( + &'a self, + _: &'a FleetInspectionRequest, + _: Instant, + ) -> FleetAdapterFuture<'a, Arc> { + self.effects.fetch_add(1, Ordering::SeqCst); + Box::pin(async { Err(invalid("unexpected source-reader inspection")) }) + } +} +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_reconstructed_journal_and_public_reconciler_preserve_native_inputs() { + let source = SourceFixture::new().await; + let client = SqliteJournal::open( + source.fixture.root.path().join("evacuation.sqlite"), + scope(), + FleetProfile::default(), + clock().unwrap(), + ) + .await + .unwrap(); + let snapshot = client.load_snapshot(scope()).await.unwrap(); + client + .set_scheduling(snapshot.registry(), true) + .await + .unwrap(); + let snapshot = client.load_snapshot(scope()).await.unwrap(); + let roster = FleetRoster::collect(&client, &snapshot, deadline()) + .await + .unwrap(); + let original = FleetMaintenanceEnrollments::collect(&client, &roster, deadline(), clock) + .await + .unwrap(); + let provider = Provider::stable(source.inputs.clone()); + let policies = source + .verifier() + .collect_source_readers(&client, &original, &roster, &provider, deadline(), clock) + .await + .unwrap(); + assert_eq!(policies.checks().len(), 1); + drop((policies, provider, original, roster)); + client.close().await.unwrap(); + let observer = Arc::new(Observer { + journal: source.fixture.journal.clone(), + verifier: source.verifier(), + provider: Provider::stable(source.inputs.clone()), + directory: source.fixture.directory.clone(), + effects: AtomicUsize::new(0), + }); + let driver = FleetReconciler::new( + scope(), + SessionId::from_bytes([206; 16]), + FleetProfile::default(), + source.fixture.journal.clone(), + observer.clone(), + observer.clone(), + ) + .unwrap(); + let report = driver.reconcile_once(clock, deadline()).await.unwrap(); + let progress = report.maintenance_policy.unwrap(); + assert_eq!( + ( + progress.required, + progress.checked, + progress.source_successors + ), + (1, 1, 0) + ); + assert_eq!(progress.registry, report.snapshot.registry()); + assert_eq!(report.allocated, 0); + assert!( + report + .blockers + .contains(&cellule_runtime::fleet::operations::DrainBlocker::IncompleteObservation) + ); + assert_eq!(observer.effects.load(Ordering::SeqCst), 0); + assert_eq!( + source + .fixture + .journal + .load_snapshot(scope()) + .await + .unwrap() + .head() + .maintenance() + .unwrap() + .phase(), + cellule_runtime::fleet::operations::MaintenancePhase::Evacuating + ); + drop((driver, observer)); + source.finish().await; +} diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/races.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/races.rs new file mode 100644 index 00000000..f4b55d29 --- /dev/null +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/races.rs @@ -0,0 +1,240 @@ +//! Changes behind actual signed native replies cannot produce checked policy. +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_refuses_policy_head_and_writer_changes_behind_native_reply() { + for change in 0..3 { + let source = SourceFixture::new().await; + let provider = Provider::stable(source.inputs.clone()); + let (roster, original) = source.basis().await; + let verifier = source.verifier(); + let (entered, resume) = source.fixture.transport.pause_probe(2); + let mut work = Box::pin(verifier.collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + )); + tokio::select! { result = &mut work => panic!("unexpected source completion {}", result.is_ok()), result = entered => result.unwrap() } + match change { + 0 => { + source.fixture.managers[1] + .set_target(&source.fixture.target, 1, 0) + .await + .unwrap() + .unwrap(); + } + 1 => { + source + .fixture + .journal + .set_scheduling(roster.snapshot().registry(), true) + .await + .unwrap(); + } + _ => { + source.successor.drain().await.unwrap(); + } + } + resume.send(()).unwrap(); + assert!(work.await.is_err()); + assert_eq!( + source.fixture.original_row().await, + *source.inputs.retirement.retired() + ); + assert!( + source + .fixture + .reader + .lifecycle_observation() + .await + .locally_joined() + ); + drop(provider); + source.finish().await; + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_cancelled_probe_and_deadline_leave_original_joined_evidence_retained() +{ + let source = SourceFixture::new().await; + let provider = Provider::stable(source.inputs.clone()); + let (roster, original) = source.basis().await; + let verifier = source.verifier(); + let (entered, resume) = source.fixture.transport.pause_probe(1); + let mut work = Box::pin(verifier.collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + )); + tokio::select! { result = &mut work => panic!("unexpected source completion {}", result.is_ok()), result = entered => result.unwrap() } + drop(work); + let _ = resume.send(()); + assert!(matches!( + verifier + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + Instant::now(), + clock + ) + .await, + Err(Error::Deadline) + )); + assert_eq!( + source.fixture.original_row().await, + *source.inputs.retirement.retired() + ); + assert_eq!(source.collect(&provider).await.unwrap().checks().len(), 1); + drop(provider); + source.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_whole_capture_refuses_backward_negative_and_overlong_clocks() { + let source = SourceFixture::new().await; + let provider = Provider::stable(source.inputs.clone()); + let (roster, original) = source.basis().await; + for drift in [-1, 30_001] { + let started = clock().unwrap(); + let mut calls = 0; + let result = source + .verifier() + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + || { + calls += 1; + Ok(if calls == 1 { started } else { started + drift }) + }, + ) + .await; + assert!(matches!(result, Err(Error::Deadline))); + } + assert!(matches!( + source + .verifier() + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + || Ok(-1) + ) + .await, + Err(Error::Deadline) + )); + assert_eq!(provider.calls.load(Ordering::SeqCst), 0); + drop(provider); + source.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_cannot_replace_exact_origin_with_equal_counters_from_another_backend() +{ + let source = SourceFixture::new().await; + let empty = CellStorageLayout::new( + Store::new(Arc::new(InMemory::new())), + ObjectPath::from("missing-origin"), + [3; 16], + ); + let replica = CellReplica::new( + empty, + *source.fixture.target.cell_id().as_bytes(), + *source.fixture.description.incarnation.as_bytes(), + Limits::default(), + ) + .unwrap(); + let inputs = Arc::new(FleetSourceReaderInputs { + retirement: source.inputs.retirement.clone(), + node: node_id(2), + host: source.inputs.host.clone(), + catalog: source.inputs.catalog.clone(), + authority: source.inputs.authority.clone(), + replica, + }); + let provider = Provider::stable(inputs); + assert!(source.collect(&provider).await.is_err()); + assert_eq!(provider.calls.load(Ordering::SeqCst), 1); + assert_eq!( + source.fixture.original_row().await, + *source.inputs.retirement.retired() + ); + drop(provider); + source.finish().await; +} + +struct FailedLookup { + inputs: Arc, + fail: usize, + calls: AtomicUsize, +} +impl FleetSourceReaderSuccessors for FailedLookup { + fn successor<'a>( + &'a self, + _: &'a EnrollmentRecord, + _: &'a FleetJournalSnapshot, + ) -> FleetAdapterFuture<'a, Option>> { + Box::pin(async move { + let call = self.calls.fetch_add(1, Ordering::SeqCst) + 1; + if call == self.fail { + return Err( + Box::new(std::io::Error::other("original successor lookup failure")) + as Box, + ); + } + Ok(Some(self.inputs.clone())) + }) + } +} +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_preserves_initial_and_global_lookup_error_sources() { + let source = SourceFixture::new().await; + for fail in [1, 2] { + let provider = FailedLookup { + inputs: source.inputs.clone(), + fail, + calls: AtomicUsize::new(0), + }; + let (roster, original) = source.basis().await; + let result = source + .verifier() + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + ) + .await; + let Err(Error::Facility { + name, + source: error, + }) = result + else { + panic!("source error lost") + }; + assert_eq!(name, "fleet-source-reader-successors"); + let original_error = error.downcast_ref::().unwrap(); + assert_eq!(original_error.kind(), std::io::ErrorKind::Other); + assert_eq!( + original_error.to_string(), + "original successor lookup failure" + ); + assert_eq!(provider.calls.load(Ordering::SeqCst), fail); + } + source.finish().await; +} diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/tests.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/tests.rs new file mode 100644 index 00000000..d353bd4b --- /dev/null +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/tests.rs @@ -0,0 +1,325 @@ +use super::*; +use cellule_host::fleet::FleetMaintenancePolicyStatus; + +async fn ads(fixture: &Fixture) -> Vec { + let mut nodes = Vec::new(); + for index in 0..3 { + nodes.push( + fixture + .directory + .load(session(index), clock().unwrap()) + .await + .unwrap() + .unwrap() + .advertisement() + .clone(), + ); + } + nodes +} +async fn outer( + source: &SourceFixture, + registry: cellule_runtime::fleet::operations::RegistryVersion, + started: i64, +) -> FleetObservation { + FleetObservation::new( + scope(), + registry, + registry.revision(), + started, + clock().unwrap(), + false, + ads(&source.fixture).await, + Vec::new(), + ) + .unwrap() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_binds_actual_final_root_writer_and_new_request_in_both_orders() { + let source = SourceFixture::new().await; + let provider = Provider::stable(source.inputs.clone()); + for source_first in [false, true] { + let started = clock().unwrap(); + let (roster, original) = source.basis().await; + let policies = source + .verifier() + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + ) + .await + .unwrap(); + assert_eq!(policies.checks().len(), 1); + let check = &policies.checks()[0]; + assert_eq!(check.retirement().original(), &source.fixture.original); + assert_eq!(check.origin().prefix(), source.inputs.retirement.root()); + assert_eq!(check.origin().root(), source.inputs.retirement.root()); + assert_eq!(check.serving().owner().session, session(2)); + let EnrollmentRole::Reader { position, .. } = &source.fixture.original.spec().role else { + panic!("not reader") + }; + assert!(check.serving().position().epoch > position.epoch); + assert_eq!(check.desired_readers(), 1); + assert_eq!(check.policy_revision(), Some(1)); + assert_eq!(check.replacements().len(), 1); + assert_eq!(check.replacements()[0].node, node_id(1)); + assert_ne!( + check.replacements()[0].enrollment_key, + source.fixture.original.spec().key().unwrap() + ); + let observation = outer(&source, roster.snapshot().registry(), started).await; + let observation = if source_first { + observation + .with_source_reader_policies(policies) + .unwrap() + .with_maintenance_enrollments(original) + .unwrap() + } else { + observation + .with_maintenance_enrollments(original) + .unwrap() + .with_source_reader_policies(policies) + .unwrap() + } + .check_maintenance_policies(&roster, clock().unwrap()) + .unwrap(); + let coverage = observation.maintenance_policy_coverage().unwrap(); + assert!(coverage.is_complete()); + assert_eq!(coverage.progress().required, 1); + assert_eq!(coverage.progress().checked, 1); + assert_eq!(coverage.progress().source_successors, 0); + assert!(matches!( + coverage.obligations()[0].status(), + FleetMaintenancePolicyStatus::SourceReader(_) + )); + assert_eq!( + coverage.obligations()[0].original(), + Some(&source.fixture.original) + ); + assert_eq!( + coverage.obligations()[0].current(), + source.inputs.retirement.retired() + ); + } + assert_eq!(provider.calls.load(Ordering::SeqCst), 4); + drop(provider); + source.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_missing_native_capsule_keeps_complete_original_obligation_unknown() { + let source = SourceFixture::new().await; + let provider = Provider { + inputs: None, + later: None, + calls: AtomicUsize::new(0), + }; + let started = clock().unwrap(); + let (roster, original) = source.basis().await; + let policies = source + .verifier() + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + ) + .await + .unwrap(); + assert!(policies.checks().is_empty()); + assert_eq!(provider.calls.load(Ordering::SeqCst), 2); + let observation = outer(&source, roster.snapshot().registry(), started) + .await + .with_source_reader_policies(policies) + .unwrap() + .with_maintenance_enrollments(original) + .unwrap() + .check_maintenance_policies(&roster, clock().unwrap()) + .unwrap(); + let progress = observation + .maintenance_policy_coverage() + .unwrap() + .progress(); + assert_eq!( + ( + progress.required, + progress.checked, + progress.source_successors + ), + (1, 0, 1) + ); + drop(observation); + source.finish().await; +} + +fn mapping(source: &SourceFixture, node: usize) -> Arc { + Arc::new(FleetSourceReaderInputs { + retirement: source.inputs.retirement.clone(), + node: node_id(node), + host: source.inputs.host.clone(), + catalog: source.inputs.catalog.clone(), + authority: source.inputs.authority.clone(), + replica: source.inputs.replica.clone(), + }) +} +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_refuses_changed_presence_allocation_and_physical_mapping() { + let source = SourceFixture::new().await; + for (first, later) in [ + (Some(source.inputs.clone()), None), + (None, Some(source.inputs.clone())), + (Some(source.inputs.clone()), Some(mapping(&source, 2))), + ] { + let provider = Provider { + inputs: first, + later: Some(later), + calls: AtomicUsize::new(0), + }; + assert!(matches!( + source.collect(&provider).await, + Err(Error::Fenced) + )); + assert_eq!(provider.calls.load(Ordering::SeqCst), 2); + } + for node in [0, 1] { + let provider = Provider::stable(mapping(&source, node)); + assert!(matches!( + source.collect(&provider).await, + Err(Error::Fenced) + )); + } + source.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_refuses_duplicate_late_and_restamped_original_attachment() { + let source = SourceFixture::new().await; + let provider = Provider::stable(source.inputs.clone()); + for late in [false, true] { + let started = clock().unwrap(); + let (roster, original) = source.basis().await; + let verifier = source.verifier(); + let check = verifier + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + ) + .await + .unwrap(); + let other = verifier + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + ) + .await + .unwrap(); + let observation = outer(&source, roster.snapshot().registry(), started) + .await + .with_maintenance_enrollments(original) + .unwrap(); + let observation = if late { + observation + .check_maintenance_policies(&roster, clock().unwrap()) + .unwrap() + } else { + observation.with_source_reader_policies(check).unwrap() + }; + assert!(observation.with_source_reader_policies(other).is_err()); + } + let (roster, original) = source.basis().await; + let policies = source + .verifier() + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + ) + .await + .unwrap(); + let after = clock().unwrap().max(policies.interval().1 + 1); + let observation = FleetObservation::new( + scope(), + roster.snapshot().registry(), + roster.snapshot().registry().revision(), + after, + after, + false, + ads(&source.fixture).await, + Vec::new(), + ) + .unwrap(); + assert!(observation.with_source_reader_policies(policies).is_err()); + drop(provider); + source.finish().await; +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn source_reader_policy_outer_boots_and_native_writer_must_match_checked_evidence() { + let source = SourceFixture::new().await; + let provider = Provider::stable(source.inputs.clone()); + for change in 0..3 { + let started = clock().unwrap(); + let (roster, original) = source.basis().await; + let policies = source + .verifier() + .collect_source_readers( + source.fixture.journal.as_ref(), + &original, + &roster, + &provider, + deadline(), + clock, + ) + .await + .unwrap(); + let mut nodes = ads(&source.fixture).await; + let mut cells = Vec::new(); + match change { + 0 => nodes.retain(|node| node.node() != node_id(1)), + 1 => nodes.retain(|node| node.node() != node_id(2)), + _ => { + let mut observation = policies.checks()[0].serving().native().clone(); + observation.generation += 1; + cells.push(cellule_host::fleet::FleetOwnedCell { + node: node_id(2), + session: session(2), + observation, + }); + } + } + let outer = FleetObservation::new( + scope(), + roster.snapshot().registry(), + roster.snapshot().registry().revision(), + started, + clock().unwrap(), + false, + nodes, + cells, + ) + .unwrap() + .with_maintenance_enrollments(original) + .unwrap(); + assert!(outer.with_source_reader_policies(policies).is_err()); + } + drop(provider); + source.finish().await; +} diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/transport.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/transport.rs index a148b688..98099152 100644 --- a/crates/cellule-host/minion/scenario/reader_tests/evacuation/transport.rs +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/transport.rs @@ -16,6 +16,7 @@ use tokio::sync::oneshot; #[derive(Clone)] pub(super) struct NativePeers { directory: NodeDirectory, + origin: usize, dispatchers: Vec>, pub(super) probes: Arc, pause: Arc>>, @@ -25,10 +26,12 @@ struct Pause { captured: oneshot::Sender<()>, resume: oneshot::Receiver<()>, } -struct Authorizer; +struct Authorizer { + origin: SessionId, +} impl PeerAuthorizer for Authorizer { fn authorize(&self, request: &VerifiedPeerRequest) -> cellule_runtime::Result<()> { - if request.origin_session() != session(0) + if request.origin_session() != self.origin || request.principal().issuer != "managed-owner" || request.principal().subject != "live-owner" || !request @@ -50,6 +53,15 @@ impl NativePeers { managers: &[ReadReplicaManager], layout: &CellStorageLayout, directory: NodeDirectory, + ) -> Self { + Self::for_origin(nodes, managers, layout, directory, 0) + } + pub(super) fn for_origin( + nodes: &[Arc], + managers: &[ReadReplicaManager], + layout: &CellStorageLayout, + directory: NodeDirectory, + origin: usize, ) -> Self { let dispatchers = nodes .iter() @@ -64,7 +76,9 @@ impl NativePeers { layout.clone(), registry, )), - Arc::new(Authorizer), + Arc::new(Authorizer { + origin: session(origin), + }), ) .with_replica_control(Arc::new(manager.clone())) .with_replica_resolver(Arc::new(manager.clone())), @@ -73,6 +87,7 @@ impl NativePeers { .collect(); Self { directory, + origin, dispatchers, probes: Arc::new(AtomicUsize::new(0)), pause: Arc::new(Mutex::new(None)), @@ -127,7 +142,9 @@ impl PeerRoundTrip for NativePeers { .verify_peer_request( &request, Digest::from_bytes([30; 32]), - SigningKey::from_bytes(&[1; 32]).verifying_key().to_bytes(), + SigningKey::from_bytes(&[peers.origin as u8 + 1; 32]) + .verifying_key() + .to_bytes(), now, ) .await?; diff --git a/crates/cellule-host/src/fleet/failed_boot/writers/successors/collection.rs b/crates/cellule-host/src/fleet/failed_boot/writers/successors/collection.rs index ef1f7ad9..e759382c 100644 --- a/crates/cellule-host/src/fleet/failed_boot/writers/successors/collection.rs +++ b/crates/cellule-host/src/fleet/failed_boot/writers/successors/collection.rs @@ -106,56 +106,19 @@ async fn observe( directory: &NodeDirectory, now_ms: i64, ) -> Result { - if !inputs.host.is_management_ready() { - return Err(Error::CellDraining); - } - let observation = inputs - .host - .runtime() - .observe_serving( - &inputs.catalog, - &inputs.authority, - original.control.incarnation, - original.control.epoch, - ) - .await?; - if observation.owner().session != inputs.host.session { - return Err(Error::Fenced); - } - let boot = roster.boot(inputs.node, observation.owner().session)?; - let original_boot = inputs - .host - .fleet_startup - .lock() - .map_err(|_| Error::Control("CellNode fleet startup lock poisoned"))? - .as_ref() - .and_then(|startup| startup.boot.clone()) - .ok_or(Error::Fenced)?; - if original_boot.spec() != boot.enrollment().spec() - || original_boot.accepted_at_ms() != boot.enrollment().accepted_at_ms() - || original_boot.established_evidence() != boot.enrollment().established_evidence() - { - return Err(Error::Fenced); - } - let signed = directory - .load_if_live(observation.owner().session, now_ms) - .await? - .ok_or(Error::Fenced)?; - let ad = signed.advertisement(); - if ad.release() != inputs.host.application().registry().release_digest() - || ad.node() != inputs.node - || ad.endpoint() != observation.owner().endpoint - || !inputs.host.application().registry().supports_cell( - original.target.namespace(), - inputs.catalog.entry().role(), - observation.native().code, - observation.native().schema, - ) - { - return Err(Error::Fenced); - } - if !inputs.host.is_management_ready() { - return Err(Error::CellDraining); - } - Ok(observation) + crate::fleet::native_writer::observe( + &original.target, + original.control.incarnation, + original.control.epoch, + crate::fleet::native_writer::NativeWriter { + node: inputs.node, + host: &inputs.host, + catalog: &inputs.catalog, + authority: &inputs.authority, + }, + roster, + directory, + now_ms, + ) + .await } diff --git a/crates/cellule-host/src/fleet/maintenance_policies/mod.rs b/crates/cellule-host/src/fleet/maintenance_policies/mod.rs index fabd2c25..44ff2391 100644 --- a/crates/cellule-host/src/fleet/maintenance_policies/mod.rs +++ b/crates/cellule-host/src/fleet/maintenance_policies/mod.rs @@ -1,7 +1,7 @@ //! Complete original/current request matching for maintenance replacement policy. use super::{ FleetFollowerEvacuationCheck, FleetJournalSnapshot, FleetMaintenanceEnrollments, - FleetReaderEvacuationCheck, FleetRoster, operation, + FleetReaderEvacuationCheck, FleetRoster, FleetSourceReaderPolicies, operation, }; use cellule_runtime::fleet::operations::{EnrollmentRecord, EnrollmentStatus, RegistryVersion}; use cellule_runtime::identity::Digest; @@ -24,6 +24,8 @@ pub enum FleetMaintenancePolicyStatus { Follower(Digest), /// Independently joined original work whose role effect never committed. Nonexecution(Digest), + /// Exact native source reader joined with current successor and reader policy. + SourceReader(Digest), /// Original acceptance still has an unknown native outcome. Pending, /// An installed original role has not been retired with checked replacement policy. @@ -41,7 +43,7 @@ impl FleetMaintenancePolicyStatus { pub const fn is_checked(self) -> bool { matches!( self, - Self::Reader(_) | Self::Follower(_) | Self::Nonexecution(_) + Self::Reader(_) | Self::Follower(_) | Self::Nonexecution(_) | Self::SourceReader(_) ) } } @@ -119,6 +121,7 @@ impl FleetMaintenancePolicyCoverage { readers: &[FleetReaderEvacuationCheck], followers: &[FleetFollowerEvacuationCheck], nonexecution: Option<&FleetMaintenanceNonexecution>, + source_readers: Option<&FleetSourceReaderPolicies>, inputs: Digest, ) -> Result { let requests = requests::required(original, roster)?; @@ -140,6 +143,23 @@ impl FleetMaintenancePolicyCoverage { )?; } } + if let Some(checked) = source_readers { + if checked.snapshot() != roster.snapshot() + || checked.roster_digest() != roster.digest()? + || checked.original_digest() != original.digest()? + { + return Err(Error::Fenced); + } + let proof = checked.digest()?; + for check in checked.checks() { + witness_optional( + &mut witnesses, + check.retirement().retired(), + Some(enrollment_digest(check.retirement().original())?), + FleetMaintenancePolicyStatus::SourceReader(proof), + )?; + } + } for check in readers { witness_optional( &mut witnesses, @@ -182,9 +202,16 @@ impl FleetMaintenancePolicyCoverage { } FleetMaintenancePolicyStatus::Nonexecution(*proof) } - _ if row.spec().target.node != node => { - FleetMaintenancePolicyStatus::SourceSuccessor - } + _ if row.spec().target.node != node => match witnesses.get(&key) { + Some((retired, digest, FleetMaintenancePolicyStatus::SourceReader(proof))) => { + if row != *retired { + return Err(Error::Fenced); + } + requests::validate_original(accepted, *digest)?; + FleetMaintenancePolicyStatus::SourceReader(*proof) + } + _ => FleetMaintenancePolicyStatus::SourceSuccessor, + }, _ if row.established_evidence().is_none() => { FleetMaintenancePolicyStatus::UnprovenNonexecution } @@ -210,11 +237,12 @@ impl FleetMaintenancePolicyCoverage { .map(FleetReaderEvacuationCheck::interval) .chain(followers.iter().map(FleetFollowerEvacuationCheck::interval)) .chain(nonexecution.map(FleetMaintenanceNonexecution::interval)) + .chain(source_readers.map(FleetSourceReaderPolicies::interval)) .fold(original.interval(), |interval, next| { (interval.0.min(next.0), interval.1.max(next.1)) }); let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-maintenance-policy-coverage.v1\0"); + hash.update(b"cellule.fleet-maintenance-policy-coverage.v2\0"); hash.update(inputs.as_bytes()); hash.update(&(obligations.len() as u64).to_be_bytes()); for obligation in &obligations { @@ -232,6 +260,7 @@ impl FleetMaintenancePolicyCoverage { FleetMaintenancePolicyStatus::SourceSuccessor => (6, None), FleetMaintenancePolicyStatus::UnprovenNonexecution => (7, None), FleetMaintenancePolicyStatus::Nonexecution(proof) => (8, Some(proof)), + FleetMaintenancePolicyStatus::SourceReader(proof) => (9, Some(proof)), }; hash.update(&[tag]); if let Some(proof) = proof { @@ -291,7 +320,8 @@ impl FleetMaintenancePolicyCoverage { for obligation in &self.obligations { match obligation.status { FleetMaintenancePolicyStatus::Reader(_) - | FleetMaintenancePolicyStatus::Follower(_) => progress.checked += 1, + | FleetMaintenancePolicyStatus::Follower(_) + | FleetMaintenancePolicyStatus::SourceReader(_) => progress.checked += 1, FleetMaintenancePolicyStatus::Nonexecution(_) => { progress.checked += 1; progress.nonexecution += 1; diff --git a/crates/cellule-host/src/fleet/mod.rs b/crates/cellule-host/src/fleet/mod.rs index 2a7faa74..bed93550 100644 --- a/crates/cellule-host/src/fleet/mod.rs +++ b/crates/cellule-host/src/fleet/mod.rs @@ -30,12 +30,14 @@ pub use maintenance_policies::{ FleetMaintenancePolicyProgress, FleetMaintenancePolicyStatus, }; mod movement; +mod native_writer; mod reader_evacuation; mod reconciler; mod recovered; pub use reader_evacuation::{ FleetReaderEvacuationCandidate, FleetReaderEvacuationCheck, FleetReaderEvacuationJournal, - FleetReaderEvacuationPublication, FleetReaderEvacuationVerifier, + FleetReaderEvacuationPublication, FleetReaderEvacuationVerifier, FleetSourceReaderCheck, + FleetSourceReaderInputs, FleetSourceReaderPolicies, FleetSourceReaderSuccessors, }; mod references; mod roster; diff --git a/crates/cellule-host/src/fleet/native_writer.rs b/crates/cellule-host/src/fleet/native_writer.rs new file mode 100644 index 00000000..a3ffade4 --- /dev/null +++ b/crates/cellule-host/src/fleet/native_writer.rs @@ -0,0 +1,76 @@ +//! Shared authentication of an existing managed native writer. +use super::FleetRoster; +use crate::CellNode; +use cellule_runtime::{ + Error, Result, + cell::{actor::CellServingObservation, catalog::CatalogProof}, + control::authority::CellAuthority, + identity::{CellTarget, IncarnationId, NodeId}, + node::NodeDirectory, +}; + +pub(in crate::fleet) struct NativeWriter<'a> { + pub node: NodeId, + pub host: &'a CellNode, + pub catalog: &'a CatalogProof, + pub authority: &'a CellAuthority, +} + +#[allow(clippy::too_many_arguments)] +pub(in crate::fleet) async fn observe( + target: &CellTarget, + incarnation: IncarnationId, + minimum_epoch: u64, + inputs: NativeWriter<'_>, + roster: &FleetRoster, + directory: &NodeDirectory, + now_ms: i64, +) -> Result { + if !inputs.host.is_management_ready() { + return Err(Error::CellDraining); + } + let observation = inputs + .host + .runtime() + .observe_serving(inputs.catalog, inputs.authority, incarnation, minimum_epoch) + .await?; + if observation.owner().session != inputs.host.session { + return Err(Error::Fenced); + } + let boot = roster.boot(inputs.node, observation.owner().session)?; + let original_boot = inputs + .host + .fleet_startup + .lock() + .map_err(|_| Error::Control("CellNode fleet startup lock poisoned"))? + .as_ref() + .and_then(|startup| startup.boot.clone()) + .ok_or(Error::Fenced)?; + if original_boot.spec() != boot.enrollment().spec() + || original_boot.accepted_at_ms() != boot.enrollment().accepted_at_ms() + || original_boot.established_evidence() != boot.enrollment().established_evidence() + { + return Err(Error::Fenced); + } + let signed = directory + .load_if_live(observation.owner().session, now_ms) + .await? + .ok_or(Error::Fenced)?; + let ad = signed.advertisement(); + if ad.release() != inputs.host.application().registry().release_digest() + || ad.node() != inputs.node + || ad.endpoint() != observation.owner().endpoint + || !inputs.host.application().registry().supports_cell( + target.namespace(), + inputs.catalog.entry().role(), + observation.native().code, + observation.native().schema, + ) + { + return Err(Error::Fenced); + } + if !inputs.host.is_management_ready() { + return Err(Error::CellDraining); + } + Ok(observation) +} diff --git a/crates/cellule-host/src/fleet/reader_evacuation/current.rs b/crates/cellule-host/src/fleet/reader_evacuation/current.rs new file mode 100644 index 00000000..9b7c9249 --- /dev/null +++ b/crates/cellule-host/src/fleet/reader_evacuation/current.rs @@ -0,0 +1,179 @@ +//! Current policy/boot/native-prefix confirmation shared by donor and source checks. +use super::verification::FleetReaderEvacuationVerifier; +use crate::{ + fleet::{FleetJournal, FleetRoster}, + read_replicas::{ + ReaderReplacement, + maintenance::{boot_identity, same_authority, validate_replacement}, + }, +}; +use cellule_runtime::{ + Error, Result, + client::{CellDescription, Receipt}, + control::Control, + fleet::operations::ReaderReplacementWitness, + identity::CellTarget, +}; +use tokio::time::Instant; + +pub(super) struct CurrentReaderPolicy<'a> { + pub target: &'a CellTarget, + pub authority: &'a Control, + pub minimum: Receipt, + pub policy_revision: Option, + pub desired_readers: u16, + pub witnesses: &'a [ReaderReplacementWitness], + pub operation_deadline_ms: i64, +} +impl FleetReaderEvacuationVerifier { + #[allow(clippy::too_many_arguments)] + pub(super) async fn confirm_current( + &self, + journal: &dyn FleetJournal, + roster: &FleetRoster, + basis: CurrentReaderPolicy<'_>, + deadline: Instant, + clock: &mut impl FnMut() -> Result, + ) -> Result<(Control, Vec)> { + let target = basis.target; + let minimum = basis.minimum; + if basis.witnesses.len() != usize::from(basis.desired_readers) { + return Err(Error::Fenced); + } + let expected = CellDescription { + cell: minimum.cell, + incarnation: minimum.incarnation, + code: basis.authority.code, + schema: basis.authority.schema, + }; + let witnesses = basis.witnesses; + let mut replacements = Vec::with_capacity(witnesses.len()); + let mut authority = basis.authority.clone(); + for _ in 0..2 { + let now = clock()?; + if now >= basis.operation_deadline_ms { + return Err(Error::Deadline); + } + let observed = self + .authority + .load(minimum.cell) + .await? + .ok_or(Error::Fenced)?; + let fresh = observed.value(); + if !same_authority(&authority, fresh) { + return Err(Error::Fenced); + } + let policy = self.policy.load(minimum.cell).await?.map(|row| row.value()); + if policy.map(|policy| policy.revision()) != basis.policy_revision + || policy.map_or(0, |policy| policy.desired_readers()) != basis.desired_readers + || policy.is_some_and(|policy| { + policy.cell() != minimum.cell || policy.incarnation() != minimum.incarnation + }) + { + return Err(Error::Fenced); + } + let selected = self + .directory + .select_readers( + minimum.cell, + fresh.owner.as_ref().ok_or(Error::Fenced)?.session, + fresh.code, + witnesses.len(), + now, + 10_000, + ) + .await?; + if selected.len() != witnesses.len() { + return Err(Error::ReplicaUnavailable); + } + replacements.clear(); + for selected in selected { + let witness = witnesses + .iter() + .find(|entry| { + entry.node == selected.node() && entry.session == selected.session() + }) + .ok_or(Error::Fenced)?; + let node = self + .directory + .load(witness.session, clock()?) + .await? + .ok_or(Error::Fenced)?; + let node = node.advertisement(); + if node.node() != witness.node + || boot_identity(node)? != witness.boot_identity + || validate_replacement(roster, node, target, minimum)? + != (witness.enrollment_key, witness.enrollment_digest) + { + return Err(Error::Fenced); + } + let (receipt, ready) = self.peer.status(target, node.clone(), expected).await?; + if !ready + || receipt.cell != minimum.cell + || receipt.incarnation != minimum.incarnation + || receipt.commit_sequence + < minimum.commit_sequence.max(witness.commit_sequence) + { + return Err(Error::ReplicaUnavailable); + } + replacements.push(ReaderReplacement { + node: witness.node, + session: witness.session, + boot_identity: witness.boot_identity, + enrollment_key: witness.enrollment_key, + enrollment_digest: witness.enrollment_digest, + receipt, + }); + } + authority = fresh.clone(); + // Probes can suspend; no authority/policy change is hidden behind + // their replies or a stable native topology cursor. + let after = self + .authority + .load(minimum.cell) + .await? + .ok_or(Error::Fenced)?; + if !same_authority(&authority, after.value()) + || self.policy.load(minimum.cell).await?.map(|row| row.value()) != policy + { + return Err(Error::Fenced); + } + authority = after.value().clone(); + let selected = self + .directory + .select_readers( + minimum.cell, + authority.owner.as_ref().ok_or(Error::Fenced)?.session, + authority.code, + witnesses.len(), + clock()?, + 10_000, + ) + .await?; + if selected.len() != witnesses.len() { + return Err(Error::Fenced); + } + for node in selected { + let witness = witnesses + .iter() + .find(|entry| entry.node == node.node() && entry.session == node.session()) + .ok_or(Error::Fenced)?; + let fresh = self + .directory + .load(witness.session, clock()?) + .await? + .ok_or(Error::Fenced)?; + let fresh = fresh.advertisement(); + if fresh.node() != witness.node + || boot_identity(fresh)? != witness.boot_identity + || validate_replacement(roster, fresh, target, minimum)? + != (witness.enrollment_key, witness.enrollment_digest) + { + return Err(Error::Fenced); + } + } + roster.confirm(journal, deadline).await?; + } + Ok((authority, replacements)) + } +} diff --git a/crates/cellule-host/src/fleet/reader_evacuation/mod.rs b/crates/cellule-host/src/fleet/reader_evacuation/mod.rs index dd562df1..55a913fe 100644 --- a/crates/cellule-host/src/fleet/reader_evacuation/mod.rs +++ b/crates/cellule-host/src/fleet/reader_evacuation/mod.rs @@ -5,6 +5,12 @@ use cellule_runtime::{ identity::Digest, }; +mod current; +mod source; +pub use source::{ + FleetSourceReaderCheck, FleetSourceReaderInputs, FleetSourceReaderPolicies, + FleetSourceReaderSuccessors, +}; mod maintenance; mod publication; mod refresh; diff --git a/crates/cellule-host/src/fleet/reader_evacuation/source/collection.rs b/crates/cellule-host/src/fleet/reader_evacuation/source/collection.rs new file mode 100644 index 00000000..4b9dee68 --- /dev/null +++ b/crates/cellule-host/src/fleet/reader_evacuation/source/collection.rs @@ -0,0 +1,380 @@ +use super::super::{current::CurrentReaderPolicy, verification::bounded}; +use super::*; +use crate::fleet::{maintenance_policies::requests, native_writer}; +use crate::read_replicas::maintenance::{boot_identity, validate_replacement}; +use cellule_runtime::{ + Error, Result, + client::Receipt, + fleet::operations::{ + EnrollmentRole, EnrollmentStatus, MaintenancePhase, ReaderReplacementWitness, + }, + node::NodeMode, +}; +use tokio::time::Instant; + +impl FleetReaderEvacuationVerifier { + /// Composes every eligible retained source reader with its exact current + /// native successor, final-root derivation and current ready-reader policy. + /// One absolute deadline and monotonic 30-second interval bound all inputs + /// and global rechecks. Missing evidence is unknown; changed evidence refuses + /// the whole capture. No acquisition, retirement, recovery or task is started. + #[allow(clippy::too_many_arguments)] + pub async fn collect_source_readers( + &self, + journal: &dyn FleetJournal, + original: &FleetMaintenanceEnrollments, + roster: &FleetRoster, + successors: &dyn FleetSourceReaderSuccessors, + deadline: Instant, + mut clock: impl FnMut() -> Result, + ) -> Result { + let started = clock()?; + let deadline = deadline.min( + Instant::now() + .checked_add(std::time::Duration::from_secs(30)) + .ok_or(Error::Deadline)?, + ); + let mut last = started; + let mut now = || { + let next = clock()?; + if started < 0 || next < last || next - started > 30_000 { + return Err(Error::Deadline); + } + last = next; + Ok(next) + }; + bounded(deadline, async { + let required = requests::required(original, roster)?; + let operation = roster + .snapshot() + .head() + .maintenance() + .ok_or(Error::Fenced)?; + let intent = roster + .intents() + .iter() + .find(|intent| intent.node() == operation.node()) + .ok_or(Error::Fenced)?; + if self.directory.fleet() != roster.snapshot().head().scope().fleet + || !matches!( + operation.phase(), + MaintenancePhase::Evacuating | MaintenancePhase::Closing + ) + || intent.session() != operation.session() + || intent.revision() != operation.intent_revision() + || intent.mode() != NodeMode::Draining + || now()? >= operation.deadline_ms() + { + return Err(Error::Fenced); + } + roster.confirm(journal, deadline).await?; + let mut lookups = Vec::new(); + let mut checks = Vec::new(); + for request in required { + let row = request.current; + if row.status() != EnrollmentStatus::Retired + || row.established_evidence().is_none() + || row.spec().target.node == operation.node() + || row + .spec() + .source + .is_none_or(|source| source.node != operation.node()) + || !matches!(row.spec().role, EnrollmentRole::Reader { .. }) + { + continue; + } + let inputs = successors + .successor(row, roster.snapshot()) + .await + .map_err(provider)?; + now()?; + if let Some(inputs) = &inputs { + validate(row, request.original, inputs)?; + let (serving, writer_boot) = self.source_writer(inputs, roster, now()?).await?; + let root = serving.position().root.to_ltx( + inputs.retirement.receipt().cell, + serving.position().incarnation, + ); + let origin = inputs + .host + .runtime() + .verify_root_prefix( + &inputs.catalog, + &inputs.authority, + inputs.replica.clone(), + inputs.retirement.root(), + root, + 10_000, + ) + .await?; + let authority = self + .authority + .load(inputs.retirement.receipt().cell) + .await? + .ok_or(Error::Fenced)?; + let authority = authority.value().clone(); + if authority.owner.as_ref() != Some(serving.owner()) + || authority.epoch != serving.position().epoch + || authority.ltx_root() != Some(root) + || authority.code != serving.native().code + || authority.schema != serving.native().schema + { + return Err(Error::Fenced); + } + let policy = self + .policy + .load(authority.cell) + .await? + .map(|row| row.value()); + if policy.is_some_and(|policy| { + policy.cell() != authority.cell + || policy.incarnation() != authority.incarnation + }) { + return Err(Error::Fenced); + } + let desired = policy.map_or(0, |policy| policy.desired_readers()); + let minimum = Receipt { + cell: authority.cell, + incarnation: authority.incarnation, + commit_sequence: root + .commit_sequence + .max(inputs.retirement.receipt().commit_sequence), + }; + let EnrollmentRole::Reader { target, .. } = &row.spec().role else { + return Err(Error::Fenced); + }; + let selected = self + .directory + .select_readers( + authority.cell, + serving.owner().session, + authority.code, + usize::from(desired), + now()?, + 10_000, + ) + .await?; + if selected.len() != usize::from(desired) { + return Err(Error::ReplicaUnavailable); + } + let mut witnesses = Vec::with_capacity(selected.len()); + for node in selected { + if node.node() == operation.node() { + return Err(Error::Fenced); + } + let (key, digest) = validate_replacement(roster, &node, target, minimum)?; + witnesses.push(ReaderReplacementWitness { + node: node.node(), + session: node.session(), + boot_identity: boot_identity(&node)?, + enrollment_key: key, + enrollment_digest: digest, + commit_sequence: minimum.commit_sequence, + }); + } + let (authority, replacements) = self + .confirm_current( + journal, + roster, + CurrentReaderPolicy { + target, + authority: &authority, + minimum, + policy_revision: policy.map(|policy| policy.revision()), + desired_readers: desired, + witnesses: &witnesses, + operation_deadline_ms: operation.deadline_ms(), + }, + deadline, + &mut now, + ) + .await?; + let (after, after_boot) = self.source_writer(inputs, roster, now()?).await?; + if after_boot != writer_boot || !after.same_writer(&serving) { + return Err(Error::Fenced); + } + checks.push(FleetSourceReaderCheck { + inputs: inputs.clone(), + serving, + writer_boot, + origin, + authority, + policy_revision: policy.map(|policy| policy.revision()), + desired_readers: desired, + replacements, + }); + } + lookups.push((row, inputs)); + } + // Recheck presence as well as every retained native mapping. Missing + // evidence cannot arrive halfway through a supposedly complete set. + for (row, inputs) in &lookups { + let after = successors + .successor(row, roster.snapshot()) + .await + .map_err(provider)?; + match (inputs, &after) { + (None, None) => {} + (Some(before), Some(after)) if Arc::ptr_eq(before, after) => {} + _ => return Err(Error::Fenced), + } + now()?; + } + for check in &mut checks { + let inputs = &check.inputs; + let (current, current_boot) = self.source_writer(inputs, roster, now()?).await?; + if current_boot != check.writer_boot || !current.same_writer(&check.serving) { + return Err(Error::Fenced); + } + let EnrollmentRole::Reader { target, .. } = + &inputs.retirement.original().spec().role + else { + return Err(Error::Fenced); + }; + let witnesses = check + .replacements + .iter() + .map(|row| ReaderReplacementWitness { + node: row.node, + session: row.session, + boot_identity: row.boot_identity, + enrollment_key: row.enrollment_key, + enrollment_digest: row.enrollment_digest, + commit_sequence: row.receipt.commit_sequence, + }) + .collect::>(); + let (authority, replacements) = self + .confirm_current( + journal, + roster, + CurrentReaderPolicy { + target, + authority: &check.authority, + minimum: Receipt { + cell: check.authority.cell, + incarnation: check.authority.incarnation, + commit_sequence: check.origin.root().commit_sequence, + }, + policy_revision: check.policy_revision, + desired_readers: check.desired_readers, + witnesses: &witnesses, + operation_deadline_ms: operation.deadline_ms(), + }, + deadline, + &mut now, + ) + .await?; + let (after, after_boot) = self.source_writer(inputs, roster, now()?).await?; + if after_boot != check.writer_boot || !after.same_writer(&check.serving) { + return Err(Error::Fenced); + } + check.authority = authority; + check.replacements = replacements; + } + roster.confirm(journal, deadline).await?; + let finished = now()?; + if finished >= operation.deadline_ms() { + return Err(Error::Deadline); + } + Ok(FleetSourceReaderPolicies { + snapshot: roster.snapshot().clone(), + roster: roster.digest()?, + original: original.digest()?, + interval: (started, finished), + checks, + }) + }) + .await + } + + async fn source_writer( + &self, + inputs: &FleetSourceReaderInputs, + roster: &FleetRoster, + now: i64, + ) -> Result<(CellServingObservation, Digest)> { + let original = inputs.retirement.original(); + let EnrollmentRole::Reader { target, position } = &original.spec().role else { + return Err(Error::Fenced); + }; + if inputs.node + == roster + .snapshot() + .head() + .maintenance() + .ok_or(Error::Fenced)? + .node() + || roster + .boot(inputs.node, inputs.host.session)? + .intent() + .mode() + != NodeMode::Active + { + return Err(Error::Fenced); + } + let signed = self + .directory + .load_if_live(inputs.host.session, now) + .await? + .ok_or(Error::Fenced)?; + if !signed.advertisement().accepts_new_roles(now) { + return Err(Error::Fenced); + } + let serving = native_writer::observe( + target, + position.incarnation, + position.epoch, + native_writer::NativeWriter { + node: inputs.node, + host: &inputs.host, + catalog: &inputs.catalog, + authority: &inputs.authority, + }, + roster, + &self.directory, + now, + ) + .await?; + Ok((serving, boot_identity(signed.advertisement())?)) + } +} +fn validate( + row: &EnrollmentRecord, + accepted: Option<&EnrollmentRecord>, + inputs: &FleetSourceReaderInputs, +) -> Result<()> { + let original = inputs.retirement.original(); + let EnrollmentRole::Reader { target, position } = &original.spec().role else { + return Err(Error::Fenced); + }; + let digest = + Digest::from_bytes(*blake3::hash(&original.to_bytes().map_err(operation)?).as_bytes()); + requests::validate_original(accepted, Some(digest))?; + let entry = inputs.catalog.entry(); + let root = inputs.retirement.root(); + if inputs.retirement.retired() != row + || row.spec() != original.spec() + || row.accepted_at_ms() != original.accepted_at_ms() + || row.established_evidence() != original.established_evidence() + || entry.cell() != target.cell_id() + || entry.namespace() != target.namespace() + || entry.partition() != target.partition() + || inputs.replica.scope() + != ( + *target.cell_id().as_bytes(), + *position.incarnation.as_bytes(), + ) + || root.cell != *target.cell_id().as_bytes() + || root.incarnation != *position.incarnation.as_bytes() + || root.commit_sequence != inputs.retirement.receipt().commit_sequence + { + return Err(Error::Fenced); + } + Ok(()) +} +fn provider(source: Box) -> Error { + Error::Facility { + name: "fleet-source-reader-successors", + source, + } +} diff --git a/crates/cellule-host/src/fleet/reader_evacuation/source/digest.rs b/crates/cellule-host/src/fleet/reader_evacuation/source/digest.rs new file mode 100644 index 00000000..cf6ca09b --- /dev/null +++ b/crates/cellule-host/src/fleet/reader_evacuation/source/digest.rs @@ -0,0 +1,89 @@ +//! Local input identity; no persisted or signed wire contract is changed. +use super::*; +use cellule_runtime::Result; + +impl FleetSourceReaderPolicies { + /// Binds full original/current rows, exact roots, native successor identity, + /// policy, ready replacements and the original fresh collection barrier. + pub fn digest(&self) -> Result { + let mut hash = blake3::Hasher::new(); + hash.update(b"cellule.fleet-source-reader-policies.v1\0"); + for bytes in [ + self.snapshot.head().to_bytes().map_err(operation)?, + self.snapshot.registry().to_bytes().map_err(operation)?, + ] { + hash.update(&(bytes.len() as u64).to_be_bytes()); + hash.update(&bytes); + } + hash.update(self.roster.as_bytes()); + hash.update(self.original.as_bytes()); + for time in [self.interval.0, self.interval.1] { + hash.update(&time.to_be_bytes()); + } + hash.update(&(self.checks.len() as u64).to_be_bytes()); + for check in &self.checks { + let retirement = check.retirement(); + for row in [retirement.original(), retirement.retired()] { + let bytes = row.to_bytes().map_err(operation)?; + hash.update(&(bytes.len() as u64).to_be_bytes()); + hash.update(&bytes); + } + for time in [retirement.interval().0, retirement.interval().1] { + hash.update(&time.to_be_bytes()); + } + root(&mut hash, retirement.root()); + root(&mut hash, check.origin.prefix()); + root(&mut hash, check.origin.root()); + hash.update(&(check.origin.inspected_roots() as u64).to_be_bytes()); + hash.update(&(check.origin.dependency_count() as u64).to_be_bytes()); + hash.update(check.inputs.node.as_bytes()); + hash.update(check.writer_boot.as_bytes()); + hash.update( + check + .inputs + .host + .application() + .registry() + .release_digest() + .as_bytes(), + ); + let serving = &check.serving; + hash.update(serving.owner().session.as_bytes()); + hash.update(&(serving.owner().endpoint.len() as u64).to_be_bytes()); + hash.update(serving.owner().endpoint.as_bytes()); + hash.update(&serving.position().epoch.to_be_bytes()); + hash.update(&serving.native().generation.to_be_bytes()); + hash.update(serving.native().code.as_bytes()); + hash.update(&serving.native().schema.to_be_bytes()); + hash.update(&[u8::from(check.policy_revision.is_some())]); + if let Some(revision) = check.policy_revision { + hash.update(&revision.to_be_bytes()); + } + hash.update(&check.desired_readers.to_be_bytes()); + hash.update(&(check.replacements.len() as u64).to_be_bytes()); + for row in &check.replacements { + hash.update(row.node.as_bytes()); + hash.update(row.session.as_bytes()); + hash.update(row.boot_identity.as_bytes()); + hash.update(row.enrollment_key.as_bytes()); + hash.update(row.enrollment_digest.as_bytes()); + hash.update(row.receipt.cell.as_bytes()); + hash.update(row.receipt.incarnation.as_bytes()); + hash.update(&row.receipt.commit_sequence.to_be_bytes()); + } + } + Ok(Digest::from_bytes(*hash.finalize().as_bytes())) + } +} +fn root(hash: &mut blake3::Hasher, root: cellule_runtime::ltx::RootRef) { + hash.update(&root.cell); + hash.update(&root.incarnation); + hash.update(&root.digest); + for n in [ + root.position.txid, + root.position.checksum, + root.commit_sequence, + ] { + hash.update(&n.to_be_bytes()); + } +} diff --git a/crates/cellule-host/src/fleet/reader_evacuation/source/mod.rs b/crates/cellule-host/src/fleet/reader_evacuation/source/mod.rs new file mode 100644 index 00000000..b5ce15ff --- /dev/null +++ b/crates/cellule-host/src/fleet/reader_evacuation/source/mod.rs @@ -0,0 +1,151 @@ +//! Source reader joining composed with actual successor and current reader policy. +use super::*; +use crate::{ + CellNode, + read_replicas::{ReaderEnrollmentRetirement, ReaderReplacement}, +}; +use cellule_runtime::{ + cell::{actor::CellServingObservation, catalog::CatalogProof}, + control::{ + Control, + authority::{CellAuthority, VerifiedRootPrefix}, + }, + fleet::operations::EnrollmentRecord, + identity::NodeId, + ltx::CellReplica, +}; +use std::sync::Arc; + +mod collection; +mod digest; + +/// Authenticated mapping of one retained native retirement to its current writer. +/// Providers retain this exact allocation through their existing accepted-work +/// owner. Losing native evidence cannot be repaired from a terminal journal row. +pub struct FleetSourceReaderInputs { + /// Exact locally joined original reader and last installed root. + pub retirement: Arc, + /// Authenticated physical destination of the current native writer. + pub node: NodeId, + /// Actual existing managed host; collection starts no writer acquisition. + pub host: Arc, + /// Canonical catalog proof for the original target. + pub catalog: CatalogProof, + /// Application-authenticated canonical authority backend. + pub authority: CellAuthority, + /// Original Cell/incarnation origin through the existing native I/O path. + pub replica: CellReplica, +} + +/// Read-only application lookup of original native evidence and current writer. +/// Authenticate both physical endpoints and all canonical backend mappings. +/// Process restart exclusion and durable accepted-work retention remain separate +/// requirements; this interface cannot reconstruct a lost native join witness. +pub trait FleetSourceReaderSuccessors: Send + Sync { + /// None is unknown, never empty work. Return the same retained allocation on + /// global recheck; substituting a new capsule or mapping fences collection. + fn successor<'a>( + &'a self, + retired: &'a EnrollmentRecord, + expected: &'a FleetJournalSnapshot, + ) -> FleetAdapterFuture<'a, Option>>; +} + +/// Fresh current policy for one exact source-side native reader retirement. +/// This grants no original-writer/process/accepted-work or finalization rights. +pub struct FleetSourceReaderCheck { + inputs: Arc, + serving: CellServingObservation, + writer_boot: Digest, + origin: VerifiedRootPrefix, + authority: Control, + policy_revision: Option, + desired_readers: u16, + replacements: Vec, +} +impl FleetSourceReaderCheck { + /// Original charged local retirement retained throughout this observation. + #[must_use] + pub fn retirement(&self) -> &ReaderEnrollmentRetirement { + &self.inputs.retirement + } + /// Authenticated physical successor. + #[must_use] + pub fn node(&self) -> NodeId { + self.inputs.node + } + /// Immutable signed identity of the actual successor boot. + #[must_use] + pub const fn boot_identity(&self) -> Digest { + self.writer_boot + } + /// Actual compiled release checked against the signed successor. + #[must_use] + pub fn release(&self) -> Digest { + self.inputs.host.application().registry().release_digest() + } + /// Actual managed native successor observed around all I/O. + #[must_use] + pub fn serving(&self) -> &CellServingObservation { + &self.serving + } + /// Exact final installed prefix and complete current origin verification. + #[must_use] + pub fn origin(&self) -> &VerifiedRootPrefix { + &self.origin + } + /// Current canonical reader policy revision; absent differs from explicit zero. + #[must_use] + pub const fn policy_revision(&self) -> Option { + self.policy_revision + } + /// Current desired reader count. + #[must_use] + pub const fn desired_readers(&self) -> u16 { + self.desired_readers + } + /// Actual ready, enrolled replacements selected by the canonical policy. + #[must_use] + pub fn replacements(&self) -> &[ReaderReplacement] { + &self.replacements + } +} + +/// Complete lookup of eligible source reader requests at one full roster barrier. +/// Missing evidence stays absent for the full maintenance matcher to report. +/// Applications account bounded collector copies; retained retirement capsules +/// and native origin reads retain their existing runtime admission charges. +pub struct FleetSourceReaderPolicies { + snapshot: FleetJournalSnapshot, + roster: Digest, + original: Digest, + interval: (i64, i64), + checks: Vec, +} +impl FleetSourceReaderPolicies { + /// Exact full head and registry checked before and after collection. + #[must_use] + pub fn snapshot(&self) -> &FleetJournalSnapshot { + &self.snapshot + } + /// Full roster including original terminal and Pending rows. + #[must_use] + pub const fn roster_digest(&self) -> Digest { + self.roster + } + /// Exact immutable original/current request capture consumed by this lookup. + #[must_use] + pub const fn original_digest(&self) -> Digest { + self.original + } + /// Checked requests only; unknown requests remain obligations in the matcher. + #[must_use] + pub fn checks(&self) -> &[FleetSourceReaderCheck] { + &self.checks + } + /// Original fresh interval, separate from historical native retirement times. + #[must_use] + pub const fn interval(&self) -> (i64, i64) { + self.interval + } +} diff --git a/crates/cellule-host/src/fleet/reader_evacuation/verification.rs b/crates/cellule-host/src/fleet/reader_evacuation/verification.rs index 92bd4d6b..408a4d6a 100644 --- a/crates/cellule-host/src/fleet/reader_evacuation/verification.rs +++ b/crates/cellule-host/src/fleet/reader_evacuation/verification.rs @@ -1,11 +1,8 @@ use super::*; -use crate::read_replicas::{ - ReaderReplacement, - maintenance::{boot_identity, same_authority, validate_replacement}, -}; +use crate::read_replicas::ReaderReplacement; use cellule_runtime::{ Error, Result, - client::{CellDescription, Receipt}, + client::Receipt, control::{Control, authority::CellAuthority}, fleet::operations::{EnrollmentRole, MaintenancePhase}, node::{NodeDirectory, NodeMode}, @@ -165,140 +162,28 @@ impl FleetReaderEvacuationVerifier { incarnation: position.incarnation, commit_sequence: record.minimum_sequence(), }; - let expected = CellDescription { - cell: minimum.cell, - incarnation: minimum.incarnation, - code: record.authority().code, - schema: record.authority().schema, - }; - let witnesses: Vec<_> = pages.iter().flat_map(|page| page.entries()).collect(); - let mut replacements = Vec::with_capacity(witnesses.len()); - let mut authority = record.authority().clone(); - for _ in 0..2 { - let now = clock()?; - if now >= current.deadline_ms() { - return Err(Error::Deadline); - } - let observed = self - .authority - .load(minimum.cell) - .await? - .ok_or(Error::Fenced)?; - let fresh = observed.value(); - if !same_authority(&authority, fresh) { - return Err(Error::Fenced); - } - let policy = self.policy.load(minimum.cell).await?.map(|row| row.value()); - if policy.map(|policy| policy.revision()) != record.policy_revision() - || policy.map_or(0, |policy| policy.desired_readers()) != record.desired_readers() - || policy.is_some_and(|policy| { - policy.cell() != minimum.cell || policy.incarnation() != minimum.incarnation - }) - { - return Err(Error::Fenced); - } - let selected = self - .directory - .select_readers( - minimum.cell, - fresh.owner.as_ref().ok_or(Error::Fenced)?.session, - fresh.code, - witnesses.len(), - now, - 10_000, - ) - .await?; - if selected.len() != witnesses.len() { - return Err(Error::ReplicaUnavailable); - } - replacements.clear(); - for selected in selected { - let witness = witnesses - .iter() - .find(|entry| { - entry.node == selected.node() && entry.session == selected.session() - }) - .ok_or(Error::Fenced)?; - let node = self - .directory - .load(witness.session, clock()?) - .await? - .ok_or(Error::Fenced)?; - let node = node.advertisement(); - if node.node() != witness.node - || boot_identity(node)? != witness.boot_identity - || validate_replacement(&roster, node, target, minimum)? - != (witness.enrollment_key, witness.enrollment_digest) - { - return Err(Error::Fenced); - } - let (receipt, ready) = self.peer.status(target, node.clone(), expected).await?; - if !ready - || receipt.cell != minimum.cell - || receipt.incarnation != minimum.incarnation - || receipt.commit_sequence - < minimum.commit_sequence.max(witness.commit_sequence) - { - return Err(Error::ReplicaUnavailable); - } - replacements.push(ReaderReplacement { - node: witness.node, - session: witness.session, - boot_identity: witness.boot_identity, - enrollment_key: witness.enrollment_key, - enrollment_digest: witness.enrollment_digest, - receipt, - }); - } - authority = fresh.clone(); - // Probes can suspend; no authority/policy change is hidden behind - // their replies or a stable native topology cursor. - let after = self - .authority - .load(minimum.cell) - .await? - .ok_or(Error::Fenced)?; - if !same_authority(&authority, after.value()) - || self.policy.load(minimum.cell).await?.map(|row| row.value()) != policy - { - return Err(Error::Fenced); - } - authority = after.value().clone(); - let selected = self - .directory - .select_readers( - minimum.cell, - authority.owner.as_ref().ok_or(Error::Fenced)?.session, - authority.code, - witnesses.len(), - clock()?, - 10_000, - ) - .await?; - if selected.len() != witnesses.len() { - return Err(Error::Fenced); - } - for node in selected { - let witness = witnesses - .iter() - .find(|entry| entry.node == node.node() && entry.session == node.session()) - .ok_or(Error::Fenced)?; - let fresh = self - .directory - .load(witness.session, clock()?) - .await? - .ok_or(Error::Fenced)?; - let fresh = fresh.advertisement(); - if fresh.node() != witness.node - || boot_identity(fresh)? != witness.boot_identity - || validate_replacement(&roster, fresh, target, minimum)? - != (witness.enrollment_key, witness.enrollment_digest) - { - return Err(Error::Fenced); - } - } - roster.confirm(journal, deadline).await?; - } + let witnesses: Vec<_> = pages + .iter() + .flat_map(|page| page.entries()) + .cloned() + .collect(); + let (authority, replacements) = self + .confirm_current( + journal, + &roster, + super::current::CurrentReaderPolicy { + target, + authority: record.authority(), + minimum, + policy_revision: record.policy_revision(), + desired_readers: record.desired_readers(), + witnesses: &witnesses, + operation_deadline_ms: current.deadline_ms(), + }, + deadline, + clock, + ) + .await?; let finished = clock()?; if finished < started || finished - started > 30_000 || finished >= current.deadline_ms() { return Err(Error::Deadline); diff --git a/crates/cellule-host/src/fleet/reconciler/observation/evacuations.rs b/crates/cellule-host/src/fleet/reconciler/observation/evacuations.rs index f5ce1b61..b619f4cf 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/evacuations.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/evacuations.rs @@ -215,7 +215,7 @@ impl FleetObservation { Ok(()) } - fn role_boot( + pub(super) fn role_boot( &self, node: NodeId, session: SessionId, diff --git a/crates/cellule-host/src/fleet/reconciler/observation/maintenance_policies.rs b/crates/cellule-host/src/fleet/reconciler/observation/maintenance_policies.rs index c5ca8e85..f090d9b9 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/maintenance_policies.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/maintenance_policies.rs @@ -22,6 +22,7 @@ impl FleetObservation { self.reader_evacuations().unwrap_or(&[]), self.follower_evacuations().unwrap_or(&[]), self.maintenance_nonexecution.as_ref(), + self.source_reader_policies.as_ref(), self.maintenance_policy_inputs()?, )?; self.maintenance_policies = Some(coverage); @@ -51,9 +52,10 @@ impl FleetObservation { fn maintenance_policy_inputs(&self) -> Result { let original = self.maintenance_enrollments.as_ref().ok_or(Error::Fenced)?; let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-maintenance-policy-inputs.v2\0"); + hash.update(b"cellule.fleet-maintenance-policy-inputs.v3\0"); hash.update(original.digest()?.as_bytes()); self.hash_role_evacuations(&mut hash)?; + self.hash_source_reader_policies(&mut hash)?; hash.update(&[u8::from(self.maintenance_nonexecution.is_some())]); if let Some(nonexecution) = &self.maintenance_nonexecution { hash.update(nonexecution.digest().as_bytes()); diff --git a/crates/cellule-host/src/fleet/reconciler/observation/mod.rs b/crates/cellule-host/src/fleet/reconciler/observation/mod.rs index ac19aa08..e6413d84 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/mod.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/mod.rs @@ -3,7 +3,7 @@ use std::collections::{HashMap, HashSet}; use crate::fleet::{ FleetFailedBootClosure, FleetMaintenanceEnrollments, FleetMaintenanceNonexecution, FleetMaintenancePolicyCoverage, FleetOriginalWriterSuccessorInventory, FleetRoleCoverage, - FleetRoster, + FleetRoster, FleetSourceReaderPolicies, }; use cellule_runtime::cell::actor::OwnedCellObservation; use cellule_runtime::fleet::operations::{FleetScope, RegistryVersion}; @@ -49,6 +49,7 @@ pub struct FleetObservation { maintenance_enrollments: Option, maintenance_policies: Option, maintenance_nonexecution: Option, + source_reader_policies: Option, } impl FleetObservation { @@ -84,6 +85,7 @@ impl FleetObservation { maintenance_enrollments: None, maintenance_policies: None, maintenance_nonexecution: None, + source_reader_policies: None, }; observation.placements(capture_finished_at_ms)?; Ok(observation) @@ -112,6 +114,7 @@ impl FleetObservation { self.validate_maintenance_policies()?; self.validate_maintenance_enrollments()?; self.validate_maintenance_nonexecution()?; + self.validate_source_reader_policies()?; self.validate_original_writer_successors()?; self.validate_failed_boot_closures()?; self.validate_role_evacuations()?; @@ -217,7 +220,7 @@ impl FleetObservation { pub(super) fn digest(&self, now_ms: i64) -> Result { let nodes = self.placements(now_ms)?; let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-planner-inputs.v11\0"); + hash.update(b"cellule.fleet-planner-inputs.v12\0"); hash.update(self.scope.fleet.as_bytes()); hash.update(self.scope.application.as_bytes()); hash.update(&self.registry.to_bytes().map_err(super::operation)?); @@ -272,6 +275,7 @@ impl FleetObservation { } } self.hash_role_evacuations(&mut hash)?; + self.hash_source_reader_policies(&mut hash)?; hash.update(&[u8::from(self.maintenance_nonexecution.is_some())]); if let Some(nonexecution) = &self.maintenance_nonexecution { hash.update(nonexecution.digest().as_bytes()); @@ -384,5 +388,6 @@ mod maintenance_enrollments; mod maintenance_policies; mod nonexecution; mod original_writers; +mod source_readers; #[cfg(test)] mod tests; diff --git a/crates/cellule-host/src/fleet/reconciler/observation/original_writers.rs b/crates/cellule-host/src/fleet/reconciler/observation/original_writers.rs index 27add42c..cdee14b5 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/original_writers.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/original_writers.rs @@ -51,40 +51,50 @@ impl FleetObservation { } for proof in inventory.proofs() { let serving = proof.serving(); - if !self.nodes.iter().any(|node| { - node.node() == proof.node() - && node.session() == serving.owner().session - && node.endpoint() == serving.owner().endpoint - && node.release() == proof.release() - }) { - return Err(Error::Node("original writer successor boot differs")); - } - let observed = self.cells.iter().find(|owned| { - owned.observation.target.cell_id() == proof.original().target.cell_id() - }); - // A partial planner scan can omit a scoped row; the complete - // original proof remains retained and cannot certify that scan. - if observed.is_none() - && self.complete - && proof.original().target.application() == self.scope.application + self.current_writer(proof.node(), proof.release(), serving)?; + } + Ok(()) + } + pub(super) fn current_writer( + &self, + node: NodeId, + release: Digest, + serving: &cellule_runtime::cell::actor::CellServingObservation, + ) -> Result<()> { + if !self.nodes.iter().any(|ad| { + ad.node() == node + && ad.session() == serving.owner().session + && ad.endpoint() == serving.owner().endpoint + && ad.release() == release + }) { + return Err(Error::Node("original writer successor boot differs")); + } + let observed = self + .cells + .iter() + .find(|owned| owned.observation.target.cell_id() == serving.native().target.cell_id()); + // A partial planner scan can omit a scoped row; the complete + // original proof remains retained and cannot certify that scan. + if observed.is_none() + && self.complete + && serving.native().target.application() == self.scope.application + { + return Err(Error::Node("original writer successor row is missing")); + } + if let Some(owned) = observed { + let row = &owned.observation; + let native = serving.native(); + if owned.node != node + || owned.session != serving.owner().session + || row.target != native.target + || row.incarnation != native.incarnation + || row.generation != native.generation + || row.code != native.code + || row.schema != native.schema + || row.role != native.role + || row.position.as_ref() != Some(serving.position()) { - return Err(Error::Node("original writer successor row is missing")); - } - if let Some(owned) = observed { - let row = &owned.observation; - let native = serving.native(); - if owned.node != proof.node() - || owned.session != serving.owner().session - || row.target != native.target - || row.incarnation != native.incarnation - || row.generation != native.generation - || row.code != native.code - || row.schema != native.schema - || row.role != native.role - || row.position.as_ref() != Some(serving.position()) - { - return Err(Error::Node("original writer successor row differs")); - } + return Err(Error::Node("original writer successor row differs")); } } Ok(()) diff --git a/crates/cellule-host/src/fleet/reconciler/observation/source_readers.rs b/crates/cellule-host/src/fleet/reconciler/observation/source_readers.rs new file mode 100644 index 00000000..c46253c9 --- /dev/null +++ b/crates/cellule-host/src/fleet/reconciler/observation/source_readers.rs @@ -0,0 +1,103 @@ +//! Exact source-reader policy capture inside the full original observation. +use super::*; + +impl FleetObservation { + /// Retains exact source native joining, final-root derivation and current + /// successor/reader policy before complete request matching. Attachment + /// starts no effect and cannot upgrade `complete` or grant finalization. + pub fn with_source_reader_policies( + mut self, + policies: FleetSourceReaderPolicies, + ) -> Result { + if self.source_reader_policies.is_some() { + return Err(Error::Control("source reader policies already retained")); + } + self.source_reader_policies = Some(policies); + self.validate_role_coverage()?; + Ok(self) + } + /// Original source checks, with their retained charged native capsules. + #[must_use] + pub fn source_reader_policies(&self) -> Option<&FleetSourceReaderPolicies> { + self.source_reader_policies.as_ref() + } + pub(super) fn validate_source_reader_policies(&self) -> Result<()> { + let Some(checks) = &self.source_reader_policies else { + return Ok(()); + }; + let snapshot = checks.snapshot(); + let interval = checks.interval(); + if snapshot.head().scope() != self.scope + || snapshot.registry() != self.registry + || interval.0 < self.capture_started_at_ms + || interval.1 > self.capture_finished_at_ms + { + return Err(Error::Node( + "source reader policy observation barrier differs", + )); + } + if let Some(original) = &self.maintenance_enrollments + && (snapshot != original.snapshot() + || checks.roster_digest() != original.roster_digest() + || checks.original_digest() != original.digest()?) + { + return Err(Error::Node("source reader policy original capture differs")); + } + if let Some(roster) = &self.roster + && (snapshot != roster.snapshot() || checks.roster_digest() != roster.digest()?) + { + return Err(Error::Node("source reader policy roster differs")); + } + if self.role_coverage.as_ref().is_some_and(|coverage| { + snapshot != coverage.snapshot() || checks.roster_digest() != coverage.roster_digest() + }) || self + .original_writer_successors + .as_ref() + .is_some_and(|inventory| snapshot != inventory.original().snapshot()) + || self + .failed_boot_closures + .as_ref() + .is_some_and(|rows| rows.iter().any(|row| row.snapshot() != snapshot)) + || self.maintenance_nonexecution.as_ref().is_some_and(|proof| { + snapshot != proof.snapshot() || checks.roster_digest() != proof.roster_digest() + }) + || self.reader_evacuations().is_some_and(|rows| { + rows.iter().any(|row| { + row.snapshot() != snapshot || checks.roster_digest() != row.roster_digest() + }) + }) + || self.follower_evacuations().is_some_and(|rows| { + rows.iter().any(|row| { + row.snapshot() != snapshot || checks.roster_digest() != row.roster_digest() + }) + }) + { + return Err(Error::Node("source reader policy evidence barrier differs")); + } + for check in checks.checks() { + self.current_writer(check.node(), check.release(), check.serving())?; + self.role_boot( + check.node(), + check.serving().owner().session, + check.boot_identity(), + crate::read_replicas::maintenance::boot_identity, + )?; + for replacement in check.replacements() { + self.role_boot( + replacement.node, + replacement.session, + replacement.boot_identity, + crate::read_replicas::maintenance::boot_identity, + )?; + } + } + Ok(()) + } + pub(super) fn hash_source_reader_policies(&self, hash: &mut blake3::Hasher) -> Result<()> { + hash.update(&[u8::from(self.source_reader_policies.is_some())]); + if let Some(policies) = &self.source_reader_policies { + hash.update(policies.digest()?.as_bytes()); + } + Ok(()) + } +} diff --git a/docs/fleet-operations-plan.md b/docs/fleet-operations-plan.md index 0671e30f..8dc17d36 100644 --- a/docs/fleet-operations-plan.md +++ b/docs/fleet-operations-plan.md @@ -37,7 +37,7 @@ same action and evidence contracts. | Which external facilities are required? | A linearizable journal, complete enrollment registry, authenticated management transport, and trusted local Cell inputs. The reference example supplies these behind the same contracts. | | What keeps operations simple? | One operation ID, one status contract, one application loop, bounded defaults, and one canonical host drain lane. | | What proves success? | Receipt-preserving actor activation on another node. Maintenance additionally requires settled role obligations, host Stopped, and session withdrawal. | -| What is the next concrete change? | Qualify the current head, including both routing profiles. Complete source/failed-owner successor policy for installed roles and production nonexecution evidence retention. Exact reader removal now guards the original request under the native activation lane and retains its joined prefix; current policy and durable process evidence remain separate. Fresh nonexecution confirmation now consumes independently joined original witnesses; terminal rows alone remain unknown. Native donor policy lookup discovers latest history through the existing authenticated reader/follower verifiers. Complete observation around original-writer successors, failed-boot closures and durable unknown work. Connect reader/follower replacement policy and accepted-work barriers to SettleRoles/Finalize. Keep cross-session recovery, process/provider and remaining failure/inspection gates in scope. | +| What is the next concrete change? | Qualify the current head, including both routing profiles. Complete durable provider/process retention and failed-owner reader/follower succession. Native source reader policy now composes exact joining/final-root verification with an actual managed writer and current reader policy. Exact reader removal now guards the original request under the native activation lane and retains its joined prefix; the local source collector supplies current policy while durable process evidence remains separate. Fresh nonexecution confirmation now consumes independently joined original witnesses; terminal rows alone remain unknown. Native donor policy lookup discovers latest history through the existing authenticated reader/follower verifiers. Complete observation around original-writer successors, failed-boot closures and durable unknown work. Connect reader/follower replacement policy and accepted-work barriers to SettleRoles/Finalize. Keep cross-session recovery, process/provider and remaining failure/inspection gates in scope. | Start with the [first slice commit sequence](#first-slice-commit-sequence). Every increment must expose a reviewable public behavior and retain its test @@ -290,10 +290,11 @@ present API; the pseudocode below specifies the remaining orchestration. | `FollowerEvacuationRecord` / `FleetFollowerEvacuationJournal` / `FleetFollowerEvacuationVerifier` | Present bounded durable live-owner ensemble history, revisioned application policy, atomic latest pointers and canonical/native revalidation/refresh. | Bind every original Retired member and replacement request, confirm current canonical ensemble plus the actual source/member native owners through `FleetSnapshotTransport`. Refresh after policy/epoch/operation changes without repeating original retirement. Complete failed-owner successor policy, full observer and SettleRoles/Finalize integration; this per-owner evidence cannot finish maintenance. | | `FleetObservation::with_role_coverage` | Present retained graph attachment and planner-input binding. | Preserve the original interval and reject replacement or scope/registry mismatches. The reconciler rechecks the graph's exact head/registry and full roster digest; unchanged registry alone cannot admit an earlier controller head. Attachment never upgrades partial adapter coverage. | | `MaintenanceEnrollmentInventory`, `FleetJournal::maintenance_enrollments` and `FleetMaintenanceEnrollments::collect` | Present bounded original Pending/Established reader/follower pages frozen with the first BeginEvacuation CAS, at either physical endpoint including earlier boots. | Retain exact pretransition head, registry, operation and acceptance times. Traverse every page, match original acceptances against the fresh full roster, and attach with `FleetObservation::with_maintenance_enrollments`. Missing history remains unknown; zero requires a committed empty manifest. New source replacement work after capture remains in the current graph. All original and current policy/work obligations remain required before settlement. | -| `FleetObservation::with_role_evacuations` | Present retention of freshly rechecked durable reader/follower policy history, current authority/prefixes and native source/member evidence. Planner digest v11 binds complete original checks and the immutable maintenance enrollment manifest. | Require one exact full head, registry and roster, with every original fresh interval inside the outer observation. Compare signed replacement boots with the existing producer identities and refuse duplicate obligations. Public minion reconciliation consumes actual ready readers and rotated ensembles without upgrading partial coverage. Complete authenticated policy coverage, failed-owner successor policy and accepted-work/finalization barriers remain required. | -| `FleetMaintenanceNonexecution` / `FleetEnrollmentNonexecution` | Present independent read-only confirmation of retained original nonexecution, exact terminal witnesses and the complete original/current request set. Whole-capture clocks/deadlines and full head/registry rechecks refuse missing or changed evidence; planner digest v11 binds checked results. | Applications authenticate original endpoints, earlier native/external joining and durable exclusion evidence. Actual reader/follower cases consume joined original witnesses; production provider/process retention, installed-role succession, other accepted work and finalization remain required. | -| `FleetObservation::check_maintenance_policies` / `FleetReconcileReport::maintenance_policy` | Present bounded matching of every immutable original request and every currently unresolved related reader/follower against all supplied native policy checks. Exact current rows and Established donor original digests are required. Planner digest v11 binds source inputs and every explicit result. | Attach original capture and all policy checks before matching. Missing history refuses matching; missing policy, Pending/Established, source succession and unproven nonexecution remain explicit gaps. Advisory counts retain their own full head/registry/interval; absent coverage stays unknown. Complete source/failed-owner and nonexecution evidence, native/external accepted work and finalization remain required. | +| `FleetObservation::with_role_evacuations` | Present retention of freshly rechecked durable reader/follower policy history, current authority/prefixes and native source/member evidence. Planner digest v12 binds complete original checks and the immutable maintenance enrollment manifest. | Require one exact full head, registry and roster, with every original fresh interval inside the outer observation. Compare signed replacement boots with the existing producer identities and refuse duplicate obligations. Public minion reconciliation consumes actual ready readers and rotated ensembles without upgrading partial coverage. Complete authenticated policy coverage, failed-owner successor policy and accepted-work/finalization barriers remain required. | +| `FleetMaintenanceNonexecution` / `FleetEnrollmentNonexecution` | Present independent read-only confirmation of retained original nonexecution, exact terminal witnesses and the complete original/current request set. Whole-capture clocks/deadlines and full head/registry rechecks refuse missing or changed evidence; planner digest v12 binds checked results. | Applications authenticate original endpoints, earlier native/external joining and durable exclusion evidence. Actual reader/follower cases consume joined original witnesses; production provider/process retention, installed-role succession, other accepted work and finalization remain required. | +| `FleetObservation::check_maintenance_policies` / `FleetReconcileReport::maintenance_policy` | Present bounded matching of every immutable original request and every currently unresolved related reader/follower against all supplied native policy checks. Exact current rows and Established donor original digests are required. Planner digest v12 binds source inputs and every explicit result. | Attach original capture and all policy checks before matching. Missing history refuses matching; missing policy, Pending/Established, source succession and unproven nonexecution remain explicit gaps. Advisory counts retain their own full head/registry/interval; absent coverage stays unknown. Complete source/failed-owner and nonexecution evidence, native/external accepted work and finalization remain required. | | `FleetReaderEvacuationVerifier::collect_maintenance` / `FleetFollowerEvacuationVerifier::collect_maintenance` | Present complete eligible donor history lookup from the canonical original/current request set, followed by existing authenticated native/current-policy confirmation. Exact original digest/current row and full head/registry/roster checks are retained. | Collect original history first and construct the outer observation after both role collections. Missing latest history supplies no check and stays MissingPolicy in the matcher. One monotonic 30-second capture and caller deadline bound collection; changed heads/policies during native capture fail closed. Source/failed-owner succession, nonexecution, unknown native/external work and finalization remain required. | +| `FleetReaderEvacuationVerifier::collect_source_readers` / `FleetSourceReaderPolicies` / `FleetObservation::with_source_reader_policies` | Present exact locally joined source reader retirement composed with an actual managed native writer successor, canonical final-root/origin verification, current reader policy and ready enrolled replacement prefixes. The complete original/current request set, full head/registry/roster, repeated provider allocation/presence and monotonic capture are retained in planner identity v12. | Authenticate both endpoints and canonical backend mappings. Retain the original charged native capsule through the application's existing accepted-work owner. Missing evidence stays SourceSuccessor; checked exact native source requests become SourceReader. Local SQLite client reconstruction/public reconciliation cannot establish OS/process restart exclusion or durable provider retention. Complete failed-owner reader/follower succession, other accepted work and SettleRoles/Finalize remain required. | | `CellNode::follower_evacuation` | Present per-live-owner replacement and retirement check after the original requested rotation. | Require the declared member minimum, donor exclusion, complete Established replacement rows, pinned signed boots, current authority and full journal rechecks. Retain the original completion/error history and publish/revalidate its interval. This does not settle failed owners, Pending producers or the physical node's other obligations. | | Recovered-log authorization, member retirement and canonical Retired CAS | Present runtime tail closure after canonical recovery pinning. | Use `RecoveredNodeLogTransport`, receiver-side `authorize_recovered_log_retire` and `FollowerStore::retire_recovered`; confirm every original member before `retire_recovered_log`. Adopt exact committed closure with `retired_recovered_log` before repeating effects. Publish original enrollments through the host capsule, then complete failed-process barriers and maintenance orchestration; native closure alone cannot finish W7. | | `RecoveryManifestStore::load_manifest` | Complete digest-verified recovered-suffix metadata across every original application, retained after successor materialization. | Read the identity from canonical sealed-log authority, then verify each bundle through its application's ordinary store. Retain the complete original writer set separately, including object-covered Cells without a suffix; check current successor authority, acknowledged prefixes and serving. This metadata read cannot settle roles or finalize a node. | diff --git a/docs/fleet-operations-progress.md b/docs/fleet-operations-progress.md index f3c99b1f..9bc76c30 100644 --- a/docs/fleet-operations-progress.md +++ b/docs/fleet-operations-progress.md @@ -4,6 +4,79 @@ The [implementation plan](fleet-operations-plan.md) remains the full scope. This page records focused checkpoints; it does not establish complete fleet balancing, maintenance, or deployment qualification. +## October 4 2026 source reader policy checkpoint + +The canonical executable remains `crates/cellule-host/minion`, Cargo target +`fleet_operations`. The full W1–W10 plan remains active. + +`FleetReaderEvacuationVerifier::collect_source_readers` now enumerates the exact +original/current request set and composes retained native retirement with the +actual managed writer successor, canonical final-root derivation/current origin, +current reader policy and actual ready replacement requests. It uses the same +managed-writer validator as original failed-boot writers and the same current +reader confirmation loop as donor history. Whole-capture clocks/deadlines and +full roster/head checks bound the collection. Provider absence remains unknown; +changed allocations/presence and original backend errors refuse the capture. + +`FleetObservation::with_source_reader_policies` retains those native capsules and +checks their original capture, full barrier, signed successor/replacement boots +and captured writer rows. Planner identity v12 binds the new inputs. The complete +maintenance matcher marks only the exact checked source reader `SourceReader`; +omitted evidence remains `SourceSuccessor`. Attachment preserves incomplete +observation and grants no SettleRoles/Finalize rights. Persisted enrollment, +evacuation and signed peer formats are unchanged. + +Eleven real minion cases pass on the isolated source snapshot. They refresh past +the original opening, drain the original writer, restore on another managed boot, +join the exact original reader and install a new request on the same Active +receiver. They cover missing/changed evidence, foreign physical/origin mappings, +clock bounds, cancellation, deadline, source error preservation, policy/head/ +writer changes behind actual signed native replies, duplicate/late attachment, +outer boot/native-row mismatch and independent SQLite client/public reconciliation. +Every successful fixture joins all native owners and checks released ledgers. +An initial fixture activation used the departed writer's signer and was correctly +refused; the fixture now authenticates the actual successor boot through the same +signature/principal path. Initial compile/setup failures are retained as diagnostics. + +All 13 fresh Rust commands pass on the frozen Rust 1.99 source: 1,711 framework +passes with 37 documented ignores, all 331 minion cases, local LTX 54, Axum without +default features 16 and cookbook 256 plus check, Clippy, docs and binary builds. +The framework/minion total is 2,042 distinct passes; focused repeats and overlapping +local LTX are excluded. Workspace features/targets, Clippy, warning-denied API docs +and the minion build pass. Nine static gates pass; all 137 Rust snippets parse and +1,307 local links resolve. All 1,133 Rust/Cargo paths match manifest SHA256 +`2e382bfbcbe4682a383628848c3187d6c06007e8d0e89661ef488785d17760a5`. +Exact source/manifests, commands/results and initial failures are retained under +`/Users/haipingfu/Workspace/crabbuild-target/cellule-source-reader-9110095/evidence`. +A one-variable negative control verifies the opening root instead of the final +joined root. The exact-prefix regression fails as intended, distinguishing both +digests, transaction positions and sequences. The production source is restored +byte-identically; initial control-root/LTX-root type failure is retained and excluded +from regression evidence. The restored source also passed broad qualification. + +PR #37 was merged as `fa548bb` while qualification ran. Its Git tree is byte-identical +to qualified baseline `9110095`; the native source reader increment continues on +`codex/native-source-reader-policy` from that new main. Full follow-up head CI, +physical process/provider retention and the complete W1–W10 scope remain required. + +The new [source reader recipe](../crates/cellule-host/docs/source-readers.md) documents +provider ownership, collection/attachment and the runnable cases. + +### Highest remaining work + +1. Qualify the follow-up head in CI and both routing campaigns. + Earlier follower-observation and overload failures still need independent + diagnosis if reproduced; a later green run alone does not establish their cause. +2. Complete durable provider/process retention and failed-owner reader/follower + succession, full native/external accepted-work coverage and unknown outcomes. +3. Implement SettleRoles/Finalize through original action joining, native Stopped, + withdrawal, retired boots and committed completion; finish Cron/Blob owners + and primitive fault matrices. +4. Complete receiver-session loss/recovery/adoption and sustained convergence; + deliver canonical minion maintenance and receiver-loss commands. +5. Complete W9 physical process/provider faults, mixed binaries and load/soak; + W10 exercised runbooks and staged rollout/rollback. + ## October 4 2026 replica retry routing checkpoint The canonical executable remains `crates/cellule-host/minion`, Cargo target From c52ac385c2b5ee1ebaa2df03988e6d6341faa040 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 4 Oct 2026 02:17:01 -0700 Subject: [PATCH 2/5] fix failed reader source succession --- crates/cellule-host/docs/lifecycle.md | 19 +- crates/cellule-host/docs/source-readers.md | 43 +-- .../reader_tests/evacuation/fixture.rs | 176 ++++++++---- .../scenario/reader_tests/evacuation/mod.rs | 6 +- .../reader_tests/evacuation/source/fixture.rs | 268 +++++++++++++----- .../reader_tests/evacuation/source/mod.rs | 6 +- .../reader_tests/evacuation/source/tests.rs | 46 ++- .../minion/scenario/startup/mod.rs | 47 ++- .../src/fleet/failed_boot/readers/mod.rs | 48 +++- .../fleet/failed_boot/readers/publication.rs | 4 +- .../src/fleet/failed_boot/readers/records.rs | 4 +- crates/cellule-host/src/fleet/mod.rs | 3 +- .../src/fleet/reader_evacuation/mod.rs | 2 +- .../reader_evacuation/source/collection.rs | 55 +++- .../fleet/reader_evacuation/source/digest.rs | 31 +- .../src/fleet/reader_evacuation/source/mod.rs | 103 ++++++- .../src/fleet/reconciler/observation/mod.rs | 2 +- docs/fleet-operations-plan.md | 10 +- docs/fleet-operations-progress.md | 23 +- 19 files changed, 690 insertions(+), 206 deletions(-) diff --git a/crates/cellule-host/docs/lifecycle.md b/crates/cellule-host/docs/lifecycle.md index d03fdfea..f155fcca 100644 --- a/crates/cellule-host/docs/lifecycle.md +++ b/crates/cellule-host/docs/lifecycle.md @@ -968,7 +968,7 @@ ensembles. Signed replacement boots must match the existing producer-specific identity; a changed writer row or follower epoch/ensemble invalidates the input. Attachment order with role coverage, writer successors or failed-boot closure does not change these checks. The reconciler repeats the roster comparison. -Planner digest v12 binds collection presence, canonical record order, full +Planner digest v13 binds collection presence, canonical record order, full barriers, fresh intervals, current authority, reader prefixes and retained native follower inventories. Persisted history, enrollment and transport codecs are unchanged. @@ -993,7 +993,7 @@ fresh complete roster, and reconfirms the full journal snapshot. Retiring rows, extending a deadline or adopting a successor boot preserves the first set. `FleetObservation::with_maintenance_enrollments` retains this evidence inside the outer interval and compares the exact head, registry and roster with other checks, -in either attachment order. Planner digest v12 binds its presence and full digest. +in either attachment order. Planner digest v13 binds its presence and full digest. A missing manifest or page remains unknown. A committed empty manifest proves only that the original unresolved role set was empty at the phase transaction. @@ -1049,11 +1049,12 @@ starts no role effects or tasks and grants no settlement/finalization rights. ### Confirm source reader succession Use the [source reader collector](source-readers.md) after actual writer handoff -and exact native reader joining. It consumes the same complete original/current -request set and checks current writer origin, policy and native ready readers. +and native or failed-process reader closure. It consumes the same complete +original/current request set and checks current writer origin, policy and native +ready readers. Retain `with_source_reader_policies` before full maintenance policy matching. -Missing native/provider evidence stays unknown; complete accepted-work/process -coverage and finalization remain separate. +Missing native/provider evidence stays unknown. Production accepted-work/process +qualification and finalization remain separate. ### Confirm original nonexecution @@ -1104,7 +1105,7 @@ async fn attach_nonexecution( The observation's outer interval must include both collections. Attach all native policy and nonexecution checks before matching; later input changes refuse. -Checked nonexecution is reported separately and binds planner digest v12. It +Checked nonexecution is reported separately and binds planner digest v13. It cannot upgrade incomplete node observation, prove installed-role redundancy, join other accepted work, or grant SettleRoles/Finalize rights. @@ -1136,7 +1137,7 @@ supply complete request coverage. | `Reader` / `Follower` | Exact current role and fresh native replacement policy checked. | | `Pending` / `Established` | Accepted native outcome or installed role still unresolved. | | `MissingPolicy` | Retired donor lacks current replacement policy evidence. | -| `SourceReader` | Exact original native source reader joined, final root verified against an actual managed successor, current ready-reader policy checked. | +| `SourceReader` | Exact original source reader closed natively or by confirmed process evidence, minimum prefix verified against an actual managed successor, current ready-reader policy checked. | | `SourceSuccessor` | Remaining source-side or failed-owner policy still needs separate evidence. | | `Nonexecution` | Independently confirmed original work joining and exclusion; no role effect committed. | | `UnprovenNonexecution` | Closed unknown acceptance lacks checked nonexecution or policy evidence. | @@ -1147,7 +1148,7 @@ empty original manifest and no currently unresolved related requests. `FleetReconcileReport::maintenance_policy` supplies bounded advisory counts with their own head revision, registry and original interval. `None` means unknown. Later allocations cannot restamp these counts to the report's newer snapshot. -Planner digest v12 binds the coverage digest and source inputs; changing the +Planner digest v13 binds the coverage digest and source inputs; changing the attached policy collections after matching is refused. `is_complete()` describes only enumerated request-policy coverage. It does not diff --git a/crates/cellule-host/docs/source-readers.md b/crates/cellule-host/docs/source-readers.md index a1e41bea..9639f640 100644 --- a/crates/cellule-host/docs/source-readers.md +++ b/crates/cellule-host/docs/source-readers.md @@ -8,8 +8,8 @@ This uses the existing reader manager, actor FIFO, origin verifier and peer path | Input | Required evidence | | --- | --- | | Original request set | Every immutable original page and the full current roster. | -| Reader retirement | The retained opaque `ReaderEnrollmentRetirement` from exact native removal. | -| Final prefix | The last installed root after accepted reader work joined, including later refreshes. | +| Reader retirement | Either the retained `ReaderEnrollmentRetirement` from exact native removal or a `FleetFailedReaderClosure` bound to the original process and accepted-work proof. | +| Minimum prefix | Native removal requires its last installed root after accepted reader work joined. A failed receiver requires at least its original enrollment root. | | Writer successor | An actual managed native writer on an Active physical node outside maintenance, at a strictly newer epoch. | | Origin | Canonical derivation of the exact final root and availability of all current dependencies. | | Reader policy | Exact current revision, incarnation and desired count; absent differs from explicit zero. | @@ -19,14 +19,19 @@ This uses the existing reader manager, actor FIFO, origin verifier and peer path ## Retain and collect The embedding application authenticates both endpoints and canonical backend -mappings. Its existing accepted-work owner retains each native retirement and -returns the same `Arc` on repeated lookup. `None` means -unknown. A terminal enrollment row cannot reconstruct a lost native join witness. +mappings. Its existing accepted-work owner retains each native retirement or +failed-process closure and returns the same `Arc` on +repeated lookup. `None` means unknown. A terminal enrollment row cannot +reconstruct either proof. -After writer handoff, join the original reader with -`ReadReplicaManager::remove_enrolled`. Ordinary recruitment installs current -replacements. The same still-Active receiver may hold a new request under the -successor; that request cannot replace the original retirement witness. +After writer handoff, join a live original reader with +`ReadReplicaManager::remove_enrolled`. If its receiver process has stopped, +publish `FleetFailedReaderRetirement` only after the application confirms the +original process and all accepted work are durably closed. Both paths still +require the actual current writer, verified origin and current ready-reader +policy. Ordinary recruitment installs current replacements. The same +still-Active receiver may hold a new request under the successor; that request +cannot replace the original retirement proof. ```rust use cellule_host::fleet::{ @@ -62,14 +67,17 @@ Build the outer `FleetObservation` after all native collection finishes. Retain `check_maintenance_policies`. Both attachment orders validate the same original capture, full head, registry, roster and interval. Signed successor/replacement boots and any captured writer row must match the checked native evidence. -Planner identity v12 binds these inputs. Missing evidence stays `SourceSuccessor`; -a checked exact source reader is `SourceReader`. +Planner identity v13 binds native and failed-process inputs. Missing evidence +stays `SourceSuccessor`; a checked exact source reader is `SourceReader`. ## Scope and runnable evidence The canonical minion suite exercises managed writer handoff, refresh past the -original opening, exact closure, a new reader request on the same receiver, -independent SQLite journal clients and public reconciliation. It also exercises +original opening, exact native closure, a new reader request on the same +receiver, and failed receiver closure. A failed receiver with desired readers +still set to one remains blocked when no replacement is available; reducing the +canonical policy to zero permits the checked closure. Independent SQLite +journal clients and public reconciliation are also covered. The suite exercises missing/changed provider evidence, foreign origins, clock/deadline/cancellation and policy, journal or writer changes behind actual signed native replies. @@ -78,7 +86,8 @@ cargo test -p cellule-host --example fleet_operations --all-features --locked \ scenario::reader_tests::evacuation::source:: -- --test-threads=2 ``` -These local checks do not upgrade an incomplete observation or authorize -SettleRoles/Finalize. Durable process/provider retention, original writer and -failed-owner proof, follower succession, native/external accepted work and full -maintenance remain required by the [fleet plan](../../../docs/fleet-operations-plan.md). +These local checks do not qualify production process/provider retention or +upgrade an incomplete observation, and they do not authorize SettleRoles or +Finalize. Original writer proof, follower succession, native/external accepted +work and full maintenance remain required by the +[fleet plan](../../../docs/fleet-operations-plan.md). diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/fixture.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/fixture.rs index 55005262..dd0dfcb3 100644 --- a/crates/cellule-host/minion/scenario/reader_tests/evacuation/fixture.rs +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/fixture.rs @@ -6,6 +6,17 @@ impl Fixture { } pub(super) async fn for_maintenance(maintenance_node: usize) -> Self { + Self::for_maintenance_with_reader_enrollment(maintenance_node, true).await + } + + pub(super) async fn for_failed_reader_receiver(maintenance_node: usize) -> Self { + Self::for_maintenance_with_reader_enrollment(maintenance_node, false).await + } + + async fn for_maintenance_with_reader_enrollment( + maintenance_node: usize, + reader_enrollment: bool, + ) -> Self { let root = tempfile::tempdir().unwrap(); let journal = Arc::new( SqliteJournal::open( @@ -79,27 +90,29 @@ impl Fixture { ) .await .unwrap(); - let node = Arc::new( - CellNodeBuilder::new(app.clone()) - .with_runtime( - SqlWorkerPool::new(2, 8) - .unwrap() - .with_native_memory_limit(128 << 20) - .unwrap(), - if maintenance_node == 0 { - 64 << 20 - } else { - 16 << 20 - }, - ) - .with_replica_host( - Host::default().with_local_disk_budget(DiskBudget::new(8 << 30)), - ) - .with_session(session(index)) - .with_fleet_startup_intent(intent.clone()) - .build() - .unwrap(), - ); + let builder = CellNodeBuilder::new(app.clone()) + .with_runtime( + SqlWorkerPool::new(2, 8) + .unwrap() + .with_native_memory_limit(128 << 20) + .unwrap(), + if maintenance_node == 0 { + 64 << 20 + } else { + 16 << 20 + }, + ) + .with_replica_host(Host::default().with_local_disk_budget(DiskBudget::new(8 << 30))) + .with_session(session(index)); + // The failed-receiver fixture keeps this one receiver outside the + // graceful reader-retirement owner so process evidence, rather + // than native shutdown, must close the original enrollment. + let builder = if reader_enrollment || index != 1 { + builder.with_fleet_startup_intent(intent.clone()) + } else { + builder + }; + let node = Arc::new(builder.build().unwrap()); node.install_task_group(CancellationToken::new(), CancellationToken::new()) .unwrap(); node.install_fleet_actions( @@ -121,9 +134,16 @@ impl Fixture { limits, ) .unwrap(); - node.install_fleet_reader_enrollment(scope(), node_id(index), journal.clone()) - .unwrap(); - let ad = startup::advertisement(index, &node, &intent).await.unwrap(); + if reader_enrollment || index != 1 { + node.install_fleet_reader_enrollment(scope(), node_id(index), journal.clone()) + .unwrap(); + } + let ad = if !reader_enrollment && index == 1 { + startup::advertisement_with_read_capacity(index, &node, &intent).await + } else { + startup::advertisement(index, &node, &intent).await + } + .unwrap(); let spec = startup::spec(&intent).unwrap(); let original = startup::enroll(&journal, &directory, &spec, ad.clone(), clock().unwrap()) @@ -131,21 +151,23 @@ impl Fixture { .unwrap(); let guard = NodeLeaseGuard::new(clock().unwrap(), ad.expires_at_ms()).unwrap(); node.install_node_lease_for_startup(guard.clone()).unwrap(); - node.confirm_fleet_startup(journal.as_ref(), spec.key().unwrap()) - .await - .unwrap(); let observed = directory .load(session(index), clock().unwrap()) .await .unwrap() .unwrap(); - node.install_fleet_boot_withdrawal( - directory.clone(), - observed, - original, - journal.clone(), - ) - .unwrap(); + if reader_enrollment || index != 1 { + node.confirm_fleet_startup(journal.as_ref(), spec.key().unwrap()) + .await + .unwrap(); + node.install_fleet_boot_withdrawal( + directory.clone(), + observed, + original, + journal.clone(), + ) + .unwrap(); + } node.start().unwrap(); boots.push(startup::BootOwner { node: node.clone(), @@ -179,10 +201,12 @@ impl Fixture { .await .unwrap() .unwrap(); - boots[1] - .refresh_capacity(1, journal.as_ref(), Instant::now() + Duration::from_secs(3)) - .await - .unwrap(); + if reader_enrollment { + boots[1] + .refresh_capacity(1, journal.as_ref(), Instant::now() + Duration::from_secs(3)) + .await + .unwrap(); + } let transport = Arc::new(transport::NativePeers::new( &nodes, &managers, @@ -214,20 +238,68 @@ impl Fixture { code, schema: 1, }; - peer.activate(&target, &directory, ad.advertisement().clone(), description) - .await - .unwrap(); - let reader = managers[1].resolve(target.clone()).await.unwrap(); - let completion = managers[1] - .enrollment_completion(target.cell_id()) - .await - .unwrap() - .unwrap(); - let original = journal - .load_enrollment(scope(), completion.spec.key().unwrap()) - .await - .unwrap() - .unwrap(); + let (reader, original) = if reader_enrollment { + peer.activate(&target, &directory, ad.advertisement().clone(), description) + .await + .unwrap(); + let reader = managers[1].resolve(target.clone()).await.unwrap(); + let completion = managers[1] + .enrollment_completion(target.cell_id()) + .await + .unwrap() + .unwrap(); + let original = journal + .load_enrollment(scope(), completion.spec.key().unwrap()) + .await + .unwrap() + .unwrap(); + (reader, original) + } else { + let prepared = managers[1] + .prepare_source(target.clone(), session(0)) + .await + .unwrap(); + let spec = EnrollmentSpec { + scope: scope(), + request: Digest::from_bytes([190; 32]), + source: Some(EnrollmentEndpoint { + node: node_id(0), + session: session(0), + intent_revision: 1, + }), + target: EnrollmentEndpoint { + node: node_id(1), + session: session(1), + intent_revision: 1, + }, + role: EnrollmentRole::Reader { + target: target.clone(), + position: PublishedPosition { + incarnation, + epoch: prepared.epoch(), + root: prepared.root().clone(), + }, + }, + }; + let FleetEnrollmentAcceptance::New(pending) = journal + .accept_enrollment(&spec, clock().unwrap()) + .await + .unwrap() + else { + panic!("new reader request expected") + }; + managers[1].activate_source(prepared).await.unwrap(); + let original = journal + .publish_enrollment_result( + &pending, + EnrollmentEvent::Established(Digest::from_bytes([191; 32])), + clock().unwrap(), + ) + .await + .unwrap(); + let reader = managers[1].resolve(target.clone()).await.unwrap(); + (reader, original) + }; assert_eq!(original.status(), EnrollmentStatus::Established); let snapshot = journal.load_snapshot(scope()).await.unwrap(); journal diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/mod.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/mod.rs index 9baf6b47..598e3e49 100644 --- a/crates/cellule-host/minion/scenario/reader_tests/evacuation/mod.rs +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/mod.rs @@ -1,10 +1,14 @@ //! Actual managed boots and nonzero reader replacement under retained intent. use super::super::startup; use super::*; +use cellule_host::fleet::FleetEnrollmentAcceptance; use cellule_host::read_replicas::ReaderEvacuation; use cellule_runtime::{ client::{CellDescription, CellReadReplica}, - fleet::operations::{JournalTransition, MaintenanceEvent, MaintenanceOperation, OperationId}, + fleet::operations::{ + EnrollmentEndpoint, EnrollmentEvent, EnrollmentRole, EnrollmentSpec, JournalTransition, + MaintenanceEvent, MaintenanceOperation, OperationId, PublishedPosition, + }, peer::{PeerPrincipal, PeerSigner, ReplicaPeerClient}, }; use ed25519_dalek::SigningKey; diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/fixture.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/fixture.rs index 00677241..e38ef147 100644 --- a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/fixture.rs +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/fixture.rs @@ -2,7 +2,19 @@ use super::*; impl SourceFixture { pub(super) async fn new() -> Self { - let mut fixture = Fixture::for_maintenance(0).await; + Self::build(false).await + } + + pub(super) async fn failed_receiver() -> Self { + Self::build(true).await + } + + async fn build(failed_receiver: bool) -> Self { + let mut fixture = if failed_receiver { + Fixture::for_failed_reader_receiver(0).await + } else { + Fixture::for_maintenance(0).await + }; assert_eq!(fixture.original.spec().source.unwrap().node, node_id(0)); let opened = fixture.reader.lifecycle_observation().await.root(); let now = clock().unwrap(); @@ -72,92 +84,177 @@ impl SourceFixture { .refresh_capacity(2, fixture.journal.as_ref(), deadline()) .await .unwrap(); - let retirement = Arc::new( - fixture.managers[1] - .remove_enrolled(&fixture.original, deadline()) + let retirement = if failed_receiver { + let observed = fixture + .directory + .load(session(1), clock().unwrap()) .await - .unwrap(), - ); - assert_eq!( - retirement.root(), - fixture.reader.lifecycle_observation().await.root() - ); - assert!( + .unwrap() + .unwrap(); fixture - .reader - .lifecycle_observation() + .directory + .withdraw(&observed, clock().unwrap()) .await - .locally_joined() - ); - // Activation is authorized by the current writer, not the departed - // source. Pin that actual successor boot's signing key in the same - // authenticated dispatch path; existing donor fixtures retain origin 0. - fixture.transport = Arc::new(transport::NativePeers::for_origin( - &fixture.nodes, - &fixture.managers, - &fixture.layout, - fixture.directory.clone(), - 2, - )); - fixture.peer = ReplicaPeerClient::new( - fixture.nodes[2].application().registry(), - Arc::new(PeerSigner::new( - session(2), - fixture.nodes[2].application().registry().release_digest(), - SigningKey::from_bytes(&[3; 32]), - )), - PeerPrincipal { - issuer: "managed-owner".into(), - subject: "live-owner".into(), - actions: vec!["replica-maintenance".into()], - }, - fixture.transport.clone(), - ); - // The same Active receiver can install a new request under the new - // writer. Its new acceptance must never substitute the original join. - fixture.boots[1] - .refresh_capacity(1, fixture.journal.as_ref(), deadline()) - .await - .unwrap(); - let ad = fixture - .directory - .load(session(1), clock().unwrap()) - .await - .unwrap() - .unwrap(); - fixture - .peer - .activate( - &fixture.target, + .unwrap(); + fixture.nodes[1].shutdown().await.unwrap(); + let snapshot = fixture.journal.load_snapshot(scope()).await.unwrap(); + let roster = FleetRoster::collect(fixture.journal.as_ref(), &snapshot, deadline()) + .await + .unwrap(); + let boot = roster + .enrollments() + .iter() + .find(|row| { + matches!(row.spec().role, EnrollmentRole::Node { .. }) + && row.spec().target.session == session(1) + }) + .unwrap() + .clone(); + let capture = FleetFailedReaderRetirement::capture( + fixture.journal.as_ref(), &fixture.directory, - ad.advertisement().clone(), - fixture.description, + &roster, + &boot, + &fixture.original, + session(0), + deadline(), + clock, ) .await .unwrap(); - let new_reader = fixture.managers[1] - .resolve(fixture.target.clone()) + let processes = StoppedProcesses { + node: fixture.nodes[1].clone(), + }; + let publication = capture + .publish( + fixture.journal.as_ref(), + &fixture.directory, + &processes, + session(0), + deadline(), + clock, + ) + .await + .unwrap(); + let closure = publication.confirmed().unwrap().clone(); + let snapshot = fixture.journal.load_snapshot(scope()).await.unwrap(); + let roster = FleetRoster::collect(fixture.journal.as_ref(), &snapshot, deadline()) + .await + .unwrap(); + let boot_retirement = FleetFailedBootRetirement::capture( + fixture.journal.as_ref(), + &fixture.directory, + &roster, + &boot, + session(0), + deadline(), + clock, + ) .await .unwrap(); - assert_eq!(new_reader.receipt().await, retirement.receipt()); - assert_ne!( - fixture.managers[1] - .enrollment_completion(fixture.target.cell_id()) + boot_retirement + .publish( + fixture.journal.as_ref(), + &fixture.directory, + &processes, + session(0), + deadline(), + clock, + ) .await .unwrap() - .unwrap() - .spec, - *retirement.original().spec() - ); - assert_eq!( - new_reader - .query::(Some(retirement.receipt()), 0) + .confirmed() + .unwrap(); + FleetSourceReaderRetirement::Failed(Arc::new(closure)) + } else { + let retirement = Arc::new( + fixture.managers[1] + .remove_enrolled(&fixture.original, deadline()) + .await + .unwrap(), + ); + assert_eq!( + retirement.root(), + fixture.reader.lifecycle_observation().await.root() + ); + assert!( + fixture + .reader + .lifecycle_observation() + .await + .locally_joined() + ); + // Activation is authorized by the current writer, not the departed + // source. Pin that actual successor boot's signing key in the same + // authenticated dispatch path; existing donor fixtures retain origin 0. + fixture.transport = Arc::new(transport::NativePeers::for_origin( + &fixture.nodes, + &fixture.managers, + &fixture.layout, + fixture.directory.clone(), + 2, + )); + fixture.peer = ReplicaPeerClient::new( + fixture.nodes[2].application().registry(), + Arc::new(PeerSigner::new( + session(2), + fixture.nodes[2].application().registry().release_digest(), + SigningKey::from_bytes(&[3; 32]), + )), + PeerPrincipal { + issuer: "managed-owner".into(), + subject: "live-owner".into(), + actions: vec!["replica-maintenance".into()], + }, + fixture.transport.clone(), + ); + // The same Active receiver can install a new request under the new + // writer. Its new acceptance must never substitute the original join. + fixture.boots[1] + .refresh_capacity(1, fixture.journal.as_ref(), deadline()) + .await + .unwrap(); + let ad = fixture + .directory + .load(session(1), clock().unwrap()) .await .unwrap() - .output, - 17 - ); - drop(new_reader); + .unwrap(); + fixture + .peer + .activate( + &fixture.target, + &fixture.directory, + ad.advertisement().clone(), + fixture.description, + ) + .await + .unwrap(); + let new_reader = fixture.managers[1] + .resolve(fixture.target.clone()) + .await + .unwrap(); + assert_eq!(new_reader.receipt().await, retirement.receipt()); + assert_ne!( + fixture.managers[1] + .enrollment_completion(fixture.target.cell_id()) + .await + .unwrap() + .unwrap() + .spec, + *retirement.original().spec() + ); + assert_eq!( + new_reader + .query::(Some(retirement.receipt()), 0) + .await + .unwrap() + .output, + 17 + ); + drop(new_reader); + FleetSourceReaderRetirement::Native(retirement) + }; let inputs = Arc::new(FleetSourceReaderInputs { retirement, node: node_id(2), @@ -173,3 +270,26 @@ impl SourceFixture { } } } + +struct StoppedProcesses { + node: Arc, +} +impl FleetFailedBootProcesses for StoppedProcesses { + fn confirm_stopped<'a>( + &'a self, + request: &'a FleetFailedBootProcessRequest, + ) -> FleetAdapterFuture<'a, FleetFailedBootProcessEvidence> { + Box::pin(async move { + if self.node.state() != NodeState::Stopped { + return Err(std::io::Error::other("receiver process is still running").into()); + } + let mut witness = blake3::Hasher::new(); + witness.update(b"source-reader-process-closed\0"); + witness.update(request.digest().as_bytes()); + Ok(FleetFailedBootProcessEvidence::new( + request, + Digest::from_bytes(*witness.finalize().as_bytes()), + )?) + }) + } +} diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/mod.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/mod.rs index 65fbeb3d..51937741 100644 --- a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/mod.rs +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/mod.rs @@ -1,9 +1,11 @@ //! Actual managed writer handoff, exact native joining and current source policy. use super::*; use cellule_host::fleet::{ - FleetAdapterFuture, FleetJournalSnapshot, FleetMaintenanceEnrollments, FleetObservation, + FleetAdapterFuture, FleetFailedBootProcessEvidence, FleetFailedBootProcessRequest, + FleetFailedBootProcesses, FleetFailedBootRetirement, FleetFailedReaderRetirement, + FleetJournalSnapshot, FleetMaintenanceEnrollments, FleetObservation, FleetReaderEvacuationVerifier, FleetRoster, FleetSourceReaderInputs, FleetSourceReaderPolicies, - FleetSourceReaderSuccessors, + FleetSourceReaderRetirement, FleetSourceReaderSuccessors, }; use cellule_runtime::{fleet::operations::EnrollmentRole, read_policy::ReadPolicyStore}; use std::sync::atomic::{AtomicUsize, Ordering}; diff --git a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/tests.rs b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/tests.rs index d353bd4b..fcaf0201 100644 --- a/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/tests.rs +++ b/crates/cellule-host/minion/scenario/reader_tests/evacuation/source/tests.rs @@ -57,8 +57,14 @@ async fn source_reader_policy_binds_actual_final_root_writer_and_new_request_in_ assert_eq!(policies.checks().len(), 1); let check = &policies.checks()[0]; assert_eq!(check.retirement().original(), &source.fixture.original); - assert_eq!(check.origin().prefix(), source.inputs.retirement.root()); - assert_eq!(check.origin().root(), source.inputs.retirement.root()); + assert_eq!( + check.origin().prefix(), + source.inputs.retirement.root().unwrap() + ); + assert_eq!( + check.origin().root(), + source.inputs.retirement.root().unwrap() + ); assert_eq!(check.serving().owner().session, session(2)); let EnrollmentRole::Reader { position, .. } = &source.fixture.original.spec().role else { panic!("not reader") @@ -111,6 +117,42 @@ async fn source_reader_policy_binds_actual_final_root_writer_and_new_request_in_ source.finish().await; } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn failed_receiver_closure_still_requires_current_source_and_replacement_policy() { + let source = SourceFixture::failed_receiver().await; + let blocked = Provider::stable(source.inputs.clone()); + assert!(matches!( + source.collect(&blocked).await, + Err(Error::ReplicaUnavailable) + )); + assert_eq!(blocked.calls.load(Ordering::SeqCst), 1); + + let policies = ReadPolicyStore::new(source.fixture.layout.clone()); + let observed = policies + .load(source.fixture.target.cell_id()) + .await + .unwrap() + .unwrap(); + let updated = policies.update(&observed, 0).await.unwrap(); + assert_eq!(updated.value().desired_readers(), 0); + + let provider = Provider::stable(source.inputs.clone()); + let checked = source.collect(&provider).await.unwrap(); + assert_eq!(checked.checks().len(), 1); + let check = &checked.checks()[0]; + assert!(check.retirement().failed_process().is_some()); + assert_eq!(check.serving().owner().session, session(2)); + assert_eq!(check.desired_readers(), 0); + assert!(check.replacements().is_empty()); + assert_eq!( + check.origin().prefix(), + source.inputs.retirement.root().unwrap() + ); + assert!(check.origin().root().commit_sequence >= check.origin().prefix().commit_sequence); + assert_eq!(provider.calls.load(Ordering::SeqCst), 2); + source.finish().await; +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn source_reader_policy_missing_native_capsule_keeps_complete_original_obligation_unknown() { let source = SourceFixture::new().await; diff --git a/crates/cellule-host/minion/scenario/startup/mod.rs b/crates/cellule-host/minion/scenario/startup/mod.rs index 7776a5a8..a67bd484 100644 --- a/crates/cellule-host/minion/scenario/startup/mod.rs +++ b/crates/cellule-host/minion/scenario/startup/mod.rs @@ -35,6 +35,24 @@ pub(super) async fn advertisement( index: usize, node: &CellNode, intent: &NodeIntent, +) -> JournalResult { + advertisement_with_receive_capacity(index, node, intent, false).await +} + +#[cfg(test)] +pub(super) async fn advertisement_with_read_capacity( + index: usize, + node: &CellNode, + intent: &NodeIntent, +) -> JournalResult { + advertisement_with_receive_capacity(index, node, intent, true).await +} + +async fn advertisement_with_receive_capacity( + index: usize, + node: &CellNode, + intent: &NodeIntent, + receive_capacity: bool, ) -> JournalResult { // Boot discovery publishes no receive capacity before readiness. The // observer separately captures real signed capacity after startup. @@ -49,6 +67,14 @@ pub(super) async fn advertisement( .await??; let now = clock()?; let stats = node.stats(); + let memory_capacity = + u64::try_from(stats.resident_capacity_bytes() + stats.retained_capacity_bytes())?; + let memory_used = u64::try_from(stats.resident_bytes() + stats.retained_bytes())?; + let disk_free = stats + .local_disk_capacity_bytes() + .saturating_sub(stats.local_disk_reserved_bytes()); + let job_capacity = stats.placement_job_capacity(); + let jobs_running = stats.placement_running_jobs(); let key = SigningKey::from_bytes(&[index as u8 + 1; 32]); Ok(NodeAdvertisement::sign( intent.node(), @@ -66,20 +92,29 @@ pub(super) async fn advertisement( vec![1], NodeFailureDomain::default(), NodeCapacity { + free_memory_bytes: if receive_capacity { + memory_capacity.saturating_sub(memory_used) + } else { + 0 + }, + free_disk_bytes: if receive_capacity { disk_free } else { 0 }, + job_credits: if receive_capacity { + job_capacity.saturating_sub(jobs_running) + } else { + 0 + }, log_protocol: 1, - ..NodeCapacity::default() + ..Default::default() }, )? .with_operational_placement( cellule_runtime::node::NodePlacementCapacity { - memory_capacity_bytes: u64::try_from( - stats.resident_capacity_bytes() + stats.retained_capacity_bytes(), - )?, + memory_capacity_bytes: memory_capacity, disk_capacity_bytes: stats.local_disk_capacity_bytes(), active_cells: stats.placement_active_cells(), max_active_cells: stats.placement_active_cell_capacity(), - running_jobs: stats.placement_running_jobs(), - job_capacity: stats.placement_job_capacity(), + running_jobs: jobs_running, + job_capacity, ..Default::default() }, sample, diff --git a/crates/cellule-host/src/fleet/failed_boot/readers/mod.rs b/crates/cellule-host/src/fleet/failed_boot/readers/mod.rs index d34d088f..81ecedde 100644 --- a/crates/cellule-host/src/fleet/failed_boot/readers/mod.rs +++ b/crates/cellule-host/src/fleet/failed_boot/readers/mod.rs @@ -10,6 +10,7 @@ mod records; /// Source failure cannot authorize retirement on a live receiver. pub struct FleetFailedReaderRetirement { request: FleetFailedBootProcessRequest, + original: EnrollmentRecord, reader: EnrollmentRecord, } impl FleetFailedReaderRetirement { @@ -32,7 +33,11 @@ impl FleetFailedReaderRetirement { ) .await?; let reader = records::select(roster, &request, original)?; - Ok(Self { request, reader }) + Ok(Self { + request, + original: original.clone(), + reader, + }) } /// Full original head and registry barrier. #[must_use] @@ -45,6 +50,11 @@ impl FleetFailedReaderRetirement { pub const fn request(&self) -> &FleetFailedBootProcessRequest { &self.request } + /// Exact original reader acceptance supplied to this closure attempt. + #[must_use] + pub const fn original(&self) -> &EnrollmentRecord { + &self.original + } /// Original reader request, acceptance and retained establishment history. #[must_use] pub const fn reader(&self) -> &EnrollmentRecord { @@ -91,8 +101,11 @@ impl FleetFailedReaderPublication { /// Interval evidence that one original reader's receiver cannot execute again. /// This does not prove replacement redundancy, writer relocation, other role /// closure, boot retirement or completed physical maintenance. +#[derive(Clone)] pub struct FleetFailedReaderClosure { snapshot: FleetJournalSnapshot, + request: FleetFailedBootProcessRequest, + original: EnrollmentRecord, reader: EnrollmentRecord, process: FleetFailedBootProcessEvidence, digest: Digest, @@ -110,12 +123,43 @@ impl FleetFailedReaderClosure { pub const fn reader(&self) -> &EnrollmentRecord { &self.reader } + /// Exact original reader acceptance whose failed receiver was joined. + #[must_use] + pub const fn original(&self) -> &EnrollmentRecord { + &self.original + } + /// Exact process request confirmed with retained application evidence. + #[must_use] + pub const fn request(&self) -> &FleetFailedBootProcessRequest { + &self.request + } + /// Rechecks the original receiver boot against its current journal row. + /// An already Retired boot must carry the canonical event from this exact + /// process witness; a different witness cannot be substituted later. + pub fn validate_boot(&self, current: &EnrollmentRecord) -> Result<()> { + let original = self.request.boot(); + current + .validate_replay(original.spec()) + .map_err(operation)?; + if current.accepted_at_ms() != original.accepted_at_ms() + || current.established_evidence() != original.established_evidence() + || !matches!( + current.status(), + EnrollmentStatus::Established | EnrollmentStatus::Retired + ) + || (current.status() == EnrollmentStatus::Retired + && current.settlement_evidence() != Some(super::records::retirement(&self.process))) + { + return Err(Error::Fenced); + } + Ok(()) + } /// Application-authenticated original receiver lifetime evidence. #[must_use] pub const fn process(&self) -> &FleetFailedBootProcessEvidence { &self.process } - /// Stable identity of the original terminal row and process witness. + /// Stable identity of the exact original request, terminal row and process witness. #[must_use] pub const fn digest(&self) -> Digest { self.digest diff --git a/crates/cellule-host/src/fleet/failed_boot/readers/publication.rs b/crates/cellule-host/src/fleet/failed_boot/readers/publication.rs index 96b08376..836a3598 100644 --- a/crates/cellule-host/src/fleet/failed_boot/readers/publication.rs +++ b/crates/cellule-host/src/fleet/failed_boot/readers/publication.rs @@ -124,7 +124,9 @@ impl FleetFailedReaderRetirement { let finished_at_ms = clock()?; Ok(FleetFailedReaderClosure { snapshot, - digest: records::closure_digest(&reader, process)?, + request: self.request.clone(), + digest: records::closure_digest(&self.original, &reader, process)?, + original: self.original.clone(), reader, process: process.clone(), started_at_ms: self.request.started_at_ms, diff --git a/crates/cellule-host/src/fleet/failed_boot/readers/records.rs b/crates/cellule-host/src/fleet/failed_boot/readers/records.rs index c08c35c6..d0c8cc71 100644 --- a/crates/cellule-host/src/fleet/failed_boot/readers/records.rs +++ b/crates/cellule-host/src/fleet/failed_boot/readers/records.rs @@ -75,11 +75,13 @@ pub(super) fn result( } pub(super) fn closure_digest( + original: &EnrollmentRecord, reader: &EnrollmentRecord, process: &FleetFailedBootProcessEvidence, ) -> Result { let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-failed-reader-closure.v1\0"); + hash.update(b"cellule.fleet-failed-reader-closure.v2\0"); + hash.update(&original.to_bytes().map_err(operation)?); hash.update(&reader.to_bytes().map_err(operation)?); hash.update(retirement(reader, process)?.as_bytes()); Ok(Digest::from_bytes(*hash.finalize().as_bytes())) diff --git a/crates/cellule-host/src/fleet/mod.rs b/crates/cellule-host/src/fleet/mod.rs index bed93550..c0cab4d9 100644 --- a/crates/cellule-host/src/fleet/mod.rs +++ b/crates/cellule-host/src/fleet/mod.rs @@ -37,7 +37,8 @@ mod recovered; pub use reader_evacuation::{ FleetReaderEvacuationCandidate, FleetReaderEvacuationCheck, FleetReaderEvacuationJournal, FleetReaderEvacuationPublication, FleetReaderEvacuationVerifier, FleetSourceReaderCheck, - FleetSourceReaderInputs, FleetSourceReaderPolicies, FleetSourceReaderSuccessors, + FleetSourceReaderInputs, FleetSourceReaderPolicies, FleetSourceReaderRetirement, + FleetSourceReaderSuccessors, }; mod references; mod roster; diff --git a/crates/cellule-host/src/fleet/reader_evacuation/mod.rs b/crates/cellule-host/src/fleet/reader_evacuation/mod.rs index 55a913fe..e16e9246 100644 --- a/crates/cellule-host/src/fleet/reader_evacuation/mod.rs +++ b/crates/cellule-host/src/fleet/reader_evacuation/mod.rs @@ -9,7 +9,7 @@ mod current; mod source; pub use source::{ FleetSourceReaderCheck, FleetSourceReaderInputs, FleetSourceReaderPolicies, - FleetSourceReaderSuccessors, + FleetSourceReaderRetirement, FleetSourceReaderSuccessors, }; mod maintenance; mod publication; diff --git a/crates/cellule-host/src/fleet/reader_evacuation/source/collection.rs b/crates/cellule-host/src/fleet/reader_evacuation/source/collection.rs index 4b9dee68..98ac8e69 100644 --- a/crates/cellule-host/src/fleet/reader_evacuation/source/collection.rs +++ b/crates/cellule-host/src/fleet/reader_evacuation/source/collection.rs @@ -89,12 +89,17 @@ impl FleetReaderEvacuationVerifier { .map_err(provider)?; now()?; if let Some(inputs) = &inputs { - validate(row, request.original, inputs)?; + validate(row, request.original, inputs, roster)?; let (serving, writer_boot) = self.source_writer(inputs, roster, now()?).await?; - let root = serving.position().root.to_ltx( - inputs.retirement.receipt().cell, - serving.position().incarnation, - ); + let EnrollmentRole::Reader { target, position } = + &inputs.retirement.original().spec().role + else { + return Err(Error::Fenced); + }; + let root = serving + .position() + .root + .to_ltx(target.cell_id(), position.incarnation); let origin = inputs .host .runtime() @@ -102,14 +107,14 @@ impl FleetReaderEvacuationVerifier { &inputs.catalog, &inputs.authority, inputs.replica.clone(), - inputs.retirement.root(), + inputs.retirement.root()?, root, 10_000, ) .await?; let authority = self .authority - .load(inputs.retirement.receipt().cell) + .load(target.cell_id()) .await? .ok_or(Error::Fenced)?; let authority = authority.value().clone(); @@ -138,10 +143,7 @@ impl FleetReaderEvacuationVerifier { incarnation: authority.incarnation, commit_sequence: root .commit_sequence - .max(inputs.retirement.receipt().commit_sequence), - }; - let EnrollmentRole::Reader { target, .. } = &row.spec().role else { - return Err(Error::Fenced); + .max(inputs.retirement.minimum_commit_sequence()), }; let selected = self .directory @@ -342,6 +344,7 @@ fn validate( row: &EnrollmentRecord, accepted: Option<&EnrollmentRecord>, inputs: &FleetSourceReaderInputs, + roster: &FleetRoster, ) -> Result<()> { let original = inputs.retirement.original(); let EnrollmentRole::Reader { target, position } = &original.spec().role else { @@ -351,7 +354,8 @@ fn validate( Digest::from_bytes(*blake3::hash(&original.to_bytes().map_err(operation)?).as_bytes()); requests::validate_original(accepted, Some(digest))?; let entry = inputs.catalog.entry(); - let root = inputs.retirement.root(); + let root = inputs.retirement.root()?; + let minimum_commit_sequence = inputs.retirement.minimum_commit_sequence(); if inputs.retirement.retired() != row || row.spec() != original.spec() || row.accepted_at_ms() != original.accepted_at_ms() @@ -366,10 +370,35 @@ fn validate( ) || root.cell != *target.cell_id().as_bytes() || root.incarnation != *position.incarnation.as_bytes() - || root.commit_sequence != inputs.retirement.receipt().commit_sequence + || root.commit_sequence < minimum_commit_sequence + || inputs + .retirement + .native() + .is_some_and(|native| root.commit_sequence != native.receipt().commit_sequence) { return Err(Error::Fenced); } + if let Some(closure) = inputs.retirement.failed_process() + && (closure.reader() != row + || closure.original() != original + || closure.snapshot().head().scope() != row.spec().scope + || closure.request().boot().spec().scope != row.spec().scope + || closure.request().boot().spec().target.node != original.spec().target.node + || closure.request().boot().spec().target.session != original.spec().target.session + || closure.process().request_digest() != closure.request().digest()) + { + return Err(Error::Fenced); + } + if let Some(closure) = inputs.retirement.failed_process() { + let request_boot = closure.request().boot(); + let boot_key = request_boot.spec().key().map_err(operation)?; + let boot = roster + .enrollments() + .iter() + .find(|record| record.spec().key().is_ok_and(|key| key == boot_key)) + .ok_or(Error::Fenced)?; + closure.validate_boot(boot)?; + } Ok(()) } fn provider(source: Box) -> Error { diff --git a/crates/cellule-host/src/fleet/reader_evacuation/source/digest.rs b/crates/cellule-host/src/fleet/reader_evacuation/source/digest.rs index cf6ca09b..bbbc6d87 100644 --- a/crates/cellule-host/src/fleet/reader_evacuation/source/digest.rs +++ b/crates/cellule-host/src/fleet/reader_evacuation/source/digest.rs @@ -3,11 +3,11 @@ use super::*; use cellule_runtime::Result; impl FleetSourceReaderPolicies { - /// Binds full original/current rows, exact roots, native successor identity, - /// policy, ready replacements and the original fresh collection barrier. + /// Binds full original/current rows, closure evidence, exact roots, native + /// successor identity, policy, ready replacements and the original barrier. pub fn digest(&self) -> Result { let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-source-reader-policies.v1\0"); + hash.update(b"cellule.fleet-source-reader-policies.v2\0"); for bytes in [ self.snapshot.head().to_bytes().map_err(operation)?, self.snapshot.registry().to_bytes().map_err(operation)?, @@ -23,6 +23,29 @@ impl FleetSourceReaderPolicies { hash.update(&(self.checks.len() as u64).to_be_bytes()); for check in &self.checks { let retirement = check.retirement(); + match retirement { + FleetSourceReaderRetirement::Native(_) => { + hash.update(&[1]); + } + FleetSourceReaderRetirement::Failed(closure) => { + hash.update(&[2]); + hash.update(closure.digest().as_bytes()); + hash.update(closure.request().digest().as_bytes()); + hash.update(closure.process().request_digest().as_bytes()); + hash.update(closure.process().witness().as_bytes()); + for bytes in [ + closure.snapshot().head().to_bytes().map_err(operation)?, + closure + .snapshot() + .registry() + .to_bytes() + .map_err(operation)?, + ] { + hash.update(&(bytes.len() as u64).to_be_bytes()); + hash.update(&bytes); + } + } + } for row in [retirement.original(), retirement.retired()] { let bytes = row.to_bytes().map_err(operation)?; hash.update(&(bytes.len() as u64).to_be_bytes()); @@ -31,7 +54,7 @@ impl FleetSourceReaderPolicies { for time in [retirement.interval().0, retirement.interval().1] { hash.update(&time.to_be_bytes()); } - root(&mut hash, retirement.root()); + root(&mut hash, retirement.root()?); root(&mut hash, check.origin.prefix()); root(&mut hash, check.origin.root()); hash.update(&(check.origin.inspected_roots() as u64).to_be_bytes()); diff --git a/crates/cellule-host/src/fleet/reader_evacuation/source/mod.rs b/crates/cellule-host/src/fleet/reader_evacuation/source/mod.rs index b5ce15ff..0260640a 100644 --- a/crates/cellule-host/src/fleet/reader_evacuation/source/mod.rs +++ b/crates/cellule-host/src/fleet/reader_evacuation/source/mod.rs @@ -2,15 +2,17 @@ use super::*; use crate::{ CellNode, + fleet::FleetFailedReaderClosure, read_replicas::{ReaderEnrollmentRetirement, ReaderReplacement}, }; use cellule_runtime::{ + Error, Result, cell::{actor::CellServingObservation, catalog::CatalogProof}, control::{ Control, authority::{CellAuthority, VerifiedRootPrefix}, }, - fleet::operations::EnrollmentRecord, + fleet::operations::{EnrollmentRecord, EnrollmentRole}, identity::NodeId, ltx::CellReplica, }; @@ -19,12 +21,89 @@ use std::sync::Arc; mod collection; mod digest; -/// Authenticated mapping of one retained native retirement to its current writer. +/// Durable native or failed-process closure for one original reader request. +/// A failed-process closure proves nonexecution only; current source lineage and +/// replacement policy still have to pass the same successor checks. +#[derive(Clone)] +pub enum FleetSourceReaderRetirement { + /// Native removal joined all local query/refresh work and retained its final root. + Native(Arc), + /// Application-confirmed original process and accepted-work closure. + Failed(Arc), +} +impl FleetSourceReaderRetirement { + /// Exact first accepted reader row used by the retirement proof. + #[must_use] + pub fn original(&self) -> &EnrollmentRecord { + match self { + Self::Native(retirement) => retirement.original(), + Self::Failed(closure) => closure.original(), + } + } + /// Exact terminal row confirmed by the retirement proof. + #[must_use] + pub fn retired(&self) -> &EnrollmentRecord { + match self { + Self::Native(retirement) => retirement.retired(), + Self::Failed(closure) => closure.reader(), + } + } + /// Original bounded proof interval, distinct from the current lookup interval. + #[must_use] + pub fn interval(&self) -> (i64, i64) { + match self { + Self::Native(retirement) => retirement.interval(), + Self::Failed(closure) => closure.interval(), + } + } + /// Exact native final root, or the original failed reader's pinned opening root. + pub fn root(&self) -> Result { + match self { + Self::Native(retirement) => Ok(retirement.root()), + Self::Failed(closure) => { + let EnrollmentRole::Reader { target, position } = &closure.original().spec().role + else { + return Err(Error::Fenced); + }; + Ok(position.root.to_ltx(target.cell_id(), position.incarnation)) + } + } + } + /// Lowest prefix this exact retirement requires from its current successor. + pub fn minimum_commit_sequence(&self) -> u64 { + match self { + Self::Native(retirement) => retirement.receipt().commit_sequence, + Self::Failed(closure) => match &closure.original().spec().role { + EnrollmentRole::Reader { position, .. } => position.root.commit_sequence, + _ => 0, + }, + } + } + /// Failed-process evidence when the old receiver lifetime no longer exists. + #[must_use] + pub fn failed_process(&self) -> Option<&FleetFailedReaderClosure> { + match self { + Self::Native(_) => None, + Self::Failed(closure) => Some(closure.as_ref()), + } + } + /// Native join capsule when the old receiver remains available. + #[must_use] + pub fn native(&self) -> Option<&ReaderEnrollmentRetirement> { + match self { + Self::Native(retirement) => Some(retirement.as_ref()), + Self::Failed(_) => None, + } + } +} + +/// Authenticated mapping of one retained retirement to its current writer. /// Providers retain this exact allocation through their existing accepted-work -/// owner. Losing native evidence cannot be repaired from a terminal journal row. +/// owner. Failed receiver closures are not a substitute for current source or +/// replacement evidence. pub struct FleetSourceReaderInputs { - /// Exact locally joined original reader and last installed root. - pub retirement: Arc, + /// Exact native join or application-authenticated failed receiver closure. + pub retirement: FleetSourceReaderRetirement, /// Authenticated physical destination of the current native writer. pub node: NodeId, /// Actual existing managed host; collection starts no writer acquisition. @@ -37,10 +116,10 @@ pub struct FleetSourceReaderInputs { pub replica: CellReplica, } -/// Read-only application lookup of original native evidence and current writer. +/// Read-only application lookup of original closure evidence and current writer. /// Authenticate both physical endpoints and all canonical backend mappings. -/// Process restart exclusion and durable accepted-work retention remain separate -/// requirements; this interface cannot reconstruct a lost native join witness. +/// Process restart exclusion and durable accepted-work retention remain the +/// provider's responsibility; None remains unknown and cannot infer closure. pub trait FleetSourceReaderSuccessors: Send + Sync { /// None is unknown, never empty work. Return the same retained allocation on /// global recheck; substituting a new capsule or mapping fences collection. @@ -51,8 +130,8 @@ pub trait FleetSourceReaderSuccessors: Send + Sync { ) -> FleetAdapterFuture<'a, Option>>; } -/// Fresh current policy for one exact source-side native reader retirement. -/// This grants no original-writer/process/accepted-work or finalization rights. +/// Fresh current policy for one exact native or failed-process reader closure. +/// This grants no original-writer, accepted-work or finalization rights. pub struct FleetSourceReaderCheck { inputs: Arc, serving: CellServingObservation, @@ -64,9 +143,9 @@ pub struct FleetSourceReaderCheck { replacements: Vec, } impl FleetSourceReaderCheck { - /// Original charged local retirement retained throughout this observation. + /// Exact native/process retirement evidence retained throughout this observation. #[must_use] - pub fn retirement(&self) -> &ReaderEnrollmentRetirement { + pub fn retirement(&self) -> &FleetSourceReaderRetirement { &self.inputs.retirement } /// Authenticated physical successor. diff --git a/crates/cellule-host/src/fleet/reconciler/observation/mod.rs b/crates/cellule-host/src/fleet/reconciler/observation/mod.rs index e6413d84..32ca0a0b 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/mod.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/mod.rs @@ -220,7 +220,7 @@ impl FleetObservation { pub(super) fn digest(&self, now_ms: i64) -> Result { let nodes = self.placements(now_ms)?; let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-planner-inputs.v12\0"); + hash.update(b"cellule.fleet-planner-inputs.v13\0"); hash.update(self.scope.fleet.as_bytes()); hash.update(self.scope.application.as_bytes()); hash.update(&self.registry.to_bytes().map_err(super::operation)?); diff --git a/docs/fleet-operations-plan.md b/docs/fleet-operations-plan.md index 8dc17d36..f3fa251d 100644 --- a/docs/fleet-operations-plan.md +++ b/docs/fleet-operations-plan.md @@ -290,11 +290,11 @@ present API; the pseudocode below specifies the remaining orchestration. | `FollowerEvacuationRecord` / `FleetFollowerEvacuationJournal` / `FleetFollowerEvacuationVerifier` | Present bounded durable live-owner ensemble history, revisioned application policy, atomic latest pointers and canonical/native revalidation/refresh. | Bind every original Retired member and replacement request, confirm current canonical ensemble plus the actual source/member native owners through `FleetSnapshotTransport`. Refresh after policy/epoch/operation changes without repeating original retirement. Complete failed-owner successor policy, full observer and SettleRoles/Finalize integration; this per-owner evidence cannot finish maintenance. | | `FleetObservation::with_role_coverage` | Present retained graph attachment and planner-input binding. | Preserve the original interval and reject replacement or scope/registry mismatches. The reconciler rechecks the graph's exact head/registry and full roster digest; unchanged registry alone cannot admit an earlier controller head. Attachment never upgrades partial adapter coverage. | | `MaintenanceEnrollmentInventory`, `FleetJournal::maintenance_enrollments` and `FleetMaintenanceEnrollments::collect` | Present bounded original Pending/Established reader/follower pages frozen with the first BeginEvacuation CAS, at either physical endpoint including earlier boots. | Retain exact pretransition head, registry, operation and acceptance times. Traverse every page, match original acceptances against the fresh full roster, and attach with `FleetObservation::with_maintenance_enrollments`. Missing history remains unknown; zero requires a committed empty manifest. New source replacement work after capture remains in the current graph. All original and current policy/work obligations remain required before settlement. | -| `FleetObservation::with_role_evacuations` | Present retention of freshly rechecked durable reader/follower policy history, current authority/prefixes and native source/member evidence. Planner digest v12 binds complete original checks and the immutable maintenance enrollment manifest. | Require one exact full head, registry and roster, with every original fresh interval inside the outer observation. Compare signed replacement boots with the existing producer identities and refuse duplicate obligations. Public minion reconciliation consumes actual ready readers and rotated ensembles without upgrading partial coverage. Complete authenticated policy coverage, failed-owner successor policy and accepted-work/finalization barriers remain required. | -| `FleetMaintenanceNonexecution` / `FleetEnrollmentNonexecution` | Present independent read-only confirmation of retained original nonexecution, exact terminal witnesses and the complete original/current request set. Whole-capture clocks/deadlines and full head/registry rechecks refuse missing or changed evidence; planner digest v12 binds checked results. | Applications authenticate original endpoints, earlier native/external joining and durable exclusion evidence. Actual reader/follower cases consume joined original witnesses; production provider/process retention, installed-role succession, other accepted work and finalization remain required. | -| `FleetObservation::check_maintenance_policies` / `FleetReconcileReport::maintenance_policy` | Present bounded matching of every immutable original request and every currently unresolved related reader/follower against all supplied native policy checks. Exact current rows and Established donor original digests are required. Planner digest v12 binds source inputs and every explicit result. | Attach original capture and all policy checks before matching. Missing history refuses matching; missing policy, Pending/Established, source succession and unproven nonexecution remain explicit gaps. Advisory counts retain their own full head/registry/interval; absent coverage stays unknown. Complete source/failed-owner and nonexecution evidence, native/external accepted work and finalization remain required. | +| `FleetObservation::with_role_evacuations` | Present retention of freshly rechecked durable reader/follower policy history, current authority/prefixes and native source/member evidence. Planner digest v13 binds complete original checks and the immutable maintenance enrollment manifest. | Require one exact full head, registry and roster, with every original fresh interval inside the outer observation. Compare signed replacement boots with the existing producer identities and refuse duplicate obligations. Public minion reconciliation consumes actual ready readers and rotated ensembles without upgrading partial coverage. Complete authenticated policy coverage, failed-owner successor policy and accepted-work/finalization barriers remain required. | +| `FleetMaintenanceNonexecution` / `FleetEnrollmentNonexecution` | Present independent read-only confirmation of retained original nonexecution, exact terminal witnesses and the complete original/current request set. Whole-capture clocks/deadlines and full head/registry rechecks refuse missing or changed evidence; planner digest v13 binds checked results. | Applications authenticate original endpoints, earlier native/external joining and durable exclusion evidence. Actual reader/follower cases consume joined original witnesses; production provider/process retention, installed-role succession, other accepted work and finalization remain required. | +| `FleetObservation::check_maintenance_policies` / `FleetReconcileReport::maintenance_policy` | Present bounded matching of every immutable original request and every currently unresolved related reader/follower against all supplied native policy checks. Exact current rows and Established donor original digests are required. Planner digest v13 binds source inputs and every explicit result. | Attach original capture and all policy checks before matching. Missing history refuses matching; missing policy, Pending/Established, source succession and unproven nonexecution remain explicit gaps. Advisory counts retain their own full head/registry/interval; absent coverage stays unknown. Complete source/failed-owner and nonexecution evidence, native/external accepted work and finalization remain required. | | `FleetReaderEvacuationVerifier::collect_maintenance` / `FleetFollowerEvacuationVerifier::collect_maintenance` | Present complete eligible donor history lookup from the canonical original/current request set, followed by existing authenticated native/current-policy confirmation. Exact original digest/current row and full head/registry/roster checks are retained. | Collect original history first and construct the outer observation after both role collections. Missing latest history supplies no check and stays MissingPolicy in the matcher. One monotonic 30-second capture and caller deadline bound collection; changed heads/policies during native capture fail closed. Source/failed-owner succession, nonexecution, unknown native/external work and finalization remain required. | -| `FleetReaderEvacuationVerifier::collect_source_readers` / `FleetSourceReaderPolicies` / `FleetObservation::with_source_reader_policies` | Present exact locally joined source reader retirement composed with an actual managed native writer successor, canonical final-root/origin verification, current reader policy and ready enrolled replacement prefixes. The complete original/current request set, full head/registry/roster, repeated provider allocation/presence and monotonic capture are retained in planner identity v12. | Authenticate both endpoints and canonical backend mappings. Retain the original charged native capsule through the application's existing accepted-work owner. Missing evidence stays SourceSuccessor; checked exact native source requests become SourceReader. Local SQLite client reconstruction/public reconciliation cannot establish OS/process restart exclusion or durable provider retention. Complete failed-owner reader/follower succession, other accepted work and SettleRoles/Finalize remain required. | +| `FleetReaderEvacuationVerifier::collect_source_readers` / `FleetSourceReaderPolicies` / `FleetObservation::with_source_reader_policies` | Present exact native reader joins or failed-process reader closures composed with an actual managed native writer successor, minimum-prefix/origin verification, current reader policy and ready enrolled replacement prefixes. The complete original/current request set, full head/registry/roster, repeated provider allocation/presence and monotonic capture are retained in planner identity v13. | Authenticate both endpoints and canonical backend mappings. Retain the original native capsule or durable process closure through the application's existing accepted-work owner. Missing evidence stays SourceSuccessor; checked exact source requests become SourceReader. Local SQLite client reconstruction/public reconciliation cannot qualify OS/process restart exclusion or durable provider retention. Complete failed-owner follower succession, other accepted work and SettleRoles/Finalize remain required. | | `CellNode::follower_evacuation` | Present per-live-owner replacement and retirement check after the original requested rotation. | Require the declared member minimum, donor exclusion, complete Established replacement rows, pinned signed boots, current authority and full journal rechecks. Retain the original completion/error history and publish/revalidate its interval. This does not settle failed owners, Pending producers or the physical node's other obligations. | | Recovered-log authorization, member retirement and canonical Retired CAS | Present runtime tail closure after canonical recovery pinning. | Use `RecoveredNodeLogTransport`, receiver-side `authorize_recovered_log_retire` and `FollowerStore::retire_recovered`; confirm every original member before `retire_recovered_log`. Adopt exact committed closure with `retired_recovered_log` before repeating effects. Publish original enrollments through the host capsule, then complete failed-process barriers and maintenance orchestration; native closure alone cannot finish W7. | | `RecoveryManifestStore::load_manifest` | Complete digest-verified recovered-suffix metadata across every original application, retained after successor materialization. | Read the identity from canonical sealed-log authority, then verify each bundle through its application's ordinary store. Retain the complete original writer set separately, including object-covered Cells without a suffix; check current successor authority, acknowledged prefixes and serving. This metadata read cannot settle roles or finalize a node. | @@ -305,7 +305,7 @@ present API; the pseudocode below specifies the remaining orchestration. | `FleetRecoveredFollowerRetirement` | Present checked publication of recovered ensemble enrollment closure. | Capture the exact canonically Retired physical leader/epoch against the full roster, publish all original member requests through the existing journal, then confirm the complete new roster and canonical authority. Retain every original response/error. Fresh recapture adopts lost replies without native RPCs or timestamp refresh; failed boots, replacement policy and operation integration remain required. | | `FleetFailedBootRetirement` / `FleetFailedBootProcesses` | Present original failed-boot closure against canonical terminal fencing and durable application process evidence. `confirm` freshly rechecks an already committed retirement without effects. Process confirmation pins the committed request/witness binding. `FleetObservation::with_failed_boot_closures` retains the exact full barrier, original row/process and capture interval inside planner inputs. | Complete every original related request first. The read-only provider proves original process/accepted external work joining and session nonexecution; publication and confirmation share terminal roster/authority/reference/process checks. Public minion observation consumes both an empty-writer surviving graph and combined original-writer/closure evidence across two applications in either attachment order. Complete authenticated observation, actual provider qualification, replacement policy and maintenance actions remain required. | | `NodeDirectory::fenced_session` / `FleetFailedBootProcessRequest::capture_fenced` / `confirm` | Present process confirmation before leader-log recovery/retirement, using the existing application provider. | Authenticate and join the exact original process and all accepted work, then retain every affected Cell before recovery effects. The v2 identity survives recovery and claim adoption; `capture_retained` reuses it only after strict terminal boot/role checks. Existing terminal v1 identities remain unchanged. Process evidence cannot substitute for original writer inventory or current successor/policy evidence. | -| `FleetFailedReaderRetirement` / `FleetFailedBootProcessRequest::capture` | Present original failed receiver reader publication, including Pending and Established histories, through the existing journal/provider boundary. | Capture the original boot/process request before role settlement, then publish each exact reader only under joined receiver lifetime evidence. Source failure cannot close a live receiver. Persist replacement-policy evidence, handle live receivers through ordinary closure, and qualify actual OS/process/external-job providers before completing W7. | +| `FleetFailedReaderRetirement` / `FleetFailedBootProcessRequest::capture` | Present original failed receiver reader publication, including Pending and Established histories, through the existing journal/provider boundary. The source-reader collector now consumes this closure only alongside current writer lineage and current ready-reader policy; missing replacements remain blocking. | Retain production process and accepted-work evidence durably across controller/provider restarts. Handle live receivers through ordinary closure, complete follower succession, and qualify actual OS/process/external-job providers before completing W7. | | `FleetActionJournal` | Present in host `fleet/journal.rs`. | Implement acceptance, result publication, original-action lookup, and acquisition/recovery basis and evidence recording/lookup with the specified atomic and durable semantics. | | `FleetCellProvider::cell_inputs(spec)` | Present in host `fleet/cells.rs`. | Resolve metadata and canonical local inputs without performing an effect. | | `FleetCellProvider::recovery_inputs(spec)` | Present in host `fleet/cells.rs`. | Supply existing canonical failed-session proof and manifest access. Ordinary recovery establishes fencing and tail sealing independently. | diff --git a/docs/fleet-operations-progress.md b/docs/fleet-operations-progress.md index 9bc76c30..39dce403 100644 --- a/docs/fleet-operations-progress.md +++ b/docs/fleet-operations-progress.md @@ -62,13 +62,32 @@ physical process/provider retention and the complete W1–W10 scope remain requi The new [source reader recipe](../crates/cellule-host/docs/source-readers.md) documents provider ownership, collection/attachment and the runnable cases. +### Failed receiver source succession continuation + +`FleetSourceReaderRetirement` now accepts either the retained native reader +join or a `FleetFailedReaderClosure`. The collector checks the failed proof's +original reader, process request and current boot row, then verifies the original +opening root through the actual writer successor. Current replacement policy +still applies: desired count 1 with no available replacement blocks collection; +an explicit update to zero allows it. The source-policy subdigest moved to v2, +and planner input identity moved to v13 to bind the new proof variant. + +All 12 source-reader minion cases pass, including the new failed-receiver case; +all 10 failed-reader lifetime cases pass. The host crate passes all-target, +all-feature `cargo check`, and formatting passes. The fixtures prove the API +composition using process evidence tied to a stopped native reader; production +OS termination and external accepted-work providers remain unqualified. PR #56 +is currently mergeable with no reported conflicts. These follow-up edits are +local and have not been pushed yet. + ### Highest remaining work 1. Qualify the follow-up head in CI and both routing campaigns. Earlier follower-observation and overload failures still need independent diagnosis if reproduced; a later green run alone does not establish their cause. -2. Complete durable provider/process retention and failed-owner reader/follower - succession, full native/external accepted-work coverage and unknown outcomes. +2. Qualify production durable provider/process retention; complete failed-owner + follower succession, full native/external accepted-work coverage and unknown + outcomes. 3. Implement SettleRoles/Finalize through original action joining, native Stopped, withdrawal, retired boots and committed completion; finish Cron/Blob owners and primitive fault matrices. From 29a8d5f73b4c7807ebe0dafda0146686c1450d05 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 4 Oct 2026 02:22:55 -0700 Subject: [PATCH 3/5] update fleet operations progress --- docs/fleet-operations-progress.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/fleet-operations-progress.md b/docs/fleet-operations-progress.md index 39dce403..dd3e4ddc 100644 --- a/docs/fleet-operations-progress.md +++ b/docs/fleet-operations-progress.md @@ -76,9 +76,8 @@ All 12 source-reader minion cases pass, including the new failed-receiver case; all 10 failed-reader lifetime cases pass. The host crate passes all-target, all-feature `cargo check`, and formatting passes. The fixtures prove the API composition using process evidence tied to a stopped native reader; production -OS termination and external accepted-work providers remain unqualified. PR #56 -is currently mergeable with no reported conflicts. These follow-up edits are -local and have not been pushed yet. +OS termination and external accepted-work providers remain unqualified. The +follow-up is published on PR #56; hosted qualification is pending. ### Highest remaining work From 4d1020ecc40d253877f9117477fcaebadaa9b49f Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 4 Oct 2026 03:11:40 -0700 Subject: [PATCH 4/5] wire recovered follower maintenance evidence --- .../successors/observation/combined.rs | 175 ++++++++++++++- .../src/fleet/maintenance_policies/mod.rs | 116 +++++++++- .../reconciler/observation/evacuations.rs | 6 + .../reconciler/observation/failed_boots.rs | 10 + .../observation/maintenance_enrollments.rs | 8 + .../observation/maintenance_policies.rs | 23 +- .../src/fleet/reconciler/observation/mod.rs | 11 +- .../reconciler/observation/nonexecution.rs | 8 + .../observation/original_writers.rs | 8 + .../observation/recovered_followers.rs | 200 ++++++++++++++++++ .../reconciler/observation/source_readers.rs | 4 + .../cellule-host/src/fleet/recovered/mod.rs | 1 + docs/fleet-operations-plan.md | 13 +- docs/fleet-operations-progress.md | 40 ++++ 14 files changed, 604 insertions(+), 19 deletions(-) create mode 100644 crates/cellule-host/src/fleet/reconciler/observation/recovered_followers.rs diff --git a/crates/cellule-host/minion/scenario/recovered_followers/failed_boot/writer_tests/successors/observation/combined.rs b/crates/cellule-host/minion/scenario/recovered_followers/failed_boot/writer_tests/successors/observation/combined.rs index 5bbac1ae..95f441c4 100644 --- a/crates/cellule-host/minion/scenario/recovered_followers/failed_boot/writer_tests/successors/observation/combined.rs +++ b/crates/cellule-host/minion/scenario/recovered_followers/failed_boot/writer_tests/successors/observation/combined.rs @@ -1,10 +1,51 @@ //! Original acknowledged prefixes and stopped-boot proof share one full barrier. //! The scoped planner remains partial: this is no replacement/finalization grant. use super::*; -use cellule_host::fleet::{FleetFailedBootClosure, FleetFailedBootRetirement}; +use cellule_host::fleet::{ + FleetFailedBootClosure, FleetFailedBootRetirement, FleetMaintenanceEnrollments, + FleetMaintenancePolicyStatus, FleetRecoveredFollowerClosure, FleetRecoveredFollowerRetirement, +}; +use cellule_runtime::fleet::operations::MaintenanceEvent; impl SuccessorFixture { async fn retire_original(&self) { + self.retire_original_base().await; + } + + async fn retire_original_evidence( + &self, + ) -> (FleetRecoveredFollowerClosure, FleetFailedBootClosure) { + self.retire_original_base().await; + let boot = self.original_closure().await; + let base = &self.original.base; + let recovered = FleetRecoveredFollowerRetirement::capture( + base.journal.as_ref(), + &base.directory, + &base.roster().await, + &base.sealed, + session(1), + deadline(), + crate::scenario::clock, + ) + .await + .unwrap() + .publish( + base.journal.as_ref(), + &base.directory, + session(1), + deadline(), + crate::scenario::clock, + ) + .await + .unwrap() + .confirmed() + .unwrap() + .clone(); + assert_eq!(boot.snapshot(), recovered.snapshot()); + (recovered, boot) + } + + async fn retire_original_base(&self) { let base = &self.original.base; let proof = retire_recovered_members(base.transport.clone(), &base.sealed) .await @@ -54,6 +95,31 @@ impl SuccessorFixture { assert_eq!(base.failed_boot().await.status(), EnrollmentStatus::Retired); } + async fn begin_maintenance_evacuation(&self) { + let journal = &self.original.base.journal; + let mut snapshot = journal.load_snapshot(scope()).await.unwrap(); + let epoch = snapshot.head().controller().unwrap().epoch; + snapshot = journal + .compare_exchange( + &snapshot, + epoch, + crate::scenario::clock().unwrap(), + &JournalTransition::Maintenance(MaintenanceEvent::Cordoned), + ) + .await + .unwrap(); + let epoch = snapshot.head().controller().unwrap().epoch; + journal + .compare_exchange( + &snapshot, + epoch, + crate::scenario::clock().unwrap(), + &JournalTransition::Maintenance(MaintenanceEvent::BeginEvacuation), + ) + .await + .unwrap(); + } + async fn original_retirement(&self) -> FleetFailedBootRetirement { let base = &self.original.base; FleetFailedBootRetirement::capture_retained( @@ -221,6 +287,113 @@ async fn combined_original_observation_reconciles_both_attachment_orders_without } } +#[tokio::test] +async fn failed_owner_follower_maintenance_matches_recovered_epoch_and_joined_process() { + let fixture = fixture().await; + fixture.begin_maintenance_evacuation().await; + let (recovered, boot) = fixture.retire_original_evidence().await; + let recovered_members = recovered.members().len(); + let inventory = fixture.collect_current().await.unwrap(); + assert_eq!(recovered.snapshot(), inventory.original().snapshot()); + let (nodes, cells) = fixture.observation_parts(&inventory).await; + let snapshot = fixture + .original + .base + .journal + .load_snapshot(scope()) + .await + .unwrap(); + let roster = FleetRoster::collect( + fixture.original.base.journal.as_ref(), + &snapshot, + deadline(), + ) + .await + .unwrap(); + let original = FleetMaintenanceEnrollments::collect( + fixture.original.base.journal.as_ref(), + &roster, + deadline(), + crate::scenario::clock, + ) + .await + .unwrap(); + let observation = fixture + .combined_observation(&inventory, &boot, nodes, cells) + .with_failed_boot_closures(vec![boot]) + .unwrap() + .with_recovered_follower_closures(vec![recovered]) + .unwrap() + .with_original_writer_successors(inventory) + .unwrap() + .with_maintenance_enrollments(original) + .unwrap() + .check_maintenance_policies(&roster, crate::scenario::clock().unwrap()) + .unwrap(); + let coverage = observation.maintenance_policy_coverage().unwrap(); + let progress = coverage.progress(); + assert!(coverage.is_complete()); + assert_eq!(progress.required, recovered_members); + assert_eq!(progress.checked, recovered_members); + assert_eq!(progress.recovered_followers, recovered_members); + assert_eq!(progress.source_successors, 0); + assert!(coverage.obligations().iter().all(|obligation| matches!( + obligation.status(), + FleetMaintenancePolicyStatus::RecoveredFollower(_) + ))); + fixture.close().await; +} + +#[tokio::test] +async fn recovered_follower_alone_does_not_close_failed_owner_policy_obligations() { + let fixture = fixture().await; + fixture.begin_maintenance_evacuation().await; + let (recovered, boot) = fixture.retire_original_evidence().await; + let recovered_members = recovered.members().len(); + let inventory = fixture.collect_current().await.unwrap(); + let (nodes, cells) = fixture.observation_parts(&inventory).await; + let snapshot = fixture + .original + .base + .journal + .load_snapshot(scope()) + .await + .unwrap(); + let roster = FleetRoster::collect( + fixture.original.base.journal.as_ref(), + &snapshot, + deadline(), + ) + .await + .unwrap(); + let original = FleetMaintenanceEnrollments::collect( + fixture.original.base.journal.as_ref(), + &roster, + deadline(), + crate::scenario::clock, + ) + .await + .unwrap(); + let observation = fixture + .combined_observation(&inventory, &boot, nodes, cells) + .with_recovered_follower_closures(vec![recovered]) + .unwrap() + .with_original_writer_successors(inventory) + .unwrap() + .with_maintenance_enrollments(original) + .unwrap() + .check_maintenance_policies(&roster, crate::scenario::clock().unwrap()) + .unwrap(); + let coverage = observation.maintenance_policy_coverage().unwrap(); + let progress = coverage.progress(); + assert!(!coverage.is_complete()); + assert_eq!(progress.required, recovered_members); + assert_eq!(progress.checked, 0); + assert_eq!(progress.recovered_followers, 0); + assert_eq!(progress.source_successors, recovered_members); + fixture.close().await; +} + #[tokio::test] async fn combined_original_observation_refuses_changed_full_head_with_same_registry_in_both_orders() { diff --git a/crates/cellule-host/src/fleet/maintenance_policies/mod.rs b/crates/cellule-host/src/fleet/maintenance_policies/mod.rs index 44ff2391..bec32635 100644 --- a/crates/cellule-host/src/fleet/maintenance_policies/mod.rs +++ b/crates/cellule-host/src/fleet/maintenance_policies/mod.rs @@ -1,7 +1,8 @@ //! Complete original/current request matching for maintenance replacement policy. use super::{ - FleetFollowerEvacuationCheck, FleetJournalSnapshot, FleetMaintenanceEnrollments, - FleetReaderEvacuationCheck, FleetRoster, FleetSourceReaderPolicies, operation, + FleetFailedBootClosure, FleetFollowerEvacuationCheck, FleetJournalSnapshot, + FleetMaintenanceEnrollments, FleetReaderEvacuationCheck, FleetRecoveredFollowerClosure, + FleetRoster, FleetSourceReaderPolicies, operation, }; use cellule_runtime::fleet::operations::{EnrollmentRecord, EnrollmentStatus, RegistryVersion}; use cellule_runtime::identity::Digest; @@ -26,6 +27,9 @@ pub enum FleetMaintenancePolicyStatus { Nonexecution(Digest), /// Exact native source reader joined with current successor and reader policy. SourceReader(Digest), + /// Original follower owner failed, its canonical log was recovered and + /// retired, and the exact failed process boot was joined. + RecoveredFollower(Digest), /// Original acceptance still has an unknown native outcome. Pending, /// An installed original role has not been retired with checked replacement policy. @@ -43,7 +47,11 @@ impl FleetMaintenancePolicyStatus { pub const fn is_checked(self) -> bool { matches!( self, - Self::Reader(_) | Self::Follower(_) | Self::Nonexecution(_) | Self::SourceReader(_) + Self::Reader(_) + | Self::Follower(_) + | Self::Nonexecution(_) + | Self::SourceReader(_) + | Self::RecoveredFollower(_) ) } } @@ -88,6 +96,8 @@ pub struct FleetMaintenancePolicyProgress { pub checked: usize, /// Requests independently confirmed never to have installed a native role. pub nonexecution: usize, + /// Failed-owner follower requests closed by canonical recovery and process proof. + pub recovered_followers: usize, /// Unknown accepted native outcomes. pub pending: usize, /// Installed roles awaiting checked retirement/replacement. @@ -114,16 +124,30 @@ pub struct FleetMaintenancePolicyCoverage { obligations: Vec, digest: Digest, } + +/// Checked supplemental evidence retained by one maintenance-policy matcher. +pub(super) struct FleetMaintenancePolicyEvidence<'a> { + pub(super) nonexecution: Option<&'a FleetMaintenanceNonexecution>, + pub(super) source_readers: Option<&'a FleetSourceReaderPolicies>, + pub(super) failed_boots: &'a [FleetFailedBootClosure], + pub(super) recovered_followers: &'a [FleetRecoveredFollowerClosure], +} + impl FleetMaintenancePolicyCoverage { pub(in crate::fleet) fn check( original: &FleetMaintenanceEnrollments, roster: &FleetRoster, readers: &[FleetReaderEvacuationCheck], followers: &[FleetFollowerEvacuationCheck], - nonexecution: Option<&FleetMaintenanceNonexecution>, - source_readers: Option<&FleetSourceReaderPolicies>, + evidence: FleetMaintenancePolicyEvidence<'_>, inputs: Digest, ) -> Result { + let FleetMaintenancePolicyEvidence { + nonexecution, + source_readers, + failed_boots, + recovered_followers, + } = evidence; let requests = requests::required(original, roster)?; let node = original.original().operation().node(); let mut witnesses = BTreeMap::new(); @@ -183,6 +207,69 @@ impl FleetMaintenancePolicyCoverage { )?; } } + for closure in recovered_followers { + if closure.leader_node() != node { + continue; + } + if closure.snapshot() != roster.snapshot() { + return Err(Error::Fenced); + } + let leader_session = closure.retired().session(); + let Some(boot) = failed_boots.iter().find(|proof| { + let target = proof.boot().spec().target; + target.node == node && target.session == leader_session + }) else { + // Retired log authority does not prove the old process stopped. + // Keep its source-side requests in SourceSuccessor until joined. + continue; + }; + if boot.snapshot() != roster.snapshot() + || boot.boot().status() != EnrollmentStatus::Retired + || boot.canonical().node() != node + || boot.canonical().session() != leader_session + || boot.canonical().log() != Some(closure.retired().log()) + { + return Err(Error::Fenced); + } + let mut proof_hash = blake3::Hasher::new(); + proof_hash.update(b"cellule.fleet-maintenance-recovered-follower.v1\0"); + proof_hash.update(closure.digest().as_bytes()); + proof_hash.update(boot.digest().as_bytes()); + let proof = Digest::from_bytes(*proof_hash.finalize().as_bytes()); + for row in closure.members() { + let source = row.spec().source.ok_or(Error::Fenced)?; + if source.node != node || source.session != leader_session { + return Err(Error::Fenced); + } + let key = row.spec().key().map_err(operation)?; + if !roster.enrollments().iter().any(|current| { + current + .spec() + .key() + .is_ok_and(|current_key| current_key == key) + && current == row + }) { + return Err(Error::Fenced); + } + let accepted = original.entries().find(|accepted| { + accepted + .spec() + .key() + .is_ok_and(|accepted_key| accepted_key == key) + }); + let accepted_digest = accepted + .filter(|accepted| accepted.status() == EnrollmentStatus::Established) + .map(enrollment_digest) + .transpose()?; + requests::validate_original(accepted, accepted_digest)?; + witness_optional( + &mut witnesses, + row, + accepted_digest, + FleetMaintenancePolicyStatus::RecoveredFollower(proof), + )?; + } + } let mut obligations = Vec::with_capacity(requests.len()); for request in requests { let accepted = request.original; @@ -210,6 +297,17 @@ impl FleetMaintenancePolicyCoverage { requests::validate_original(accepted, *digest)?; FleetMaintenancePolicyStatus::SourceReader(*proof) } + Some(( + retired, + digest, + status @ FleetMaintenancePolicyStatus::RecoveredFollower(_), + )) => { + if row != *retired { + return Err(Error::Fenced); + } + requests::validate_original(accepted, *digest)?; + *status + } _ => FleetMaintenancePolicyStatus::SourceSuccessor, }, _ if row.established_evidence().is_none() => { @@ -242,7 +340,7 @@ impl FleetMaintenancePolicyCoverage { (interval.0.min(next.0), interval.1.max(next.1)) }); let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-maintenance-policy-coverage.v2\0"); + hash.update(b"cellule.fleet-maintenance-policy-coverage.v3\0"); hash.update(inputs.as_bytes()); hash.update(&(obligations.len() as u64).to_be_bytes()); for obligation in &obligations { @@ -261,6 +359,7 @@ impl FleetMaintenancePolicyCoverage { FleetMaintenancePolicyStatus::UnprovenNonexecution => (7, None), FleetMaintenancePolicyStatus::Nonexecution(proof) => (8, Some(proof)), FleetMaintenancePolicyStatus::SourceReader(proof) => (9, Some(proof)), + FleetMaintenancePolicyStatus::RecoveredFollower(proof) => (10, Some(proof)), }; hash.update(&[tag]); if let Some(proof) = proof { @@ -311,6 +410,7 @@ impl FleetMaintenancePolicyCoverage { required: self.obligations.len(), checked: 0, nonexecution: 0, + recovered_followers: 0, pending: 0, established: 0, missing_policy: 0, @@ -322,6 +422,10 @@ impl FleetMaintenancePolicyCoverage { FleetMaintenancePolicyStatus::Reader(_) | FleetMaintenancePolicyStatus::Follower(_) | FleetMaintenancePolicyStatus::SourceReader(_) => progress.checked += 1, + FleetMaintenancePolicyStatus::RecoveredFollower(_) => { + progress.checked += 1; + progress.recovered_followers += 1; + } FleetMaintenancePolicyStatus::Nonexecution(_) => { progress.checked += 1; progress.nonexecution += 1; diff --git a/crates/cellule-host/src/fleet/reconciler/observation/evacuations.rs b/crates/cellule-host/src/fleet/reconciler/observation/evacuations.rs index b619f4cf..4b495d14 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/evacuations.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/evacuations.rs @@ -83,6 +83,12 @@ impl FleetObservation { .and_then(|proofs| proofs.first()) .map(FleetFailedBootClosure::snapshot) }) + .or_else(|| { + self.recovered_follower_closures + .as_ref() + .and_then(|proofs| proofs.first()) + .map(FleetRecoveredFollowerClosure::snapshot) + }) .or_else(|| first.map(|(snapshot, _)| snapshot)); let roster = match &self.roster { Some(roster) => Some(roster.digest()?), diff --git a/crates/cellule-host/src/fleet/reconciler/observation/failed_boots.rs b/crates/cellule-host/src/fleet/reconciler/observation/failed_boots.rs index 43242c64..449cb602 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/failed_boots.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/failed_boots.rs @@ -45,6 +45,12 @@ impl FleetObservation { .as_ref() .map(|inventory| inventory.original().snapshot()) }) + .or_else(|| { + self.recovered_follower_closures + .as_ref() + .and_then(|proofs| proofs.first()) + .map(FleetRecoveredFollowerClosure::snapshot) + }) .or_else(|| closures.first().map(FleetFailedBootClosure::snapshot)); for closure in closures { let snapshot = closure.snapshot(); @@ -64,6 +70,10 @@ impl FleetObservation { .cells .iter() .any(|owned| owned.session == endpoint.session) + || self + .recovered_follower_closures + .as_ref() + .is_some_and(|proofs| proofs.iter().any(|proof| proof.snapshot() != snapshot)) { return Err(Error::Node("failed boot observation barrier differs")); } diff --git a/crates/cellule-host/src/fleet/reconciler/observation/maintenance_enrollments.rs b/crates/cellule-host/src/fleet/reconciler/observation/maintenance_enrollments.rs index dc26a930..3293dcc3 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/maintenance_enrollments.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/maintenance_enrollments.rs @@ -49,6 +49,14 @@ impl FleetObservation { .iter() .any(|closure| closure.snapshot() != snapshot) }) + || self + .recovered_follower_closures + .as_ref() + .is_some_and(|closures| { + closures + .iter() + .any(|closure| closure.snapshot() != snapshot) + }) || self .reader_evacuations() .is_some_and(|checks| checks.iter().any(|check| check.snapshot() != snapshot)) diff --git a/crates/cellule-host/src/fleet/reconciler/observation/maintenance_policies.rs b/crates/cellule-host/src/fleet/reconciler/observation/maintenance_policies.rs index f090d9b9..b24a83de 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/maintenance_policies.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/maintenance_policies.rs @@ -21,8 +21,12 @@ impl FleetObservation { roster, self.reader_evacuations().unwrap_or(&[]), self.follower_evacuations().unwrap_or(&[]), - self.maintenance_nonexecution.as_ref(), - self.source_reader_policies.as_ref(), + super::super::super::maintenance_policies::FleetMaintenancePolicyEvidence { + nonexecution: self.maintenance_nonexecution.as_ref(), + source_readers: self.source_reader_policies.as_ref(), + failed_boots: self.failed_boot_closures.as_deref().unwrap_or(&[]), + recovered_followers: self.recovered_follower_closures.as_deref().unwrap_or(&[]), + }, self.maintenance_policy_inputs()?, )?; self.maintenance_policies = Some(coverage); @@ -52,10 +56,23 @@ impl FleetObservation { fn maintenance_policy_inputs(&self) -> Result { let original = self.maintenance_enrollments.as_ref().ok_or(Error::Fenced)?; let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-maintenance-policy-inputs.v3\0"); + hash.update(b"cellule.fleet-maintenance-policy-inputs.v4\0"); hash.update(original.digest()?.as_bytes()); self.hash_role_evacuations(&mut hash)?; self.hash_source_reader_policies(&mut hash)?; + hash.update(&[u8::from(self.failed_boot_closures.is_some())]); + if let Some(closures) = &self.failed_boot_closures { + let mut ordered = closures.iter().collect::>(); + ordered.sort_by_key(|closure| { + let target = closure.boot().spec().target; + (*target.node.as_bytes(), *target.session.as_bytes()) + }); + hash.update(&(ordered.len() as u64).to_be_bytes()); + for closure in ordered { + hash.update(closure.digest().as_bytes()); + } + } + self.hash_recovered_follower_closures(&mut hash)?; hash.update(&[u8::from(self.maintenance_nonexecution.is_some())]); if let Some(nonexecution) = &self.maintenance_nonexecution { hash.update(nonexecution.digest().as_bytes()); diff --git a/crates/cellule-host/src/fleet/reconciler/observation/mod.rs b/crates/cellule-host/src/fleet/reconciler/observation/mod.rs index 32ca0a0b..383a0277 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/mod.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/mod.rs @@ -2,8 +2,8 @@ use std::collections::{HashMap, HashSet}; use crate::fleet::{ FleetFailedBootClosure, FleetMaintenanceEnrollments, FleetMaintenanceNonexecution, - FleetMaintenancePolicyCoverage, FleetOriginalWriterSuccessorInventory, FleetRoleCoverage, - FleetRoster, FleetSourceReaderPolicies, + FleetMaintenancePolicyCoverage, FleetOriginalWriterSuccessorInventory, + FleetRecoveredFollowerClosure, FleetRoleCoverage, FleetRoster, FleetSourceReaderPolicies, }; use cellule_runtime::cell::actor::OwnedCellObservation; use cellule_runtime::fleet::operations::{FleetScope, RegistryVersion}; @@ -45,6 +45,7 @@ pub struct FleetObservation { role_coverage: Option, original_writer_successors: Option, failed_boot_closures: Option>, + recovered_follower_closures: Option>, role_evacuations: Option, maintenance_enrollments: Option, maintenance_policies: Option, @@ -81,6 +82,7 @@ impl FleetObservation { role_coverage: None, original_writer_successors: None, failed_boot_closures: None, + recovered_follower_closures: None, role_evacuations: None, maintenance_enrollments: None, maintenance_policies: None, @@ -117,6 +119,7 @@ impl FleetObservation { self.validate_source_reader_policies()?; self.validate_original_writer_successors()?; self.validate_failed_boot_closures()?; + self.validate_recovered_follower_closures()?; self.validate_role_evacuations()?; if let Some(coverage) = &self.role_coverage { let (started, finished) = coverage.interval(); @@ -220,7 +223,7 @@ impl FleetObservation { pub(super) fn digest(&self, now_ms: i64) -> Result { let nodes = self.placements(now_ms)?; let mut hash = blake3::Hasher::new(); - hash.update(b"cellule.fleet-planner-inputs.v13\0"); + hash.update(b"cellule.fleet-planner-inputs.v14\0"); hash.update(self.scope.fleet.as_bytes()); hash.update(self.scope.application.as_bytes()); hash.update(&self.registry.to_bytes().map_err(super::operation)?); @@ -274,6 +277,7 @@ impl FleetObservation { } } } + self.hash_recovered_follower_closures(&mut hash)?; self.hash_role_evacuations(&mut hash)?; self.hash_source_reader_policies(&mut hash)?; hash.update(&[u8::from(self.maintenance_nonexecution.is_some())]); @@ -388,6 +392,7 @@ mod maintenance_enrollments; mod maintenance_policies; mod nonexecution; mod original_writers; +mod recovered_followers; mod source_readers; #[cfg(test)] mod tests; diff --git a/crates/cellule-host/src/fleet/reconciler/observation/nonexecution.rs b/crates/cellule-host/src/fleet/reconciler/observation/nonexecution.rs index d29cd91d..589587be 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/nonexecution.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/nonexecution.rs @@ -61,6 +61,14 @@ impl FleetObservation { .iter() .any(|closure| closure.snapshot() != snapshot) }) + || self + .recovered_follower_closures + .as_ref() + .is_some_and(|closures| { + closures + .iter() + .any(|closure| closure.snapshot() != snapshot) + }) || self.reader_evacuations().is_some_and(|roles| { roles.iter().any(|check| { check.snapshot() != snapshot || check.roster_digest() != checks.roster_digest() diff --git a/crates/cellule-host/src/fleet/reconciler/observation/original_writers.rs b/crates/cellule-host/src/fleet/reconciler/observation/original_writers.rs index cdee14b5..9e40938d 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/original_writers.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/original_writers.rs @@ -46,6 +46,14 @@ impl FleetObservation { .role_coverage .as_ref() .is_some_and(|coverage| coverage.snapshot() != original.snapshot()) + || self + .recovered_follower_closures + .as_ref() + .is_some_and(|closures| { + closures + .iter() + .any(|closure| closure.snapshot() != original.snapshot()) + }) { return Err(Error::Node("original writer observation barrier differs")); } diff --git a/crates/cellule-host/src/fleet/reconciler/observation/recovered_followers.rs b/crates/cellule-host/src/fleet/reconciler/observation/recovered_followers.rs new file mode 100644 index 00000000..e7fe266d --- /dev/null +++ b/crates/cellule-host/src/fleet/reconciler/observation/recovered_followers.rs @@ -0,0 +1,200 @@ +//! Canonical recovered follower retirement retained at the maintenance barrier. +use super::*; +use crate::fleet::{FleetFollowerEvacuationCheck, FleetReaderEvacuationCheck}; +use cellule_runtime::fleet::operations::{EnrollmentRole, EnrollmentStatus}; +use cellule_runtime::node::log_state::NodeLogPhase; + +impl FleetObservation { + /// Retains exact canonical recovered follower closures for maintenance + /// matching. A failed source additionally needs its matching failed-boot + /// process closure before the source-side request is considered checked. + /// This does not upgrade `complete` or grant finalization rights. + pub fn with_recovered_follower_closures( + mut self, + closures: Vec, + ) -> Result { + if self.recovered_follower_closures.is_some() { + return Err(Error::Control( + "recovered follower closures already retained", + )); + } + self.recovered_follower_closures = Some(closures); + self.validate_role_coverage()?; + Ok(self) + } + + /// Canonically retired recovered epochs at this observation barrier. + #[must_use] + pub fn recovered_follower_closures(&self) -> Option<&[FleetRecoveredFollowerClosure]> { + self.recovered_follower_closures.as_deref() + } + + pub(super) fn validate_recovered_follower_closures(&self) -> Result<()> { + let Some(closures) = &self.recovered_follower_closures else { + return Ok(()); + }; + if closures.len() > 10_000 { + return Err(Error::Capacity("recovered follower closure bound exceeded")); + } + let first = closures.first(); + let expected = self + .roster + .as_ref() + .map(FleetRoster::snapshot) + .or_else(|| self.role_coverage.as_ref().map(FleetRoleCoverage::snapshot)) + .or_else(|| { + self.original_writer_successors + .as_ref() + .map(|proof| proof.original().snapshot()) + }) + .or_else(|| { + self.failed_boot_closures + .as_ref() + .and_then(|proofs| proofs.first()) + .map(FleetFailedBootClosure::snapshot) + }) + .or_else(|| { + self.maintenance_enrollments + .as_ref() + .map(FleetMaintenanceEnrollments::snapshot) + }) + .or_else(|| { + self.maintenance_nonexecution + .as_ref() + .map(FleetMaintenanceNonexecution::snapshot) + }) + .or_else(|| { + self.source_reader_policies + .as_ref() + .map(FleetSourceReaderPolicies::snapshot) + }) + .or_else(|| { + self.reader_evacuations() + .and_then(|rows| rows.first()) + .map(FleetReaderEvacuationCheck::snapshot) + .or_else(|| { + self.follower_evacuations() + .and_then(|rows| rows.first()) + .map(FleetFollowerEvacuationCheck::snapshot) + }) + }) + .or_else(|| first.map(FleetRecoveredFollowerClosure::snapshot)); + let mut epochs = HashSet::new(); + for closure in closures { + let sealed = closure.retired(); + let interval = closure.interval(); + if closure.snapshot().head().scope() != self.scope + || closure.snapshot().registry() != self.registry + || expected != Some(closure.snapshot()) + || interval.0 < self.capture_started_at_ms + || interval.1 > self.capture_finished_at_ms + || interval.1 < interval.0 + || sealed.log().phase() != NodeLogPhase::Retired + || closure.members().is_empty() + || closure.members().len() != sealed.log().members().len() + || !epochs.insert(( + closure.leader_node(), + sealed.session(), + sealed.log().epoch(), + )) + { + return Err(Error::Node( + "recovered follower observation barrier differs", + )); + } + let mut targets = HashSet::new(); + for row in closure.members() { + let source = row.spec().source.ok_or(Error::Fenced)?; + let EnrollmentRole::Follower { log_epoch } = row.spec().role else { + return Err(Error::Fenced); + }; + if row.spec().scope != self.scope + || source.node != closure.leader_node() + || source.session != sealed.session() + || log_epoch != sealed.log().epoch() + || row.status() != EnrollmentStatus::Retired + || row.settlement_evidence().is_none() + || !sealed.log().members().contains(&row.spec().target.node) + || !targets.insert(row.spec().target.node) + { + return Err(Error::Fenced); + } + if let Some(roster) = &self.roster { + let key = row.spec().key().map_err(super::super::operation)?; + if roster + .enrollments() + .iter() + .find(|current| current.spec().key().is_ok_and(|value| value == key)) + != Some(row) + { + return Err(Error::Node( + "recovered follower row differs from current roster", + )); + } + } + } + if sealed + .log() + .members() + .iter() + .any(|member| !targets.contains(member)) + { + return Err(Error::Fenced); + } + if let Some(boot) = self.failed_boot_closures.as_ref().and_then(|boots| { + boots.iter().find(|proof| { + let target = proof.boot().spec().target; + target.node == closure.leader_node() && target.session == sealed.session() + }) + }) && (boot.snapshot() != closure.snapshot() + || boot.canonical().node() != closure.leader_node() + || boot.canonical().session() != sealed.session() + || boot.canonical().log() != Some(sealed.log())) + { + return Err(Error::Node( + "recovered follower and failed boot evidence differ", + )); + } + } + Ok(()) + } + + pub(super) fn hash_recovered_follower_closures(&self, hash: &mut blake3::Hasher) -> Result<()> { + let Some(closures) = &self.recovered_follower_closures else { + hash.update(&[0]); + return Ok(()); + }; + hash.update(&[1]); + let mut ordered = closures.iter().collect::>(); + ordered.sort_by_key(|closure| { + ( + *closure.leader_node().as_bytes(), + *closure.retired().session().as_bytes(), + closure.retired().log().epoch(), + ) + }); + hash.update(&(ordered.len() as u64).to_be_bytes()); + for closure in ordered { + hash.update(closure.digest().as_bytes()); + for bytes in [ + closure + .snapshot() + .head() + .to_bytes() + .map_err(super::super::operation)?, + closure + .snapshot() + .registry() + .to_bytes() + .map_err(super::super::operation)?, + ] { + hash.update(&(bytes.len() as u64).to_be_bytes()); + hash.update(&bytes); + } + for time in [closure.interval().0, closure.interval().1] { + hash.update(&time.to_be_bytes()); + } + } + Ok(()) + } +} diff --git a/crates/cellule-host/src/fleet/reconciler/observation/source_readers.rs b/crates/cellule-host/src/fleet/reconciler/observation/source_readers.rs index c46253c9..b88599dc 100644 --- a/crates/cellule-host/src/fleet/reconciler/observation/source_readers.rs +++ b/crates/cellule-host/src/fleet/reconciler/observation/source_readers.rs @@ -58,6 +58,10 @@ impl FleetObservation { .failed_boot_closures .as_ref() .is_some_and(|rows| rows.iter().any(|row| row.snapshot() != snapshot)) + || self + .recovered_follower_closures + .as_ref() + .is_some_and(|rows| rows.iter().any(|row| row.snapshot() != snapshot)) || self.maintenance_nonexecution.as_ref().is_some_and(|proof| { snapshot != proof.snapshot() || checks.roster_digest() != proof.roster_digest() }) diff --git a/crates/cellule-host/src/fleet/recovered/mod.rs b/crates/cellule-host/src/fleet/recovered/mod.rs index 1811a043..29e54cc1 100644 --- a/crates/cellule-host/src/fleet/recovered/mod.rs +++ b/crates/cellule-host/src/fleet/recovered/mod.rs @@ -151,6 +151,7 @@ impl FleetRecoveredFollowerPublication { /// Complete original ensemble enrollment closure at one checked journal barrier. /// This settles only these follower rows. The failed leader's boot, reader roles, /// affected Cells and native process lifetimes still require their own proofs. +#[derive(Clone)] pub struct FleetRecoveredFollowerClosure { snapshot: FleetJournalSnapshot, leader_node: NodeId, diff --git a/docs/fleet-operations-plan.md b/docs/fleet-operations-plan.md index f3fa251d..77a3f82d 100644 --- a/docs/fleet-operations-plan.md +++ b/docs/fleet-operations-plan.md @@ -37,7 +37,7 @@ same action and evidence contracts. | Which external facilities are required? | A linearizable journal, complete enrollment registry, authenticated management transport, and trusted local Cell inputs. The reference example supplies these behind the same contracts. | | What keeps operations simple? | One operation ID, one status contract, one application loop, bounded defaults, and one canonical host drain lane. | | What proves success? | Receipt-preserving actor activation on another node. Maintenance additionally requires settled role obligations, host Stopped, and session withdrawal. | -| What is the next concrete change? | Qualify the current head, including both routing profiles. Complete durable provider/process retention and failed-owner reader/follower succession. Native source reader policy now composes exact joining/final-root verification with an actual managed writer and current reader policy. Exact reader removal now guards the original request under the native activation lane and retains its joined prefix; the local source collector supplies current policy while durable process evidence remains separate. Fresh nonexecution confirmation now consumes independently joined original witnesses; terminal rows alone remain unknown. Native donor policy lookup discovers latest history through the existing authenticated reader/follower verifiers. Complete observation around original-writer successors, failed-boot closures and durable unknown work. Connect reader/follower replacement policy and accepted-work barriers to SettleRoles/Finalize. Keep cross-session recovery, process/provider and remaining failure/inspection gates in scope. | +| What is the next concrete change? | Qualify the current head, including both routing profiles. Complete durable provider/process retention and remaining failed-owner role succession. Native source reader policy now composes exact joining/final-root verification with an actual managed writer and current reader policy. Exact reader removal now guards the original request under the native activation lane and retains its joined prefix; the local source collector supplies current policy while durable process evidence remains separate. Fresh nonexecution confirmation now consumes independently joined original witnesses; terminal rows alone remain unknown. Native donor policy lookup discovers latest history through the existing authenticated reader/follower verifiers. Canonical recovered-follower retirement plus exact failed-boot closure now closes the matching source-side follower request, but a recovered log alone stays blocked. Complete observation around original-writer successors, failed-boot closures and durable unknown work. Connect reader/follower replacement policy and accepted-work barriers to SettleRoles/Finalize. Keep cross-session recovery, process/provider and remaining failure/inspection gates in scope. | Start with the [first slice commit sequence](#first-slice-commit-sequence). Every increment must expose a reviewable public behavior and retain its test @@ -287,14 +287,14 @@ present API; the pseudocode below specifies the remaining orchestration. | `FleetRoleCoverage` | Present cross-node original producer, native lane and current foreign authority matching. | Supply every required original boot and physical reference scan; finish initial collection, global native rechecks, then exact foreign rechecks and full journal confirmation. Observe enrolled lanes before their first append through the complete original producer/ensemble witness. Authentication, unexpected advertisements, current Cell authority, replacement policy and failed-process closure remain required; coverage does not grant settlement or finalization. | | `ReadReplicaManager::remove_enrolled` / `ReaderEnrollmentRetirement` | Present exact original native joining and confirmed terminal enrollment through the existing activation/removal/producer path. | Authenticate both endpoints and establish writer succession/current replacement policy before source maintenance removal. Retain original request/source, exact final installed root, joined receipt, terminal timestamps and bounded interval; cancelled/lost replies resume the same retained view/event. Verify the exact final root through canonical successor origin/lineage, rather than sequence counters. Newer requests for the same Cell are refused before closure. Durable process retention and fleet policy/work matching remain required. | | `ReaderEvacuationRecord` / `FleetReaderEvacuationJournal` / `FleetReaderEvacuationVerifier` | Present bounded durable reader policy history, complete replacement pages, atomic latest pointers and independent native revalidation/refresh. | Commit through the shared registry transaction; confirm exact policy, authority, enrolled selected boots and ready prefixes after reconstruction. Refresh from the original committed retirement after policy/boot/owner changes, including Closing. Failed-owner follower policy, failed receiver integration, complete fleet observation and aggregate maintenance actions remain required. | -| `FollowerEvacuationRecord` / `FleetFollowerEvacuationJournal` / `FleetFollowerEvacuationVerifier` | Present bounded durable live-owner ensemble history, revisioned application policy, atomic latest pointers and canonical/native revalidation/refresh. | Bind every original Retired member and replacement request, confirm current canonical ensemble plus the actual source/member native owners through `FleetSnapshotTransport`. Refresh after policy/epoch/operation changes without repeating original retirement. Complete failed-owner successor policy, full observer and SettleRoles/Finalize integration; this per-owner evidence cannot finish maintenance. | +| `FollowerEvacuationRecord` / `FleetFollowerEvacuationJournal` / `FleetFollowerEvacuationVerifier` | Present bounded durable live-owner ensemble history, revisioned application policy, atomic latest pointers and canonical/native revalidation/refresh. | Bind every original Retired member and replacement request, confirm current canonical ensemble plus the actual source/member native owners through `FleetSnapshotTransport`. Refresh after policy/epoch/operation changes without repeating original retirement. Complete failed-owner cases outside the recovered-log path, full observer and SettleRoles/Finalize integration; this per-owner evidence cannot finish maintenance. | | `FleetObservation::with_role_coverage` | Present retained graph attachment and planner-input binding. | Preserve the original interval and reject replacement or scope/registry mismatches. The reconciler rechecks the graph's exact head/registry and full roster digest; unchanged registry alone cannot admit an earlier controller head. Attachment never upgrades partial adapter coverage. | | `MaintenanceEnrollmentInventory`, `FleetJournal::maintenance_enrollments` and `FleetMaintenanceEnrollments::collect` | Present bounded original Pending/Established reader/follower pages frozen with the first BeginEvacuation CAS, at either physical endpoint including earlier boots. | Retain exact pretransition head, registry, operation and acceptance times. Traverse every page, match original acceptances against the fresh full roster, and attach with `FleetObservation::with_maintenance_enrollments`. Missing history remains unknown; zero requires a committed empty manifest. New source replacement work after capture remains in the current graph. All original and current policy/work obligations remain required before settlement. | -| `FleetObservation::with_role_evacuations` | Present retention of freshly rechecked durable reader/follower policy history, current authority/prefixes and native source/member evidence. Planner digest v13 binds complete original checks and the immutable maintenance enrollment manifest. | Require one exact full head, registry and roster, with every original fresh interval inside the outer observation. Compare signed replacement boots with the existing producer identities and refuse duplicate obligations. Public minion reconciliation consumes actual ready readers and rotated ensembles without upgrading partial coverage. Complete authenticated policy coverage, failed-owner successor policy and accepted-work/finalization barriers remain required. | -| `FleetMaintenanceNonexecution` / `FleetEnrollmentNonexecution` | Present independent read-only confirmation of retained original nonexecution, exact terminal witnesses and the complete original/current request set. Whole-capture clocks/deadlines and full head/registry rechecks refuse missing or changed evidence; planner digest v13 binds checked results. | Applications authenticate original endpoints, earlier native/external joining and durable exclusion evidence. Actual reader/follower cases consume joined original witnesses; production provider/process retention, installed-role succession, other accepted work and finalization remain required. | -| `FleetObservation::check_maintenance_policies` / `FleetReconcileReport::maintenance_policy` | Present bounded matching of every immutable original request and every currently unresolved related reader/follower against all supplied native policy checks. Exact current rows and Established donor original digests are required. Planner digest v13 binds source inputs and every explicit result. | Attach original capture and all policy checks before matching. Missing history refuses matching; missing policy, Pending/Established, source succession and unproven nonexecution remain explicit gaps. Advisory counts retain their own full head/registry/interval; absent coverage stays unknown. Complete source/failed-owner and nonexecution evidence, native/external accepted work and finalization remain required. | +| `FleetObservation::with_role_evacuations` | Present retention of freshly rechecked durable reader/follower policy history, current authority/prefixes and native source/member evidence. Planner digest v14 binds complete original checks and the immutable maintenance enrollment manifest. | Require one exact full head, registry and roster, with every original fresh interval inside the outer observation. Compare signed replacement boots with the existing producer identities and refuse duplicate obligations. Public minion reconciliation consumes actual ready readers and rotated ensembles without upgrading partial coverage. Complete authenticated policy coverage and accepted-work/finalization barriers remain required. | +| `FleetMaintenanceNonexecution` / `FleetEnrollmentNonexecution` | Present independent read-only confirmation of retained original nonexecution, exact terminal witnesses and the complete original/current request set. Whole-capture clocks/deadlines and full head/registry rechecks refuse missing or changed evidence; planner digest v14 binds checked results. | Applications authenticate original endpoints, earlier native/external joining and durable exclusion evidence. Actual reader/follower cases consume joined original witnesses; production provider/process retention, installed-role succession, other accepted work and finalization remain required. | +| `FleetObservation::check_maintenance_policies` / `FleetReconcileReport::maintenance_policy` | Present bounded matching of every immutable original request and every currently unresolved related reader/follower against all supplied native policy checks. Exact current rows and Established donor original digests are required. Planner digest v14 binds source inputs and every explicit result. An exact canonical recovered-follower closure joined with its failed-boot process closure closes only the corresponding failed-owner source requests. | Attach original capture and all policy checks before matching. Missing history refuses matching; missing policy, Pending/Established, source succession and unproven nonexecution remain explicit gaps. Advisory counts retain their own full head/registry/interval; absent coverage stays unknown. Complete accepted-work and finalization evidence remain required. | | `FleetReaderEvacuationVerifier::collect_maintenance` / `FleetFollowerEvacuationVerifier::collect_maintenance` | Present complete eligible donor history lookup from the canonical original/current request set, followed by existing authenticated native/current-policy confirmation. Exact original digest/current row and full head/registry/roster checks are retained. | Collect original history first and construct the outer observation after both role collections. Missing latest history supplies no check and stays MissingPolicy in the matcher. One monotonic 30-second capture and caller deadline bound collection; changed heads/policies during native capture fail closed. Source/failed-owner succession, nonexecution, unknown native/external work and finalization remain required. | -| `FleetReaderEvacuationVerifier::collect_source_readers` / `FleetSourceReaderPolicies` / `FleetObservation::with_source_reader_policies` | Present exact native reader joins or failed-process reader closures composed with an actual managed native writer successor, minimum-prefix/origin verification, current reader policy and ready enrolled replacement prefixes. The complete original/current request set, full head/registry/roster, repeated provider allocation/presence and monotonic capture are retained in planner identity v13. | Authenticate both endpoints and canonical backend mappings. Retain the original native capsule or durable process closure through the application's existing accepted-work owner. Missing evidence stays SourceSuccessor; checked exact source requests become SourceReader. Local SQLite client reconstruction/public reconciliation cannot qualify OS/process restart exclusion or durable provider retention. Complete failed-owner follower succession, other accepted work and SettleRoles/Finalize remain required. | +| `FleetReaderEvacuationVerifier::collect_source_readers` / `FleetSourceReaderPolicies` / `FleetObservation::with_source_reader_policies` | Present exact native reader joins or failed-process reader closures composed with an actual managed native writer successor, minimum-prefix/origin verification, current reader policy and ready enrolled replacement prefixes. The complete original/current request set, full head/registry/roster, repeated provider allocation/presence and monotonic capture are retained in planner identity v14. | Authenticate both endpoints and canonical backend mappings. Retain the original native capsule or durable process closure through the application's existing accepted-work owner. Missing evidence stays SourceSuccessor; checked exact source requests become SourceReader. Local SQLite client reconstruction/public reconciliation cannot qualify OS/process restart exclusion or durable provider retention. Other accepted work and SettleRoles/Finalize remain required. | | `CellNode::follower_evacuation` | Present per-live-owner replacement and retirement check after the original requested rotation. | Require the declared member minimum, donor exclusion, complete Established replacement rows, pinned signed boots, current authority and full journal rechecks. Retain the original completion/error history and publish/revalidate its interval. This does not settle failed owners, Pending producers or the physical node's other obligations. | | Recovered-log authorization, member retirement and canonical Retired CAS | Present runtime tail closure after canonical recovery pinning. | Use `RecoveredNodeLogTransport`, receiver-side `authorize_recovered_log_retire` and `FollowerStore::retire_recovered`; confirm every original member before `retire_recovered_log`. Adopt exact committed closure with `retired_recovered_log` before repeating effects. Publish original enrollments through the host capsule, then complete failed-process barriers and maintenance orchestration; native closure alone cannot finish W7. | | `RecoveryManifestStore::load_manifest` | Complete digest-verified recovered-suffix metadata across every original application, retained after successor materialization. | Read the identity from canonical sealed-log authority, then verify each bundle through its application's ordinary store. Retain the complete original writer set separately, including object-covered Cells without a suffix; check current successor authority, acknowledged prefixes and serving. This metadata read cannot settle roles or finalize a node. | @@ -304,6 +304,7 @@ present API; the pseudocode below specifies the remaining orchestration. | `CellAuthority::owner_history` | Canonical owner departure retains full original controls before the existing owner CAS; complete per-incarnation epoch reads refuse missing or changing history. | Use the original capture/journal path to retain the complete authenticated operation set, then verify current successor prefix/serving. Missing legacy, restored or older-binary history is a typed blocker. These rows grant no authority or root retention pin and cannot settle roles or finalize a node. | | `FleetRecoveredFollowerRetirement` | Present checked publication of recovered ensemble enrollment closure. | Capture the exact canonically Retired physical leader/epoch against the full roster, publish all original member requests through the existing journal, then confirm the complete new roster and canonical authority. Retain every original response/error. Fresh recapture adopts lost replies without native RPCs or timestamp refresh; failed boots, replacement policy and operation integration remain required. | | `FleetFailedBootRetirement` / `FleetFailedBootProcesses` | Present original failed-boot closure against canonical terminal fencing and durable application process evidence. `confirm` freshly rechecks an already committed retirement without effects. Process confirmation pins the committed request/witness binding. `FleetObservation::with_failed_boot_closures` retains the exact full barrier, original row/process and capture interval inside planner inputs. | Complete every original related request first. The read-only provider proves original process/accepted external work joining and session nonexecution; publication and confirmation share terminal roster/authority/reference/process checks. Public minion observation consumes both an empty-writer surviving graph and combined original-writer/closure evidence across two applications in either attachment order. Complete authenticated observation, actual provider qualification, replacement policy and maintenance actions remain required. | +| `FleetRecoveredFollowerClosure` / `FleetObservation::with_recovered_follower_closures` | Present retention of exact canonical recovered-log retirement and the complete original follower-member rows under one full snapshot and capture interval. The observation digest binds these closures. | A recovered log alone remains `SourceSuccessor`. Maintenance matching requires the exact failed-boot closure for the same physical node, session, snapshot and retired log before marking its original source-side follower requests checked. This proof does not replace live-owner follower evacuation policy or SettleRoles/Finalize. | | `NodeDirectory::fenced_session` / `FleetFailedBootProcessRequest::capture_fenced` / `confirm` | Present process confirmation before leader-log recovery/retirement, using the existing application provider. | Authenticate and join the exact original process and all accepted work, then retain every affected Cell before recovery effects. The v2 identity survives recovery and claim adoption; `capture_retained` reuses it only after strict terminal boot/role checks. Existing terminal v1 identities remain unchanged. Process evidence cannot substitute for original writer inventory or current successor/policy evidence. | | `FleetFailedReaderRetirement` / `FleetFailedBootProcessRequest::capture` | Present original failed receiver reader publication, including Pending and Established histories, through the existing journal/provider boundary. The source-reader collector now consumes this closure only alongside current writer lineage and current ready-reader policy; missing replacements remain blocking. | Retain production process and accepted-work evidence durably across controller/provider restarts. Handle live receivers through ordinary closure, complete follower succession, and qualify actual OS/process/external-job providers before completing W7. | | `FleetActionJournal` | Present in host `fleet/journal.rs`. | Implement acceptance, result publication, original-action lookup, and acquisition/recovery basis and evidence recording/lookup with the specified atomic and durable semantics. | diff --git a/docs/fleet-operations-progress.md b/docs/fleet-operations-progress.md index dd3e4ddc..5906234a 100644 --- a/docs/fleet-operations-progress.md +++ b/docs/fleet-operations-progress.md @@ -6493,3 +6493,43 @@ checked source release and acquisition/recovery evidence, and adopt unknown outcomes across controller replacement. Complete host shutdown/role barriers and concurrent ordinary activation/reader admission scenarios. W5–W10 and the plan's full acceptance matrix remain pending. + +## October 4 2026 failed-owner follower maintenance checkpoint + +The public maintenance-policy matcher now consumes an exact canonical +recovered-follower closure together with the matching failed-boot process +closure. At one full snapshot and capture barrier, it verifies the recovered +log was retired, every original member row is present in the current roster, +and the old source process is confirmed retired against that same log. Only +those exact failed-owner source-side follower requests move from +`SourceSuccessor` to `RecoveredFollower`. A recovered log without the process +closure remains unchecked. Planner inputs bind these closures; the new policy +status has its own versioned coverage digest. + +The runnable minion scenario exercises successful composition and the +negative case where canonical log retirement alone leaves the maintenance +obligations open. Both preserve the original maintenance capture and do not +grant settlement or finalization. + +| Command | Observed result | +| --- | --- | +| `cargo test -p cellule-host --all-features --locked --example fleet_operations scenario::recovered_followers::failed_boot::writer_tests::successors::observation::combined::` | 4 passed; 330 filtered out. | +| `cargo test -p cellule-host --all-features --locked --example fleet_operations` | 334 passed; 0 failed; 69.81 seconds. | + +The final source shape also passed workspace all-target/all-feature `cargo +check`, warning-denied workspace Clippy, and warning-denied workspace API docs. +Format, boundary, module-layout, Rust-fence, Markdown-link, SQL/peer-contract, +and whitespace gates passed: 137 Rust snippets, 1307 Markdown links, and 28 +schema/protocol assertions with 570 validator links. Hosted CI has not yet run +for these uncommitted changes. + +### Remaining work at the failed-owner follower checkpoint + +This closes one source-side policy gap only. Complete the aggregate +authenticated observation and connect role policy, native/external accepted +work and provider/process retention to `SettleRoles`/`Finalize`. Implement the +host action executor beyond Cordon, remaining live-owner/failed-owner role +evacuation, receiver adoption, Cron/Blob owner handling, broad fault and load +qualification, and the W9–W10 rollout/runbooks. PR #37 is already merged; +the continuation PR #56 was mergeable at this checkpoint, before these local +changes were committed or pushed. From 631e0420562dc0cc7edc021952e78bd4c0582ade Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 4 Oct 2026 03:25:56 -0700 Subject: [PATCH 5/5] inspect local fleet maintenance admission --- crates/cellule-host/docs/lifecycle.md | 2 +- crates/cellule-host/minion/README.md | 4 + crates/cellule-host/src/fleet/actions/mod.rs | 11 ++- crates/cellule-host/src/fleet/inspection.rs | 59 +++++++++---- crates/cellule-host/src/node/fleet.rs | 7 +- .../tests/node/fleet_maintenance.rs | 82 ++++++++++++++++--- docs/fleet-operations-plan.md | 2 +- docs/fleet-operations-progress.md | 28 +++++++ 8 files changed, 162 insertions(+), 33 deletions(-) diff --git a/crates/cellule-host/docs/lifecycle.md b/crates/cellule-host/docs/lifecycle.md index f155fcca..3465442b 100644 --- a/crates/cellule-host/docs/lifecycle.md +++ b/crates/cellule-host/docs/lifecycle.md @@ -744,7 +744,7 @@ complete primitive maintenance coverage and role finalization remain under imple | Source preflight refusal | The actor returns `Error::CellReleaseRefused` only before this request begins canonical deactivation. Record Rejected with its original source error; the reconciler then cancels and joins unused receiver credit. Independent local eviction grants no release proof for this attempt. | | Prepare receiver | Validate catalog/Cell/incarnation and current release/schema support; reserve actual runtime/LTX resources before returning Reserved. A refusal preserves the original error and leaves the source serving. | | Maintenance Cordon | Apply retained Draining intent through the shared admission gate. Replays retain the exact acceptance/result; a dropped waiter leaves publication owned. This preserves existing obligations and uses no shutdown lane. | -| Other maintenance effects | Refuse role settlement, finalization and maintenance inspection until their inventory and host barriers are implemented. A cordon receipt cannot establish Stopped or withdrawal. | +| Other maintenance effects | Refuse role settlement and finalization until their inventory and host barriers are implemented. Fresh maintenance inspection reads the exact local admission gate: only Draining reports Cordoned, while Active/Cordoned stays blocked. This inspection cannot establish role settlement, Stopped, or withdrawal. | | Activate receiver | Confirm the exact Idle acquisition basis is durably retained before ordinary ownership CAS; consume prepared credit through canonical restore and actor activation. | | Acquisition-basis reply lost | Keep authority untouched and the prepared credit charged. Repeating the same accepted action confirms the original basis and capture time before takeover. | | Unused credit after release | Journal CleaningReceiver, cancel/join that credit, and retain the source position and fleet permit. Cleanup returns to Released, with no claim of pre-release cancellation. After confirmed cleanup, ordinary admitted acquisition may resume. | diff --git a/crates/cellule-host/minion/README.md b/crates/cellule-host/minion/README.md index 3b738d9b..8562c51d 100644 --- a/crates/cellule-host/minion/README.md +++ b/crates/cellule-host/minion/README.md @@ -435,6 +435,10 @@ Fresh inspection records bind a nonce and original capture interval. Use the host's `inspect_fleet_action` and validate its response against the complete request and a finite age bound. Legacy retained Inspect results are historical. The journal authorizes a capture; it does not supply current actor evidence. +Movement inspection checks current actor/authority serving. Maintenance +inspection checks only whether the exact local admission gate is Draining, so +it can recover a lost Cordon reply. It does not prove role settlement, Stopped, +or withdrawal. Before enabling fleet execution, the embedding application must provide the controlled bootstrap/import barrier, wire every enrollment producer, authenticate diff --git a/crates/cellule-host/src/fleet/actions/mod.rs b/crates/cellule-host/src/fleet/actions/mod.rs index d534addb..35f51ff1 100644 --- a/crates/cellule-host/src/fleet/actions/mod.rs +++ b/crates/cellule-host/src/fleet/actions/mod.rs @@ -282,9 +282,16 @@ impl FleetActionExecutor { if request.node() != self.node || request.session() != self.session { return Err(Arc::new(Error::Fenced)); } - if !matches!(request.action().kind(), FleetActionKind::Movement { .. }) { + if !matches!( + request.action().kind(), + FleetActionKind::Movement { .. } + | FleetActionKind::Maintenance { + action: MaintenanceAction::Inspect, + .. + } + ) { return Err(Arc::new(Error::Control( - "fleet maintenance inspection requires its host inventory barrier", + "fleet inspection requires an Inspect action", ))); } let completion = self.submit(JobRequest::Inspection(request)).await?; diff --git a/crates/cellule-host/src/fleet/inspection.rs b/crates/cellule-host/src/fleet/inspection.rs index f58298f5..896ab0bc 100644 --- a/crates/cellule-host/src/fleet/inspection.rs +++ b/crates/cellule-host/src/fleet/inspection.rs @@ -2,9 +2,10 @@ use std::sync::Arc; use cellule_runtime::Error; use cellule_runtime::fleet::operations::{ - FleetActionKind, FleetActionOutcome, FleetInspectionObservation, FleetInspectionRequest, - MovementAction, OperationError, + DrainBlocker, FleetActionKind, FleetActionOutcome, FleetInspectionObservation, + FleetInspectionRequest, MaintenanceAction, MovementAction, OperationError, }; +use cellule_runtime::node::NodeMode; use super::actions::{FleetActionExecutor, journal_error, operation, wall_time_ms}; @@ -22,19 +23,47 @@ impl FleetActionExecutor { .await .map_err(journal_error) .map_err(Arc::new)?; - let FleetActionKind::Movement { - action: MovementAction::Inspect, - attempt, - } = request.action().kind() - else { - return Err(Arc::new(Error::Control("fleet inspection is not movement"))); + let inspected = match request.action().kind() { + FleetActionKind::Movement { + action: MovementAction::Inspect, + attempt, + } => { + // This path never reads a cached Inspect result and never + // invokes acquisition or cleanup. Current serving comes from + // actor + authority. + let result = self.inspect(attempt).await.map_err(Arc::new)?; + if let Some(error) = result.error { + return Err(Arc::new(error)); + } + result.outcome + } + FleetActionKind::Maintenance { + action: MaintenanceAction::Inspect, + operation, + } => { + if operation.node() != self.node || operation.session() != self.session { + return Err(Arc::new(Error::Fenced)); + } + // This is deliberately a local admission check only. It can + // confirm local Cordon after a lost reply, but cannot claim role settlement, + // drained facilities, Stopped, or session withdrawal. + match self.runtime.node_admission().mode().map_err(Arc::new)? { + NodeMode::Draining => { + cellule_runtime::fleet::operations::FleetOutcome::Cordoned + } + NodeMode::Active | NodeMode::Cordoned => { + cellule_runtime::fleet::operations::FleetOutcome::Blocked( + DrainBlocker::IncompleteObservation, + ) + } + } + } + _ => { + return Err(Arc::new(Error::Control( + "fleet inspection requires an Inspect action", + ))); + } }; - // This path never reads a cached Inspect result and never invokes - // acquisition or cleanup. Current serving comes from actor + authority. - let result = self.inspect(attempt).await.map_err(Arc::new)?; - if let Some(error) = result.error { - return Err(Arc::new(error)); - } let finished = wall_time_ms().map_err(Arc::new)?; if finished >= request.deadline_ms() { return Err(Arc::new(operation(OperationError::Deadline))); @@ -49,7 +78,7 @@ impl FleetActionExecutor { node: self.node, session: self.session, observed_at_ms: finished, - outcome: result.outcome, + outcome: inspected, }; FleetInspectionObservation::new(request, started, outcome) .map(Arc::new) diff --git a/crates/cellule-host/src/node/fleet.rs b/crates/cellule-host/src/node/fleet.rs index 9a582b00..2fbae00e 100644 --- a/crates/cellule-host/src/node/fleet.rs +++ b/crates/cellule-host/src/node/fleet.rs @@ -379,8 +379,11 @@ impl CellNode { /// Captures read-only request-bound evidence through the owned fleet lane. /// Authenticate the caller first. The node checks current journal state and - /// actual authority/actor readiness; cached effect replies are not consulted - /// as proof of current serving. A dropped waiter leaves the finite job owned. + /// actual authority/actor readiness for movement, or the exact local + /// admission gate for a maintenance Cordon. Cached effect replies are not + /// consulted as proof of current serving; maintenance inspection does not + /// prove role settlement or finalization. A dropped waiter leaves the finite + /// job owned. pub async fn inspect_fleet_action( &self, request: cellule_runtime::fleet::operations::FleetInspectionRequest, diff --git a/crates/cellule-host/tests/node/fleet_maintenance.rs b/crates/cellule-host/tests/node/fleet_maintenance.rs index b01106f0..116c3ad7 100644 --- a/crates/cellule-host/tests/node/fleet_maintenance.rs +++ b/crates/cellule-host/tests/node/fleet_maintenance.rs @@ -254,6 +254,20 @@ fn cordon(fixture: &Fixture) -> FleetAction { ) } +fn maintenance_inspection(fixture: &Fixture, nonce: u8) -> FleetInspectionRequest { + FleetInspectionRequest::new( + fixture + .journal + .maintenance_action(MaintenanceAction::Inspect), + fixture.journal.registry(), + Digest::from_bytes([nonce; 32]), + NodeId::from_bytes([201; 16]), + SessionId::from_bytes([201; 16]), + clock() + 30_000, + ) + .unwrap() +} + async fn check_existing_owner(fixture: &Fixture) { let value = fixture .handle @@ -301,6 +315,53 @@ async fn committed_cordon_closes_shared_role_gate_keeps_owner_and_is_idempotent( close(fixture).await; } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn fresh_maintenance_inspection_confirms_cordon_but_never_claims_role_settlement() { + let fixture = fixture().await; + let _cordon = cordon(&fixture); + + let before = fixture + .node + .inspect_fleet_action(maintenance_inspection(&fixture, 211)) + .await + .unwrap(); + assert_eq!( + before.outcome().outcome, + FleetOutcome::Blocked(DrainBlocker::IncompleteObservation) + ); + assert_eq!(fixture.journal.accepted_count(), 0); + assert_eq!( + fixture.node.runtime().node_admission().mode().unwrap(), + NodeMode::Active + ); + + let action = fixture + .journal + .maintenance_action(MaintenanceAction::Cordon); + fixture.journal.lose_next_result_reply(); + let lost = fixture + .node + .apply_fleet_action(action, clock()) + .await + .unwrap(); + assert!(!lost.committed); + assert!(lost.journal_error.is_some()); + + let after = fixture + .node + .inspect_fleet_action(maintenance_inspection(&fixture, 212)) + .await + .unwrap(); + assert_eq!(after.outcome().outcome, FleetOutcome::Cordoned); + assert_eq!(fixture.journal.accepted_count(), 1); + assert_eq!( + fixture.node.runtime().node_admission().mode().unwrap(), + NodeMode::Draining + ); + check_existing_owner(&fixture).await; + close(fixture).await; +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn lost_publication_reply_replays_original_cordon_and_keeps_gate_closed() { let fixture = fixture().await; @@ -394,18 +455,15 @@ async fn wrong_boot_stale_intent_and_unsupported_roles_do_not_accept_effects() { .await .is_err() ); - let inspection = FleetInspectionRequest::new( - fixture - .journal - .maintenance_action(MaintenanceAction::Inspect), - fixture.journal.registry(), - Digest::from_bytes([211; 32]), - NodeId::from_bytes([201; 16]), - SessionId::from_bytes([201; 16]), - clock() + 30_000, - ) - .unwrap(); - assert!(fixture.node.inspect_fleet_action(inspection).await.is_err()); + let inspection = fixture + .node + .inspect_fleet_action(maintenance_inspection(&fixture, 213)) + .await + .unwrap(); + assert_eq!( + inspection.outcome().outcome, + FleetOutcome::Blocked(DrainBlocker::IncompleteObservation) + ); assert_eq!(fixture.journal.accepted_count(), 0); assert_eq!( fixture.node.runtime().node_admission().mode().unwrap(), diff --git a/docs/fleet-operations-plan.md b/docs/fleet-operations-plan.md index 77a3f82d..7b2d72c8 100644 --- a/docs/fleet-operations-plan.md +++ b/docs/fleet-operations-plan.md @@ -279,7 +279,7 @@ present API; the pseudocode below specifies the remaining orchestration. | `CellNode::fleet_durability_supervisor` | Present fixed-metadata advisory capture of the retained native supervisor and its bounded rotation bank. | Preserve distinct unstarted/running/returned/joined states and separate original supervisor/request-stop failures, including capture during drain after byte admission closes. Missing or failed capture supplies no coverage; joined state and local rotation completion still require producer/native lanes, current authority and replacement-policy evidence. | | `NodeDirectory::prepare_log_enrollment` / `prepare_log_enrollment_attempt` / `commit_log_enrollment` | Present opaque follower selection and CAS bridge. | Revalidate exact boots before acceptance; journal all selected members Pending before dispatch, retain the original attempt, and reconcile that same scope. `fence_log_enrollment` proves only its original-token refusal; absence cannot settle unknown work. Managed host producer and journal retirement remain required. | | `CellNode::apply_fleet_action(action, now_ms)` | Present; returns a retained `FleetActionCompletion`. | Movement actions use this path. Check `committed`, retain unknown outcomes, and obtain fresh serving evidence. Maintenance integration remains required. | -| `CellNode::inspect_fleet_action(request)` | Present; returns `FleetInspectionObservation`. | Use a fresh nonce, exact head/registry and endpoint, and exclusive capture deadline. Authenticate and validate the full request plus original capture interval; recheck head/registry when committing dependent decisions. Inspection starts no acquisition or cleanup effect; the shared lane can settle prior completed work. | +| `CellNode::inspect_fleet_action(request)` | Present; returns `FleetInspectionObservation`. Movement inspection checks current serving; maintenance inspection reports only the exact local admission gate. | Use a fresh nonce, exact head/registry and endpoint, and exclusive capture deadline. Authenticate and validate the full request plus original capture interval; recheck head/registry when committing dependent decisions. Inspection starts no acquisition or cleanup effect; the shared lane can settle prior completed work. A Draining gate may recover a lost Cordon reply. Active/Cordoned maintenance observations remain blocked, and this endpoint never proves role settlement, Stopped, or withdrawal. | | `CellNode::fleet_action_work()` / `FleetNodeSnapshot::action_work()` | Present bounded original finite job metadata, including response versus join state, canonical results, original failures, admission closure and effect/inspection turnover. Native snapshots exclude only their exact executing request and refuse changed work; full traversal and global rechecks bind the work fingerprint. | Authenticate the original endpoint, preserve original errors and intervals, and account copied collector buffers. Joined tasks, committed results and known outcomes are independent; unknown durable actions, native/external work and terminal action handoff still need their own barriers. No local count or fingerprint grants finalization. | | `CellNode::fleet_snapshot(request)` | Present request-bound native page capture; aggregate observation remains incomplete. | Pin the full head/registry, physical boot, nonce, category, original continuation, limit and deadline. Authorize before and after capture through the same finite action bank. Retain original page charges and errors; authenticate and combine every required producer/native page with fresh authority and replacement-policy evidence. Unbound is never empty coverage. | | `FleetNodeInventoryScan` and `FleetNodeInventory::recheck` | Present complete native traversal and all-category rechecks at a full roster barrier. | Preserve original requests, native lifetimes, transitions and shared errors. Validate enrollment matching, then recheck every category after fleet-wide authority/policy discovery and reconfirm the journal. Native interval evidence cannot settle failed processes or grant finalization. | diff --git a/docs/fleet-operations-progress.md b/docs/fleet-operations-progress.md index 5906234a..610956b5 100644 --- a/docs/fleet-operations-progress.md +++ b/docs/fleet-operations-progress.md @@ -6533,3 +6533,31 @@ evacuation, receiver adoption, Cron/Blob owner handling, broad fault and load qualification, and the W9–W10 rollout/runbooks. PR #37 is already merged; the continuation PR #56 was mergeable at this checkpoint, before these local changes were committed or pushed. + +## October 4 2026 maintenance inspection checkpoint + +`CellNode::inspect_fleet_action` now accepts fresh maintenance `Inspect` +requests through the existing owned finite-action lane. It authorizes the exact +head, registry and endpoint, reads the live local admission gate, and returns +`Cordoned` only for `Draining`; `Active` and `Cordoned` return the explicit +`IncompleteObservation` blocker. The read performs no action acceptance, +release, role settlement or shutdown. A dropped Cordon result can therefore be +recovered from local state without treating a historical result as current. + +Public node tests cover the active and draining responses, a lost durable +Cordon-result reply, preservation of the existing Cell owner, and the rule that +maintenance inspection never reports `RolesSettled` or `Stopped`. + +| Command | Observed result | +| --- | --- | +| `cargo test -p cellule-host --all-features --locked --test node fleet_maintenance::` | 6 passed; 109 filtered out. | + +The full `node` integration target also passed (115 tests). Workspace check, +warning-denied Clippy, and warning-denied API docs passed. Format, boundary, +module-layout, Rust-fence, Markdown-link, SQL/peer-contract, and whitespace +gates passed: 137 Rust snippets, 1307 Markdown links, and 28 schema/protocol +assertions with 570 validator links. + +The remaining maintenance inspection gap is the full role, accepted-work, +facility, Stopped, and withdrawal barrier. This local check is only a Cordon +recovery step and cannot move the operation into Closing or Completed.