Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions crates/cellule-host/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
| --- | --- |
| [Lifecycle](lifecycle.md) | Start and stop one node safely. |
| [Read replicas](read-replicas.md) | Install and supervise immutable views. |
| [Source reader succession](source-readers.md) | Check exact native retirement against the current writer and reader policy. |
| [Original failed-boot writers](original-writers.md) | Retain complete original ownership metadata before dependent effects. |
| [Fleet journal example](../minion/README.md) | Run the durable local journal foundation and inspect its integration limits. |
| [Crate entry](../README.md) | Ownership and test command. |
Expand Down
23 changes: 17 additions & 6 deletions crates/cellule-host/docs/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -744,7 +744,7 @@ complete primitive maintenance coverage and role finalization remain under imple
| Source preflight refusal | The actor returns `Error::CellReleaseRefused` only before this request begins canonical deactivation. Record Rejected with its original source error; the reconciler then cancels and joins unused receiver credit. Independent local eviction grants no release proof for this attempt. |
| Prepare receiver | Validate catalog/Cell/incarnation and current release/schema support; reserve actual runtime/LTX resources before returning Reserved. A refusal preserves the original error and leaves the source serving. |
| Maintenance Cordon | Apply retained Draining intent through the shared admission gate. Replays retain the exact acceptance/result; a dropped waiter leaves publication owned. This preserves existing obligations and uses no shutdown lane. |
| Other maintenance effects | Refuse role settlement, finalization and maintenance inspection until their inventory and host barriers are implemented. A cordon receipt cannot establish Stopped or withdrawal. |
| Other maintenance effects | Refuse role settlement and finalization until their inventory and host barriers are implemented. Fresh maintenance inspection reads the exact local admission gate: only Draining reports Cordoned, while Active/Cordoned stays blocked. This inspection cannot establish role settlement, Stopped, or withdrawal. |
| Activate receiver | Confirm the exact Idle acquisition basis is durably retained before ordinary ownership CAS; consume prepared credit through canonical restore and actor activation. |
| Acquisition-basis reply lost | Keep authority untouched and the prepared credit charged. Repeating the same accepted action confirms the original basis and capture time before takeover. |
| Unused credit after release | Journal CleaningReceiver, cancel/join that credit, and retain the source position and fleet permit. Cleanup returns to Released, with no claim of pre-release cancellation. After confirmed cleanup, ordinary admitted acquisition may resume. |
Expand Down Expand Up @@ -968,7 +968,7 @@ ensembles. Signed replacement boots must match the existing producer-specific
identity; a changed writer row or follower epoch/ensemble invalidates the input.
Attachment order with role coverage, writer successors or failed-boot closure
does not change these checks. The reconciler repeats the roster comparison.
Planner digest v11 binds collection presence, canonical record order, full
Planner digest v13 binds collection presence, canonical record order, full
barriers, fresh intervals, current authority, reader prefixes and retained native
follower inventories. Persisted history, enrollment and transport codecs are unchanged.

Expand All @@ -993,7 +993,7 @@ fresh complete roster, and reconfirms the full journal snapshot. Retiring rows,
extending a deadline or adopting a successor boot preserves the first set.
`FleetObservation::with_maintenance_enrollments` retains this evidence inside the
outer interval and compares the exact head, registry and roster with other checks,
in either attachment order. Planner digest v11 binds its presence and full digest.
in either attachment order. Planner digest v13 binds its presence and full digest.

A missing manifest or page remains unknown. A committed empty manifest proves
only that the original unresolved role set was empty at the phase transaction.
Expand Down Expand Up @@ -1046,6 +1046,16 @@ Pending/Established work, source succession and unknown nonexecution stay in the
full matcher even when they are ineligible for donor lookup. This read-only path
starts no role effects or tasks and grants no settlement/finalization rights.

### Confirm source reader succession

Use the [source reader collector](source-readers.md) after actual writer handoff
and native or failed-process reader closure. It consumes the same complete
original/current request set and checks current writer origin, policy and native
ready readers.
Retain `with_source_reader_policies` before full maintenance policy matching.
Missing native/provider evidence stays unknown. Production accepted-work/process
qualification and finalization remain separate.

### Confirm original nonexecution

A terminal request without establishment history remains unknown. An independent
Expand Down Expand Up @@ -1095,7 +1105,7 @@ async fn attach_nonexecution(

The observation's outer interval must include both collections. Attach all native
policy and nonexecution checks before matching; later input changes refuse.
Checked nonexecution is reported separately and binds planner digest v11. It
Checked nonexecution is reported separately and binds planner digest v13. It
cannot upgrade incomplete node observation, prove installed-role redundancy,
join other accepted work, or grant SettleRoles/Finalize rights.

Expand Down Expand Up @@ -1127,7 +1137,8 @@ supply complete request coverage.
| `Reader` / `Follower` | Exact current role and fresh native replacement policy checked. |
| `Pending` / `Established` | Accepted native outcome or installed role still unresolved. |
| `MissingPolicy` | Retired donor lacks current replacement policy evidence. |
| `SourceSuccessor` | Source-side or failed-owner policy still needs separate evidence. |
| `SourceReader` | Exact original source reader closed natively or by confirmed process evidence, minimum prefix verified against an actual managed successor, current ready-reader policy checked. |
| `SourceSuccessor` | Remaining source-side or failed-owner policy still needs separate evidence. |
| `Nonexecution` | Independently confirmed original work joining and exclusion; no role effect committed. |
| `UnprovenNonexecution` | Closed unknown acceptance lacks checked nonexecution or policy evidence. |

Expand All @@ -1137,7 +1148,7 @@ empty original manifest and no currently unresolved related requests.
`FleetReconcileReport::maintenance_policy` supplies bounded advisory counts with
their own head revision, registry and original interval. `None` means unknown.
Later allocations cannot restamp these counts to the report's newer snapshot.
Planner digest v11 binds the coverage digest and source inputs; changing the
Planner digest v13 binds the coverage digest and source inputs; changing the
attached policy collections after matching is refused.

`is_complete()` describes only enumerated request-policy coverage. It does not
Expand Down
93 changes: 93 additions & 0 deletions crates/cellule-host/docs/source-readers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Source reader succession

A reader can live on another node while its original writer enters maintenance.
The host checks that exact reader's native retirement against the actual writer
successor and current reader policy before marking its policy obligation checked.
This uses the existing reader manager, actor FIFO, origin verifier and peer path.

| Input | Required evidence |
| --- | --- |
| Original request set | Every immutable original page and the full current roster. |
| Reader retirement | Either the retained `ReaderEnrollmentRetirement` from exact native removal or a `FleetFailedReaderClosure` bound to the original process and accepted-work proof. |
| Minimum prefix | Native removal requires its last installed root after accepted reader work joined. A failed receiver requires at least its original enrollment root. |
| Writer successor | An actual managed native writer on an Active physical node outside maintenance, at a strictly newer epoch. |
| Origin | Canonical derivation of the exact final root and availability of all current dependencies. |
| Reader policy | Exact current revision, incarnation and desired count; absent differs from explicit zero. |
| Replacements | Canonically selected Active boots, exact Established requests and actual ready native prefixes. |
| Capture | One full journal/roster barrier, absolute deadline and monotonic interval of at most 30 seconds. |

## Retain and collect

The embedding application authenticates both endpoints and canonical backend
mappings. Its existing accepted-work owner retains each native retirement or
failed-process closure and returns the same `Arc<FleetSourceReaderInputs>` on
repeated lookup. `None` means unknown. A terminal enrollment row cannot
reconstruct either proof.

After writer handoff, join a live original reader with
`ReadReplicaManager::remove_enrolled`. If its receiver process has stopped,
publish `FleetFailedReaderRetirement` only after the application confirms the
original process and all accepted work are durably closed. Both paths still
require the actual current writer, verified origin and current ready-reader
policy. Ordinary recruitment installs current replacements. The same
still-Active receiver may hold a new request under the successor; that request
cannot replace the original retirement proof.

```rust
use cellule_host::fleet::{
FleetJournal, FleetMaintenanceEnrollments, FleetReaderEvacuationVerifier,
FleetRoster, FleetSourceReaderPolicies, FleetSourceReaderSuccessors,
};
use cellule_runtime::Result;
use tokio::time::Instant;

async fn source_policies(
verifier: &FleetReaderEvacuationVerifier,
journal: &dyn FleetJournal,
original: &FleetMaintenanceEnrollments,
roster: &FleetRoster,
successors: &dyn FleetSourceReaderSuccessors,
deadline: Instant,
clock: impl FnMut() -> Result<i64>,
) -> Result<FleetSourceReaderPolicies> {
verifier
.collect_source_readers(journal, original, roster, successors, deadline, clock)
.await
}
```

The collector visits the complete original/current request set. It verifies
current native writers around origin and reader probes, then globally rechecks
policy, selected boots, native prefixes, provider presence/identity and the full
roster. Changed inputs refuse the capture. Native origin I/O uses the runtime's
existing reservations; applications account bounded metadata copies.

Build the outer `FleetObservation` after all native collection finishes. Retain
`with_source_reader_policies` and `with_maintenance_enrollments` before
`check_maintenance_policies`. Both attachment orders validate the same original
capture, full head, registry, roster and interval. Signed successor/replacement
boots and any captured writer row must match the checked native evidence.
Planner identity v13 binds native and failed-process inputs. Missing evidence
stays `SourceSuccessor`; a checked exact source reader is `SourceReader`.

## Scope and runnable evidence

The canonical minion suite exercises managed writer handoff, refresh past the
original opening, exact native closure, a new reader request on the same
receiver, and failed receiver closure. A failed receiver with desired readers
still set to one remains blocked when no replacement is available; reducing the
canonical policy to zero permits the checked closure. Independent SQLite
journal clients and public reconciliation are also covered. The suite exercises
missing/changed provider evidence, foreign origins, clock/deadline/cancellation
and policy, journal or writer changes behind actual signed native replies.

```sh
cargo test -p cellule-host --example fleet_operations --all-features --locked \
scenario::reader_tests::evacuation::source:: -- --test-threads=2
```

These local checks do not qualify production process/provider retention or
upgrade an incomplete observation, and they do not authorize SettleRoles or
Finalize. Original writer proof, follower succession, native/external accepted
work and full maintenance remain required by the
[fleet plan](../../../docs/fleet-operations-plan.md).
17 changes: 17 additions & 0 deletions crates/cellule-host/minion/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,10 @@ Fresh inspection records bind a nonce and original capture interval. Use the
host's `inspect_fleet_action` and validate its response against the complete
request and a finite age bound. Legacy retained Inspect results are historical.
The journal authorizes a capture; it does not supply current actor evidence.
Movement inspection checks current actor/authority serving. Maintenance
inspection checks only whether the exact local admission gate is Draining, so
it can recover a lost Cordon reply. It does not prove role settlement, Stopped,
or withdrawal.

Before enabling fleet execution, the embedding application must provide the
controlled bootstrap/import barrier, wire every enrollment producer, authenticate
Expand Down Expand Up @@ -578,6 +582,19 @@ Complete authenticated lookup, source/failed-owner successor policy, checked
nonexecution, accepted-work joining and SettleRoles/Finalize remain delivery
requirements. See the [host matching recipe](../docs/lifecycle.md#match-every-required-maintenance-policy).

## Native source reader succession

The [source reader recipe](../docs/source-readers.md) composes exact retained
native retirement with an actual managed writer successor and current ready
reader policy. It consumes the complete original/current request set and is
retained through the public observer and matcher. Missing evidence remains an
explicit source obligation. A checked local request does not finish maintenance.

```sh
cargo test -p cellule-host --example fleet_operations --all-features --locked \
scenario::reader_tests::evacuation::source:: -- --test-threads=2
```

## Original nonexecution confirmation

The actual reader/follower suites consume `FleetMaintenanceNonexecution` through
Expand Down
Loading
Loading