Close the two remaining P0 gaps: process termination and store wipe

[mo4islona]

Closes the last two P0 rows in the spec/13 gap register. Both are remote-triggerable
by whoever publishes an assignment document.

GAP-2 — externally supplied content could terminate the process

Under ADR-14's supervision tree any panic in a loop is a whole-process outage, so a
scheduler bug or a registry blip becomes a fleet outage. Four paths, one per subsystem:

Path	Fix
A malformed file address panicked the reconciler	list_files returning None defers that chunk and alarms; the rest of the document applies (FM-11)
An unparseable roster peer id panicked the reader	DatasetsIndex::new is a panic-catching boundary (ADR-21) → the document is rejected whole (FM-12); the spawn_blocking join repeats the catch instead of .expect-ing
A pathological per-chunk file count overflowed the download watchdog	watchdog_timeout saturates, and a zero-file chunk gets one timeout instead of expiring immediately
A registry error at startup was fatal	AllocationsChecker resolves its on-chain id lazily in the polling loop and alarms while unresolved (FM-52)

Deferred chunks are held out of the pending set only for the duration of a pass, so the
download loop still terminates, and applying a new index notifies the loop even when the
chunk set is unchanged — otherwise a repaired address would wait for an unrelated event.

GAP-3 — no reconciliation deletion floor

Reconciliation deleted whatever the assignment stopped naming, so one short document
wiped a multi-terabyte store in a single pass. ADR-17 is ratified and implemented: an
application that would evict more than P-DEL-FLOOR (default 50 %, --deletion-floor)
of the stored chunks evicts none of them, raises an alarm, and re-runs the test every
pass, so a restoring assignment releases the hold by itself. Held data stays queryable.
The batch is held whole rather than trimmed to the bar, because trimming still wipes the
store, just over several passes.

The hold is a bounded delay, not a veto. An indefinite one breaks two MUSTs outright:
a withheld chunk meets LIV-4's precondition exactly (in the available set, not in the
assignment, unpinned) and never leaves, and its bytes belong to no term of RS-3's excess
bound. It also strands disk on a worker the network legitimately shrank — and since
eviction precedes fetching in the loop, a worker that cannot free space needs twice the
headroom to converge on a large rebalance. So the batch goes through after
P-DEL-HOLD-MAX (default 1 h, DELETION_HOLD_MAX_SEC), off a timer the reconciliation
loop owns, which is what LIV-4's "without requiring any further input event" demands.

What the window buys is machine self-correction, not human response: a scheduler glitch
is corrected by the next publication, while a real shrink keeps being republished for the
whole window, so the two separate themselves. A bug that outlives the window still wipes
— this guards against a glitch, not a sustained fault. Raising the floor to 1 is the
operator override.

The floor counts every chunk the assignment dropped, pinned or not. A pin lasts one
query, so letting it shrink the batch would deal a wipe out in instalments instead of
holding it. Pins keep deciding which permitted removals run now (RS-2 precedence).

Spec consequences recorded rather than hand-waved: RS-3 gains an explicit H term for
held bytes bounded by P-DEL-HOLD-MAX, LIV-4's bound becomes P-DEL-HOLD-MAX + P-EVICT-BOUND for the hold path, and ADR-17 states plainly what the guard does not
cover.

Alarms

Both closures needed somewhere to report, which is OB-12's first real binding:
worker_alarms{reason} plus a per-instance Alarms snapshot. The conformance tests
assert the instance state — the gauges are process-global and would race across tests.

Testing

New tests/assignment_faults.rs (CT-4 assignment-fault corpus, wired into the CI
conformance job) with 7 tests; the unit tier grows to 47. Every fix was verified by
reverting it and confirming its test fails — including the hold's self-wake (the loop
hangs 30 s without the timer arm) and the pin-counting fix — so none of them pass
vacuously. Green on the unit tier, the conformance tier under both CI feature configs,
cargo fmt, the clippy gate, and check_spec.py.

Judgment calls worth a look

• The spec linter is tuned. Closing a gap deletes its register row, but ADR-14 cites
  GAP-2 and accepted ADRs are append-only, so the id dangles. id-undefined now exempts
  decisions/ for the GAP- prefix, and only for ADRs whose status is Accepted —
  proposed ones stay checked, which is how ADR-18's now-stale claim about the reader panic
  got caught and corrected. The alternative is keeping closed rows with a status column.
• A rejected document is not retried under the same assignment id (pre-existing: the
  id is recorded before application). Every rejection cause is a property of the document
  bytes, so re-fetching identical bytes is waste, and acknowledging the id only on success
  would loop forever on a genuinely bad document — GAP-9's head-of-line blocking. Recovery
  is bounded by the next published id. Say the word if you want it tracked as a gap row.
• The conformance job builds a 4th test binary against the 10-minute P-GATE-PR-TIME
  timeout. The tests run in under a second; the cost is one more ~300 MB link per matrix
  config. The timeout is unchanged — worth watching on the first run.

🤖 Generated with Claude Code

[define-null]

@mo4islona I think GAP-3 becomes problematic, because the local decision of the worker tries to be smarter than decision on the high-level (scheduler). If worker had a period of failing to fetch/apply old assignment, and now with a new assignment is asked to reshuffle the data there is less value keeping it, given that the portal might not even continue routing for those chunks in the first place. In that case such a worker would be just stuck with the old data, without possibility to recover.

Instead of having this on the worker the practical approach would be to have something similar on the scheduler, but that's again depends on the shape of the data. If suddenly couple of workers leave and the reshuffling change significantly the picture of chunk distributions it's not realistic to keep worker decide what it should and shouldn't keep.

AlsoLIV-4 is problematic on it's own, because of the eventual nature. With multistep scheduler it should leave immediatly before applying the new assignment, because otherwise we have a risk of overcomitting.