unii: add Phase 2 durable storage and checkpoint replay

unii/docs/HANDOFF.md

@@ -1,4 +1,4 @@
-# unii handoff: Phase 0 and Phase 1
+# unii handoff: Phase 0 through Phase 2
 
 unii is a provisional name. It is the memory engine for an endless
 conversation agent, built from the plan in this repository's task, *Shen and
@@ -74,7 +74,10 @@ unii/
     core.lua            boots shen-lua, checks the build stamp, calls the typed core
     supervisor.lua      single writer: validate, transition, journal and fsync, adopt; replay
     storage.lua         framed journal adapter (interface in docs/contracts/storage.md)
-    posix.lua           FFI open/write-all/fsync/flock/ftruncate/rename/atomic write
+    posix.lua           FFI durability operations and fault boundaries
+    storage_blobs.lua   content-addressed message/summary bodies
+    storage_checkpoint.lua durable checkpoint framing and fallback
+    storage_index.lua   rebuildable journal/message/blob/node/job indexes
     sha256.lua          pure-Lua SHA-256
     models.lua          provider interface, adapter-result classification
     network/            REAL libcurl adapter (PR #75; its README covers TLS, logging, limits)
@@ -132,14 +135,17 @@ The oracle's canonical-text function and the byte-hysteresis fixture were
 changed to the reconciled policy. The oracle remains a separate
 implementation.
 
-**Phase 2 storage.** It is being built on a branch off this one. The
-`host/storage.lua` interface is unchanged. The dispatch record is an
-ordinary payload, so a new backend needs nothing extra.
+**Phase 2 storage.** Content-addressed blobs, daily shards, checkpoints,
+indexes and streaming replay are integrated behind the unchanged
+`host/storage.lua` interface. Checkpoints preserve outstanding commands and
+their dispatch intents, complete round/candidate state and provisional view
+lines. An in-flight command restored from a checkpoint becomes uncertain and
+is not resent.
 
 ## Acceptance gates and how each was verified
 
 Every row is an automated test in `luajit unii/test/run.lua`. The last run
-passed 93 tests, with 0 failed and 0 skipped. `--with-upstream` adds the
+passed 108 tests, with 0 failed and 0 skipped. `--with-upstream` adds the
 pinned port specs (1,088 checks across 22 specs) and the kernel suite
 (134 of 134).
 
@@ -199,6 +205,20 @@ same property at the API level, plus crash tails at every byte offset,
 corruption refusal, lock contention across processes, tampered
 transactions, foreign bundles and configuration changes.
 
+### Phase 2
+
+`test_storage_phase2` proves that journals hold blob hashes rather than
+message/summary bodies, sequence order crosses daily shards, indexes rebuild,
+valid checkpoints skip earlier records with identical state/view hashes,
+stale checkpoints fall back, bad blobs refuse without journal mutation, and
+recovery does not read whole journal files into RAM. It also covers
+hash-only `summary-uncertain` raw content and checkpoint restoration of the
+round's best candidate and provisional line.
+
+`test_storage_faults` injects process death at every observed write, sync and
+rename boundary, and separately covers commit ordering, torn writes and
+ENOSPC. See `docs/STORAGE_PHASE2.md` for the outcome matrix.
+
 ## Decisions worth knowing
 
 * **Leaf cap.** The 512-byte cap applies to `kind ": " text`. A user
@@ -270,11 +290,9 @@ been reported upstream from this branch.
   * policy epoch change;
   * message chunking (events longer than `chunk_max` are rejected);
   * compaction context and a search fallback.
-* **Storage.** One journal file holds everything. It is read entirely into
-  RAM, replay starts from record 1, and message text is held in RAM. There
-  are no blobs, checkpoints, indexes, day shards or backup/restore. Fault
-  injection covers only a failed append. On macOS, `fsync` has weaker
-  guarantees, and macOS has not been tested.
+* **Storage limits.** Backup/restore and blob garbage collection do not
+  exist. Index metadata is proportional to history. Linux x86_64 is tested;
+  macOS uses `F_FULLFSYNC` for regular files but is not exercised in CI.
 * **Complexity.** View, live-node and job collections are Shen lists, so
   each transition is O(view + live + jobs). That is a few hundred lines at
   the default thresholds. The one-million-message gate in Phase 7 has not
@@ -287,13 +305,9 @@ been reported upstream from this branch.
 
 ## Suggested next steps
 
-1. Phase 2 storage: content-addressed blobs for message text, checkpoints
-   (view, state, journal position) so replay does not start at record 1,
-   rebuildable indexes, and fault injection around every write, fsync and
-   rename.
-2. Run the chat-completions provider against a real endpoint with a key,
+1. Run the chat-completions provider against a real endpoint with a key,
    and replace the provisional prompt with the plan's (Phase 3).
-3. Add timers as host events (`TimerObserved`) for retry backoff.
-4. Add message chunking on UTF-8 boundaries, upstream of `message-appended`.
-5. Turns and prompts (Phase 4) on top of `unii.view-ready?` and the
+2. Add timers as host events (`TimerObserved`) for retry backoff.
+3. Add message chunking on UTF-8 boundaries, upstream of `message-appended`.
+4. Turns and prompts (Phase 4) on top of `unii.view-ready?` and the
    rendered view.

unii/docs/STORAGE_PHASE2.md

@@ -0,0 +1,55 @@
+# Phase 2 storage and recovery
+
+The journal remains the sole authority. Blobs hold message and accepted
+summary text; checkpoints and indexes can always be discarded and rebuilt.
+
+## Fault outcomes
+
+| Fault | Defined outcome | Automated evidence |
+|---|---|---|
+| Crash before/during blob write, sync or rename | No journal record; any temp/orphan is ignored | `test_storage_faults` boundary sweep |
+| Crash after blob durability, before journal completion | Blob may be orphaned; incomplete journal bytes are set aside; no record is fabricated | boundary sweep and torn-write case |
+| Crash after a complete frame write, before/after sync | On the test filesystem the checksum-valid frame recovers; callers must still treat a failed append as unacknowledged | boundary sweep |
+| Incomplete final frame at every byte | Preserve `journal.tail-*`, truncate to the committed prefix, continue | `test_storage` exhaustive cut test |
+| Bad header/checksum/separator/sequence in committed data | Refuse with `repair required`; do not rewrite the shard | `test_storage` corruption cases |
+| Missing or corrupted committed blob | Refuse; journal remains byte-identical | `test_storage_phase2` |
+| Corrupt or stale checkpoint | Rename `.invalid-*`, try an older valid checkpoint, otherwise replay from record one | `test_storage_phase2` |
+| Stale/corrupt/missing index | Rebuild from validated journal and blobs | indexes are replaced on open |
+| ENOSPC before blob commit | Refuse append; no journal or supervisor state mutation | `test_storage_faults` and `test_storage` |
+| ENOSPC/torn journal append | Refuse append; isolate partial final frame on restart | `test_storage_faults` |
+| Second owner | Refuse nonblocking OS `flock`; no PID-file inference | `test_storage` same- and cross-process cases |
+
+The boundary sweeps discover every write, file/directory sync and rename
+callback exercised by blob, shard-manifest, journal, index and checkpoint
+commits, then repeat the operation with a simulated process death at each
+callback. Journal recovery must produce either zero records or the one
+complete transaction, never a partial or invented transaction; checkpoint
+recovery must preserve the journal-derived state.
+
+After any mutating append error, the store handle is poisoned and refuses
+later appends until it is closed and recovery reopens it. This prevents
+sequence reuse or appending after an unknown partial frame.
+
+## Checkpoint proof
+
+A checkpoint binds:
+
+* physical journal sequence and physical-payload hash;
+* rule bundle and complete policy configuration;
+* canonical serialized Shen state and state hash;
+* exact rendered view hash and revision;
+* pending summary commands.
+
+Recovery validates all fields, restores the state, and starts transition
+replay at `checkpoint.seq + 1`. Tests compare final state and view bytes/hashes
+with uninterrupted execution and assert the number of replayed records.
+
+## Remaining limits
+
+* Blob garbage collection and backup/restore are not implemented.
+* Indexes are periodically/close-time replaced rather than incrementally
+  paged; their in-memory metadata is proportional to record/message count.
+* Checkpoint files retain the newest four manifest entries; invalid and
+  orphaned diagnostic files require operator cleanup.
+* Linux x86_64 is tested. macOS uses `F_FULLFSYNC` for regular files, but
+  filesystem/device durability still needs platform testing.

unii/docs/contracts/storage.md

@@ -1,8 +1,9 @@
-# Storage (storage_format 1)
+# Storage (storage_format 2)
 
-A chat is stored in one directory, which one process owns at a time. Phase 1
-has one storage adapter: a framed append journal (`host/storage.lua`) with
-POSIX helpers in `host/posix.lua`, which use the LuaJIT FFI.
+A chat is stored in one directory, which one process owns at a time. Existing
+callers keep the same storage interface, while Phase 2 adds content-addressed
+blobs, daily journal shards, checkpoints and rebuildable indexes. POSIX
+durability operations remain isolated in `host/posix.lua`.
 
 ## Interface
 
@@ -13,19 +14,29 @@ store:append(bytes) -> seq              -- write all bytes, then fsync, before r
 store:close()                           -- release the lock
 ```
 
-Another backend (with blobs, checkpoints or shards) can replace this one
-without changing the supervisor, as long as it keeps the same contract:
-records come back in the order they were appended, `append` is durable
-when it returns, and the store has a single owner.
+`records()` remains compatible, but its payload fields are lazy: one framed
+record is read and hydrated at a time. `iter_records(after_seq)` is the
+streaming replay path. `append` does not return until blob, journal and
+directory durability boundaries have succeeded.
 
 ## Directory layout
 
 | File | Purpose |
 |---|---|
 | `LOCK` | holds the `flock(LOCK_EX \| LOCK_NB)` lock, open for the lifetime of the store. A second owner, in this process or another, gets `chat <dir> is owned by another process` |
-| `journal.uj` | the journal |
+| `journals/MANIFEST` | ordered `USM1` list of daily shards |
+| `journals/YYYY-MM-DD.uj` | framed records; sequence numbers remain global across shards |
+| `blobs/<sha256>` | immutable message and accepted-summary UTF-8 bytes |
+| `checkpoints/checkpoint-<seq>-<digest>.uc` | exact serialized core state and recovery metadata |
+| `checkpoints/MANIFEST` | newest four checkpoint candidates |
+| `indexes/{journal,messages,blobs,nodes,jobs}.idx` | rebuildable projections, never authority |
 | `journal.tail-<offset>-<n>.bin` | uncommitted crash tail kept for diagnostics: the `n` bytes that followed committed data at `offset` |
 
+The init record uses the `0000-00-00` shard. A `message-appended` host event
+selects its recorded UTC day; records without a date remain on the active
+shard. Creating a shard syncs the empty file, journal directory and manifest
+before a frame can be appended.
+
 ## Frame
 
 ```
@@ -40,7 +51,13 @@ when it returns, and the store has a single owner.
 
 When the journal is opened:
 
-* **Complete frames:** read in order.
+* **Complete frames:** stream in manifest order. Only one bounded frame is
+  resident. Global sequence continuity and every frame checksum are checked.
+* **Blob references:** message text, summary output and the raw leaf carried
+  by `summary-uncertain` are absent from physical frames. Their byte count
+  and SHA-256 fields name the blob. Every committed reference is size- and
+  hash-verified before replay. Missing or corrupted blobs refuse open with
+  `repair required`; journal and blob bytes are not changed.
 * **Incomplete final frame:** treated as an uncommitted crash tail. This
   covers a partial header shorter than 64 bytes and a frame that ends before
   its checksum line. The tail bytes are written to a `journal.tail-*` file
@@ -52,6 +69,11 @@ When the journal is opened:
   (repair required)`, and the file is left unchanged. This covers a bad
   header, a checksum mismatch, a missing separator or a sequence gap
   anywhere before the end. No history is discarded.
+* **Checkpoints:** newest to oldest, verify framing, journal sequence and
+  physical-record anchor, bundle/configuration, serialized state hash,
+  rendered view hash/revision, pending commands and their dispatch intents.
+  A bad or stale checkpoint is renamed `.invalid-*`; recovery tries an older
+  one, then record one.
 
 ## Transaction payloads
 
@@ -97,13 +119,23 @@ storage interface is unchanged: this is one more payload through `append`.
 
 ## Commit order (supervisor)
 
-1. The boundary validates the event.
-2. The pure transition runs.
+1. The boundary validates the event and the pure transition runs.
+2. Each new message or summary blob is written to a temporary file, checked,
+   file-synced, atomically renamed and its directory synced.
 3. The view is rendered and its bytes are checked against the core's
    accounting. The view and the state are hashed.
-4. The transaction is appended, using a write-all loop followed by fsync.
+4. The hash-only transaction is appended with a write-all loop, the shard is
+   synced, then the journal directory is synced.
 5. Only after that does the supervisor adopt the new state and dispatch
    commands. If step 4 fails, the previous state is kept.
+6. Checkpoints and indexes are written by temp/write/sync/rename/directory
+   sync. Their failure cannot undo or compete with an authoritative journal
+   commit.
+
+Any failed mutating append poisons that store handle. It refuses every later
+append until the process closes and reopens the chat, because the failure may
+have left either a torn frame or a complete frame whose sync result is
+uncertain.
 
 ## Replay
 
@@ -113,7 +145,10 @@ When the store is opened:
   unsupported format version, or a configuration different from the one
   passed in. Changing the policy epoch needs an epoch event that does not
   exist yet.
-* Every transaction is replayed through the same transition, and the
+* The latest valid checkpoint restores exact core state (including summary
+  round candidates and provisional lines), view, pending commands, dispatch
+  intents and journal position. Only later transactions are replayed.
+* Every replayed transaction runs through the same transition, and the
   commands, decisions, view hash and state hash must match exactly.
   Otherwise the open fails with `replay divergence at seq N: <what>`.
 * The view is never re-fitted from the current policy. It comes only from
@@ -129,15 +164,18 @@ When the store is opened:
 
 ## Durability and limits
 
-* Durability depends on `fsync` of the journal and, when the journal is
-  created, of the directory. On macOS, `fsync` does not flush the drive
-  cache (`F_FULLFSYNC` is not used), so its guarantee is weaker. Only Linux
-  x86_64 has been verified.
-* The whole journal is read into memory on open, and replay starts from
-  record 1. Checkpoints, indexes, blobs, day shards and backup/restore do
-  not exist yet.
-* Message text is stored only inside journal transactions, and the
-  supervisor keeps a map from message id to text in RAM. The plan requires
-  bounded memory and paging, which is still to do.
-* Fault injection exists only for a failed append (`test_storage.lua`). It
-  does not yet cover every write, sync and rename boundary.
+* Linux uses `fsync`. macOS regular files use `fcntl(F_FULLFSYNC)` because
+  macOS `fsync` may not flush volatile drive caches; directories still use
+  `fsync`. Actual guarantees remain subject to filesystem, device firmware
+  and mount settings. Linux x86_64 is the verified platform; macOS behavior
+  is implemented but not exercised in CI.
+* Recovery stores bounded frame metadata and rebuildable indexes in memory,
+  not journal payloads. Core state still contains its bounded active view and
+  queues. Message source text is loaded from blobs on demand.
+* Checkpoints are taken every 32 records by default and explicitly on demand.
+  At most four are named by the checkpoint manifest.
+* `test_storage_faults.lua` injects process death at every observed write,
+  sync and rename boundary, plus torn writes and ENOSPC. Complete durable
+  frames recover; incomplete final frames are set aside; committed
+  corruption/missing blobs refuse without rewriting authority.
+* Backup/restore is not implemented.