feat(cloud): cloud status reports the seen-txn ledger size per store (T13317)

.changeset/t12343-genesis-checkpoint.md

@@ -0,0 +1,38 @@
+---
+id: t12343-genesis-checkpoint
+tasks: [T12343, T13300, T13302]
+kind: feat
+summary: "`cleo sync enable push` - genesis cut plus genesis checkpoint (checkpoint/v3), behind the unreleased sync.push flag"
+---
+
+`cleo sync enable push [--scope project|global]` starts a store's change-journal push (journal spec §2.11 §10; T12343 S4-1b). It is
+refused while `sync.push` is unreleased; the core call `enableSyncPush` takes an `allowUnreleased` opt-in for tests and staging
+only.
+
+- **What it checks first:**
+  - the store's bound replica must be the one the stream knows (the link);
+  - this machine must have synced the stream's head (pull first);
+  - a stream whose head is already a journal checkpoint is refused with `E_NEXUS_SYNC_STREAM_JOURNALED`, so this store joins it by
+    pulling instead of writing a second genesis.
+- **Under the write lease:**
+  - It commits the genesis cut, then exports the store as the cut left it through the vault's bundle export
+    (`cutGenesisWithSnapshot`; the genesis marker holds every other writer off meanwhile). A failed export, or a write that raced
+    it, undoes the cut, and nothing is saved or pushed.
+  - It saves the bundle beside the store, then pushes a **checkpoint/v3** genesis: the vault manifest with empty
+    `pending`/`voided`/`revived`/`pruned` and the store's replay pin. The parent is the current head (null only on an empty
+    stream), and a count-delta segment is appended when the rows differ from the parent plus the window.
+  - A window that crosses a sync-schema rise is refused (its pin would need that rise's journal hash).
+- **After the checkpoint is stored:**
+  - `completeGenesis` raises the store's replicaSeq high-water mark to the replica's last segment on the stream (the vault's delta
+    segments spend this replica's seqs), and clears `genesis_pending`.
+  - The saved bundle is removed.
+  - A push that fails after the cut keeps the bundle and resumes on the next run (`status: 'resumed'`), without a second cut. A
+    cut that crashed before its bundle was saved is resumed at that cut, or re-cut if the store moved on (T13301).
+- **A genesis the server already stored is adopted, never cut twice (T13302).** `genesis_pending` means "not confirmed stored". If
+  a run died after the upload was accepted, the next run finds that checkpoint on the stream by the id recorded beside the bundle
+  before each upload (`Journal.pushCheckpoint` now takes a caller-minted `checkpointId`). It verifies the checkpoint, records it
+  (high-water mark, pending cleared), and pushes no second genesis, even after a local write.
+- **`cleo cloud push` refuses a store past its genesis cut.** Its changes travel as journal segments; a vault delta would count them
+  twice.
+- **New error codes:** `E_NEXUS_SYNC_REFUSED`, `E_NEXUS_SYNC_STREAM_JOURNALED`.
+- **Refactor:** the vault's replay, delta and checkpoint loop is now `appendCheckpoint`, shared by `cloud push` and the genesis push.

.changeset/t12343-genesis-snapshot.md

@@ -1,22 +1,36 @@
 ---
 id: t12343-genesis-snapshot
-tasks: [T12343]
+tasks: [T12343, T13296, T13297]
 kind: feat
-summary: Sync genesis cut snapshots its checkpoint bundle at the cut; other cleo processes wait or refuse cleanly
+summary: Sync genesis cut commits, then snapshots its checkpoint bundle under a genesis marker; writers wait or refuse cleanly
 ---
 
-The genesis checkpoint's bundle must be the store exactly at the cut (journal spec §2.11 §10). Otherwise a write made after the
-cut would travel both in the bundle and in a segment, and restore would apply it twice (T12343 S4-1b).
+The genesis checkpoint's bundle must be the store exactly as the cut left it (journal spec §2.11 §10; T12343 S4-1b). Otherwise a
+write made after the cut would travel both in the bundle and in a segment, and restore would apply it twice.
 
-- **`cutGenesisWithSnapshot(db, {…, dbPath}, snapshot)`** runs `snapshot(cut)` while the cut's `BEGIN IMMEDIATE` is held, before the
-  cut commits. A snapshot taken on another connection sees exactly the cut state, because nothing else can commit. A failing snapshot
-  rolls the cut back. `cutGenesis` shares the same two phases.
-- **A `genesis` store marker.** The export can outlast another writer's 30 s busy timeout, so for its duration the store carries a
-  `genesis` marker (the restore-in-progress marker, T13258). Another cleo process:
-  - waits at its store open, then refuses with `E_STORE_GENESIS` and a remedy;
-  - waits at the write chokepoint too (`assertExodusWriteSafe`, which now calls `awaitStoreWritable`, without blocking the event
-    loop), then refuses the same way. A write that waits it out lands after the snapshot.
+- **`cutGenesisWithSnapshot(db, {…, dbPath}, snapshot)`** commits the cut FIRST, then runs `snapshot(cut)` (T13296). The bundle
+  therefore carries everything the cut wrote: genesis row meta, the genesis keys, `undo_enabled`, `sync.push` and the folded
+  transactions. A device restored from it has full row meta and never pushes a folded transaction again.
+- **A failed snapshot undoes the cut.** It drops the stream's genesis keys, turns push off, returns the folded transactions to
+  `sealed`, and turns undo off (dropping undo written since) when no other stream is cut. The same happens when a write reached the
+  store during the snapshot (the capture position moved past the cut): `GenesisRacedError` (`E_SYNC_GENESIS_RACED`) is thrown, so no
+  bundle holding a post-cut write is ever pushed.
+- **A `genesis` store marker** (the restore-in-progress marker, T13258) covers the whole run:
+  - another process waits at its store open, then refuses with `E_STORE_GENESIS` and a remedy;
+  - every writer, in this process too (T13297), waits at the write chokepoint without blocking the event loop, then refuses the same
+    way. A write that waits it out lands after the snapshot.
 
-  The holder itself is exempt, and the marker is always released.
+  The write chokepoint is `assertExodusWriteSafe`, which now calls `awaitStoreWritable`. Its callers are the task accessor's write
+  transactions, session creation, `insertIdempotent` and `upsertIdempotent`. A writer that bypasses those is not paused; the race
+  check above catches its write instead.
+- **A marker that lapses undoes the cut.** The genesis cut checks its marker after the snapshot. The cut is undone and
+  `GenesisRacedError` (`E_SYNC_GENESIS_MARKER_LOST`) is thrown in either case:
+  - the marker is older than the stale window, so other processes may already have been writing;
+  - another process replaced the marker.
+
+  Nothing is recorded or uploaded.
+- **Marker window and release.** The stale window stays at the 1 h floor. The worst measured hold is about 63 s, on a scratch copy of
+  the 1.3 GB cleocode store: cut 2.4 s, export 48.7 s, extract and manifest 11.8 s. A marker is released only while it is still the
+  holder's own (`holdRestoreMarker`), so a lapsed holder never removes the marker of the process that took over.
 - **The restore-in-progress marker** (`store/restore-marker.ts`, `store/pid-alive.ts` and the open-path checks) comes from #1902
   (T13258), cherry-picked unchanged.

.changeset/t12343-pull.md

@@ -0,0 +1,34 @@
+---
+id: t12343-pull
+tasks: [T12343, T13306, T13307]
+kind: feat
+summary: Sync pull - stage and apply a stream's verified segments, never a transaction twice (S5-1)
+---
+
+A store can now pull its stream's journal (journal spec §3.1, §2.8; T12343 S5-1).
+
+- **`pullStream(db, …)`** (`store/sync/pull.ts`) refuses while `sync.pull` is off. Otherwise it pages through the stream after the
+  store's persisted cursor. For each page, in one
+  local transaction:
+  - **Decode:** `deflate-raw` canonical JSON of `LedgerTxn[]`, parsed with the ledger schema. A vault count-delta segment
+    (`cleo-vault-delta/v1`) carries no ops and is passed over.
+  - **Verify:** every transaction's signature is checked against the segment's device key, before anything of the page is written.
+    A bad signature or a malformed body refuses the segment. The pull stops there: what precedes it is staged and applied, the
+    cursor stops just before it, and the report's `refused` names its seq, replica and device (T13307).
+  - **Deduplicate:** every transaction id already staged from the stream is skipped (`_sync_seen_txn`), so a re-delivered
+    transaction is never staged or applied twice and a counter delta never doubles.
+  - **Stage and advance:** the rest are staged, and the cursor advances (`_sync_cursor`).
+
+  Then the applier applies, in stream order, own echoes included (they feed the fast path and the scoped rebase).
+- **`_sync_seen_txn` is not pruned yet.** A transaction already seen can come back in a new segment above any checkpoint, for
+  example a retired replica's late segment or a rebind's re-push. If its first-delivery row had been dropped, it would apply
+  twice. `pruneSeenTxns` exists, but the cloud pull calls it only once the receive watermark (T13256) provides a floor the
+  applier refuses to go below.
+- **`pullSyncStream(opts)`** (`cloud/nexus-vault.ts`) wires `pullStream` to the journal client and the trusted device keys. It refuses
+  a store bound to another replica than the link. With `sync.pull` off it refuses before any network call. A store whose last synced checkpoint is a vault snapshot from
+  before the stream's journal genesis is refused, with nothing pulled (T13306). The changes folded into that genesis never travel
+  as segments, so pulling across it would silently diverge. The remedy is to restore the journal checkpoint, then join with
+  `cleo sync enable push`. A store with no pull position starts after the checkpoint it last synced (its
+  genesis, or the snapshot it restored), seeded from that checkpoint's signed replica map.
+- **New sync-journal migration** `t12343-pull-cursor`, local-only: `_sync_cursor` and `_sync_seen_txn`. Both are cleared from vault
+  bundles, so a restored store resumes from its own checkpoint.

.changeset/t12343-push.md

@@ -0,0 +1,32 @@
+---
+id: t12343-push
+tasks: [T12343, T13304]
+kind: feat
+summary: Sync push - seal, persist and upload a stream's journal segments, behind the unreleased sync.push flag (S4-2)
+---
+
+A store past its genesis can now push its change journal (journal spec §2.8, §2.9, §1.3; T12343 S4-2).
+
+- **`pushStream(db, …)`** (`store/sync/push.ts`) runs these steps in order:
+  1. **Preconditions:** push is on, the stream is cut, and its genesis checkpoint is stored. No segment ever precedes the checkpoint
+     it extends.
+  2. **Local clock check:** if this device's clock runs more than `MAX_DRIFT` (5 min) ahead of the server's, push pauses and
+     records `sync.clock_ahead` for `cleo doctor`. The offset is measured when the server's `Date` arrives, so later time never
+     counts. Local writes are never refused, and the key clears once the clock recovers.
+  3. **Server-authoritative rollback check:** a later segment from this replica on the server than the store ever persisted means
+     the store was restored or copied behind it. Nothing is sent; it must rebind (T12753).
+  4. **Seal and pack:** pending captures are sealed (at most 8 passes, so steady writers never hold the push back), then packed
+     into persisted segments. The device registry's high-water mark
+     advances after each persist.
+  5. **Upload:** the persisted bytes are uploaded, lowest `replicaSeq` first, one at a time. A duplicate counts as stored, and a
+     failed upload is resent with the same bytes next run.
+- **`pushSyncStream(opts)`** (`cloud/nexus-vault.ts`) wires `pushStream` to the journal client:
+  - `sealSegment` and `push` with the persisted bytes;
+  - the device's transaction signatures;
+  - the server's clock offset (the new `Http.lastServerOffsetMs`, measured at receipt);
+  - E3's `replica.lastReplicaSeq` (project streams);
+  - the device registry.
+
+  It refuses, with nothing sealed or sent, when the store's bound replica is not the one the stream knows (T13304); push would
+  otherwise seal as one replica and never select the other's transactions. A segment conflict maps to `E_NEXUS_SYNC_REFUSED`
+  with the rebind remedy. `cleo cloud sync` (T12996) will call it.

.changeset/t12343-replay-pin.md

@@ -0,0 +1,22 @@
+---
+id: t12343-replay-pin
+tasks: [T12343, T13298]
+kind: feat
+summary: Sync checkpoints compute their replay pin - the ordered migration journal hash and a literal-safe trigger-set hash
+---
+
+A checkpoint/v3 manifest carries a `replayPin` (journal spec §2.11 §7). It records what an endorsing device must replay under to
+recompute the checkpoint's tallies (T12343 S4-1b).
+
+- **`migrationJournalHash(db)`** is sha256 over the whole ordered `__drizzle_migrations` journal, as `[created_at, name, hash]` in
+  `(created_at, name)` order. A drifted row below the head changes it.
+- **`triggerSetHash(db)`** is sha256 over the DDL of the store's owned triggers (`OWNED_TRIGGERS`: the guard and side-effect
+  triggers a replay runs under), sorted by name. Capture triggers (generated per build) and device-local maintenance triggers (the
+  twin-collapse docs freeze, whose message names the build that froze that store, and the legacy track triggers) are excluded.
+  Otherwise two devices with the same replay semantics would compute different pins and never endorse each other (T13298).
+- **`normalizeTriggerDdl(sql)`** normalizes that DDL literal-safely:
+  - comments are dropped;
+  - outside literals, whitespace collapses and words are lower-cased;
+  - string literals are kept byte for byte;
+  - every quoted identifier is written `"x"`.
+- **`replayPinOf(db, transitions)`** assembles the wire `ReplayPin`. A genesis checkpoint passes no transitions.

.changeset/t12996-cloud-sync.md

@@ -0,0 +1,24 @@
+---
+id: t12996-cloud-sync
+tasks: [T12996]
+kind: feat
+summary: "`cleo cloud sync` - seal, push, pull and apply each attached stream in one call, one result per stream"
+---
+
+`cleo cloud sync [--scope project|global]` runs the change journal end to end for each attached stream: the project's and the
+account's global store, or just the one `--scope` names (T12996).
+
+- **Per stream:** it seals pending writes, pushes them as segments (`pushSyncStream`), pulls the stream's new segments, and applies
+  them (`pullSyncStream`).
+- **One LAFS envelope** (`CloudSyncResult`) reports, per stream:
+  - `status`: `synced`, `disabled`, `paused` (device clock ahead), `not-attached`, `refused`, or `failed` (an error stopped this
+    stream, reported, never thrown, so the next stream still syncs);
+  - `skipped`: legs with nothing to do (a flag off, or push before this store has its own genesis), which never count as refused;
+  - sealed, built, sent, duplicates, received, staged, redelivered, applied, held and conflict counts;
+  - the staged position against the server's head.
+- **`E_SYNC_DISABLED`** is returned, naming `cleo sync enable push`, when no attached stream has `sync.push` or `sync.pull` on. A store
+  with `sync.pull` off now refuses its pull before it looks for a pull position.
+- **An interrupted run resumes on the next:** a segment is resent with the same bytes, and a transaction is never staged or applied
+  twice.
+- **One connection and one account-key unlock serve every stream of a run.**
+- `sync.push` and `sync.pull` stay unreleased: the CLI never opts in, so stores can't turn them on yet.

.changeset/t13256-idle-no-pingpong.md

@@ -0,0 +1,16 @@
+---
+id: t13256-idle-no-pingpong
+tasks: [T13256]
+kind: test
+summary: "Two idle replicas syncing in a loop exchange no segments after the first round (journal spec §3.5 R7-7)"
+---
+
+A new two-device test runs on the fake server. Device B joins the journal device A started, and each device makes one real
+write. One round of `cleo cloud sync` carries both writes across. Then four more rounds with no writes run on each device.
+
+- No device sends a segment in those rounds, and the stream's segment count does not move.
+- Applying the other device's segment never makes a device seal or send one back: the apply's echo is subtracted (§3.3).
+- Both stores end with the same rows.
+
+This test guards the R7-7 rule for when receive watermarks land: a watermark rides on an ordinary segment and is never sent
+alone after a pull.

.changeset/t13301-genesis-crash-resume.md

@@ -0,0 +1,16 @@
+---
+id: t13301-genesis-crash-resume
+tasks: [T13301]
+kind: fix
+summary: A genesis cut that crashed before its snapshot resumes on the next run; undoing a cut un-folds only its own transactions
+---
+
+The genesis cut commits before its snapshot (T13296), so a crash, kill or sleep mid-export used to leave the stream cut and
+`genesis_pending` set with no snapshot, and the next run returned `already` without ever producing the checkpoint.
+
+- **`cutGenesisWithSnapshot` now resumes such a cut.**
+  - With no capture since the crash, the store is still exactly the cut, and the snapshot runs at it (`resumed: true`).
+  - Otherwise the stale cut is undone and the store is cut again.
+  - A stored cut (`genesis_pending` cleared) is still `already`, and no snapshot runs.
+- **Undoing a cut reverts only the transactions that cut folded** (`genesis_folded_from` to `genesis_folded_upto`). A store feeding
+  two streams no longer un-folds the other stream's earlier fold.

.changeset/t13312-journal-join.md

@@ -0,0 +1,30 @@
+---
+id: t13312-journal-join
+tasks: [T13312, T12999, T12996]
+kind: feat
+summary: "`cleo sync enable push` joins a stream another device journaled, from the journal checkpoint this store restored"
+---
+
+A second device can now enter a stream whose change journal another device already started (T13312, T12999). Before this, a
+store restored from the journal checkpoint carried no row meta and no join state, so `cleo sync enable push` refused it and
+there was no way in.
+
+- **Restore keeps the journal's row state.** On a journal checkpoint, the restore keeps the snapshot's `_sync_row_meta` and
+  `_sync_field_leave` rows instead of clearing them as machine-local.
+- **The join.** On a stream whose head is a journal checkpoint this store restored, `cleo sync enable push` returns status
+  `joined`:
+  - the store's cut is placed at its current capture position;
+  - the ledgers are baselined;
+  - push, pull and undo are turned on;
+  - pulling resumes right after the restored checkpoint.
+  No second genesis is pushed.
+- **A changed store is refused, never folded.** The join is refused with `E_NEXUS_SYNC_REFUSED` when:
+  - any table's rows differ from the checkpoint's manifest;
+  - any row no longer matches its restored row meta;
+  - a capture is still waiting to seal;
+  - a write landed after the check, which is caught inside the cut's own transaction, so it is never baselined as
+    checkpoint state.
+  The remedy is to restore the checkpoint again (`cleo cloud restore --force`), then join. A refused join changes nothing: capture is left as it was found, and sealing stays off.
+- **A store that never restored the journal checkpoint** gets `E_NEXUS_SYNC_STREAM_JOURNALED`, naming the restore-then-join
+  path. The pull refusal for a store behind the journal genesis (T13306) names the same path.
+- `sync.push` and `sync.pull` stay unreleased.

.changeset/t13315-typed-skip.md

@@ -0,0 +1,19 @@
+---
+id: t13315-typed-skip
+tasks: [T13315]
+kind: refactor
+summary: "`cleo cloud sync` sorts skipped legs by a typed refusal kind instead of matching refusal message text"
+---
+
+`cleo cloud sync` used to decide whether a push or pull leg was skipped or refused by matching the refusal's message
+against regexes. Rewording a refusal would silently turn a skip (`sync.push` off, no genesis yet) into a reported refusal.
+
+- **Push results** (`PushStreamReport`) now carry `refusedKind`, which is one of:
+  - `schema-missing`
+  - `push-off`
+  - `no-genesis`
+  - `genesis-pending`
+  - `store-behind`
+- **Pull results** (`PullStreamReport`) now carry `refusedKind`, which is `pull-off` or `segment`.
+- **Classification** is done by `classifySyncLegs`, using only these kinds. The message regexes are removed. A stream is
+  `disabled` when both legs are off. The pull position is reported unless the pull leg was skipped.