Read agent sessions from cmux, with a registry fallback for sessions cmux missed

CHANGELOG.md

@@ -9,6 +9,81 @@ The version is carried in `pyproject.toml` and `src/assistant/__init__.py`
 
 ## [Unreleased]
 
+### Changed
+- Read agent sessions from cmux first, with mukul-memory's registry as a
+  fallback. cmux 0.64.25 records agent sessions itself, and
+  `cmux sessions list --json --all` prints those records without the cmux
+  socket. The new `bin/cmux_sessions.py` is the one place that runs it. It
+  waits up to 15 seconds, and when cmux can't answer it returns no records and
+  writes one stderr line. `world-scanner.py`, `session-context-watcher.py`,
+  `build-ws-context.py`, `transcript-tail.py --closed-cwd`, and
+  `cmux-restore-sessions` read it instead of `~/.cmux-session-ledger.jsonl`.
+  - cmux misses a session whose SessionStart never reached it. On 2026-10-19
+    that was ee4ee641 in workspace:27. For such sessions every reader also
+    takes the row from `~/.claude/cmux-registry.json`, written by mukul-memory's
+    `cmux-registry.py` hook.
+  - `world.json` `_meta.session_record_fallback` counts the sessions that came
+    from that fallback, and the scanner log shows `fallback_hits=N`. Retire the
+    fallback after a soak period with zero hits.
+  - A registry row with a live pid must still be a `claude` or `droid` process.
+    This drops sessions like aea1111b, whose recorded pid now belongs to another
+    program.
+  - On live state the scanner verifies the same 10 sessions as before. Only
+    the aea1111b ghost drops.
+  - Only the session cmux marks active on a surface counts as running there, so
+    a session left behind by `/clear` in the same process no longer looks live.
+  - Output fields keep their names, and `registry_live_pid` and
+    `session_start_registry` keep their values. Some values now come from
+    different sources:
+    - **First recorded** comes from cmux's `started_at`, and from a registry
+      row's SessionStart time for fallback sessions. A session restored into
+      another workspace shows its original start, not when it arrived there.
+    - A session's `cwd` is its launch directory, and `ts` is the record's
+      latest update.
+    - `transcript-tail.py --closed-cwd` picks the most recently active session.
+    - Sessions of agents the fleet doesn't run, such as Copilot, are left out.
+- Make `hooks/cmux-auto-resume.py` forward reliably. It's what makes cmux
+  record agents started without cmux's Claude wrapper, such as an absolute
+  `~/.local/bin/claude` like the warm comms session, and the scheduled fleet
+  spawns.
+  - The hook now starts a detached worker and returns at once.
+  - The worker checks each `cmux hooks <provider> session-start` exit code.
+    It then confirms the record with `cmux sessions list --session <id>`.
+  - It retries after 0, 2, 5, 15, and 30 seconds.
+  - A session cmux still lacks gets one `cmux-session-forward-fail` entry in
+    `~/.assistant/actions-ledger.jsonl`.
+  - Before, one try was made, with a 5-second timeout and its errors ignored.
+- `install/patch-settings.py` registers `cmux-auto-resume.py` on SessionStart in
+  Claude settings and Factory hooks when it's missing. A fresh machine used to
+  link the hook without registering it. The file is written without escaping
+  non-ASCII text.
+- `cmux-restore-sessions` finds a session for a panel with no resume binding in
+  cmux's records, then in the registry. The ledger is no longer used. Native
+  `cmux restore --surface` needs a binding, so this CLI stays.
+- Measure Python under `install/` and `hooks/` in the new-code coverage gate,
+  so changes there can pass it.
+
+### Removed
+- Retire `hooks/cmux-session-ledger.py`. For this release it's a no-op, so a
+  registration that outlives `git pull` still exits 0.
+  `install/patch-settings.py` now removes its SessionStart and SessionEnd
+  registrations from `~/.claude/settings.json` and `~/.factory/hooks.json`
+  instead of adding them. Delete the stub and its install.sh link next release.
+  `~/.cmux-session-ledger.jsonl` stays on disk, but nothing writes to it any
+  more. mukul-memory's `cmux-registry.py` registration is left alone.
+
+### Rollback
+- Revert the merge commit, then run `install.sh --apply`. That restores the
+  readers on registry and ledger, `cmux-restore-sessions` on the ledger, and
+  the single fire-and-forget forward. The base `patch-settings.py` also
+  re-registers the ledger hooks on SessionStart and SessionEnd, and the ledger
+  file resumes from where it stopped.
+- The `cmux-auto-resume.py` registration this release added on a fresh machine
+  stays, because the base installer never removes hooks. It's harmless: the
+  base hook makes the same single forward.
+- Ledger events between deploy and rollback are lost, so **First recorded**
+  for sessions started in that window is unknown until they restart.
+
 ### Added
 - Add the `cmux-ws-numberer` LaunchAgent as an opt-in feature. It adds a `[N]`
   suffix to cmux workspace titles and gives each workspace an unused sidebar

README.md

@@ -224,8 +224,9 @@ and next action back to the original session. Copying never sends it automatical
 
 Saved context must match the current workspace and session identities.
 Missing or unsupported session bindings stay unknown instead of borrowing another
-session's notes. **First recorded** means the earliest matching start in your
-local session history, not the session's original creation date. Pending task
+session's notes. **First recorded** is the session's start as cmux recorded it,
+or its SessionStart in `~/.claude/cmux-registry.json` when cmux has no record; a
+session restored into another workspace keeps its original start. Pending task
 dates come from their creation records. The updated `/back-off` skill saves the
 workspace identity so the overview can confirm **Parked**. If you run the dashboard
 from a separate checkout, use its displayed confirmation command for older pause

bin/build-ws-context.py

@@ -12,7 +12,8 @@
     "title": "...",
     "cwd": "...",
     "transcript_path": "/Users/.../<session>.jsonl" | null,
-    "transcript_source": "screen_session_id" | "registry_live_pid" | null,
+    "transcript_source": "screen_session_id" | "cmux_resume_binding"
+                         | "registry_live_pid" | null,
     "session_id8": "<8hex>" | null,
     "agent_surface": "surface:N" | null,
     "observation_identity": {
@@ -44,10 +45,13 @@
     one carrying a Claude status bar / boot banner. Its screen is `screen_text`.
   - `transcript_path` resolves from the session-id the agent stamps on its OWN
     status bar (`… │ #<8hex>`), then confirms the file's internal `sessionId`
-    matches — exact, self-verifying. Fallback: a cmux-registry row for the
-    agent pane's surface UUID, but ONLY when its `claude_pid` is still alive
-    AND it agrees with the live screen (the registry goes stale on surface
-    reuse — the ws:12 trap). `transcript_source` records which gate passed.
+    matches — exact, self-verifying. Next: the live cmux resume binding for
+    the pane. Last: cmux's own session record for the agent pane's surface
+    UUID (`cmux sessions list`), or the cmux-registry.json row for a session
+    cmux missed, but ONLY when its pid is still alive AND it
+    agrees with the live screen (a record goes stale on surface reuse — the
+    ws:12 trap). `transcript_source` records which gate passed; that last gate
+    keeps its historical name, `registry_live_pid`.
 
 The Observer is still told to trust the screen over transcript-derived signals
 on conflict — the screen is what the agent is showing RIGHT NOW.
@@ -63,7 +67,11 @@
 import time
 from pathlib import Path
 
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import cmux_sessions  # noqa: E402
+
 HOME = Path(os.environ["HOME"])
+CMUX_REGISTRY = HOME / ".claude/cmux-registry.json"
 PROTECTED_REFS = {"workspace:3", "workspace:108", "workspace:7"}
 CMUX_BIN = os.environ.get(
     "CMUX_BIN", "/Applications/cmux.app/Contents/Resources/bin/cmux")
@@ -119,10 +127,10 @@
 #   - `read-screen --workspace` reads only the FOCUSED pane; in a split
 #     workspace the focused pane is often a shell, not the agent. We enumerate
 #     ALL panes and pick the agent.
-#   - The cmux registry's surface→session map goes STALE when a surface is
-#     reused (ws:12: registry pointed at a dead headless pulse's session while
-#     the live agent ran a different one). Registry entries are trusted only
-#     when their claude_pid is still alive AND they agree with the live screen.
+#   - A surface→session record goes STALE when a surface is reused (ws:12: the
+#     record pointed at a dead headless pulse's session while the live agent
+#     ran a different one). cmux session records are trusted only when their
+#     pid is still alive AND they agree with the live screen.
 
 # The Claude Code status bar's last cell is the running session-id prefix,
 # rendered as `… │ #<8hex>` at the END of a box-drawing line. Anchoring to that
@@ -340,44 +348,47 @@ def transcript_from_session_id(sid8: str | None) -> str | None:
     return None
 
 
-def registry_transcript_for_surface(surface_uuid: str, expect_sid8: str | None = None) -> dict | None:
-    """Transcript for a surface via the cmux registry — SAFE variant.
+def recorded_transcript_for_surface(surface_uuid: str, expect_sid8: str | None = None) -> dict | None:
+    """Transcript for a surface via its session record — SAFE variant.
 
-    The registry maps surface UUID → {session_id, claude_pid, transcript_path},
-    but the mapping goes stale when a surface is reused. We therefore return an
-    entry ONLY when its claude_pid is still ALIVE (a dead pid means the entry
-    describes a session that no longer occupies the surface — the ws:12 trap).
-    When `expect_sid8` is given (a live status-bar id), the registry must AGREE
-    with it; on disagreement we trust the screen and return None here.
+    cmux records which session is active on each surface UUID, with its pid
+    and transcript; for a session cmux missed, the registry's row for the
+    surface stands in (the newest, if several). A record goes stale when a
+    surface is reused, so we return one ONLY when its pid is still ALIVE (a
+    dead pid means the record describes a session that no longer occupies the
+    surface — the ws:12 trap). Two cmux records for the surface are ambiguous
+    and return None. When
+    `expect_sid8` is given (a live status-bar id), the record must AGREE with
+    it; on disagreement we trust the screen and return None here.
 
-    Returns {transcript_path, session_id8} or None.
+    Returns {transcript_path, session_id8, agent} or None.
     """
     if not surface_uuid:
         return None
-    try:
-        reg = json.load(open(HOME / ".claude/cmux-registry.json"))
-    except Exception:
-        return None
     want = surface_uuid.upper()
-    entries = [e for e in reg.values()
-               if (e.get("surface_id") or "").upper() == want]
-    if not entries:
+    cmux = cmux_sessions.list_sessions()
+    records = [r for r in cmux if r.active_for_surface and r.provider
+               and (r.surface_id or "").upper() == want]
+    if not records:
+        rows = [r for r in cmux_sessions.registry_fallback(CMUX_REGISTRY, cmux)
+                if (r.surface_id or "").upper() == want]
+        records = [max(rows, key=lambda r: r.updated_at or 0)] if rows else []
+    if len(records) != 1:
         return None
-    entry = max(entries, key=lambda e: e.get("ts", 0))
-    if not _pid_alive(entry.get("agent_pid") or entry.get("claude_pid")):
-        return None  # stale registry row — do NOT trust
-    sid = (entry.get("session_id") or "")
-    sid8 = sid[:8] if sid else None
-    if expect_sid8 and sid8 and sid8 != expect_sid8:
-        return None  # registry disagrees with the live screen → screen wins
-    tp = entry.get("transcript_path")
+    record = records[0]
+    if not _pid_alive(record.pid):
+        return None  # stale record — do NOT trust
+    sid8 = record.session_id[:8]
+    if expect_sid8 and sid8 != expect_sid8:
+        return None  # the record disagrees with the live screen → screen wins
+    tp = record.transcript_path
     if not tp or not os.path.exists(tp):
         return None
     # Final identity gate: the file must self-identify as this session.
     internal = _transcript_internal_sid(tp)
-    if internal and sid8 and not internal.startswith(sid8):
+    if internal and not internal.startswith(sid8):
         return None
-    return {"transcript_path": tp, "session_id8": sid8}
+    return {"transcript_path": tp, "session_id8": sid8, "agent": record.provider}
 
 
 def resume_binding_for_surface(surface_ref: str, ws_ref: str) -> dict | None:
@@ -498,10 +509,13 @@ def resolve_workspace_screen_and_transcript(ws_ref: str) -> dict:
     Resolution priority (first verified wins; otherwise None):
       1. screen_session_id — id from the agent's own status bar, glob+verified.
          Strongest: it's what the running agent says about itself, right now.
-      2. registry_live_pid — registry row for the agent pane's surface UUID,
-         but only with a LIVE pid and agreement with the screen. Covers a
-         booting agent whose status bar hasn't rendered yet.
-      3. None — no verified signal. The Observer falls back to screen_text.
+      2. cmux_resume_binding — the live cmux resume binding for the pane.
+      3. registry_live_pid — cmux's session record for the agent pane's
+         surface UUID (or the registry's row, for a session cmux missed),
+         but only with a LIVE pid and agreement with the screen
+         and pane agent. Covers a booting agent whose status bar hasn't
+         rendered yet.
+      4. None — no verified signal. The Observer falls back to screen_text.
          (We deliberately do NOT guess by mtime/cwd/title — that was the bug.)
     """
     pane = find_agent_pane(ws_ref)
@@ -530,11 +544,11 @@ def resolve_workspace_screen_and_transcript(ws_ref: str) -> dict:
                 source = "cmux_resume_binding"
 
     if transcript is None and surface_uuid:
-        reg = registry_transcript_for_surface(surface_uuid, expect_sid8=sid8)
-        if reg:
-            transcript = reg["transcript_path"]
-            sid8 = sid8 or reg["session_id8"]
-            agent = agent or "claude"
+        recorded = recorded_transcript_for_surface(surface_uuid, expect_sid8=sid8)
+        if recorded and (not agent or recorded["agent"] == agent):
+            transcript = recorded["transcript_path"]
+            sid8 = sid8 or recorded["session_id8"]
+            agent = agent or recorded["agent"]
             source = "registry_live_pid"
 
     return {
@@ -641,8 +655,8 @@ def main() -> int:
     args = ap.parse_args()
 
     # Find the agent pane, read its screen, and resolve its transcript using
-    # ONLY verified signals (status-bar session id, or a live-pid registry row
-    # that agrees with the screen). transcript_path is None when nothing
+    # ONLY verified signals (status-bar session id, cmux resume binding, or a
+    # live-pid cmux session record that agrees with the screen). transcript_path is None when nothing
     # verifies — by design: a wrong transcript is worse than none.
     observed_at = time.time()
     identity_before = workspace_identity_snapshot(args.ws_ref)

bin/cmux-restore-sessions.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 """
-Layer 3: cmux-restore-sessions — one-shot session recovery CLI.
+cmux-restore-sessions — one-shot session recovery CLI.
 
 Run after a manual cmux restart (or at boot) to respawn Claude Code or
 Factory Droid panels whose sessions did not auto-resume.
@@ -10,8 +10,10 @@
 2. For each panel that has a resumeBinding but NO live agent PID:
    a. If autoResume=true, cmux handles it, so skip.
    b. If the resumeBinding has a checkpointId (session_id), respawn.
-3. For panels with NO resumeBinding at all, fall back to the ledger
-   (~/.cmux-session-ledger.jsonl) to find the last session for that panel.
+3. For panels with NO resumeBinding at all, fall back to the session last
+   recorded for that panel: cmux's own record (`cmux sessions list`), or the
+   ~/.claude/cmux-registry.json row for a session cmux never recorded. Native
+   `cmux restore --surface` needs a binding, so this covers what it can't.
 4. Respawn by sending the resume command to the panel's surface.
 
 Usage:
@@ -25,7 +27,10 @@
 import sys
 from pathlib import Path
 
-LEDGER = Path.home() / ".cmux-session-ledger.jsonl"
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+import cmux_sessions  # noqa: E402
+
+REGISTRY = Path.home() / ".claude/cmux-registry.json"
 CMUX_STATE = Path.home() / "Library/Application Support/cmux/session-com.cmuxterm.app.json"
 
 
@@ -45,33 +50,28 @@ def load_state():
         return json.load(f)
 
 
-def load_ledger():
-    """Returns {panel_id: last_start_entry} from the ledger."""
-    if not LEDGER.exists():
-        return {}
-    starts = {}
-    with open(LEDGER) as f:
-        for line in f:
-            line = line.strip()
-            if not line:
-                continue
-            try:
-                entry = json.loads(line)
-            except Exception:
-                continue
-            if entry.get("event") == "start":
-                panel_id = entry.get("panel_id") or entry.get("surface_id", "")
-                if panel_id:
-                    existing = starts.get(panel_id)
-                    if not existing or entry.get("ts", 0) > existing.get("ts", 0):
-                        starts[panel_id] = entry
-            elif entry.get("event") == "end":
-                # Mark the session as ended
-                sid = entry.get("session_id", "")
-                for pid, e in list(starts.items()):
-                    if e.get("session_id") == sid:
-                        starts[pid]["ended"] = True
-    return starts
+def load_recorded():
+    """Returns {panel_id: entry} for the newest session recorded on each panel."""
+    records = cmux_sessions.list_sessions()
+    records += cmux_sessions.registry_fallback(REGISTRY, records)
+    newest = {}
+    for record in records:
+        if not (record.active_for_surface and record.provider and record.surface_id):
+            continue
+        current = newest.get(record.surface_id)
+        if current is None or (record.updated_at or 0) > (current.updated_at or 0):
+            newest[record.surface_id] = record
+    return {
+        surface_id: {
+            "session_id": record.session_id,
+            "cwd": record.launch_cwd or record.cwd,
+            "provider": "factory" if record.provider == "droid" else "claude",
+            "launch_argv": list(record.launch_argv),
+            "transcript_path": record.transcript_path,
+            "pid": record.pid,
+        }
+        for surface_id, record in newest.items()
+    }
 
 
 def is_agent_alive(pid: int) -> bool:
@@ -342,9 +342,9 @@ def main():
         print(f"Error reading cmux state: {e}", file=sys.stderr)
         sys.exit(1)
 
-    ledger = load_ledger()
+    ledger = load_recorded()
     if args.verbose:
-        print(f"Ledger entries: {len(ledger)}")
+        print(f"Recorded sessions: {len(ledger)}")
 
     restored = 0
     skipped = 0