Sweep leaked ROS processes from dead agent worktrees

CLAUDE.md

@@ -32,7 +32,8 @@ pixi run setup          # One-time Pi setup: udev + wifi-powersave + systemd (ne
 pixi run udev           # Install udev rules + dialout group (needs sudo)
 pixi run wifi-powersave # Disable WiFi power save via NetworkManager (needs sudo)
 pixi run setup-ids      # Guided servo ID assignment tool
-pixi run kill           # Kill stale ROS processes and reset daemon
+pixi run kill           # Kill this checkout's ROS processes and reset daemon
+pixi run sweep          # Report ROS processes leaked by dead agent jobs (--kill to reap)
 pixi run identity       # Fleet identity CLI: show / id / set --id --name --site
 pixi run tailnet        # Join this machine to the Tailscale overlay (needs sudo)
 pixi run provision      # Render cloud-init user-data for a clean Pi
@@ -200,6 +201,8 @@ The C++ that runs *inside* other people's processes: a Nav2 plugin and two compo
 ### `mote_bringup` (Python/ament)
 Launch files, config, udev rules, NetworkManager drop-ins, systemd services, and the fleet foundation: `mote_home.py` (per-robot state root), `identity.py` (`identity` console script), `provision.py` + `provisioning/user-data.template` (`provision`), `dds_participants.py` (`dds_participants`), `twist_relay.py` (`twist_relay`, the Foxglove teleop seam), `explore.py` (`explore`, `pixi run explore` — autonomous mapping coverage: left-wall following with a Nav2 frontier-seek escape and a stuck detector for lidar-invisible obstacles like rug edges; run beside `pixi run mapping` **on the Pi**, so a wifi drop cannot end the mission; `--sim-time` gates it on /clock for the sim, and `--cruise`/`--obstacle`/`--desired-left`/`--follow-band` tighten the corridor-scale defaults for domestic layouts), the `foxglove/` layout, and `tailscale/install.sh`, plus `bundle.py` — the site bundle's *content*: a ROS-free reader/validator/packer for a map revision, shared with the fleet server's registry (M4) so both ends check a revision with the same code. See Fleet above.
 
+**Stray ROS processes** are `sweep_orphans.py`'s subject, from both ends (`mote_bringup/README.md` "Clearing stray ROS processes"). *Cure* is `pixi run sweep`: agent worktrees leave nodes behind that reparent to init and outlive the job by days — 44 of them, 371 MB, were found on the dev box — and they are the exact process names a benchmark measures, against system-wide counters no `overhead.py`-style scoping can isolate. **Matching is on identity in /proc, never on the command line**, which is not fastidiousness: `pixi run kill`'s old `pkill -9 -f '<driver names>'` matched the shell running the task (the names are in its own command line), SIGKILLed it, and so **never reached the `ros2 daemon` reset that followed it** — reproduced, exit 137 — besides matching every other checkout on the box. Both modes now require a ROS environment, a path under the directory in question, and absence from the sweeper's own ancestry; `kill` keeps that scope (`--scope .`) and drops the orphan/age tests, since a wedged stack is live by definition. The sweep adds them, because a `setsid`-detached run — the sim smoke test's — is indistinguishable from a leak by ancestry alone, so `--min-age` (30 min) is what separates them and reporting is the default. *Prevention* is `spawn_reapable`/`reap_group` in the same module: **`ros2 run` is a wrapper that `Popen`s the real executable and handles no SIGTERM** (only `KeyboardInterrupt`, assuming a terminal Ctrl-C reached the whole group), so `proc.terminate()` on it kills the wrapper and hands the node to init — once per run, *on the success path*. Measured on `test_twist_mux_arbitration.py`: 6 tests pass, 1 `twist_mux` survives; 0 after. Every `ros2 run` spawn goes through it (`test_twist_mux_arbitration.py`, `test_foxglove_teleop.py`, `tools/icp_gate_replay.py`; `bench.py`/`replay.py`/`camera_layer_decay.py` already did it by hand). The other half — a job killed outright, taking pytest with it before any teardown — no fixture can fix, and is what the sweep exists for.
+
 **On-robot reliability** (see `mote_bringup/README.md`): `pixi run robot`/`mapping` include the health monitor, so a manual run publishes `/health` too; the systemd units are installed by `pixi run setup` but **not enabled** (autostart would drain the battery on a desk — opt in with `systemctl enable --now mote-bringup mote-health`). the systemd services restart with backoff and never permanently give up (`Restart=always`, `RestartSec`/`RestartSteps`/`RestartMaxDelaySec`, `StartLimitIntervalSec=0`), order after the udev-tagged `dev-mote_*.device` units, and bound the journal. A pre-flight self-check (`self_check.py`, run as `mote-bringup`'s `ExecStartPre`; `pixi run self-check`) gates bringup on servo ping + lidar/camera/disk/clock/config and keeps the robot idle with a clear reason on failure. A health monitor (`health_monitor.py`, `mote-health.service` with a `Type=notify` watchdog; `pixi run health`) publishes per-subsystem `diagnostic_msgs/DiagnosticArray` on `/diagnostics_agg` and a single OK/DEGRADED/FAULT summary on `/health`. **Wheel slip needs no IMU**: kinematic_icp takes wheel odometry as its prior and corrects it against the scan, so the correction already measures how wrong the wheels were — `slip_monitor.py` (started by `mote_launch.py` beside `system_monitor`) compares the two over a 1 s sliding window and publishes `slip` / `stuck` / `icp_fault` as the `slip` status on `/diagnostics`, plus the raw residual on `slip/residual`. All three are DEGRADED, never FAULT: each is a reason to stop and re-plan, not to refuse to drive, and a monitor that can halt the robot on a threshold is a worse failure than the slip. The maths is ROS-free in `odom_residual.py` and **shared with `tools/slip_replay.py`**, which is what set `config/slip.yaml`'s thresholds from the residual distribution over the real mapping bags — a threshold calibrated offline only means something if the robot computes the same number. Two things are load-bearing. **Only translation is thresholded**: the yaw residual is published but its p99 reaches the yaw rate itself, so no threshold survives a hard turn — and a lag sweep puts the two streams within ±10 ms, so this is scan-match jitter, *not* the stamp skew task 165 suspected. **A stalled lidar must not read as slip**: the window would freeze at the last pose while the wheels keep turning, growing without bound, so a source older than `max_lag` yields no verdict at all. `health_monitor` lifts named statuses off the shared `/diagnostics` by exact name, listed in `health.yaml`'s `diagnostic_statuses`. Six real events (two stuck robots, four scan-match excursions) were found in the existing "known-good" bags; the derivation and the sim demonstration are in `docs/tuning/2026-07-28-slip-detection.md`. Driver and nav2 nodes are `respawn=True` for per-node recovery under the whole-service systemd restart. Battery voltage is **not** software-measurable (the power bank exposes no telemetry); `system_monitor` reports the Pi's `get_throttled` flags as the only power signal — read via **`vcgencmd`**, since the Pi 4 sysfs node does not exist on a Pi 5, alongside the Active Cooler's `fan_rpm` from the `pwmfan` hwmon.
 
 **Launch hierarchy:** the two mission launches (`mapping_launch.py`, `robot_launch.py`) each take a `base` arg (default true) that includes the hardware base, and a `use_sim_time` arg they forward to everything they include. The sim runs these *same* files with `base:=false`, supplying a Gazebo base in place of the drivers — so the missions are defined once and the sim exercises the real launch files.

mote_bringup/README.md

@@ -330,6 +330,54 @@ The monitor is also the systemd watchdog feeder: it sends `READY=1` once up and
 pets the watchdog on every publish (`sd_notify.py`, a dependency-free
 `$NOTIFY_SOCKET` client that no-ops outside systemd).
 
+## Clearing stray ROS processes — `sweep_orphans.py`
+
+Two different messes, one module, and the difference is worth knowing before
+reaching for either.
+
+`pixi run kill` clears **this checkout's** ROS processes and resets the daemon —
+the "my stack is wedged" reset. `pixi run sweep` clears **other jobs'** leftovers:
+processes an agent worktree started and never reaped, which reparent to init and
+run until the box is rebooted. It reports by default and only acts with `--kill`:
+
+```bash
+pixi run sweep                 # what would go, grouped by the job that left it
+pixi run sweep -- --kill       # reap them
+pixi run sweep -- --json       # for a script
+```
+
+That second mess is not just untidiness. Leftovers are the exact process names a
+benchmark measures, and the system-wide counters a benchmark sits in — context
+switches, interrupts, memory pressure, CPU contention — cannot be scoped the way
+`overhead.py` scopes its own match. A drifting background makes every
+measurement on the box a little less comparable than it looks.
+
+**Matching is on process identity, never on the command line.** The `pkill -9 -f
+'<driver names>'` that `kill` used to run matched the shell running the task
+itself — those names are in its own command line — SIGKILLed it, and so never
+reached the `ros2 daemon` reset that followed; it also matched every other
+checkout and worktree on the machine. Both modes now read `/proc` and require a
+ROS environment, a path under the directory in question, and absence from the
+sweeper's own ancestry. The sweep additionally requires that the process be
+orphaned and older than `--min-age` (30 min), because a deliberately
+session-detached run — the sim smoke test `setsid`s its launch — is
+indistinguishable from a leak by ancestry alone.
+
+### Why they escape
+
+`ros2 run` is a wrapper: it `Popen`s the real executable and installs no SIGTERM
+handler, tolerating only `KeyboardInterrupt` on the assumption that the signal
+reached the whole process group — true of a Ctrl-C at a terminal, false of a
+`proc.terminate()` from a test fixture. Terminating the wrapper therefore kills
+the wrapper and hands the node to init, **once per run, on the path where the run
+succeeded**. Measured on `test_twist_mux_arbitration.py`: six tests pass and one
+`twist_mux` survives.
+
+So anything spawning `ros2 run` uses `spawn_reapable` / `reap_group` from
+`sweep_orphans`, which put the child in its own session and signal the group.
+The other half — a job killed outright, taking pytest with it before any teardown
+runs — no fixture can fix, and that is what the sweep is for.
+
 ## Known gap: battery voltage
 
 The USB-C power bank exposes **no state-of-charge or voltage telemetry**, so the