Skip to content

Sweep leaked ROS processes from dead agent worktrees - #88

Merged
MJohnson459 merged 1 commit into
mainfrom
sweep-orphan-ros-procs
Jul 31, 2026
Merged

Sweep leaked ROS processes from dead agent worktrees#88
MJohnson459 merged 1 commit into
mainfrom
sweep-orphan-ros-procs

Conversation

@MJohnson459

Copy link
Copy Markdown
Contributor

Adds pixi run sweep to reap ROS processes leaked by dead agent jobs, fixes the
pixi run kill foot-gun that motivated it, and closes the leak at source.

WHAT WAS THERE: 44 orphaned ROS processes, 299 MB, up to 4 days old, from five
ended jobs -- 19 twist_mux, 7 icp_odom_gate, 5 foxglove_bridge, 5 ros2 wrappers,
3 mosquitto, 3 twist_relay, task_server, a benchmark recorder.

THE SWEEP (mote_bringup/sweep_orphans.py, stdlib-only, pixi run sweep, reports
by default, --kill to act, --json). Matching is on identity in /proc: a ROS
environment, a path under an agent root, absence from the sweeper's own
ancestry, orphaned by ancestry, and older than --min-age (30 min). Two findings
changed the design from the task's sketch:

  • /proc//exe alone misses a third of them. A worktree running against the
    primary checkout's pixi env has its interpreter and binaries outside the job
    dir; only the ament/colcon/pixi prefixes implicate it. Provenance is exe +
    cwd + argv paths + those prefixes.
  • ancestry alone would kill live work. run_sim_smoke.sh setsids its launch,
    so a healthy smoke test has ppid 1 and looks orphaned. --min-age separates
    them, and reporting is the default.

pixi run kill HAD THE FOOT-GUN THE TASK WARNED ABOUT. Its pkill -9 -f 'robot_state_publisher|...' matched the shell running the task -- those names
are in its own command line -- SIGKILLed it (reproduced: exit 137), so the
ros2 daemon stop/start; rm -f /dev/shm/fast* half NEVER RAN. It also matched
every other checkout and worktree on the box, not just the current checkout as
the task assumed. It now uses the same /proc path scoped to the checkout
(--scope .), without the orphan/age tests since a wedged stack is live.

HOW THEY ESCAPED -- not the guessed cause. Not interactive pixi run robot/
sim: it is the test fixtures, on the SUCCESS path. ros2 run Popens the real
executable and installs no SIGTERM handler (only KeyboardInterrupt, assuming a
terminal Ctrl-C reached the whole group), so proc.terminate() on the wrapper
kills the wrapper and hands the node to init -- one leaked node per run.
Confirmed in ros2run/api/init.py and reproduced live. spawn_reapable/
reap_group put the child in its own session and signal the group; applied to
test_twist_mux_arbitration.py, test_foxglove_teleop.py and
tools/icp_gate_replay.py (bench.py/replay.py/camera_layer_decay.py already did
this by hand). A job killed outright, taking pytest with it, no fixture can fix
-- that is the sweep's half.

VERIFIED

  • 28 new tests pass (mote_bringup/test/test_sweep_orphans.py). Mostly about
    what the sweep DECLINES to kill: a run owned by a shell, a young detached
    run, a file manager holding a job path, the sweeper's own ancestry, another
    checkout. Selection is a pure function of a process table so those are
    fabricated; two cases spawn a real orphan via double-fork and reap it.
  • Leak fix measured on the real test: pre-fix fixture, 6 pass and 1 twist_mux
    survives; post-fix, 6 pass and 0 survive.
  • Dry run on the live box finds all 44 leaks and correctly skips claude bg-spare, cosmic-files and the shells that a pkill -f would have hit.
  • pre-commit clean; pixi arg passthrough confirmed for the task shape.
  • test_twist_mux.py::test_the_base_launches_the_mux fails identically on
    unmodified main under this ad-hoc pytest invocation -- pre-existing, an
    install-space path, not from this change.

NOT DONE: the 44 processes are still running. The permission classifier blocked
the actual kill, which is the right call for a destructive machine-wide action.
Run pixi run sweep to review, then pixi run sweep -- --kill to reap. Note
that pixi run kill cannot be exercised until this branch is in the checkout
the task runs from.

44 orphaned ROS processes (299 MB, up to 4 days old) were running on the dev
box: 19 twist_mux, 7 icp_odom_gate, 5 foxglove_bridge, 3 mosquitto and more,
from five agent jobs that had long since ended. They are the exact process
names a benchmark measures, and unlike overhead.py's own match, the system-wide
counters a benchmark sits in cannot be scoped around them.

`pixi run sweep` reports them, grouped by the job that left them, and reaps
them with --kill. Matching is on identity in /proc -- a ROS environment, a path
under an agent root, and absence from the sweeper's own ancestry -- never on the
command line. Matching on /proc/<pid>/exe alone, as first suggested, would have
missed a third of them: a worktree running against the primary checkout's pixi
environment has its interpreter and binaries outside the job directory, and only
the ament/colcon/pixi prefixes implicate it.

That identity rule also fixes `pixi run kill`, whose `pkill -9 -f
'robot_state_publisher|...'` matched the shell running the task itself -- those
names are in its own command line -- SIGKILLed it (reproduced: exit 137), and so
never reached the `ros2 daemon` reset that followed. It also matched every other
checkout and worktree on the machine. It now takes the same /proc path scoped to
the checkout (--scope .), minus the orphan and age tests, since a wedged stack is
live by definition.

How they escaped is not what it looked like. `ros2 run` is a wrapper that Popens
the real executable and installs no SIGTERM handler -- it tolerates only
KeyboardInterrupt, assuming the signal reached the whole process group, which is
true of a terminal Ctrl-C and false of a fixture's proc.terminate(). So
terminating the wrapper kills the wrapper and hands the node to init, once per
run, on the path where the run *succeeded*. Measured on
test_twist_mux_arbitration.py: 6 tests pass and 1 twist_mux survives; 0 after.
spawn_reapable/reap_group put the child in its own session and signal the group;
the three sites that lacked it now use it (bench.py, replay.py and
camera_layer_decay.py already did this by hand). The other half -- a job killed
outright, taking pytest with it before any teardown runs -- no fixture can fix,
and is what the sweep is for.

The sweep spares more than it takes, so the tests are mostly about what it
declines to kill: a run still owned by a shell, a young session-detached run
(the sim smoke test setsids its launch, so ancestry alone cannot tell it from a
leak -- hence --min-age, and reporting as the default), a file manager holding a
job path, the sweeper's own ancestry. The selection rule is a pure function of a
process table so those cases are fabricated rather than staged; two cases spawn
a real orphan through a double-fork and reap it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MHoWpoZUtVD1ynehcg3Pgr
@MJohnson459
MJohnson459 merged commit c3f5e62 into main Jul 31, 2026
3 checks passed
@MJohnson459
MJohnson459 deleted the sweep-orphan-ros-procs branch July 31, 2026 22:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant