Sweep leaked ROS processes from dead agent worktrees - #88
Merged
Conversation
44 orphaned ROS processes (299 MB, up to 4 days old) were running on the dev box: 19 twist_mux, 7 icp_odom_gate, 5 foxglove_bridge, 3 mosquitto and more, from five agent jobs that had long since ended. They are the exact process names a benchmark measures, and unlike overhead.py's own match, the system-wide counters a benchmark sits in cannot be scoped around them. `pixi run sweep` reports them, grouped by the job that left them, and reaps them with --kill. Matching is on identity in /proc -- a ROS environment, a path under an agent root, and absence from the sweeper's own ancestry -- never on the command line. Matching on /proc/<pid>/exe alone, as first suggested, would have missed a third of them: a worktree running against the primary checkout's pixi environment has its interpreter and binaries outside the job directory, and only the ament/colcon/pixi prefixes implicate it. That identity rule also fixes `pixi run kill`, whose `pkill -9 -f 'robot_state_publisher|...'` matched the shell running the task itself -- those names are in its own command line -- SIGKILLed it (reproduced: exit 137), and so never reached the `ros2 daemon` reset that followed. It also matched every other checkout and worktree on the machine. It now takes the same /proc path scoped to the checkout (--scope .), minus the orphan and age tests, since a wedged stack is live by definition. How they escaped is not what it looked like. `ros2 run` is a wrapper that Popens the real executable and installs no SIGTERM handler -- it tolerates only KeyboardInterrupt, assuming the signal reached the whole process group, which is true of a terminal Ctrl-C and false of a fixture's proc.terminate(). So terminating the wrapper kills the wrapper and hands the node to init, once per run, on the path where the run *succeeded*. Measured on test_twist_mux_arbitration.py: 6 tests pass and 1 twist_mux survives; 0 after. spawn_reapable/reap_group put the child in its own session and signal the group; the three sites that lacked it now use it (bench.py, replay.py and camera_layer_decay.py already did this by hand). The other half -- a job killed outright, taking pytest with it before any teardown runs -- no fixture can fix, and is what the sweep is for. The sweep spares more than it takes, so the tests are mostly about what it declines to kill: a run still owned by a shell, a young session-detached run (the sim smoke test setsids its launch, so ancestry alone cannot tell it from a leak -- hence --min-age, and reporting as the default), a file manager holding a job path, the sweeper's own ancestry. The selection rule is a pure function of a process table so those cases are fabricated rather than staged; two cases spawn a real orphan through a double-fork and reap it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHoWpoZUtVD1ynehcg3Pgr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
pixi run sweepto reap ROS processes leaked by dead agent jobs, fixes thepixi run killfoot-gun that motivated it, and closes the leak at source.WHAT WAS THERE: 44 orphaned ROS processes, 299 MB, up to 4 days old, from five
ended jobs -- 19 twist_mux, 7 icp_odom_gate, 5 foxglove_bridge, 5 ros2 wrappers,
3 mosquitto, 3 twist_relay, task_server, a benchmark recorder.
THE SWEEP (mote_bringup/sweep_orphans.py, stdlib-only,
pixi run sweep, reportsby default, --kill to act, --json). Matching is on identity in /proc: a ROS
environment, a path under an agent root, absence from the sweeper's own
ancestry, orphaned by ancestry, and older than --min-age (30 min). Two findings
changed the design from the task's sketch:
primary checkout's pixi env has its interpreter and binaries outside the job
dir; only the ament/colcon/pixi prefixes implicate it. Provenance is exe +
cwd + argv paths + those prefixes.
run_sim_smoke.shsetsids its launch,so a healthy smoke test has ppid 1 and looks orphaned. --min-age separates
them, and reporting is the default.
pixi run killHAD THE FOOT-GUN THE TASK WARNED ABOUT. Itspkill -9 -f 'robot_state_publisher|...'matched the shell running the task -- those namesare in its own command line -- SIGKILLed it (reproduced: exit 137), so the
ros2 daemon stop/start; rm -f /dev/shm/fast*half NEVER RAN. It also matchedevery other checkout and worktree on the box, not just the current checkout as
the task assumed. It now uses the same /proc path scoped to the checkout
(--scope .), without the orphan/age tests since a wedged stack is live.
HOW THEY ESCAPED -- not the guessed cause. Not interactive
pixi run robot/sim: it is the test fixtures, on the SUCCESS path.ros2 runPopens the realexecutable and installs no SIGTERM handler (only KeyboardInterrupt, assuming a
terminal Ctrl-C reached the whole group), so proc.terminate() on the wrapper
kills the wrapper and hands the node to init -- one leaked node per run.
Confirmed in ros2run/api/init.py and reproduced live. spawn_reapable/
reap_group put the child in its own session and signal the group; applied to
test_twist_mux_arbitration.py, test_foxglove_teleop.py and
tools/icp_gate_replay.py (bench.py/replay.py/camera_layer_decay.py already did
this by hand). A job killed outright, taking pytest with it, no fixture can fix
-- that is the sweep's half.
VERIFIED
what the sweep DECLINES to kill: a run owned by a shell, a young detached
run, a file manager holding a job path, the sweeper's own ancestry, another
checkout. Selection is a pure function of a process table so those are
fabricated; two cases spawn a real orphan via double-fork and reap it.
survives; post-fix, 6 pass and 0 survive.
claude bg-spare, cosmic-files and the shells that apkill -fwould have hit.unmodified main under this ad-hoc pytest invocation -- pre-existing, an
install-space path, not from this change.
NOT DONE: the 44 processes are still running. The permission classifier blocked
the actual kill, which is the right call for a destructive machine-wide action.
Run
pixi run sweepto review, thenpixi run sweep -- --killto reap. Notethat
pixi run killcannot be exercised until this branch is in the checkoutthe task runs from.