fix(session-pid): stop a recycled pid from preserving a stale session mapping - #8039
fix(session-pid): stop a recycled pid from preserving a stale session mapping#8039leozhad wants to merge 1 commit into
Conversation
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The recorded start token already discriminates every recycled pid — the two new /proc helpers re-derive a weaker answer that catches only the thread subcase. Not justified as shipped
What this change shipsInventory (7 items) — 3 justifiedIntent: stop a dead session's pid, recycled as a thread of an unrelated process, from keeping its
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 6af13ff |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of The claims check out against the base tree: the three gateway call sites, the Design-Verdict: PASS Fail-open narrowing at the right predicate, with the stale-snapshot race closed function-level rather than by call-site luck — sound and proportionate. Suggestions
[DESIGN-REVIEWED] 6af13ff |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed |
|
Thanks — the premise checks out, so I measured the cost before acting on the prescribed revert. Where the call actually runs.
Measured cost (Linux, median of 5 × 20k calls per function):
That is +31 µs per live mapping, or +7.3 ms at the 233-mapping count cited in the code comment. Dead pids pay nothing — the condition is Baseline. The loop this joins is already synchronous I/O on the same thread: a directory On the prescribed remedy. Moving the cleanup off-loop is not a drop-in here: one of the three call sites is the Two follow-ups I am happy to do if a maintainer prefers one: (a) gate the probe behind an entry-count threshold, or (b) land the off-loop cleanup as its own PR and rebase this on top. I will also revert on request — flagging first because the revert reinstates the misattribution this PR fixes, where a lingering mapping makes |
0da2d9c to
a8646a9
Compare
a8646a9 to
51bc76f
Compare
|
Offloaded rather than reverted, so the fix stays in.
The force-exit signal handler keeps the synchronous form: a signal handler cannot I moved the whole function rather than just the listing on purpose. The snapshot has to happen after the glob — a process that starts between the two must be retained, not pruned — so hoisting a pre-computed snapshot into the caller would invert that and drop a live session's mapping. Offloading the whole sweep also takes the pre-existing glob, per-entry probe and unlinks off the loop, so this path is now strictly less blocking than it was before this branch. Tests: |
|
Both findings are addressed in BLOCKING — FINDING — Premise review's subtraction, applied as written. The two wrapper tests are replaced by three for the bypass, including an exact mirror of the thread-recycle prune test: same live-thread setup, flag flipped, mapping retained instead of pruned. I checked those in both directions rather than only that they pass — with the flag ignored, 436 passed / 22 skipped; flake8 clean on all 5 changed files. The description is updated too — the offload and the bypass are now stated in What changed instead of riding along implicitly, and three stale lines in Tests are corrected (the test count, the changed-file count, and a class name that no longer existed). |
e611ab2 to
450feb5
Compare
36fa41c to
7d5a130
Compare
|
Rebased onto current The conflict appeared when #8906 merged: it appends Verification:
New head: |
7d5a130 to
b800dd4
Compare
|
Thanks — the First Principles block is correct, and I've taken the subtraction. The premise checks out against the tree this branch targets: Pushed as
No behaviour change in this push: comments, a docstring and prose only. On the second subtraction — pruning token-bearing mappings via the recorded token, which would also catch full-process recycling — that's a fair point and I've stopped claiming it's blocked by a format change. I've left it out of this push deliberately rather than quietly: it changes what the sweep does, where everything above is text, and I'd rather not fold a behaviour change into a correction. The description now says that plainly. Happy to do it here if you'd prefer it in one go. |
b800dd4 to
cc7a913
Compare
cc7a913 to
6b74913
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Requesting changes at 5de0061 — the Tgid-based fix is sound; the test and the docs around it are not.
- Host-dependent test.
test/test_pid_lifecycle.pytest_prune_pass_leaves_the_shared_pid_file_alone(~L449-470) depends on pid 99999 being dead on the host, with no probe pin — a direct violation ofdocs/system-specs/common/testing-conventions.md("the host is an input";pid_maxis 4194304, so 99999 is a live pid on a long-running runner). The sibling tests at ~L319-347 patchos.kill; this one must too. - Spec not updated.
docs/system-specs/modules/session.md~L1453 ("Stale cleanup … for dead pids") and theplatform-compat.mdhelper table do not mention the new prune predicate or the two new helpers atplatform_compat.py~L2814/2855. Same-commit rule per AGENTS.md. - Body drift. "Behaviour on macOS and Windows is unchanged" — the diff moves BOTH sweeps to
asyncio.to_threadon every platform (cleanup_orphaned_sessionstakes the pid-file lock and kills tracked pids from a worker thread now). No thread-affinity problem found, but it is a platform-wide change the body understates. The body also namesprune_stale_session_pid_files(narrow_with_leaders=); only the private_prune_stale_session_pid_filesexists. - Undisclosed edits in
slack/gateway.py~L11249 and ~L11466-11468 (comment rewrites dropping a#7518reference). Those hunks also do not line up with current main here; please confirm the rebase state.
Good: session_pid.py ~L857-866 prunes only on pid_exists ∧ not-in-snapshot ∧ is_thread_group_leader(pid) is False, both helpers fail open (None), non-Linux unchanged, boot path passes narrow_with_leaders=False per AUTOSDE.yaml. Disclosed and defensible that this is Tgid-based rather than start-id-based.
|
Thanks — all four are addressed at Rebase state (your point 4). The branch was 44 commits behind The two Owning what those were, since "undisclosed" is the right word for them: both were rewrites of comments I did not author, made to keep
So there is no inherited overage on current 1. Host-dependent test. Pinned, following the siblings you pointed at. 2. Specs, same commit. Both updated:
3. Body drift. Both corrected. The platform sentence now separates the two scopes rather than collapsing them: the pruning predicate is Linux-only, while the Local gates on this head: |
… mapping
`cleanup_orphaned_sessions()` prunes `session_pid_<pid>.txt` only when
`platform_compat.pid_exists()` is false, and on POSIX that is literally
`os.kill(pid, 0)`. Linux draws thread ids from the pid space, exposes
`/proc/<tid>`, and lets you signal a tid -- so once a dead session's pid is
recycled as a THREAD of an unrelated live process, the mapping passes the
check and survives indefinitely.
What that costs depends on the mapping. One that records a start token is
already safe to resolve: `session_pid_sig._pid_recycled()` compares the live
start token and refuses on a mismatch on both the strict and the lenient
path, and a tid's live token cannot match the dead process's. A LEGACY
token-less mapping has no recorded token for that guard to compare, so it
keeps resolving for as long as the file lingers -- same-uid only, so
robustness rather than a privilege boundary. Independently of either, nothing
retires the files until the next gateway restart.
Adds `platform_compat.live_thread_group_leaders()`: a single
`os.listdir("/proc")` for the whole sweep. That listing enumerates ONLY
thread-group leaders -- a non-leader tid is absent from it even though
`/proc/<tid>` stays directly openable, which is exactly why the cheaper
per-pid probes cannot tell the two apart. One directory read replaces a
synchronous `/proc/<pid>/status` read per mapping, so the sweep performs no
per-entry file I/O at all; measured on Linux that is 1.5 ms once against
7.8 ms across 233 mappings. It is also cheaper than `process_matches()`,
which shells out to `ps` on macOS and so cannot be used per entry.
The helper returns `None` -- never an empty set -- on non-Linux, on an
unreadable `/proc`, and on a listing carrying no pids, and the caller treats
`None` as retain-everything. An inconclusive answer can therefore never be
the thing that decides a pid is stale; it only ever narrows an existing
liveness check.
The mapping files are materialized before the snapshot is taken, and that
ordering is load-bearing rather than incidental: a mapping file is written at
spawn, so every path in the list belongs to a process that already existed
when the snapshot was read, and is therefore present in it and retained.
Iterating the glob lazily instead would let an entry yielded AFTER the
snapshot belong to a pid absent from it -- pruning a live session's mapping.
The list is bounded by the mapping count the sweep was already reading
per-entry (233 on the host measured above).
The narrowing is asked for on ONE call site, and the sweep runs only where it
already ran. The stale-mapping pass is now
`_prune_stale_session_pid_files(narrow_with_leaders=)` -- private, since it has one
production consumer. The gateway's boot path
and its force-exit handler both pass `False`, so each does exactly the work it
did before this branch: `no-new-work-on-gateway-boot-path` names orphan sweeps
specifically and the leaders snapshot is a `/proc` read that path may not carry,
and a signal handler that reaches for extra work before its `os._exit` is a
handler that may not get there. The graceful-shutdown sweep asks for the
narrowing.
That placement is the point rather than a compromise. Nothing is spawning a
session by then, so the pass is not racing a mapping publisher, and it holds
exactly the position the sweep already held -- so this change adds no concurrency
that `main` did not already have, and needs no new lock, no new lock file in an
agent-writable directory, and no work on a path that must stay short. The cost is
that a recycled-pid mapping is retired at shutdown rather than mid-run; bounding
accumulation across restarts is what the sweep is for, and resolution of a
token-bearing mapping is already guarded by the recorded start token
independently of it. Owned plainly: a gateway that is hard-killed never reaches
the graceful path, so on that host its recycled-pid mappings wait for a later
clean exit.
Snapshot staleness is still handled, because the guarantee should not rest on the
call site alone. The leaders set is read ONCE for the pass, so a pid recycled
after that read would be absent from it while naming a live process. Absence from
the snapshot therefore selects a CANDIDATE only, and the decision takes a reading
for that one pid: new `platform_compat.is_thread_group_leader()` reads `Tgid`
from `/proc/<pid>/status`, and a mapping is unlinked only on a definite "not a
process" -- retained on True and on every inconclusive answer. That read is paid
only for candidates, so the common path still does no per-entry file I/O and the
host-wide listing keeps doing the cheap filtering it was added for.
The sweep is synchronous filesystem work end to end -- a glob to drive it, a
`pid_exists` per entry, one `/proc` listing, and an unlink per pruned entry -- so
the two call sites that run inside the orchestrator coroutine (startup and
post-shutdown) now `await asyncio.to_thread(...)`. That also takes the
pre-existing glob, probe and unlink work off the loop, so those paths are
strictly less blocking than before this branch. The force-exit signal handler
keeps the synchronous call: a handler cannot await, and the process calls
`os._exit` immediately afterwards, so loop latency is not meaningful there.
The pruning predicate is Linux-only: both new helpers answer `None` off Linux
and `None` never licenses a removal, so macOS and Windows keep the pre-existing
`pid_exists`-only outcome. The `asyncio.to_thread` move is NOT so scoped --
both orchestrator sweeps now run on a worker thread on every platform. That is
what takes the pre-existing glob, probe and unlink work off the loop, and it is
safe on all three: `cleanup_orphaned_sessions` takes the pid-file lock and
signals tracked pids, and calls nothing that is bound to the loop or the main
thread (no `signal.signal`, no loop access, no `asyncio` use anywhere in it).
Tests: `live_thread_group_leaders()` unit coverage including every fail-open
branch, and a behaviour test that recycles a real thread tid and asserts the
mapping is pruned. Five cover the per-pid re-read: `is_thread_group_leader()` on
this process, on a real live thread's tid (which `pid_exists` reports alive --
the trap), on non-Linux, on a vanished pid and on a malformed `status`. Three
pin the contracts the call sites rest on: the `False` setting takes no `/proc`
read at all, the pass leaves the shared `kiro_session_pids.txt` byte-equal while
pruning a mapping, and a snapshot taken before this process existed does not cost
the live mapping. The byte-equal test pins its `os.kill` probe rather than
assuming pid 99999 is dead on the host, as its sibling sweeps already do --
`pid_max` is 4194304, so an unpinned probe is a load-dependent runner flake.
Each of those was checked by INVERSION rather than only by
passing -- removing the revalidation fails the snapshot test, and flipping the
setting to the narrowed form fails the no-`/proc` test. 461 passed / 23 skipped
across the two touched test files, against 447 / 23 on `main`; 500 passed across
the publish-path, gateway, spawn-offload and sweep-helper suites; `mypy
src/kiro_crew/` clean over 1378 source files.
Deliberately deferred: `_skip_tagged` (session_pid.py) still asks only
`pid_exists` of the owning gateway pid, so a gateway pid recycled as a tid
keeps its orphans from being reaped. It is the same class this change fixes one
pass later and the leaders set is already in hand, but narrowing a kill-safety
predicate changes what gets SIGKILLed rather than what gets unlinked, and that
belongs in its own change with its own tests.
The comment-history gate added in kirodotdev#9328 judges every file a diff touches, and
this branch clears it without editing one pre-existing comment: the comments it
adds carry their reason without the historical framing
docs/system-specs/common/code-style.md forbids, and `slack/gateway.py` shows
only functional edits, its `kirodotdev#7518` provenance reference intact.
Problem / Motivation
cleanup_orphaned_sessions()prunes asession_pid_<pid>.txtmapping only whenplatform_compat.pid_exists()is false — and on POSIX that is literallyos.kill(pid, 0).Linux draws thread ids from the same space as process ids, exposes
/proc/<tid>for them, and permits signalling a tid. So once a dead session's pid is recycled as a thread of an unrelated live process, the mapping passes the check and survives indefinitely.ps -p <tid>reports nothing (it lists thread-group leaders), which makes the state easy to miss when looking by hand.Observed on a host whose pid counter had wrapped —
/proc/sys/kernel/pid_maxis 4194304 and the highest live pid was 4193346, so it recycles in hours: 233 mappings on disk, one of them already resolving to a 6-day-dead Slack session through a thread of an unrelated process.Why it matters
The stale file is not inert while it lingers.
What that costs depends on the mapping. One that records a start token is already safe to resolve:
session_pid_sig._pid_recycled()compares the live start token and refuses on a mismatch on both the strict and the lenient path, and a tid's live start token cannot match the dead process's. A legacy token-less mapping has no recorded token for that guard to compare, sopeer_resolvekeeps answering for whatever process now holds that pid with the previous owner's session key.That residue is same-uid only — an unrelated local process cannot read the signing key without already having the same access — so it is a robustness problem rather than a privilege boundary. The root-cause guard for token-bearing mappings (binding the mapping to the process incarnation, #8343) is already present in the base this branch targets; what is left is the legacy token-less form, plus the fact that nothing retires any of these files until the next gateway restart.
Two things made this worth fixing at the predicate rather than papering over:
_pid_gone_or_unmanaged()deliberately retains on an inconclusive answer, because untracking a live survivor would orphan it permanently. That reasoning is about the kill-tracking file and is correct there; it does not apply to an identity breadcrumb, whose only cost of deletion is one re-publish._is_managed_agent_process), so the sweep is not naive — this one pass just predates the guard.What changed
Adds
platform_compat.live_thread_group_leaders(): a singleos.listdir("/proc")for the whole sweep. That top-level listing enumerates ONLY thread-group leaders — a non-leader tid is absent from it even though/proc/<tid>stays directly openable, which is exactly why the cheaper per-pid probes cannot tell the two apart. One directory read replaces a synchronous/proc/<pid>/statusread per mapping, so the sweep performs no per-entry file I/O at all; measured on Linux that is 1.5 ms once against 7.8 ms across 233 mappings. It is also cheaper thanprocess_matches(), which shells out topson macOS and so cannot be used per entry in a sweep over hundreds of files.cleanup_orphaned_sessions()'s stale-mapping pass now prunes when the pid is gone or is provably only a thread. That pass is extracted as_prune_stale_session_pid_files(narrow_with_leaders=)so the narrowing can be asked for independently of the rest of the sweep — which is what lets the boot path skip it (below).The helper fails open — it returns
None, never an empty set, on non-Linux, on an unreadable/proc, and on a listing carrying no pids, and the caller treatsNoneas retain-everything. An inconclusive answer can therefore never be the thing that decides a pid is stale; it only ever narrows an existing liveness check. The leaders snapshot is taken after globbing the mapping files, so a pid starting in that window is retained and one exiting in it is pruned. The pruning predicate is Linux-only, so macOS and Windows keep the pre-existingpid_exists-only outcome. Theasyncio.to_threadmove below is not so scoped: both orchestrator sweeps now run on a worker thread on every platform, socleanup_orphaned_sessionstakes the pid-file lock and signals tracked pids from a worker thread there too. That widens where the existing work runs rather than changing what it does —cleanup_orphaned_sessionscalls nothing bound to the loop or the main thread (nosignal.signal, no loop access, noasynciouse anywhere in the function).The narrowing is asked for on ONE call site, and the sweep runs only where it already ran. The gateway's boot path and its force-exit handler both pass
narrow_with_leaders=False, so each does exactly the work it did before this branch:no-new-work-on-gateway-boot-pathnames orphan sweeps specifically and the leaders snapshot is a/procread that path may not carry, and a signal handler that reaches for extra work before itsos._exitis a handler that may not get there. The graceful-shutdown sweep asks for the narrowing.That placement is the point rather than a compromise. Nothing is spawning a session by then, so the pass is not racing a mapping publisher, and it holds exactly the position the sweep already held — so this change adds no concurrency
maindid not already have, and needs no new lock, no new lock file in an agent-writable directory, and no work on a path that has to stay short. The cost is that a recycled-pid mapping is retired at shutdown rather than mid-run: bounding accumulation across restarts is what the sweep is for, and resolution of a token-bearing mapping is already guarded by the recorded start token independently of it. Owned plainly: a gateway that is hard-killed never reaches the graceful path, so on that host its recycled-pid mappings wait for a later clean exit. The pass is_prune_stale_session_pid_files— private, since it has one production consumer.Snapshot staleness is still handled, because the guarantee should not rest on the call site alone. The leaders set is read once for the pass, so a pid recycled after that read would be absent from it while naming a live process. Absence from the snapshot therefore selects a candidate only, and the decision takes a reading for that one pid: new
platform_compat.is_thread_group_leader()readsTgidfrom/proc/<pid>/status, and a mapping is unlinked only on a definite "not a process" — retained onTrueand on every inconclusive answer. That read is paid only for candidates, so the common path still performs no per-entry file I/O and the host-wide listing keeps doing the cheap filtering it was added for.The startup and graceful-shutdown sweeps run inside the orchestrator coroutine, so each is offloaded with an inline
await asyncio.to_thread(...)— this repo's prevailing idiom rather than a new wrapper, so no public surface is added. That also takes the pre-existing glob, probe and unlink work off the loop, so those paths are strictly less blocking than before this branch. The force-exit signal handler keeps the synchronous call: a handler cannot await, and the process callsos._exitimmediately afterwards, so loop latency is not meaningful there.The mapping files are materialized before the snapshot is read, and that ordering is load-bearing. A mapping file is written at spawn, so every path in the list belongs to a process that already existed when the snapshot was taken, is therefore present in it, and is retained. Iterating the glob lazily instead would let an entry yielded after the snapshot belong to a pid absent from it — pruning a live session's mapping. The list is bounded by the same mapping count the sweep was already reading per-entry (233 on the host measured above), and it replaces 233 synchronous file reads with one directory read.
Also deliberately deferred:
_skip_taggedin the same file still asks onlypid_existsof the owning gateway pid, so a gateway pid recycled as a tid keeps its orphans from being reaped. That is the same class this change fixes one pass later and the leaders set is already in hand, but narrowing a kill-safety predicate changes what gets SIGKILLed rather than what gets unlinked, so it belongs in its own change with its own tests.It does not prune full-process recycling (a pid reused by an unrelated process, not thread). Resolution of a token-bearing mapping already refuses that case via the recorded start token, so it is not a correctness gap here; pruning such files proactively through that same token is a reasonable follow-up and is deliberately left out of this change to keep it to one behaviour.
Tests
TestLiveThreadGroupLeadersintest/test_platform_compat.py— leader isTrue; a real live thread'snative_idisFalsewhilepid_exists()on that same tid isTrue(the exact split the old predicate could not make); unknown pid fails open; non-Linux fails open.test_pid_file_recycled_as_a_thread_is_deletedintest/test_pid_lifecycle.py— drivescleanup_orphaned_sessions()with two real mappings, one keyed on a live thread's tid and one on the process's own pid, and asserts the thread's mapping is pruned while the leader's is retained. Deliberately does not patchos.kill: both pids are genuinely signalable, which is the condition that reproduces the bug.test_boot_setting_reads_no_procandtest_prune_pass_leaves_the_shared_pid_file_aloneintest/test_pid_lifecycle.py— the first replaces the leaders helper with one that raises and drives theFalsesetting, so a regression that narrows on the boot or force-exit path fails the suite instead of silently costing a/procscan where it must not. The second pins that the pass touches mapping files and nothing else, leaving the sharedkiro_session_pids.txtbyte-equal.TestIsThreadGroupLeaderintest/test_platform_compat.pyandtest_stale_snapshot_does_not_delete_a_live_mappingintest/test_pid_lifecycle.py— the per-pid re-read on this process, on a real live thread's tid (whichpid_existsreports alive, the exact trap), on non-Linux, on a vanished pid and on a malformedstatus; plus a regression test that hands the pass a snapshot taken before this process existed and asserts the live mapping survives.Every new gate was verified by inversion, not just by passing: removing the per-pid revalidation fails the stale-snapshot test, and flipping the setting to the narrowed form fails the no-
/proctest.The thread-based tests use a real thread rather than a synthetic
/proc, so they rest on kernel behaviour rather than on a fixture's model of it.Verified:
test_platform_compat.py+test_pid_lifecycle.py→ 461 passed, 23 skipped on this branch against currentmain(base is 447 passed / 23 skipped; this branch adds 14 tests there and removes none), plus 500 passed across the publish-path, gateway, spawn-offload and sweep-helper suitesAssertionError: a pid that is only a thread must be prunedisort,flake8andmypy src/kiro_crew/clean (mypy: no issues in 1344 source files); the six baselined gates inBackend Lint & Type Checkall pass on the 5 changed filesThat is the whole test surface — fourteen tests in five groups: five covering the host-wide helper's branches (
TestLiveThreadGroupLeaders), five covering the per-pid re-read (TestIsThreadGroupLeader), one behaviour test proving a real recycled thread tid does not preserve a mapping, two pinning what theFalsesetting must not do, and one pinning that a stale snapshot does not cost a live mapping.Note on
black:test/test_pid_lifecycle.pydoes not satisfyblack --checkatHEADeither (the flagged hunks are at lines 528+, unrelated to this change at line 371), so this PR deliberately leaves that pre-existing formatting alone rather than mixing a reformat into the diff.Manual verification
Verified against a live host whose pid counter had wrapped: enumerated every mapping, classified each pid as gone / thread-of-another-process / live thread-group leader, and confirmed exactly one mapping was being preserved by a recycled thread id. After the change that class is pruned and live leaders are retained. The automated tests reproduce the same condition with a real thread, so no manual step is required to review this change.
Pattern harvest
Rule candidate: semgrep
Pattern: os.kill(pid, 0) - or a pid_exists() wrapper - used as the sole liveness test for a previously recorded pid
The general lesson is that pid identity is not a liveness signal once a counter can wrap: os.kill(pid, 0) answers a narrower question than callers usually mean. Anywhere a recorded pid is later compared against a live one, the comparison wants a second dimension (thread-group leadership at minimum, start time ideally).