[WIP prerequisite] finalize worker shutdown with PID identity fencing - #1202
Draft
sethkarten wants to merge 28 commits into
Draft
[WIP prerequisite] finalize worker shutdown with PID identity fencing#1202sethkarten wants to merge 28 commits into
sethkarten wants to merge 28 commits into
Conversation
… workers A worker with a durable stop intent is now reported as "stopping" and a disconnected worker is never reported as "ready". Stopping workers are excluded from live session lists and daemon-wide fan-out commands.
…y checks The list response feeds shutdownStaleDaemonIfNotBusy and probeRunningDaemonSessions, so hiding stopping workers made a tombstoned-but-running worker look idle and let a stale-daemon replace terminate it silently. Stopping workers stay listed with an honest "stopping" workerState; command fan-outs still skip them.
…ates
Revision 14 records the workerState wire-semantics change ("stopping"
state; disconnected workers no longer report "ready") so version probes
can tell old and new daemons apart. The field stays optional and
backward-tolerant, so no capability gate is needed.
… workers A worker with a durable stop intent is now reported as "stopping" and a disconnected worker is never reported as "ready". Stopping workers are excluded from live session lists and daemon-wide fan-out commands.
…y checks The list response feeds shutdownStaleDaemonIfNotBusy and probeRunningDaemonSessions, so hiding stopping workers made a tombstoned-but-running worker look idle and let a stale-daemon replace terminate it silently. Stopping workers stay listed with an honest "stopping" workerState; command fan-outs still skip them.
…ates
Revision 14 records the workerState wire-semantics change ("stopping"
state; disconnected workers no longer report "ready") so version probes
can tell old and new daemons apart. The field stays optional and
backward-tolerant, so no capability gate is needed.
…ng registrations When a worker does not exit within the stop deadline, the supervisor now keeps watching the process, escalates to SIGKILL, and completes the interrupted cleanup once the process dies. Process liveness checks also treat zombie processes as dead so cleanup is not deferred forever.
… generation The background finalizer now snapshots pid, processStartId, and stopRevision when scheduled and aborts if the stop is rescinded or the worker is relaunched, so it can never SIGKILL a retried worker or an unrelated process that reused the pid. stopWorker signalling is likewise identity-aware.
- Fail closed when a recorded processStartId cannot be observed, so a recycled pid is never signalled even if identity observation fails. - Record a schedule-time identity for workers that never had one. - Retry transient finalization cleanup failures instead of stranding the dead registration permanently. - Probe liveness with a cheap kill(0) on every poll and throttle the ps-backed zombie/identity checks so wedged workers cannot saturate the supervisor event loop.
…gone stopWorker used the identity check as a liveness predicate, so a transient getProcessStartId failure could skip signalling and delete the registration of a still-running worker. Identity verdicts are now directional: only a confirmed-current pid is signalled, only a confirmed-gone/replaced pid is cleaned up, and an unknown verdict keeps waiting.
…ched mid-await stopWorker can yield during archival while a retry rescinds the stop and relaunches the worker on the same registration. The cleanup tail now verifies the registered process is still the one it stopped before removing the registration or descriptor, so a relaunched worker is never orphaned by a stale stop invocation.
The throttled identity cache can be up to 500ms old, long enough for a pid to be recycled. Both SIGKILL sites (stopWorker force escalation and the stop finalizer) now run a fresh identity check immediately before signalling; the cache remains only for read-only wait-loop polling.
…ages A transiently unobservable identity at the escalation deadline now skips that attempt without marking the kill done, so a later pass that re-verifies the original process still escalates instead of leaving a wedged worker registered forever.
…scinded stops All stopWorker polling and signalling now use the pid and start identity captured at entry, so a retry relaunching the worker mid-stop can never be SIGKILLed through the mutable descriptor. The cleanup guard also aborts when a removeDescriptor stop lost its tombstone, catching a rescission that lands before the successor pid does.
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Draft upstream prerequisite validation for the unique #851 shutdown-finalization semantics rebased onto accepted #850. This is not a v0.8 stack deliverable and not ready for human merge review.
Exact candidate
f7ccf5e74d4d59d007ac889b3385be8eed87d798a6db0246b52cea74f2f675fea95ac43c9842d39aSafety corrections
Evidence and hold
1b8569a20a51b98c2d24f9d6c2677d52fe80b582ab3daf94e3900bd8a3b28828; baseline log SHA-256ff9ee41c2fd3d7918a37dc3cc63d8cf9a20dc09689988e0e47ad87fe07a1cab8. Earlier overlapping attempts are preserved but invalid.This Draft exists to obtain clean GitHub CI/process-smoke evidence. Do not merge, start #852, or present this as human-ready until the remote run and review gates pass.
Note
Add PID identity fencing to worker shutdown to prevent signalling recycled processes
processIdExists,isZombieProcess, andisProcessAliveutilities in child-process.ts for Unix-aware process liveness and zombie detection.signalCurrentWorkerandprocessIdentityhelpers toDaemonSupervisorthat verify a storedprocessStartIdmatches the live process before sending SIGTERM or SIGKILL, preventing signals from reaching recycled PIDs.stopWorkerto bind stop operations to an exact descriptor/stop-revision tuple; timed-out stops now schedule background finalization viafinalizeTimedOutWorkerStoprather than leaving tombstones indefinitely.effectiveWorkerStateto expose a'stopping'state and map disconnected'ready'workers to'recovering'in public summaries; daemon protocol bumped to revision 16.processStartIdare marked'failed'with a manual-recovery error instead of being silently mutated.processStartId(legacy descriptors) can no longer be automatically recovered and require manual intervention.📊 Macroscope summarized f7ccf5e. 7 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted
🗂️ Filtered Issues
No issues evaluated.