fix(k8s): bound agentenv preStop drain loop with retries and deadline - #187
Open
epicvinny wants to merge 1 commit into
Open
fix(k8s): bound agentenv preStop drain loop with retries and deadline#187epicvinny wants to merge 1 commit into
epicvinny wants to merge 1 commit into
Conversation
The preStop hook polled the local API in an unbounded while-true loop. When the endpoint was unreachable (or sandboxes never drained) the hook never exited, and with terminationGracePeriodSeconds=3600 the pod stayed Terminating for up to an hour. Observed in staging: a pod had to be force-deleted to recover, which left the old container racing its replacement on the shared hostPath state. Bound the loop two ways so termination always proceeds: - at most 5 consecutive drain-query failures with linear backoff, then exit 0 (if the API is gone there is nothing left to drain); - a global 240s deadline, after which the hook exits 0 so the kubelet can SIGTERM the agent and let its graceful shutdown persist sandboxes. Also add --max-time to curl so a hung connection cannot stall the loop.
Contributor
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s). |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Bounds the
agentenv-nodeDaemonSetpreStopdrain loop indeploy/k8s/base/agentenv-daemonset.yaml:GET /sandboxes(with linear backoff), then the hook exits 0 — if the local API is gone there is nothing left to drain.curl --max-time 5so a hung connection cannot stall the loop.Why
The hook polled the local API in an unbounded
while trueloop. When the endpoint was unreachable (or sandboxes never drained), the hook never exited and — withterminationGracePeriodSeconds: 3600— the pod stayedTerminatingfor up to an hour.Observed in staging: one
agentenv-nodepod stuck inTerminating, recovered only viakubectl delete pod --force --grace-period=0. The force-delete then raced the replacement pod on the sharedhostPath(/var/lib/aenv) state, contributing to the RocksDBLOCKcontention addressed separately in the orphan-process/lock-retry PR.Scope and non-goals
terminationGracePeriodSecondsor the graceful-shutdown semantics of the agent itself; the drain policy (who deletes sandboxes before termination) is unchanged.Compatibility and operations
Validation
yaml.safe_load) and the embedded script passessh -ncurl/jq: clean drain (exit 0 immediately), unreachable API (5 attempts with backoff, exit 0), never-draining API (global deadline, exit 0)make test-unit(N/A: manifest-only change, no Rust code touched)