Skip to content

fix(k8s): bound agentenv preStop drain loop with retries and deadline - #187

Open
epicvinny wants to merge 1 commit into
kvcache-ai:mainfrom
ResultadosDigitais:upstream/prestop-bounded-drain
Open

fix(k8s): bound agentenv preStop drain loop with retries and deadline#187
epicvinny wants to merge 1 commit into
kvcache-ai:mainfrom
ResultadosDigitais:upstream/prestop-bounded-drain

Conversation

@epicvinny

Copy link
Copy Markdown

What

Bounds the agentenv-node DaemonSet preStop drain loop in deploy/k8s/base/agentenv-daemonset.yaml:

  • At most 5 consecutive failures to query GET /sandboxes (with linear backoff), then the hook exits 0 — if the local API is gone there is nothing left to drain.
  • A global 240s deadline, after which the hook exits 0 so the kubelet can SIGTERM the agent and let its graceful shutdown persist sandboxes.
  • curl --max-time 5 so a hung connection cannot stall the loop.

Why

The hook polled the local API in an unbounded while true loop. When the endpoint was unreachable (or sandboxes never drained), the hook never exited and — with terminationGracePeriodSeconds: 3600 — the pod stayed Terminating for up to an hour.

Observed in staging: one agentenv-node pod stuck in Terminating, recovered only via kubectl delete pod --force --grace-period=0. The force-delete then raced the replacement pod on the shared hostPath (/var/lib/aenv) state, contributing to the RocksDB LOCK contention addressed separately in the orphan-process/lock-retry PR.

Scope and non-goals

  • Included: bounded retries with backoff, global drain deadline, curl timeout in the vendored base DaemonSet manifest.
  • Excluded: changing terminationGracePeriodSeconds or the graceful-shutdown semantics of the agent itself; the drain policy (who deletes sandboxes before termination) is unchanged.

Compatibility and operations

  • No API, config, or storage-format changes. Rollout only changes pod termination behavior: pods now terminate within minutes instead of hanging up to the full grace period.

Validation

  • YAML parses (yaml.safe_load) and the embedded script passes sh -n
  • Scripted dry-run of all three paths with stubbed curl/jq: clean drain (exit 0 immediately), unreachable API (5 attempts with backoff, exit 0), never-draining API (global deadline, exit 0)
  • make test-unit (N/A: manifest-only change, no Rust code touched)

The preStop hook polled the local API in an unbounded while-true loop.
When the endpoint was unreachable (or sandboxes never drained) the hook
never exited, and with terminationGracePeriodSeconds=3600 the pod stayed
Terminating for up to an hour. Observed in staging: a pod had to be
force-deleted to recover, which left the old container racing its
replacement on the shared hostPath state.

Bound the loop two ways so termination always proceeds:
- at most 5 consecutive drain-query failures with linear backoff, then
  exit 0 (if the API is gone there is nothing left to drain);
- a global 240s deadline, after which the hook exits 0 so the kubelet
  can SIGTERM the agent and let its graceful shutdown persist sandboxes.

Also add --max-time to curl so a hung connection cannot stall the loop.
@github-actions

Copy link
Copy Markdown
Contributor

OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant