Skip to content

Recover pools stuck in a rolling restart; clear stale Degraded (v0.3.2) - #18

Merged
austin-barrington merged 1 commit into
mainfrom
fix/proxy-restart-deadlock
Oct 17, 2026
Merged

austin-barrington merged 1 commit into
mainfrom
fix/proxy-restart-deadlock

Conversation

@austin-barrington

Copy link
Copy Markdown
Member

Two bugs found on shared-pool-1 after v0.3.1.

1. A pool stuck in a rolling restart never recovered. The pool's proxy was down when a rolling restart began. The restart then retried excluding a pod from the proxies forever and returned early on every pass. Because the proxy Deployment was applied only after the restart step, v0.3.1's proxy fix never landed. It needed the restart state cleared by hand.

  • The proxy Deployment is now applied before the rolling restart.
  • Exclude and include go to every proxy pod whose process is running, Ready or not, and skip pods that aren't running. Requiring Ready also deadlocked one-node pools: a proxy is Ready only with a routable backend, so excluding the only backend made every proxy un-Ready and the include never went out. A proxy that isn't running holds no exclusions (they're in memory), so it starts clean.
  • Exclude now treats 409 (already excluded) and the proxy's own "not found in pool" 404 as done, so a retry after a partial failure succeeds. Any other 404, such as a wrong port, is still an error.
  • A restart that has excluded nothing while no update is pending is cleared automatically. That's what was done by hand on the pool.

2. A stale Degraded=True stayed on healthy pools. setFailedStatus set it and nothing cleared it. It's now set to False (reason Reconciled) once a reconcile completes.

Chart and README bumped to 0.3.2; docs/compatibility.md has an upgrade note. No pod templates change.

Tests:

  • New envtest specs:
    • the stuck pool (pending update, restart waiting to exclude pod 0, only proxy crash-looping): one reconcile applies the proxy Deployment, probing admin, and moves past the exclusion;
    • an unstarted restart with no pending update is cleared;
    • a stale Degraded becomes False.
  • A spec where exclude/include reach a running but un-Ready proxy and skip a crash-looping one.
  • A unit test for the exclude response codes.
  • Mutation-checked: putting back the readiness gate, the late proxy apply, the stale state, or a Degraded left True each fails a spec.
  • go test ./... with envtest, go vet, and make lint (0 issues) pass.

A pool whose proxy was down when a rolling restart began never
recovered: the restart retried excluding a pod from the proxies forever,
it returned early on every pass, and the proxy Deployment was applied
only after it, so the proxy fix in v0.3.1 never landed.

- The proxy Deployment is applied before the rolling restart.
- Exclude and include go to every proxy whose process runs, Ready or
  not, and skip the rest. Requiring Ready also deadlocked one-node
  pools: excluding the only backend makes the proxies un-Ready, so the
  include never went out. A proxy that is not running holds no
  exclusions (they are in memory).
- Exclude treats 409 (already excluded) and the proxy's own
  not-in-pool 404 as done, so a retry after a partial failure succeeds.
- A restart that has excluded nothing while no update is pending is
  cleared, as was done by hand on the pool.
- Degraded is set False once a reconcile completes; it stayed True on
  healthy pools.

Chart 0.3.2.
@austin-barrington
austin-barrington merged commit e9e280a into main Oct 17, 2026
4 checks passed
@austin-barrington
austin-barrington deleted the fix/proxy-restart-deadlock branch October 17, 2026 22:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant