Skip to content

fix(fleet): tolerate transient watch API errors with a bounded retry - #340

Merged
joshua-temple merged 1 commit into
mainfrom
fix/fleet-watch-retry
Jun 24, 2026
Merged

joshua-temple merged 1 commit into
mainfrom
fix/fleet-watch-retry

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

Problem

The fleet dispatch-suite action and the auto-promote Release watcher used gh run watch --exit-status, which fails the whole job on a single transient HTTP 401/403/5xx mid-poll even when the watched run concludes success. This reded a fully-green fleet run (rc.4 primary) on a transient 401 under load.

Fix

Both watch sites now use a bounded gh run view --json status,conclusion poll loop that:

  • retries transient API errors (401/403/5xx/rate-limit/network), with a consecutive-error cap that fails closed after repeated failures;
  • exits 0 only on status=completed AND conclusion=success;
  • exits 1 on any other concluded conclusion (never masks a real failure);
  • fails closed on a stuck run via a ~30-minute wall-clock cap.

.github-only change (no rc cut). actionlint clean. Closes #335.

The dispatch-suite and auto-promote watchers used gh run watch --exit-status, which fails the whole job on a single transient HTTP 401/403/5xx mid-poll even when the watched run succeeds. Replace with a bounded gh run view poll that retries transient API errors, exits 0 only on completed+success, fails 1 on any other conclusion, and fails closed on a stuck run or repeated errors. Closes #335.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple
joshua-temple merged commit e72e437 into main Jun 24, 2026
13 checks passed
@joshua-temple
joshua-temple deleted the fix/fleet-watch-retry branch June 24, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flake(fleet): bound retry the run watcher against transient 401 and 403 under load

1 participant