Skip to content

fix(ci): retry HTTP 503 on Azure pipeline status reads - #969

Merged
Xuanwo merged 1 commit into
mainfrom
xuanwo/retry-azure-503
Nov 19, 2026
Merged

Xuanwo merged 1 commit into
mainfrom
xuanwo/retry-azure-503

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Nov 19, 2026

Copy link
Copy Markdown
Member

The Azure Pipelines live-test job polls the status of a queued run with GET. Job 135585811335 (run attempt 1) failed after queuing build 116 with curl: (22) The requested URL returned error: 503; the same run succeeded on attempt 2. #959 retries the observed curl 35 reset but treats HTTP 503 as permanent, so this transient status-read failure still fails the job.

Extend the same status-read retry policy to HTTP 503 only, using #959's three attempts and one-second delay. The polling loop and the cleanup status read share this helper. The HTTP code is captured with curl's write-out and the response body is written to a temporary file; failed and partial bodies are never parsed or printed. Other HTTP failures, including 401 and 403, return immediately with curl's exit status. The queue POST still runs once and the cancellation PATCH still runs at most once.

Validation: python3 .github/azure-live-tests/test_scripts.py passes 17 tests, including 503 then success, exhausted 503 retaining exit 22, and a 403 that is not retried. The synthetic suite also asserts one POST, zero or one PATCH, and no failed response body or token in output. Shell syntax and Python compilation checks pass. No live Azure credentials or endpoints were used.

The Azure live-test job 135585811335 in run 44531403273 failed on its first attempt with `curl: (22) The requested URL returned error: 503` while polling queued run 116. The job passed on attempt 2 without a code change, so the 503 was transient.

Retry status GETs on HTTP 503 within the existing three-attempt, one-second budget from #959, while keeping the curl 35 retry. Capture the HTTP code with curl's write-out and discard the response body in a temporary file on failure, so failed or partial bodies are never parsed or exposed. Other HTTP failures return immediately, the queue POST still runs once, and the cancel PATCH still runs at most once.

Validation: `python3 .github/azure-live-tests/test_scripts.py` passes 17 tests, including 503 then success, exhausted 503 retaining exit 22, and a 403 that is not retried. Shell syntax and Python compilation checks pass. No live Azure credentials or endpoints were used.
@Xuanwo
Xuanwo merged commit 2d5daad into main Nov 19, 2026
62 checks passed
@Xuanwo
Xuanwo deleted the xuanwo/retry-azure-503 branch November 19, 2026 17:10
Xuanwo added a commit that referenced this pull request Nov 19, 2026
A long-running Azure Pipelines live test can outlive the GitHub OIDC
assertion used by `azure/login`. When cleanup runs after that assertion
expires, refreshing the DevOps token or deleting the probe fails even
though the federated credential is still valid.

Renew the Azure CLI login only after a token or blob-delete failure,
using the job's existing `id-token: write` permission, the same client
and tenant, and the existing `api://AzureADTokenExchange` audience. Keep
the login helper's `::add-mask::` command on the runner's stdout while
capturing only the returned token. Retry the failed operation once. The
queue POST, run ID ownership, 3,300-second poll budget, #959/#969 curl
35 and HTTP 503 status retries, and original exit status remain
unchanged; a failed cancel still does not prevent probe deletion.

Validation on the merged branch:
- `python3 .github/azure-live-tests/test_scripts.py` (21 tests)
- Shell syntax, `cargo fmt --all -- --check`, and `hawkeye check`
- Covers normal cleanup, expired assertion cleanup for both the pipeline
run and blob-only path, Actions-mode reauthentication with exact DevOps
bearer and mask-channel checks, reauthentication failure preserving the
original exit status, the #959 cleanup probe-deletion assertion, and the
merged #969 HTTP 503/403 behavior

No new secrets, permissions, or long-lived state; no live Azure
credentials were used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants