Skip to content

fix(ci): retry transient Azure pipeline status reads - #959

Merged
Xuanwo merged 1 commit into
mainfrom
xuanwo/retry-azure-status-reads
Nov 19, 2026
Merged

Xuanwo merged 1 commit into
mainfrom
xuanwo/retry-azure-status-reads

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Nov 19, 2026

Copy link
Copy Markdown
Member

The Azure live-test job in #956 failed at job 135562203082 with curl: (35) Send failure: Connection reset by peer while polling the state of an already queued Azure Pipelines run. It failed before the Rust probe reported a result. The same status path succeeded on main and in #952, so this is a transient transport failure rather than a signing or credential regression.

Retry GET status reads for the known run ID up to three times with a one-second delay, only for curl exit 35. The cleanup status read uses the same helper. The queue POST still runs once, and the cancel PATCH still runs at most once. Status reads use --fail so an error response body is never parsed or exposed, and the final curl exit status is preserved when retries are exhausted.

Validation: python3 .github/azure-live-tests/test_scripts.py passes 14 tests, including one reset then success, exhausted retries retaining exit 35, and a cleanup reset followed by exactly one cancel. Shell syntax and Python compilation checks pass. No live Azure credentials were used.

A single curl exit 35 connection reset while reading the state of an
already queued run failed the Azure live-test job before the probe
reported anything.

Retry GET status reads up to three times with a short delay only for
curl exit 35. The cleanup status read uses the same helper. Queue POST
remains single-shot, and cancel PATCH remains at most once. Use --fail
so error response bodies are not parsed or exposed, and preserve the
final curl exit status.

Cover one reset then success, exhausted retries, and a cleanup reset
that still cancels once.
@Xuanwo
Xuanwo merged commit 724d6c7 into main Nov 19, 2026
74 of 76 checks passed
@Xuanwo
Xuanwo deleted the xuanwo/retry-azure-status-reads branch November 19, 2026 15:24
Xuanwo added a commit that referenced this pull request Nov 19, 2026
The login helper ::add-mask:: command was captured by command substitution together with the refreshed DevOps token. In Actions mode the Authorization header then contained the mask command and a newline, so cleanup status reads failed and the run was not cancelled.

Set the global access_token directly and call the helper outside command substitution so the workflow command reaches the runner on stdout. Add Actions-mode expiry coverage with exact DevOps bearer checks, and restore the #959 cleanup probe-deletion assertion.
Xuanwo added a commit that referenced this pull request Nov 19, 2026
The Azure Pipelines live-test job polls the status of a queued run with
GET. Job
[135585811335](https://github.com/apache/reqsign/actions/runs/44531403273/job/135585811335)
(run attempt 1) failed after queuing build 116 with `curl: (22) The
requested URL returned error: 503`; the same run succeeded on attempt 2.
#959 retries the observed curl 35 reset but treats HTTP 503 as
permanent, so this transient status-read failure still fails the job.

Extend the same status-read retry policy to HTTP 503 only, using #959's
three attempts and one-second delay. The polling loop and the cleanup
status read share this helper. The HTTP code is captured with curl's
write-out and the response body is written to a temporary file; failed
and partial bodies are never parsed or printed. Other HTTP failures,
including 401 and 403, return immediately with curl's exit status. The
queue POST still runs once and the cancellation PATCH still runs at most
once.

Validation: `python3 .github/azure-live-tests/test_scripts.py` passes 17
tests, including 503 then success, exhausted 503 retaining exit 22, and
a 403 that is not retried. The synthetic suite also asserts one POST,
zero or one PATCH, and no failed response body or token in output. Shell
syntax and Python compilation checks pass. No live Azure credentials or
endpoints were used.
Xuanwo added a commit that referenced this pull request Nov 19, 2026
A long-running Azure Pipelines live test can outlive the GitHub OIDC
assertion used by `azure/login`. When cleanup runs after that assertion
expires, refreshing the DevOps token or deleting the probe fails even
though the federated credential is still valid.

Renew the Azure CLI login only after a token or blob-delete failure,
using the job's existing `id-token: write` permission, the same client
and tenant, and the existing `api://AzureADTokenExchange` audience. Keep
the login helper's `::add-mask::` command on the runner's stdout while
capturing only the returned token. Retry the failed operation once. The
queue POST, run ID ownership, 3,300-second poll budget, #959/#969 curl
35 and HTTP 503 status retries, and original exit status remain
unchanged; a failed cancel still does not prevent probe deletion.

Validation on the merged branch:
- `python3 .github/azure-live-tests/test_scripts.py` (21 tests)
- Shell syntax, `cargo fmt --all -- --check`, and `hawkeye check`
- Covers normal cleanup, expired assertion cleanup for both the pipeline
run and blob-only path, Actions-mode reauthentication with exact DevOps
bearer and mask-channel checks, reauthentication failure preserving the
original exit status, the #959 cleanup probe-deletion assertion, and the
merged #969 HTTP 503/403 behavior

No new secrets, permissions, or long-lived state; no live Azure
credentials were used.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants