Repository navigation
fix(ci): retry transient Azure pipeline status reads - #959
Merged
Merged
Conversation
A single curl exit 35 connection reset while reading the state of an already queued run failed the Azure live-test job before the probe reported anything. Retry GET status reads up to three times with a short delay only for curl exit 35. The cleanup status read uses the same helper. Queue POST remains single-shot, and cancel PATCH remains at most once. Use --fail so error response bodies are not parsed or exposed, and preserve the final curl exit status. Cover one reset then success, exhausted retries, and a cleanup reset that still cancels once.
PsiACE
approved these changes
Nov 19, 2026
This was referenced Nov 19, 2026
Xuanwo
added a commit
that referenced
this pull request
Nov 19, 2026
The login helper ::add-mask:: command was captured by command substitution together with the refreshed DevOps token. In Actions mode the Authorization header then contained the mask command and a newline, so cleanup status reads failed and the run was not cancelled. Set the global access_token directly and call the helper outside command substitution so the workflow command reaches the runner on stdout. Add Actions-mode expiry coverage with exact DevOps bearer checks, and restore the #959 cleanup probe-deletion assertion.
Xuanwo
added a commit
that referenced
this pull request
Nov 19, 2026
The Azure Pipelines live-test job polls the status of a queued run with GET. Job [135585811335](https://github.com/apache/reqsign/actions/runs/44531403273/job/135585811335) (run attempt 1) failed after queuing build 116 with `curl: (22) The requested URL returned error: 503`; the same run succeeded on attempt 2. #959 retries the observed curl 35 reset but treats HTTP 503 as permanent, so this transient status-read failure still fails the job. Extend the same status-read retry policy to HTTP 503 only, using #959's three attempts and one-second delay. The polling loop and the cleanup status read share this helper. The HTTP code is captured with curl's write-out and the response body is written to a temporary file; failed and partial bodies are never parsed or printed. Other HTTP failures, including 401 and 403, return immediately with curl's exit status. The queue POST still runs once and the cancellation PATCH still runs at most once. Validation: `python3 .github/azure-live-tests/test_scripts.py` passes 17 tests, including 503 then success, exhausted 503 retaining exit 22, and a 403 that is not retried. The synthetic suite also asserts one POST, zero or one PATCH, and no failed response body or token in output. Shell syntax and Python compilation checks pass. No live Azure credentials or endpoints were used.
Xuanwo
added a commit
that referenced
this pull request
Nov 19, 2026
A long-running Azure Pipelines live test can outlive the GitHub OIDC assertion used by `azure/login`. When cleanup runs after that assertion expires, refreshing the DevOps token or deleting the probe fails even though the federated credential is still valid. Renew the Azure CLI login only after a token or blob-delete failure, using the job's existing `id-token: write` permission, the same client and tenant, and the existing `api://AzureADTokenExchange` audience. Keep the login helper's `::add-mask::` command on the runner's stdout while capturing only the returned token. Retry the failed operation once. The queue POST, run ID ownership, 3,300-second poll budget, #959/#969 curl 35 and HTTP 503 status retries, and original exit status remain unchanged; a failed cancel still does not prevent probe deletion. Validation on the merged branch: - `python3 .github/azure-live-tests/test_scripts.py` (21 tests) - Shell syntax, `cargo fmt --all -- --check`, and `hawkeye check` - Covers normal cleanup, expired assertion cleanup for both the pipeline run and blob-only path, Actions-mode reauthentication with exact DevOps bearer and mask-channel checks, reauthentication failure preserving the original exit status, the #959 cleanup probe-deletion assertion, and the merged #969 HTTP 503/403 behavior No new secrets, permissions, or long-lived state; no live Azure credentials were used.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Azure live-test job in #956 failed at job 135562203082 with
curl: (35) Send failure: Connection reset by peerwhile polling the state of an already queued Azure Pipelines run. It failed before the Rust probe reported a result. The same status path succeeded onmainand in #952, so this is a transient transport failure rather than a signing or credential regression.Retry GET status reads for the known run ID up to three times with a one-second delay, only for curl exit 35. The cleanup status read uses the same helper. The queue POST still runs once, and the cancel PATCH still runs at most once. Status reads use
--failso an error response body is never parsed or exposed, and the final curl exit status is preserved when retries are exhausted.Validation:
python3 .github/azure-live-tests/test_scripts.pypasses 14 tests, including one reset then success, exhausted retries retaining exit 35, and a cleanup reset followed by exactly one cancel. Shell syntax and Python compilation checks pass. No live Azure credentials were used.