Repository navigation
ci(azure): bound probe queue wait and execution separately - #972
Merged
Merged
Conversation
A 55-minute Azure Pipelines run can outlive the GitHub OIDC assertion used by azure/login. Token refresh then fails during cleanup, leaving the unfinished run and probe behind. Request a fresh assertion with the job existing id-token permission and audience, and log in again as the same service principal. Use it only after a failed token or blob delete, and retry that operation once. The queue POST, run ownership, polling timeout, and original exit status remain unchanged. Cover normal cleanup, expired assertion cleanup for both the pipeline run and blob-only path, and reauthentication failure.
The login helper ::add-mask:: command was captured by command substitution together with the refreshed DevOps token. In Actions mode the Authorization header then contained the mask command and a newline, so cleanup status reads failed and the run was not cancelled. Set the global access_token directly and call the helper outside command substitution so the workflow command reaches the runner on stdout. Add Actions-mode expiry coverage with exact DevOps bearer checks, and restore the #959 cleanup probe-deletion assertion.
…eanup-reauth # Conflicts: # .github/azure-live-tests/test_scripts.py
The probe had one 55-minute deadline that started when the run was queued, so a run stuck behind the single pool agent held the GitHub job for the whole deadline and the failure never said whether the probe or the queue was slow. Read the build timeline and bound the wait for an agent at 20 minutes and the execution from the observed agent start at 15 minutes. Only a Job record with a workerName counts as an agent start: a stage record starts before an agent is assigned, and a build's own startTime comes from the orchestration. A build that is still queued may have no timeline yet, which is not an execution start and not a read failure. The bounds and poll interval can be shortened for the local fixture, which covers the queue bound, the execution bound measured from a past agent start, a missing timeline, a timeline list, and a run that starts and finishes inside both bounds. The queue request, ownership check, reauthentication and cleanup are unchanged.
PsiACE
approved these changes
Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The probe used one 55-minute deadline measured from queueing. Builds 102 and
118 hit that deadline without the Azure run completing, but the read-only
DevOps access needed to read their timelines is not available yet, so the open
question is agent start versus execution: those runs cannot say how much of
the 55 minutes was queue wait.
The wait for an agent is now bounded at 20 minutes, and execution at 15
minutes from the agent start read from the build timeline: a
Jobrecordcarries
workerNameonce an agent picked it up, and itsstartTimeis whenit started there. Stage records also start before an agent is assigned, and a
build's own
startTimecomes from the orchestration, so neither is treated asagent start. A build that is still queued may not have a timeline yet, which
is still queued rather than a failed read. The timeline is read with the Build
read access this job already has.
The two budgets are a separation threshold, not a measured maximum: the probe
has finished end to end in about seven minutes when the pool was free (#952),
and the observed stalls ran past 55 minutes without completing. 20 + 15
minutes plus setup and cleanup stays inside the job's 60-minute timeout and
inside the access-token lifetime that cleanup can renew.
Verification: 26 local fixture tests (
python3 .github/azure-live-tests/test_scripts.py) cover the bounds, the 404 andtimeline-list shapes, and the record parsing; dropping the
workerNamefilter, ignoring the timeline, or removing the 404 tolerance each fails them.
shellcheckis clean.Known gap: no Azure timeline response was read while preparing this change, so
record selection is verified against the documented semantics and the fixture,
not against the real API. The
Azure Storage Testrun on this pull requestexercises it; an unexpected shape reads as still queued and fails at the queue
bound instead of holding the agent for the job timeout.