Skip to content

ci(azure): bound probe queue wait and execution separately - #972

Merged
Xuanwo merged 5 commits into
mainfrom
xuanwo/ci-azure-probe-timeouts
Nov 19, 2026
Merged

Xuanwo merged 5 commits into
mainfrom
xuanwo/ci-azure-probe-timeouts

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

The probe used one 55-minute deadline measured from queueing. Builds 102 and
118 hit that deadline without the Azure run completing, but the read-only
DevOps access needed to read their timelines is not available yet, so the open
question is agent start versus execution: those runs cannot say how much of
the 55 minutes was queue wait.

The wait for an agent is now bounded at 20 minutes, and execution at 15
minutes from the agent start read from the build timeline: a Job record
carries workerName once an agent picked it up, and its startTime is when
it started there. Stage records also start before an agent is assigned, and a
build's own startTime comes from the orchestration, so neither is treated as
agent start. A build that is still queued may not have a timeline yet, which
is still queued rather than a failed read. The timeline is read with the Build
read access this job already has.

The two budgets are a separation threshold, not a measured maximum: the probe
has finished end to end in about seven minutes when the pool was free (#952),
and the observed stalls ran past 55 minutes without completing. 20 + 15
minutes plus setup and cleanup stays inside the job's 60-minute timeout and
inside the access-token lifetime that cleanup can renew.

Verification: 26 local fixture tests (python3 .github/azure-live-tests/test_scripts.py) cover the bounds, the 404 and
timeline-list shapes, and the record parsing; dropping the workerName
filter, ignoring the timeline, or removing the 404 tolerance each fails them.
shellcheck is clean.

Known gap: no Azure timeline response was read while preparing this change, so
record selection is verified against the documented semantics and the fixture,
not against the real API. The Azure Storage Test run on this pull request
exercises it; an unexpected shape reads as still queued and fails at the queue
bound instead of holding the agent for the job timeout.

A 55-minute Azure Pipelines run can outlive the GitHub OIDC assertion used by azure/login. Token refresh then fails during cleanup, leaving the unfinished run and probe behind.

Request a fresh assertion with the job existing id-token permission and audience, and log in again as the same service principal. Use it only after a failed token or blob delete, and retry that operation once. The queue POST, run ownership, polling timeout, and original exit status remain unchanged.

Cover normal cleanup, expired assertion cleanup for both the pipeline run and blob-only path, and reauthentication failure.
The login helper ::add-mask:: command was captured by command substitution together with the refreshed DevOps token. In Actions mode the Authorization header then contained the mask command and a newline, so cleanup status reads failed and the run was not cancelled.

Set the global access_token directly and call the helper outside command substitution so the workflow command reaches the runner on stdout. Add Actions-mode expiry coverage with exact DevOps bearer checks, and restore the #959 cleanup probe-deletion assertion.
…eanup-reauth

# Conflicts:
#	.github/azure-live-tests/test_scripts.py
The probe had one 55-minute deadline that started when the run was queued,
so a run stuck behind the single pool agent held the GitHub job for the
whole deadline and the failure never said whether the probe or the queue
was slow.

Read the build timeline and bound the wait for an agent at 20 minutes and
the execution from the observed agent start at 15 minutes. Only a Job
record with a workerName counts as an agent start: a stage record starts
before an agent is assigned, and a build's own startTime comes from the
orchestration. A build that is still queued may have no timeline yet,
which is not an execution start and not a read failure.

The bounds and poll interval can be shortened for the local fixture, which
covers the queue bound, the execution bound measured from a past agent
start, a missing timeline, a timeline list, and a run that starts and
finishes inside both bounds. The queue request, ownership check,
reauthentication and cleanup are unchanged.
Base automatically changed from xuanwo/azure-probe-cleanup-reauth to main November 19, 2026 18:36
@Xuanwo
Xuanwo merged commit f4cb0a0 into main Nov 19, 2026
62 checks passed
@Xuanwo
Xuanwo deleted the xuanwo/ci-azure-probe-timeouts branch November 19, 2026 21:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants