Skip to content

MAINT: Publish the backend log from the shared test jobs - #2520

Open
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/v1.1.0-Release-Backend-Log-Artifact
Open

MAINT: Publish the backend log from the shared test jobs#2520
varunj-msft wants to merge 1 commit into
microsoft:mainfrom
varunj-msft:varunj-msft/v1.1.0-Release-Backend-Log-Artifact

Conversation

@varunj-msft

Copy link
Copy Markdown
Contributor

Description

The end-to-end and integration jobs launch pyrit_backend through ServerLauncher, which writes the server log to a file in the system temp directory and then throws it away when the agent is torn down. A failing job therefore reports only the client side of the failure, which means a backend that is alive but unresponsive is indistinguishable from one that never started at all.

That distinction is exactly what we could not make while debugging the current End to End Tests failures: every one of the failures is a client-side read timeout, and the server's own account of what it was doing is discarded along with the agent.

This collects the log and publishes it as a pipeline artifact, following the pattern adversarial-benchmark.yml already uses:

  • The collection step runs under condition: always(), so a job that fails, or times out at the 360-minute cap, still yields the log.
  • ServerLauncher writes to tempfile.gettempdir(), which honours TMPDIR/TEMP/TMP before falling back to /tmp, so the step checks the agent temp directory as well as /tmp.
  • A missing file is not an error. Jobs that never launch a backend log one line and move on.
  • The publish task is skipped unless the copy actually landed, so we never publish an empty artifact that claims a log is there when it isn't.

Only .azuredevops/test-job-template.yml changes. No product code, no test code, and no behaviour change for any job — this adds diagnostics only.

Part of the v1.1.0 release wave with #2510, #2511 and #2512.

Tests and Documentation

This is pipeline YAML, so no unit test can cover it. I validated it by extracting the script body and executing it against all three paths:

  • Log present in the temp directory: copied, backendLogFound=true, artifact published. Step exits 0.
  • No log present, i.e. a job that never starts a backend: prints "No pyrit_backend.log found", publish skipped. Step exits 0.
  • Log present but the copy fails, tested with an unwritable staging directory: flag not set, publish skipped. Step still exits 0.

The third case is worth calling out, because the first version of this step got it wrong. set -uo pipefail does not include -e, so a failing cp is silent: the step would have claimed success, set backendLogFound=true, and published an empty directory while simultaneously printing "No pyrit_backend.log found". The step now gates both the success message and the setvariable on the copy actually succeeding, so the flag is set only when the file has really landed.

No documentation changes, since this touches only pipeline YAML. JupyText was not run and is not applicable here: no notebooks, docs or public APIs are affected by this change.

@varunj-msft varunj-msft changed the title MAINT Publish the backend log from the shared test jobs MAINT: Publish the backend log from the shared test jobs Sep 1, 2026
@hannahwestra25 hannahwestra25 self-assigned this Sep 1, 2026
Comment thread .azuredevops/test-job-template.yml
Comment thread .azuredevops/test-job-template.yml Outdated
@varunj-msft
varunj-msft force-pushed the varunj-msft/v1.1.0-Release-Backend-Log-Artifact branch from 6806919 to 4ebf73f Compare September 1, 2026 18:04
Comment thread .azuredevops/test-job-template.yml
The end-to-end tests launch pyrit_backend through ServerLauncher, which writes
the server log to a file in the system temp directory and then discards it when
the agent is torn down. A failing job therefore reports only the client side of
the failure, so a backend that is alive but unresponsive is indistinguishable
from one that never started.

Collect that log and publish it as a pipeline artifact, following the pattern
already used by adversarial-benchmark.yml. The collection step tolerates a
missing file so the jobs that do not launch a backend are unaffected, and the
publish step is skipped entirely when no log was found.

Search every directory tempfile.gettempdir() honours. It reads TMPDIR, TEMP and
TMP before falling back to /tmp, so checking only TMPDIR silently dropped the
log whenever the agent set one of the other two, which is exactly the diagnostic
this change exists to capture.

Name the artifact per job attempt. Build.BuildId does not change when a failed
job is retried and pipeline artifacts cannot be overwritten, so the retry of any
failed job would fail at the publish step. System.JobAttempt keeps each attempt
distinct, and continueOnError keeps a failed upload of a diagnostic log from
changing the outcome of the test job itself.

Report a failed collection separately from a missing log. The step decided
whether a log existed by testing the destination, so a copy that failed printed
the same line as a job that never launches a backend, which on the end-to-end
pipeline is the opposite of what happened. Track whether a source was found,
warn with the reason the copy failed, and report the size of what was collected
so a zero byte log is not mistaken for a step that did nothing.
@varunj-msft
varunj-msft force-pushed the varunj-msft/v1.1.0-Release-Backend-Log-Artifact branch from 4ebf73f to 7c94d42 Compare September 1, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants