Repository navigation
[release/17.0] Fix flaky ResourceGeneratorMetadataTests: read MSBuild query results from -getResultOutputFile - #22023
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Conversation
…from -getResultOutputFile (#21920) ## Description `ResourceGeneratorMetadataTests.ResourceMetadataCoversExistingAndNewFiles` fails intermittently on `Infrastructure (ubuntu-latest)` with: ``` System.Text.Json.JsonReaderException : 'M' is an invalid start of a value. LineNumber: 0 | BytePositionInLine: 0. ``` Attempt-1 failures in the last 14 days: - https://github.com/microsoft/aspire/actions/runs/40767263273/attempts/1 (2026-10-28, [job](https://github.com/microsoft/aspire/actions/runs/40767263273/job/123096158916)) - https://github.com/microsoft/aspire/actions/runs/41708884846/attempts/1 (2026-11-03, [job](https://github.com/microsoft/aspire/actions/runs/41708884846/job/126174228961)) - https://github.com/microsoft/aspire/actions/runs/41770875499/attempts/1 (2026-11-04, [job](https://github.com/microsoft/aspire/actions/runs/41770875499/job/126380785122)) ## Failure stats (14d, attempt 1) `CI` workflow on microsoft/aspire, runs created 2026-10-25 06:28 → 2026-11-08 06:28 UTC, **attempt 1 only**. Denominator: attempt-1 jobs that ran (success or failure). Hits: the failed job's log matches the signature. Jobs that lost their runner, which have no log, are excluded. Main runs have no reruns, so their flake check is the next main run of the job. | | | |---|---| | Signature | `ResourceGeneratorMetadataTests.ResourceMetadataCoversExistingAndNewFiles` failed **and** the log has `JsonReaderException … invalid start of a value` | | Job | Infrastructure (ubuntu-latest) | | main pushes: hits / runs | 0 / 158 (0.00%) | | main pushes, 30d: hits / runs | 0 / 272 (0.00%). The 16 days before the 14d window add 0 hits. | | PR runs: hits / runs | 9 / 626 (1.44%); 0 PR-caused excluded | | Combined rate | 9 / 784 = **1.15%** | | Flake check | PR: 7/7 rerun hits passed on rerun (2 never rerun) | | First / last hit | 2026-10-26 / 2026-11-06 | | Still hitting main? | Last main hit: never. 12 main runs since the last hit (2026-11-06), with 0 hits | | Fix evidence | Local: injected MSBuild-server fallback line 3/3 fail → 5/5 pass. This PR's CI: Infrastructure (ubuntu) passed 1/1 on the head commit (plus 1/1 earlier). | Failing attempt-1 jobs, most recent first (7 of 7 log-verified hits): | Date (UTC) | Branch | Job | Signature excerpt | |---|---|---|---| | [2026-11-06 18:19](https://github.com/microsoft/aspire/actions/runs/42269717260/job/128072396605) | `eerhardt-update-appcontainers-dependency` | Infrastructure (ubuntu-latest) | `JsonReaderException : 'M' is an invalid start of a value. LineNumber: 0 \| BytePositionInL` | | [2026-11-06 00:14](https://github.com/microsoft/aspire/actions/runs/42135355787/job/127622997972) | `ievangelist-go-apphost-epic` | Infrastructure (ubuntu-latest) | `JsonReaderException : 'M' is an invalid start of a value. LineNumber: 0 \| BytePositionInL` | | [2026-11-04 04:03](https://github.com/microsoft/aspire/actions/runs/41770875499/job/126380785122) | `aspireagent/infra-tests-etxtbsy` | Infrastructure (ubuntu-latest) | `JsonReaderException : 'M' is an invalid start of a value. LineNumber: 0 \| BytePositionInL` | | [2026-11-03 20:02](https://github.com/microsoft/aspire/actions/runs/41708884846/job/126174228961) | `dev/karolz/azure-functions-v2` | Infrastructure (ubuntu-latest) | `JsonReaderException : 'M' is an invalid start of a value. LineNumber: 0 \| BytePositionInL` | | [2026-11-02 08:44](https://github.com/microsoft/aspire/actions/runs/41415300855/job/125178576561) | `mitchdenny-dashboard-testing-indicators` | Infrastructure (ubuntu-latest) | `JsonReaderException : 'M' is an invalid start of a value. LineNumber: 0 \| BytePositionInL` | | [2026-10-28 20:53](https://github.com/microsoft/aspire/actions/runs/40767263273/job/123096158916) | `danegsta-nuget-mapping-regression` | Infrastructure (ubuntu-latest) | `JsonReaderException : 'M' is an invalid start of a value. LineNumber: 0 \| BytePositionInL` | | [2026-10-26 18:53](https://github.com/microsoft/aspire/actions/runs/40381830238/job/121780795715) | `fix/template-restore-msbuild-server` | Infrastructure (ubuntu-latest) | `JsonReaderException : 'M' is an invalid start of a value. LineNumber: 0 \| BytePositionInL` | A strict re-scan of the job logs confirmed 7 of the 9 hits counted above; the table lists only confirmed hits. The rates above use the original count. **Verdict: MODERATE.** 9 hits on 9 different unrelated PR branches (1.4% of PR runs), and all 7 that were rerun passed on rerun. It has **never hit on main** in 158 runs, so main-only evidence is absent. The signature and cause are exact and match the known #21524 failure class. <!-- /failure-stats --> **Why:** each log shows the dotnet CLI's MSBuild-server client printing `MSBuild server unavailable: could not connect to the server within the timeout window; the server may have failed to start. Falling back to an in-process build.` on **stdout**. The test parsed the raw stdout of `msbuild -getItem:...` as JSON, so that line broke `JsonDocument.Parse`. This is the same failure class fixed for `Hex1bNativePublishingTests` in #21524. **Fix:** the test's `evaluate.ps1` now passes `-getResultOutputFile:<path>`, and the test parses that file instead of stdout. MSBuild-server behavior is unchanged and diagnostics stay in the captured output for failure messages. No assertions were changed and no sleeps or retries were added. The theory body moved into a `VerifyResourceMetadataAsync` helper so a new regression test, `ResourceMetadataIgnoresMSBuildServerFallbackDiagnostics`, can run the real metadata query (with all existing assertions) through a `dotnet` wrapper (`.sh` on Unix, `.cmd` on Windows) that writes the exact CI fallback line to stdout. It also asserts the line was captured, so it can't pass by skipping the noisy path. The wrapper is a shared helper (`Shared/MSBuildServerFallbackDiagnostic.cs`, written via `ExecutableFile`) now also used by `Hex1bNativePublishingTests`. ## Confidence Local runs on Linux with SDK 11.0.100-rc.2: | | Result | |---|---| | Before fix (stdout parsing, regression test kept) | **3/3 failed**: `JsonReaderException : 'M' is invalid after a single JSON value`, same exception type as CI | | After fix | **5/5 passed** for the regression test | | Related suites with the fix | `ResourceGeneratorMetadataTests` 8/8, `BuildWarningPolicyTests` 37/37, `Hex1bNativePublishingTests` 43/43 | Not run locally: Windows/macOS (the `.cmd` wrapper path is covered only by CI). The wrapper appends the diagnostic after the JSON, while in CI it appeared before it. Reading from the result file handles both orders. **How we stop this recurring:** MSBuild `-getItem`/`-getProperty` queries in tests should read `-getResultOutputFile` (as `BuildWarningPolicyTests`, `MicrosoftTestingPlatformTests` and `Hex1bNativePublishingTests` already do) rather than parsing stdout. Refs #21600 ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - [x] No **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 03b9885)
14 tasks
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
marked this pull request as ready for review
November 10, 2026 04:55
Member
Author
|
Approved for inclusion in release/17.0. Do not merge yet. |
Mitch Denny (mitchdenny)
requested review from
James Newton-King (JamesNK),
Adam Ratzman (adamint),
David Negstad (danegsta),
David Fowler (davidfowl),
Eric Erhardt (eerhardt),
Ella Hathaway (ellahathaway),
Karol Zadora-Przylecki (karolz-ms) and
Ankit Jain (radical)
as code owners
November 10, 2026 05:47
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-26-pr21920
branch
from
November 10, 2026 05:49
b7e4bc1 to
aa13403
Compare
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22023Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22023" |
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-26-pr21920
branch
from
November 10, 2026 06:47
aa13403 to
b7e4bc1
Compare
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-26-pr21920
branch
from
November 10, 2026 07:32
b7e4bc1 to
aa13403
Compare
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-26-pr21920
branch
from
November 10, 2026 08:57
aa13403 to
b7e4bc1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #21920 to
release/17.0(cherry-picked from 03b9885 with no modifications). Layer 26 of 55 in the release/17.0 backport stack; stacked on #22022.Backport recommendation: No — Test flake.
Backport of #21920
Checklist
<remarks />and<code />elements on your triple slash comments?