From 184ab69417448fbd79718739b53976049c6a060f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 6 Aug 2026 11:54:50 +0200 Subject: [PATCH 1/4] Document Azure DevOps group limitations Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- .../core/testing/microsoft-testing-platform-test-reports.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/core/testing/microsoft-testing-platform-test-reports.md b/docs/core/testing/microsoft-testing-platform-test-reports.md index 9c287949ac999..23658aade1462 100644 --- a/docs/core/testing/microsoft-testing-platform-test-reports.md +++ b/docs/core/testing/microsoft-testing-platform-test-reports.md @@ -147,6 +147,7 @@ builder.TestHost.AddAzureDevOpsProvider(); |---|---|---| | `--report-azdo` | 1.9.0 | Enables the Azure DevOps report generator. Errors and warnings are written to the output in a format that Azure DevOps understands. | | `--report-azdo-severity` | 1.9.0 | Severity to use for reported events. Valid values are `error` (default) and `warning`. | +| `--report-azdo-groups` | 2.4.0 | Enables or disables per-assembly log groups. Valid values are `on` (default) and `off`. Requires `--report-azdo`. | | `--report-azdo-flaky-history` | 2.3.0 | Queries Azure DevOps test result history for the past N days (1-90) and annotates reported failures with flakiness context. Requires `--report-azdo`. | | `--report-azdo-demote-known-flaky` | 2.3.0 | Demotes failures that are flaky enough in the Azure DevOps history window (default threshold is 25%) from errors to warnings. Requires `--report-azdo` and `--report-azdo-flaky-history`. | | `--report-azdo-quarantine-file` | 2.3.0 | Path to a text file that lists quarantined test fully qualified names or glob patterns. Matching failures are reported as warnings. Requires `--report-azdo`. | @@ -159,8 +160,11 @@ builder.TestHost.AddAzureDevOpsProvider(); | `--publish-azdo-test-results` | 2.3.0 | Publishes test results live to the Azure DevOps **Tests** tab. | | `--publish-azdo-run-name` | 2.3.0 | Sets a custom Azure DevOps test run name for live test-result publishing. Requires `--publish-azdo-test-results`. | +> [!WARNING] +> Groups aren't recommended when multiple test assemblies run in parallel. Azure DevOps `##[group]` and `##[endgroup]` formatting commands are sequential and anonymous. Concurrent assembly output can interleave, cause incorrect group nesting, and put lines under the wrong assembly. Set `--report-azdo-groups off` for parallel multi-assembly runs. Groups are suitable for one assembly or serialized assembly execution. + > [!NOTE] -> The **MTP version** column lists the MTP release in which each option first became available in a stable build. The Azure DevOps extension itself became stable in MTP 1.9.0 with `--report-azdo` and `--report-azdo-severity`; the remaining options were added in MTP 2.3.0. +> The **MTP version** column lists the MTP release in which each option first became available in a stable build. The Azure DevOps extension itself became stable in MTP 1.9.0 with `--report-azdo` and `--report-azdo-severity`; the remaining options were added in MTP 2.3.0 or 2.4.0. The extension automatically detects that it is running in continuous integration (CI) environment by checking the `TF_BUILD` environment variable. From 1abf6e4aa8f437092dc9767240ce8c42858e034c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 6 Aug 2026 11:57:29 +0200 Subject: [PATCH 2/4] Refine Azure DevOps group guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 75f9a66e-631b-4d19-9800-892cf968eefd --- .../core/testing/microsoft-testing-platform-test-reports.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/core/testing/microsoft-testing-platform-test-reports.md b/docs/core/testing/microsoft-testing-platform-test-reports.md index 23658aade1462..47585213b88d1 100644 --- a/docs/core/testing/microsoft-testing-platform-test-reports.md +++ b/docs/core/testing/microsoft-testing-platform-test-reports.md @@ -147,7 +147,7 @@ builder.TestHost.AddAzureDevOpsProvider(); |---|---|---| | `--report-azdo` | 1.9.0 | Enables the Azure DevOps report generator. Errors and warnings are written to the output in a format that Azure DevOps understands. | | `--report-azdo-severity` | 1.9.0 | Severity to use for reported events. Valid values are `error` (default) and `warning`. | -| `--report-azdo-groups` | 2.4.0 | Enables or disables per-assembly log groups. Valid values are `on` (default) and `off`. Requires `--report-azdo`. | +| `--report-azdo-groups ` | 2.4.0 | Enables or disables per-assembly log groups. Valid values are `on` (default) and `off`. Requires `--report-azdo`. | | `--report-azdo-flaky-history` | 2.3.0 | Queries Azure DevOps test result history for the past N days (1-90) and annotates reported failures with flakiness context. Requires `--report-azdo`. | | `--report-azdo-demote-known-flaky` | 2.3.0 | Demotes failures that are flaky enough in the Azure DevOps history window (default threshold is 25%) from errors to warnings. Requires `--report-azdo` and `--report-azdo-flaky-history`. | | `--report-azdo-quarantine-file` | 2.3.0 | Path to a text file that lists quarantined test fully qualified names or glob patterns. Matching failures are reported as warnings. Requires `--report-azdo`. | @@ -161,10 +161,10 @@ builder.TestHost.AddAzureDevOpsProvider(); | `--publish-azdo-run-name` | 2.3.0 | Sets a custom Azure DevOps test run name for live test-result publishing. Requires `--publish-azdo-test-results`. | > [!WARNING] -> Groups aren't recommended when multiple test assemblies run in parallel. Azure DevOps `##[group]` and `##[endgroup]` formatting commands are sequential and anonymous. Concurrent assembly output can interleave, cause incorrect group nesting, and put lines under the wrong assembly. Set `--report-azdo-groups off` for parallel multi-assembly runs. Groups are suitable for one assembly or serialized assembly execution. +> Disable groups when multiple test assemblies run in parallel. Azure DevOps `##[group]` and `##[endgroup]` formatting commands are sequential and anonymous. Concurrent assembly output can interleave, cause incorrect group nesting, and put lines under the wrong assembly. Use `--report-azdo-groups off` for parallel multi-assembly runs. Groups are suitable for one assembly or serialized assembly execution. > [!NOTE] -> The **MTP version** column lists the MTP release in which each option first became available in a stable build. The Azure DevOps extension itself became stable in MTP 1.9.0 with `--report-azdo` and `--report-azdo-severity`; the remaining options were added in MTP 2.3.0 or 2.4.0. +> The **MTP version** column lists the first MTP version that contains each option. The Azure DevOps extension itself became stable in MTP 1.9.0 with `--report-azdo` and `--report-azdo-severity`; the remaining options were added in MTP 2.3.0 or 2.4.0. The extension automatically detects that it is running in continuous integration (CI) environment by checking the `TF_BUILD` environment variable. From 9757ca4d0f96ba2a77bd3c61de42b82b532a83f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 6 Aug 2026 11:59:43 +0200 Subject: [PATCH 3/4] Align Azure DevOps group option guidance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/core/testing/microsoft-testing-platform-test-reports.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/core/testing/microsoft-testing-platform-test-reports.md b/docs/core/testing/microsoft-testing-platform-test-reports.md index 47585213b88d1..e09bc8df413e4 100644 --- a/docs/core/testing/microsoft-testing-platform-test-reports.md +++ b/docs/core/testing/microsoft-testing-platform-test-reports.md @@ -147,7 +147,7 @@ builder.TestHost.AddAzureDevOpsProvider(); |---|---|---| | `--report-azdo` | 1.9.0 | Enables the Azure DevOps report generator. Errors and warnings are written to the output in a format that Azure DevOps understands. | | `--report-azdo-severity` | 1.9.0 | Severity to use for reported events. Valid values are `error` (default) and `warning`. | -| `--report-azdo-groups ` | 2.4.0 | Enables or disables per-assembly log groups. Valid values are `on` (default) and `off`. Requires `--report-azdo`. | +| `--report-azdo-groups` | 2.4.0 | Enables or disables per-assembly log groups. Valid values are `on` (default) and `off`. Requires `--report-azdo`. | | `--report-azdo-flaky-history` | 2.3.0 | Queries Azure DevOps test result history for the past N days (1-90) and annotates reported failures with flakiness context. Requires `--report-azdo`. | | `--report-azdo-demote-known-flaky` | 2.3.0 | Demotes failures that are flaky enough in the Azure DevOps history window (default threshold is 25%) from errors to warnings. Requires `--report-azdo` and `--report-azdo-flaky-history`. | | `--report-azdo-quarantine-file` | 2.3.0 | Path to a text file that lists quarantined test fully qualified names or glob patterns. Matching failures are reported as warnings. Requires `--report-azdo`. | @@ -161,7 +161,7 @@ builder.TestHost.AddAzureDevOpsProvider(); | `--publish-azdo-run-name` | 2.3.0 | Sets a custom Azure DevOps test run name for live test-result publishing. Requires `--publish-azdo-test-results`. | > [!WARNING] -> Disable groups when multiple test assemblies run in parallel. Azure DevOps `##[group]` and `##[endgroup]` formatting commands are sequential and anonymous. Concurrent assembly output can interleave, cause incorrect group nesting, and put lines under the wrong assembly. Use `--report-azdo-groups off` for parallel multi-assembly runs. Groups are suitable for one assembly or serialized assembly execution. +> Groups aren't recommended when multiple test assemblies run in parallel. Azure DevOps `##[group]` and `##[endgroup]` formatting commands are sequential and anonymous. Concurrent assembly output can interleave, cause incorrect group nesting, and put lines under the wrong assembly. Set `--report-azdo-groups off` for parallel multi-assembly runs. Groups are suitable for one assembly or serialized assembly execution. > [!NOTE] > The **MTP version** column lists the first MTP version that contains each option. The Azure DevOps extension itself became stable in MTP 1.9.0 with `--report-azdo` and `--report-azdo-severity`; the remaining options were added in MTP 2.3.0 or 2.4.0. From 611ba01485d98bc24bebef92c674f5ae77148279 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Thu, 6 Aug 2026 12:20:12 +0200 Subject: [PATCH 4/4] Document Azure DevOps groups as opt-in Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- docs/core/testing/microsoft-testing-platform-test-reports.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/core/testing/microsoft-testing-platform-test-reports.md b/docs/core/testing/microsoft-testing-platform-test-reports.md index e09bc8df413e4..92ea1b2e5e1ba 100644 --- a/docs/core/testing/microsoft-testing-platform-test-reports.md +++ b/docs/core/testing/microsoft-testing-platform-test-reports.md @@ -147,7 +147,7 @@ builder.TestHost.AddAzureDevOpsProvider(); |---|---|---| | `--report-azdo` | 1.9.0 | Enables the Azure DevOps report generator. Errors and warnings are written to the output in a format that Azure DevOps understands. | | `--report-azdo-severity` | 1.9.0 | Severity to use for reported events. Valid values are `error` (default) and `warning`. | -| `--report-azdo-groups` | 2.4.0 | Enables or disables per-assembly log groups. Valid values are `on` (default) and `off`. Requires `--report-azdo`. | +| `--report-azdo-groups` | 2.4.0 | Enables or disables per-assembly log groups. Valid values are `on` and `off` (default). Requires `--report-azdo`. | | `--report-azdo-flaky-history` | 2.3.0 | Queries Azure DevOps test result history for the past N days (1-90) and annotates reported failures with flakiness context. Requires `--report-azdo`. | | `--report-azdo-demote-known-flaky` | 2.3.0 | Demotes failures that are flaky enough in the Azure DevOps history window (default threshold is 25%) from errors to warnings. Requires `--report-azdo` and `--report-azdo-flaky-history`. | | `--report-azdo-quarantine-file` | 2.3.0 | Path to a text file that lists quarantined test fully qualified names or glob patterns. Matching failures are reported as warnings. Requires `--report-azdo`. | @@ -161,7 +161,7 @@ builder.TestHost.AddAzureDevOpsProvider(); | `--publish-azdo-run-name` | 2.3.0 | Sets a custom Azure DevOps test run name for live test-result publishing. Requires `--publish-azdo-test-results`. | > [!WARNING] -> Groups aren't recommended when multiple test assemblies run in parallel. Azure DevOps `##[group]` and `##[endgroup]` formatting commands are sequential and anonymous. Concurrent assembly output can interleave, cause incorrect group nesting, and put lines under the wrong assembly. Set `--report-azdo-groups off` for parallel multi-assembly runs. Groups are suitable for one assembly or serialized assembly execution. +> Groups are disabled by default and aren't recommended when multiple test assemblies run in parallel. Azure DevOps `##[group]` and `##[endgroup]` formatting commands are sequential and anonymous. Concurrent assembly output can interleave, cause incorrect group nesting, and put lines under the wrong assembly. Set `--report-azdo-groups on` only for a single assembly or serialized assembly execution. > [!NOTE] > The **MTP version** column lists the first MTP version that contains each option. The Azure DevOps extension itself became stable in MTP 1.9.0 with `--report-azdo` and `--report-azdo-severity`; the remaining options were added in MTP 2.3.0 or 2.4.0.