Skip to content

Document Azure DevOps group limitations #55325

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Evangelink wants to merge 4 commits into
base: main
Choose a base branch
from
+5 −1
Open

Document Azure DevOps group limitations #55325

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
184ab69
Document Azure DevOps group limitations
Evangelink Aug 6, 2026
1abf6e4
Refine Azure DevOps group guidance
Evangelink Aug 6, 2026
9757ca4
Align Azure DevOps group option guidance
Evangelink Aug 6, 2026
611ba01
Document Azure DevOps groups as opt-in
Evangelink Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/core/testing/microsoft-testing-platform-test-reports.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ builder.TestHost.AddAzureDevOpsProvider();
|---|---|---|
| `--report-azdo` | 1.9.0 | Enables the Azure DevOps report generator. Errors and warnings are written to the output in a format that Azure DevOps understands. |
| `--report-azdo-severity` | 1.9.0 | Severity to use for reported events. Valid values are `error` (default) and `warning`. |
| `--report-azdo-groups` | 2.4.0 | Enables or disables per-assembly log groups. Valid values are `on` and `off` (default). Requires `--report-azdo`. |

@nohwnd nohwnd Aug 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not obvious to me what this does, 1 more sentence that says what the effect is would help.

| `--report-azdo-flaky-history` | 2.3.0 | Queries Azure DevOps test result history for the past N days (1-90) and annotates reported failures with flakiness context. Requires `--report-azdo`. |
| `--report-azdo-demote-known-flaky` | 2.3.0 | Demotes failures that are flaky enough in the Azure DevOps history window (default threshold is 25%) from errors to warnings. Requires `--report-azdo` and `--report-azdo-flaky-history`. |
| `--report-azdo-quarantine-file` | 2.3.0 | Path to a text file that lists quarantined test fully qualified names or glob patterns. Matching failures are reported as warnings. Requires `--report-azdo`. |
Expand All @@ -159,8 +160,11 @@ builder.TestHost.AddAzureDevOpsProvider();
| `--publish-azdo-test-results` | 2.3.0 | Publishes test results live to the Azure DevOps **Tests** tab. |
| `--publish-azdo-run-name` | 2.3.0 | Sets a custom Azure DevOps test run name for live test-result publishing. Requires `--publish-azdo-test-results`. |

> [!WARNING]
> Groups are disabled by default and aren't recommended when multiple test assemblies run in parallel. Azure DevOps `##[group]` and `##[endgroup]` formatting commands are sequential and anonymous. Concurrent assembly output can interleave, cause incorrect group nesting, and put lines under the wrong assembly. Set `--report-azdo-groups on` only for a single assembly or serialized assembly execution.

> [!NOTE]
> The **MTP version** column lists the MTP release in which each option first became available in a stable build. The Azure DevOps extension itself became stable in MTP 1.9.0 with `--report-azdo` and `--report-azdo-severity`; the remaining options were added in MTP 2.3.0.
> The **MTP version** column lists the first MTP version that contains each option. The Azure DevOps extension itself became stable in MTP 1.9.0 with `--report-azdo` and `--report-azdo-severity`; the remaining options were added in MTP 2.3.0 or 2.4.0.

The extension automatically detects that it is running in continuous integration (CI) environment by checking the `TF_BUILD` environment variable.

Expand Down
Loading