diff --git a/docs/core/testing/microsoft-testing-platform-retry.md b/docs/core/testing/microsoft-testing-platform-retry.md index 275edcf0b06df..b1bef557483e7 100644 --- a/docs/core/testing/microsoft-testing-platform-retry.md +++ b/docs/core/testing/microsoft-testing-platform-retry.md @@ -3,7 +3,7 @@ title: Microsoft.Testing.Platform (MTP) retry description: Learn about retrying failed tests with MTP. author: evangelink ms.author: amauryleve -ms.date: 06/01/2026 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -42,6 +42,10 @@ This extension is intended for integration tests where the test depends heavily Both threshold options (`--retry-failed-tests-max-percentage` and `--retry-failed-tests-max-tests`) require `--retry-failed-tests` to also be set. +Starting with MTP 2.4.0, the retry summary reports separate `flaky` and `retried` counts and lists recovered test names under **Flaky tests**. To avoid misleading totals, MTP doesn't print a full-suite summary for filtered retry attempts after the first attempt. Use the terminal reporter's [`--show-flaky-tests`](microsoft-testing-platform-terminal-output.md#options) option to show or hide flaky details. + +Each retry attempt writes its report artifacts under the `Retries` directory. For JUnit, the top-level report represents only the final filtered retry attempt, not the original full suite; collect the earlier attempt reports from `Retries` when your CI requires complete coverage. This layout applies to `--retry-failed-tests`. MSTest's in-process `[Retry]` instead collapses superseded attempts into one final JUnit or TRX result per test. + ### Examples Retry failed tests up to 3 times: diff --git a/docs/core/testing/microsoft-testing-platform-terminal-output.md b/docs/core/testing/microsoft-testing-platform-terminal-output.md index b8bc695f27000..8171110826548 100644 --- a/docs/core/testing/microsoft-testing-platform-terminal-output.md +++ b/docs/core/testing/microsoft-testing-platform-terminal-output.md @@ -3,7 +3,7 @@ title: Microsoft.Testing.Platform (MTP) terminal output description: Learn about the built-in terminal test reporter in MTP, including output modes, ANSI support, and progress indicators. author: evangelink ms.author: amauryleve -ms.date: 07/17/2026 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -60,6 +60,7 @@ If your code must write directly to the console and you need that output to rema | `--output` | — | Specifies the output verbosity when reporting tests. Valid values are `Normal` and `Detailed`. Default is `Normal`. | | `--show-stdout` | 2.2.1 | Determines when to show captured standard output of a test. Valid values are `All`, `Failed`, and `None`. Default is `All`. | | `--show-stderr` | 2.2.1 | Determines when to show captured error output of a test. Valid values are `All`, `Failed`, and `None`. Default is `All`. | +| `--show-flaky-tests` | 2.4.0 | Controls the `flaky:` summary and the **Flaky tests** list for tests that pass after a retry. Use `on` or `off`; the default is `on`. Applies to MSTest `[Retry]` and the [retry extension](microsoft-testing-platform-retry.md). | > [!NOTE] > A dash (—) in the **MTP version** column marks core options that aren't tied to a specific version because they've been available since the platform's initial releases. diff --git a/docs/core/testing/microsoft-testing-platform-test-reports.md b/docs/core/testing/microsoft-testing-platform-test-reports.md index 9c287949ac999..eb3a9ff1345ce 100644 --- a/docs/core/testing/microsoft-testing-platform-test-reports.md +++ b/docs/core/testing/microsoft-testing-platform-test-reports.md @@ -3,7 +3,7 @@ title: Microsoft.Testing.Platform (MTP) test reports description: Learn about the MTP extensions that create test report files (TRX, HTML, JUnit, CTRF, Azure DevOps, GitHub Actions). author: evangelink ms.author: amauryleve -ms.date: 06/16/2026 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -90,6 +90,8 @@ The JUnit report creates a JUnit-compatible XML file for a test session. This ex > [!NOTE] > Available in MTP starting with version 2.3.0. This extension is experimental, and its options and output format might change in a future version. +> +> Starting with MSTest.Sdk 4.3, enable this extension with `true`. The extension isn't part of the `Default` or `AllMicrosoft` MSTest.Sdk profiles. ### Manual registration @@ -126,6 +128,8 @@ builder.AddCtrfReportProvider(); | `--report-ctrf` | Generates the CTRF JSON report. | | `--report-ctrf-filename` | The name of the generated CTRF JSON report. The value must end with `.json`. The default is `____.ctrf.json`. To customize the name, see [Report file names](#report-file-names). Requires `--report-ctrf`. | +Starting with MSTest 4.4, CTRF results for retried tests include the `retries` and `retryAttempts` fields. When a test passes after an earlier failed attempt, its result also includes `flaky: true`. The terminal summary identifies flaky and retried tests. TRX and JUnit reports keep one final result per test instead of recording every attempt. + ## Azure DevOps reports Azure DevOps report plugin enhances test running for developers that host their code on GitHub, but build on Azure DevOps build agents. It adds additional information to failures to show failure directly in GitHub PR. diff --git a/docs/core/testing/mstest-analyzers/mstest0024.md b/docs/core/testing/mstest-analyzers/mstest0024.md index 564962ea42a0b..47165902c0f8f 100644 --- a/docs/core/testing/mstest-analyzers/mstest0024.md +++ b/docs/core/testing/mstest-analyzers/mstest0024.md @@ -1,7 +1,7 @@ --- title: "MSTEST0024: Do not store TestContext in a static member" description: "Learn about code analysis rule MSTEST0024: Do not store TestContext in a static member" -ms.date: 03/19/2024 +ms.date: 08/06/2026 f1_keywords: - MSTEST0024 - DoNotStoreStaticTestContextAnalyzer @@ -10,6 +10,7 @@ helpviewer_keywords: - MSTEST0024 author: Evangelink ms.author: amauryleve +ai-usage: ai-assisted --- # MSTEST0024: Do not store TestContext in a static member @@ -28,9 +29,11 @@ ms.author: amauryleve This rule raises a diagnostic when an assignment to a `static` member of a `TestContext` parameter is done. +Starting with MSTest 4.4, the rule also detects coalescing assignments, such as `s_testContext ??= testContext`, and deconstruction assignments that store `TestContext` in a static member. + ## Rule description -The `TestContext` parameter passed to each initialize method (`[AssemblyInitialize]` or `[ClassInitialize]`) is specific to the current context and is not updated on each test execution. Storing, for reuse, this `TextContext` object will most of the time lead to issues. +The `TestContext` parameter passed to each initialize method (`[AssemblyInitialize]` or `[ClassInitialize]`) is specific to the current context and is not updated on each test execution. Storing, for reuse, this `TestContext` object will most of the time lead to issues. ## How to fix violations diff --git a/docs/core/testing/mstest-analyzers/mstest0041.md b/docs/core/testing/mstest-analyzers/mstest0041.md index 5ae95618e0786..2d028ba16ca56 100644 --- a/docs/core/testing/mstest-analyzers/mstest0041.md +++ b/docs/core/testing/mstest-analyzers/mstest0041.md @@ -1,7 +1,7 @@ --- title: "MSTEST0041: Use 'ConditionBaseAttribute' on test classes" description: "Learn about code analysis rule MSTEST0041: Use 'ConditionBaseAttribute' on test classes" -ms.date: 02/13/2025 +ms.date: 08/06/2026 f1_keywords: - MSTEST0041 - UseConditionBaseWithTestClassAnalyzer @@ -10,6 +10,7 @@ helpviewer_keywords: - MSTEST0041 author: Youssef1313 ms.author: ygerges +ai-usage: ai-assisted --- # MSTEST0041: Use 'ConditionBaseAttribute' on test classes @@ -22,7 +23,7 @@ ms.author: ygerges | **Enabled by default** | Yes | | **Default severity** | Warning | | **Introduced in version** | 3.8.0 | -| **Is there a code fix** | No | +| **Is there a code fix** | Yes, starting with MSTest 4.4 | ## Cause @@ -36,6 +37,8 @@ An attribute that derives from or remove the attribute that derives from . +Starting with MSTest 4.4, the code fix adds `[TestClass]` to the affected type. + ## When to suppress warnings Do not suppress a warning from this rule. diff --git a/docs/core/testing/mstest-analyzers/mstest0050.md b/docs/core/testing/mstest-analyzers/mstest0050.md index 39af338f07f21..48473bd44751e 100644 --- a/docs/core/testing/mstest-analyzers/mstest0050.md +++ b/docs/core/testing/mstest-analyzers/mstest0050.md @@ -1,7 +1,7 @@ --- title: "MSTEST0050: Global test fixture should be valid" description: "Learn about code analysis rule MSTEST0050: Global test fixture should be valid" -ms.date: 07/29/2025 +ms.date: 08/06/2026 f1_keywords: - MSTEST0050 - GlobalTestFixtureShouldBeValidAnalyzer @@ -10,6 +10,7 @@ helpviewer_keywords: - MSTEST0050 author: Evangelink ms.author: amauryleve +ai-usage: ai-assisted --- # MSTEST0050: Global test fixture should be valid @@ -20,7 +21,7 @@ ms.author: amauryleve | **Category** | Usage | | **Fix is breaking or non-breaking** | Non-breaking | | **Enabled by default** | Yes | -| **Default severity** | Error | +| **Default severity** | Warning in MSTest 4.3 and later; error in earlier versions | | **Introduced in version** | 3.10.0 | | **Is there a code fix** | No | diff --git a/docs/core/testing/mstest-analyzers/mstest0065.md b/docs/core/testing/mstest-analyzers/mstest0065.md index b2c41bbe2375b..e8805426f24d5 100644 --- a/docs/core/testing/mstest-analyzers/mstest0065.md +++ b/docs/core/testing/mstest-analyzers/mstest0065.md @@ -1,7 +1,7 @@ --- title: "MSTEST0065: Avoid Assert.AreEqual on collection types" description: "Learn about code analysis rule MSTEST0065: Avoid Assert.AreEqual on collection types" -ms.date: 06/04/2026 +ms.date: 08/06/2026 f1_keywords: - MSTEST0065 - AvoidAssertAreEqualOnCollectionsAnalyzer @@ -29,6 +29,8 @@ dev_langs: > [!NOTE] > This rule is available starting with MSTest 4.3. +> +> Starting with MSTest 4.3.3, the rule doesn't report collection types that declare their own equality behavior. ## Cause diff --git a/docs/core/testing/mstest-analyzers/mstest0072.md b/docs/core/testing/mstest-analyzers/mstest0072.md new file mode 100644 index 0000000000000..970bd27a3f132 --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0072.md @@ -0,0 +1,85 @@ +--- +title: "MSTEST0072: '[AssemblyFixtureProvider]' isn't supported with ahead-of-time compilation" +description: "Learn about code analysis rule MSTEST0072: '[AssemblyFixtureProvider]' isn't supported with ahead-of-time compilation" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0072 +- AssemblyFixtureProviderNotSupportedWithNativeAotAnalyzer +helpviewer_keywords: +- AssemblyFixtureProviderNotSupportedWithNativeAotAnalyzer +- MSTEST0072 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0072: '\[AssemblyFixtureProvider]' isn't supported with ahead-of-time compilation + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0072 | +| **Title** | '\[AssemblyFixtureProvider]' isn't supported with ahead-of-time compilation | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Warning | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | No | + +> [!IMPORTANT] +> This analyzer is planned for MSTest 4.4 and is available only in preview builds until MSTest 4.4.0 is released. + +## Cause + +A project applies `[AssemblyFixtureProvider]`, either directly or through a referenced library, while publishing with Native AOT (`PublishAot`) or Blazor WebAssembly AOT (`RunAOTCompilation`). + +## Rule description + +`[AssemblyFixtureProvider]` discovery walks the runtime assembly reference graph, which requires the runtime to generate dynamic code. Ahead-of-time compilation flavors such as Native AOT and Blazor WebAssembly AOT can't generate dynamic code, so the runtime silently skips discovery, and the fixture's `[AssemblyInitialize]`/`[AssemblyCleanup]` methods never run. + +```csharp +// Violation: ignored at run time under Native AOT / Blazor WebAssembly AOT. +[assembly: AssemblyFixtureProvider(typeof(SharedFixtures))] +``` + +The analyzer reports this diagnostic whether the attribute is applied in the current compilation or on a referenced assembly, because either way the consuming Native AOT test project silently loses its assembly fixtures. + +## How to fix violations + +Declare the `[AssemblyInitialize]` and `[AssemblyCleanup]` methods directly in the test assembly instead of relying on a shared `[AssemblyFixtureProvider]` library. + +```csharp +public static class Fixtures +{ + [AssemblyInitialize] + public static void Init(TestContext context) { } +} +``` + +## When to suppress warnings + +Don't suppress warnings from this rule for a project that actually publishes with Native AOT or Blazor WebAssembly AOT, because the fixture methods won't run and any state they set up won't exist for your tests. Suppressing is reasonable only if the AOT-published configuration doesn't run the affected tests. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0072 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0072 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0072.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [Shared assembly fixtures with AssemblyFixtureProvider](../unit-testing-mstest-writing-tests-lifecycle.md#shared-assembly-fixtures-with-assemblyfixtureprovider) diff --git a/docs/core/testing/mstest-analyzers/mstest0073.md b/docs/core/testing/mstest-analyzers/mstest0073.md new file mode 100644 index 0000000000000..ef7fad3234102 --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0073.md @@ -0,0 +1,84 @@ +--- +title: "MSTEST0073: Prefer a constant for the '[ResourceLock]' resource key" +description: "Learn about code analysis rule MSTEST0073: Prefer a constant for the '[ResourceLock]' resource key" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0073 +- PreferConstantForResourceLockAnalyzer +helpviewer_keywords: +- PreferConstantForResourceLockAnalyzer +- MSTEST0073 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0073: Prefer a constant for the '\[ResourceLock]' resource key + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0073 | +| **Title** | Prefer a constant for the '\[ResourceLock]' resource key | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Info | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | No | + +> [!IMPORTANT] +> `ResourceLockAttribute` is planned for MSTest 4.4 and is available only in preview builds until MSTest 4.4.0 is released. + +## Cause + +A `[ResourceLock]` attribute passes its resource key as a bare string literal instead of referencing a shared constant. + +## Rule description + +`[ResourceLock]` matches tests by exact, case-sensitive string equality of the resource key. A bare string literal fails open: a typo produces a different key, so the conflicting tests are no longer serialized and race silently instead of failing with a build error. Referencing a shared constant, such as a `WellKnownResources` member or your own `const`, makes typos a compile error and lets the compiler enforce that every test contending on the same resource uses the same key. + +```csharp +[ResourceLock("database")] // Violation: bare string literal. +[TestMethod] +public void ReadsSharedSchema() { } +``` + +## How to fix violations + +Reference a `WellKnownResources` member for process-global state, or declare and reference your own `const string`. + +```csharp +private const string Database = "database"; + +[ResourceLock(Database)] +[TestMethod] +public void ReadsSharedSchema() { } +``` + +## When to suppress warnings + +It's safe to suppress this warning if you intentionally use a literal resource key and are confident no other test in the assembly needs to coordinate on the same resource. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0073 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0073 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0073.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [ResourceLockAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#resourcelockattribute) diff --git a/docs/core/testing/mstest-analyzers/mstest0074.md b/docs/core/testing/mstest-analyzers/mstest0074.md new file mode 100644 index 0000000000000..44c0ce3b3dbfb --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0074.md @@ -0,0 +1,93 @@ +--- +title: "MSTEST0074: Test mutating process-global state should declare a resource lock" +description: "Learn about code analysis rule MSTEST0074: Test mutating process-global state should declare a resource lock" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0074 +- UndeclaredProcessGlobalStateMutationAnalyzer +helpviewer_keywords: +- UndeclaredProcessGlobalStateMutationAnalyzer +- MSTEST0074 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0074: Test mutating process-global state should declare a resource lock + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0074 | +| **Title** | Test mutating process-global state should declare a resource lock | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Info | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | Yes, for C# only | + +> [!IMPORTANT] +> `ResourceLockAttribute` is planned for MSTest 4.4 and is available only in preview builds until MSTest 4.4.0 is released. + +> [!NOTE] +> This analyzer activates only when assembly parallelization is syntactically enabled, for example through `[assembly: Parallelize]` without a matching `[assembly: DoNotParallelize]`, or when a `.editorconfig` file sets `mstest_parallel_safety_mode = always`. The analyzer can't detect parallelization that's enabled only through `.runsettings` or MSBuild properties such as `MSTestParallelizeWorkers`. Set the `.editorconfig` option if you configure parallelization that way and still want this analyzer to run. + +## Cause + +A test, or a class-scoped fixture method it runs under, calls `Environment.SetEnvironmentVariable` or `Console.SetOut`/`SetError`/`SetIn` without a matching `[ResourceLock]` or `[DoNotParallelize]`. + +## Rule description + +Mutating process-global state, such as environment variables or console redirection, from a test is unsafe once in-assembly parallelization is enabled, because a sibling test running concurrently observes the mutation. Unlike `[ResourceLock]`, which fails open when a key is forgotten, this rule flags the mutation at compile time. + +```csharp +[TestMethod] +public void SetsVariable() +{ + Environment.SetEnvironmentVariable("MODE", "test"); // Violation +} +``` + +## How to fix violations + +Declare `[ResourceLock]` with the matching `WellKnownResources` key to serialize contending tests, or add `[DoNotParallelize]` to opt the test out of parallelization. + +```csharp +[ResourceLock(WellKnownResources.EnvironmentVariables)] +[TestMethod] +public void SetsVariable() +{ + Environment.SetEnvironmentVariable("MODE", "test"); +} +``` + +A C# code fix adds the `[ResourceLock]` attribute for you, at the test method or, for a class-scoped fixture such as `[TestInitialize]`, at the test class. Visual Basic code has this diagnostic but doesn't have an automatic fix; add the attribute by hand. + +## When to suppress warnings + +Don't suppress warnings from this rule without declaring a lock or opting out of parallelization, because doing so leaves the mutation racing silently against concurrently running tests. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0074 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0074 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0074.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [ResourceLockAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#resourcelockattribute) +- [DoNotParallelizeAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#donotparallelizeattribute) diff --git a/docs/core/testing/mstest-analyzers/mstest0075.md b/docs/core/testing/mstest-analyzers/mstest0075.md new file mode 100644 index 0000000000000..67b6ea96b8c3f --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0075.md @@ -0,0 +1,93 @@ +--- +title: "MSTEST0075: Avoid changing the current directory in a parallelized test" +description: "Learn about code analysis rule MSTEST0075: Avoid changing the current directory in a parallelized test" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0075 +- CurrentDirectoryMutationUnderParallelizationAnalyzer +helpviewer_keywords: +- CurrentDirectoryMutationUnderParallelizationAnalyzer +- MSTEST0075 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0075: Avoid changing the current directory in a parallelized test + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0075 | +| **Title** | Avoid changing the current directory in a parallelized test | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Info | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | Yes, for C# only | + +> [!IMPORTANT] +> `ResourceLockAttribute` is planned for MSTest 4.4 and is available only in preview builds until MSTest 4.4.0 is released. + +> [!NOTE] +> This analyzer activates only when assembly parallelization is syntactically enabled, for example through `[assembly: Parallelize]` without a matching `[assembly: DoNotParallelize]`, or when a `.editorconfig` file sets `mstest_parallel_safety_mode = always`. The analyzer can't detect parallelization that's enabled only through `.runsettings` or MSBuild properties such as `MSTestParallelizeWorkers`. Set the `.editorconfig` option if you configure parallelization that way and still want this analyzer to run. + +## Cause + +A test, or a class-scoped fixture method it runs under, calls `Directory.SetCurrentDirectory` or assigns `Environment.CurrentDirectory` without a matching `[ResourceLock]` or `[DoNotParallelize]`. + +## Rule description + +The current directory is process-global on every operating system and, unlike culture, has no thread-scoped or async-scoped equivalent. A test that changes it corrupts the working directory of every test running concurrently, and relative paths resolved elsewhere in the suite silently point at the wrong location. + +```csharp +[TestMethod] +public void ChangesDirectory() +{ + Directory.SetCurrentDirectory(tempPath); // Violation +} +``` + +## How to fix violations + +Declare `[ResourceLock(WellKnownResources.CurrentDirectory)]` to serialize contending tests, or add `[DoNotParallelize]` to opt the test out of parallelization. Even with a lock, prefer passing absolute paths so the current directory never needs to change. + +```csharp +[ResourceLock(WellKnownResources.CurrentDirectory)] +[TestMethod] +public void ChangesDirectory() +{ + Directory.SetCurrentDirectory(tempPath); +} +``` + +A C# code fix adds the `[ResourceLock]` attribute for you, at the test method or, for a class-scoped fixture such as `[TestInitialize]`, at the test class. Visual Basic code has this diagnostic but doesn't have an automatic fix; add the attribute by hand. + +## When to suppress warnings + +Don't suppress warnings from this rule without declaring a lock or opting out of parallelization, because doing so leaves the mutation racing silently against concurrently running tests. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0075 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0075 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0075.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [ResourceLockAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#resourcelockattribute) +- [DoNotParallelizeAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#donotparallelizeattribute) diff --git a/docs/core/testing/mstest-analyzers/mstest0076.md b/docs/core/testing/mstest-analyzers/mstest0076.md new file mode 100644 index 0000000000000..c7e9d871ddc5d --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0076.md @@ -0,0 +1,91 @@ +--- +title: "MSTEST0076: Avoid mutating process-wide culture in a parallelized test" +description: "Learn about code analysis rule MSTEST0076: Avoid mutating process-wide culture in a parallelized test" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0076 +- CultureMutationUnderParallelizationAnalyzer +helpviewer_keywords: +- CultureMutationUnderParallelizationAnalyzer +- MSTEST0076 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0076: Avoid mutating process-wide culture in a parallelized test + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0076 | +| **Title** | Avoid mutating process-wide culture in a parallelized test | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Info | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | No | + +> [!NOTE] +> This analyzer activates only when assembly parallelization is syntactically enabled, for example through `[assembly: Parallelize]` without a matching `[assembly: DoNotParallelize]`, or when a `.editorconfig` file sets `mstest_parallel_safety_mode = always`. The analyzer can't detect parallelization that's enabled only through `.runsettings` or MSBuild properties such as `MSTestParallelizeWorkers`. Set the `.editorconfig` option if you configure parallelization that way and still want this analyzer to run. + +## Cause + +A test, or a class-scoped fixture method it runs under, assigns or `DefaultThreadCurrentUICulture` without a matching `[ResourceLock]` or `[DoNotParallelize]`. + +## Rule description + +`CultureInfo.DefaultThreadCurrentCulture` and `DefaultThreadCurrentUICulture` set the default culture for the whole process, so every concurrently running test observes the change, and formatting or parsing can be corrupted. The per-thread and ambient forms, `Thread.CurrentThread.CurrentCulture`/`CurrentUICulture` and `CultureInfo.CurrentCulture`/`CurrentUICulture`, aren't flagged: on modern .NET they assign an `AsyncLocal`-backed value that flows with the execution context, so they don't corrupt sibling tests. + +```csharp +[TestMethod] +public void SetsCulture() +{ + CultureInfo.DefaultThreadCurrentCulture = CultureInfo.InvariantCulture; // Violation +} +``` + +Because the flagged setters are process-wide, restoring the previous value in a `finally` block doesn't help: concurrently running tests still observe the changed culture for the duration of the mutation. + +## How to fix violations + +Add `[DoNotParallelize]` on the test, or avoid mutating process-wide culture and use the per-thread or ambient culture properties instead. + +```csharp +[DoNotParallelize] +[TestMethod] +public void SetsCulture() +{ + CultureInfo.DefaultThreadCurrentCulture = CultureInfo.InvariantCulture; +} +``` + +No well-known `[ResourceLock]` key exists for culture, so a declared `[ResourceLock]` of any kind is treated as an acknowledgment that you've coordinated culture access, and this rule stays silent. + +## When to suppress warnings + +Don't suppress warnings from this rule without opting out of parallelization or switching to the per-thread culture properties, because doing so leaves the mutation racing silently against concurrently running tests. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0076 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0076 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0076.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [DoNotParallelizeAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#donotparallelizeattribute) diff --git a/docs/core/testing/mstest-analyzers/mstest0077.md b/docs/core/testing/mstest-analyzers/mstest0077.md new file mode 100644 index 0000000000000..3bea5dd82e952 --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0077.md @@ -0,0 +1,93 @@ +--- +title: "MSTEST0077: Avoid hardcoded or shared filesystem paths in a parallelized test" +description: "Learn about code analysis rule MSTEST0077: Avoid hardcoded or shared filesystem paths in a parallelized test" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0077 +- SharedFileSystemPathInTestAnalyzer +helpviewer_keywords: +- SharedFileSystemPathInTestAnalyzer +- MSTEST0077 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0077: Avoid hardcoded or shared filesystem paths in a parallelized test + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0077 | +| **Title** | Avoid hardcoded or shared filesystem paths in a parallelized test | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Info | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | No | + +> [!IMPORTANT] +> `TestContext.TestTempDirectory` is planned for MSTest 4.4 and is available only in preview builds until MSTest 4.4.0 is released. + +> [!NOTE] +> This analyzer activates only when assembly parallelization is syntactically enabled, for example through `[assembly: Parallelize]` without a matching `[assembly: DoNotParallelize]`, or when a `.editorconfig` file sets `mstest_parallel_safety_mode = always`. The analyzer can't detect parallelization that's enabled only through `.runsettings` or MSBuild properties such as `MSTestParallelizeWorkers`. Set the `.editorconfig` option if you configure parallelization that way and still want this analyzer to run. + +## Cause + +A test passes a constant absolute path, or a relative path literal, directly to a filesystem-mutating `File.*`/`Directory.*` method, such as `File.WriteAllText` or `Directory.CreateDirectory`. + +## Rule description + +A hardcoded or relative constant path targets a location shared by every other test in the assembly. Under in-assembly parallelization, two tests can then write to the same location concurrently and collide. + +```csharp +[TestMethod] +public void WritesReport() +{ + File.WriteAllText("report.txt", contents); // Violation +} +``` + +Only statically constant paths passed to a mutating API are flagged. Reads, path construction, and paths built from variables are left for you to review manually, because the analyzer can't tell whether two tests actually collide on a computed path. + +## How to fix violations + +Use a unique per-test location, such as `TestContext.TestTempDirectory`, instead of a fixed or relative path. + +```csharp +[TestMethod] +public void WritesReport() +{ + string path = Path.Combine(TestContext.TestTempDirectory!, "report.txt"); + File.WriteAllText(path, contents); +} +``` + +## When to suppress warnings + +It's safe to suppress this warning if the path intentionally targets a fixture that every test reads but no test writes concurrently, or if the tests that write to it are already coordinated, for example through `[DoNotParallelize]`. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0077 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0077 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0077.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [Per-test temporary directory](../unit-testing-mstest-writing-tests-testcontext.md#per-test-temporary-directory) +- [DoNotParallelizeAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#donotparallelizeattribute) diff --git a/docs/core/testing/mstest-analyzers/mstest0078.md b/docs/core/testing/mstest-analyzers/mstest0078.md new file mode 100644 index 0000000000000..a88480dabce4c --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0078.md @@ -0,0 +1,95 @@ +--- +title: "MSTEST0078: '[DependsOn]' arguments should be valid" +description: "Learn about code analysis rule MSTEST0078: '[DependsOn]' arguments should be valid" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0078 +- DependsOnShouldBeValidAnalyzer +helpviewer_keywords: +- DependsOnShouldBeValidAnalyzer +- MSTEST0078 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0078: '\[DependsOn]' arguments should be valid + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0078 | +| **Title** | '\[DependsOn]' arguments should be valid | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Warning | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | No | + +> [!IMPORTANT] +> Test dependencies are planned for MSTest 4.4 and are available only in preview builds until MSTest 4.4.0 is released. + +## Cause + +A `[DependsOn]` attribute references a test by name, and the reference has a problem that the test framework can decide only at run time, such as a target that doesn't exist. + +## Rule description + +The test framework deliberately treats a `[DependsOn]` target that matches no test as a non-fatal warning at run time, so that `--filter` and single-test runs keep working. That means a typo or a rename silently drops the declared ordering instead of failing the build. This analyzer reports the problems that can be decided at build time: + +- The referenced method doesn't exist on the referenced type. +- The referenced member isn't a test method, so the dependency is ignored. +- The referenced type isn't a test class, so the dependency is ignored. +- The referenced type is declared in another assembly (dependencies are resolved within a single test source). +- The referenced type is abstract, so its tests run under each derived test class instead. +- The attribute makes a test depend on itself, which is a dependency cycle that fails at run time. +- The attribute participates in a dependency cycle that's visible in the compilation, which fails every test in the cycle at run time. +- The attribute is applied where it has no effect, because the attribute target isn't a test method or runs no test. + +```csharp +[TestMethod] +public void CreateCart() { } + +[TestMethod, DependsOn("CreatCart")] // Violation: typo, no such member +public void AddItem() { } +``` + +## How to fix violations + +Fix the typo or rename, and use `nameof` so the compiler keeps the reference in sync when you rename the target. + +```csharp +[TestMethod] +public void CreateCart() { } + +[TestMethod, DependsOn(nameof(CreateCart))] +public void AddItem() { } +``` + +## When to suppress warnings + +Don't suppress warnings from this rule. Each case reported by this analyzer either fails the dependent test at run time, such as a self-reference or a cycle, or silently drops the declared ordering, such as a typo or a reference to a non-test member. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0078 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0078 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0078.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [Test dependencies](../unit-testing-mstest-writing-tests-controlling-execution.md#test-dependencies) diff --git a/docs/core/testing/mstest-analyzers/mstest0079.md b/docs/core/testing/mstest-analyzers/mstest0079.md new file mode 100644 index 0000000000000..a42caed42add9 --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0079.md @@ -0,0 +1,86 @@ +--- +title: "MSTEST0079: Use ArchitectureCondition attribute instead of runtime checks" +description: "Learn about code analysis rule MSTEST0079: Use ArchitectureCondition attribute instead of runtime checks" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0079 +- UseArchitectureConditionAttributeInsteadOfRuntimeCheckAnalyzer +helpviewer_keywords: +- UseArchitectureConditionAttributeInsteadOfRuntimeCheckAnalyzer +- MSTEST0079 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0079: Use ArchitectureCondition attribute instead of runtime checks + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0079 | +| **Title** | Use ArchitectureCondition attribute instead of runtime checks | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Info | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | Yes, for C# only | + +## Cause + +A test method's first statement compares to an value and then either returns early or calls `Assert.Inconclusive`, instead of using the `[ArchitectureCondition]` attribute. + +## Rule description + +Test methods that compare `RuntimeInformation.ProcessArchitecture` and then early return or call `Assert.Inconclusive` should use the `[ArchitectureCondition]` attribute instead. The attribute is more declarative and discoverable, and it reports the test as skipped rather than passed, which an early return doesn't. + +```csharp +[TestMethod] +public void TestMethod() +{ + if (RuntimeInformation.ProcessArchitecture != Architecture.X64) return; // Violation +} +``` + +The analyzer only recognizes the guard when it's the method's first statement, with no `else` branch, and when the referenced `Architecture` member has a matching `TestArchitectures` flag. + +## How to fix violations + +Replace the runtime check with the `[ArchitectureCondition]` attribute. + +```csharp +[TestMethod] +[ArchitectureCondition(TestArchitectures.X64)] +public void TestMethod() { } +``` + +A C# code fix replaces the guard with the attribute for you. Visual Basic code has this diagnostic but doesn't have an automatic fix; apply the attribute by hand. + +## When to suppress warnings + +You might suppress this warning if your runtime check is more complex than a simple first-statement guard, or if you need conditional logic that `[ArchitectureCondition]` can't express. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0079 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0079 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0079.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [ArchitectureConditionAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#architectureconditionattribute) +- [MSTEST0061: Use OSCondition attribute instead of runtime checks](mstest0061.md) diff --git a/docs/core/testing/mstest-analyzers/mstest0080.md b/docs/core/testing/mstest-analyzers/mstest0080.md new file mode 100644 index 0000000000000..cde1095d08fa3 --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0080.md @@ -0,0 +1,85 @@ +--- +title: "MSTEST0080: Use CICondition attribute instead of environment checks" +description: "Learn about code analysis rule MSTEST0080: Use CICondition attribute instead of environment checks" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0080 +- UseCIConditionAttributeInsteadOfEnvironmentCheckAnalyzer +helpviewer_keywords: +- UseCIConditionAttributeInsteadOfEnvironmentCheckAnalyzer +- MSTEST0080 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0080: Use CICondition attribute instead of environment checks + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0080 | +| **Title** | Use CICondition attribute instead of environment checks | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Info | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | Yes, for C# only | + +## Cause + +A test method's first statement null-checks the result of `Environment.GetEnvironmentVariable("CI")` and then either returns early or calls `Assert.Inconclusive`, instead of using the `[CICondition]` attribute. + +## Rule description + +Test methods that null-check the `CI` environment variable and then early return or call `Assert.Inconclusive` should use the `[CICondition]` attribute instead. The attribute recognizes every continuous integration provider MSTest knows about, and it reports the test as skipped rather than passed, which an early return doesn't. + +```csharp +[TestMethod] +public void TestMethod() +{ + if (Environment.GetEnvironmentVariable("CI") is null) return; // Violation +} +``` + +The analyzer is deliberately limited to the general-use `CI` variable that every major provider sets. A guard on a provider-specific variable, such as `TF_BUILD`, means "skip on Azure Pipelines", while `[CICondition]` means "skip on any CI", so the analyzer doesn't suggest replacing a provider-specific check. + +## How to fix violations + +Replace the environment-variable check with the `[CICondition]` attribute. + +```csharp +[TestMethod] +[CICondition(ConditionMode.Include)] +public void TestMethod() { } +``` + +A C# code fix replaces the guard with the attribute for you. Visual Basic code has this diagnostic but doesn't have an automatic fix; apply the attribute by hand. + +## When to suppress warnings + +You might suppress this warning if your environment check is more complex than a simple first-statement null check, or if you need conditional logic that `[CICondition]` can't express. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0080 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0080 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0080.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [CIConditionAttribute](../unit-testing-mstest-writing-tests-controlling-execution.md#ciconditionattribute) diff --git a/docs/core/testing/mstest-analyzers/mstest0081.md b/docs/core/testing/mstest-analyzers/mstest0081.md new file mode 100644 index 0000000000000..b9e6c8db835ab --- /dev/null +++ b/docs/core/testing/mstest-analyzers/mstest0081.md @@ -0,0 +1,93 @@ +--- +title: "MSTEST0081: '[TestFilterProvider]' should reference a valid test filter type" +description: "Learn about code analysis rule MSTEST0081: '[TestFilterProvider]' should reference a valid test filter type" +ms.date: 08/06/2026 +f1_keywords: +- MSTEST0081 +- TestFilterProviderShouldBeValidAnalyzer +helpviewer_keywords: +- TestFilterProviderShouldBeValidAnalyzer +- MSTEST0081 +author: evangelink +ms.author: amauryleve +ai-usage: ai-assisted +dev_langs: +- CSharp +--- +# MSTEST0081: '\[TestFilterProvider]' should reference a valid test filter type + +| Property | Value | +|-------------------------------------|----------------------------------------------------| +| **Rule ID** | MSTEST0081 | +| **Title** | '\[TestFilterProvider]' should reference a valid test filter type | +| **Category** | Usage | +| **Fix is breaking or non-breaking** | Non-breaking | +| **Enabled by default** | Yes | +| **Default severity** | Warning | +| **Introduced in version** | 4.4.0 (preview) | +| **Is there a code fix** | No | + +> [!IMPORTANT] +> This analyzer is planned for MSTest 4.4 and is available only in preview builds until MSTest 4.4.0 is released. + +## Cause + +`[assembly: TestFilterProvider(typeof(MyFilter))]` references a type that doesn't satisfy the requirements the adapter enforces at run time, or the assembly registers more than one test filter provider. + +## Rule description + +`[assembly: TestFilterProvider(typeof(MyFilter))]` passes the filter type as a `Type`, so the compiler accepts any type at all. The adapter validates the type only when it materializes the filter, and then fails the whole run. This analyzer reports the same problems at build time, where they're cheap to fix. The referenced type must: + +- Be non-generic. +- Be instantiable, so it can't be abstract, static, an interface, or a byref-like type. +- Implement `ITestFilter`. +- Declare a public parameterless constructor (every struct already satisfies this). + +At most one test filter provider can be registered per test assembly, and passing an explicit `null` filter type also fails. When targeting .NET, the generic `[assembly: TestFilterProvider]` form enforces the interface and constructor requirements through generic constraints instead, so only the generic-type and "at most one provider" checks still apply to it. + +```csharp +[assembly: TestFilterProvider(typeof(MyFilter))] +public sealed class MyFilter : ITestFilter +{ + public MyFilter(string mode) { } // Violation: no public parameterless constructor + public TestFilterResult Filter(TestFilterContext context) => TestFilterResult.Run; +} +``` + +## How to fix violations + +Add a public parameterless constructor, or otherwise adjust the type so that it's non-generic, instantiable, implements `ITestFilter`, and has a public parameterless constructor. + +```csharp +public sealed class MyFilter : ITestFilter +{ + public TestFilterResult Filter(TestFilterContext context) => TestFilterResult.Run; +} +``` + +## When to suppress warnings + +Don't suppress warnings from this rule. A violation fails the whole test run at run time rather than only the filter itself. + +## Suppress a warning + +If you just want to suppress a single violation, add preprocessor directives to your source file to disable and then re-enable the rule. + +```csharp +#pragma warning disable MSTEST0081 +// The code that's violating the rule is on this line. +#pragma warning restore MSTEST0081 +``` + +To disable the rule for a file, folder, or project, set its severity to `none` in the [configuration file](../../../fundamentals/code-analysis/configuration-files.md). + +```ini +[*.{cs,vb}] +dotnet_diagnostic.MSTEST0081.severity = none +``` + +For more information, see [How to suppress code analysis warnings](../../../fundamentals/code-analysis/suppress-warnings.md). + +## See also + +- [Programmatic test filtering with ITestFilter](../unit-testing-mstest-sdk.md#programmatic-test-filtering-with-itestfilter) diff --git a/docs/core/testing/mstest-analyzers/overview.md b/docs/core/testing/mstest-analyzers/overview.md index 21b99490cb64e..ade647d04055b 100644 --- a/docs/core/testing/mstest-analyzers/overview.md +++ b/docs/core/testing/mstest-analyzers/overview.md @@ -3,7 +3,7 @@ title: MSTest code analysis description: Learn about the MSTest code analysis. author: evangelink ms.author: amauryleve -ms.date: 12/20/2023 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -115,6 +115,9 @@ Rules that help ensure your test classes and methods are properly structured and - [MSTEST0063](mstest0063.md) - Test class should have valid constructor - [MSTEST0069](mstest0069.md) - Inherited `[TestClass]` is ignored by the MSTest source generator - [MSTEST0071](mstest0071.md) - Test method should not specify a display name equal to its name +- [MSTEST0072](mstest0072.md) - `[AssemblyFixtureProvider]` isn't supported with ahead-of-time compilation +- [MSTEST0078](mstest0078.md) - `[DependsOn]` arguments should be valid +- [MSTEST0081](mstest0081.md) - `[TestFilterProvider]` should reference a valid test filter type Related documentation: [Write tests with MSTest](../unit-testing-mstest-writing-tests.md) @@ -207,6 +210,18 @@ Rules for properly using the TestContext object: Related documentation: [TestContext](../unit-testing-mstest-writing-tests-testcontext.md) +### Parallel test safety + +Rules that help parallel tests coordinate shared resources and avoid process-wide state: + +- [MSTEST0073](mstest0073.md) - Prefer a constant for the `[ResourceLock]` resource key +- [MSTEST0074](mstest0074.md) - Test mutating process-global state should declare a resource lock +- [MSTEST0075](mstest0075.md) - Avoid changing the current directory in a parallelized test +- [MSTEST0076](mstest0076.md) - Avoid mutating process-wide culture in a parallelized test +- [MSTEST0077](mstest0077.md) - Avoid hardcoded or shared filesystem paths in a parallelized test + +Related documentation: [Test execution and control](../unit-testing-mstest-writing-tests-controlling-execution.md), [TestContext](../unit-testing-mstest-writing-tests-testcontext.md) + ### Test configuration Rules for configuring test execution, parallelization, and other test settings: @@ -221,6 +236,11 @@ Rules for configuring test execution, parallelization, and other test settings: - [MSTEST0059](mstest0059.md) - Use Parallelize attribute correctly - [MSTEST0061](mstest0061.md) - Use OSCondition attribute instead of runtime check - [MSTEST0070](mstest0070.md) - `[MemberCondition]` arguments should be valid +- [MSTEST0072](mstest0072.md) - `[AssemblyFixtureProvider]` isn't supported with ahead-of-time compilation +- [MSTEST0078](mstest0078.md) - `[DependsOn]` arguments should be valid +- [MSTEST0079](mstest0079.md) - Use ArchitectureCondition attribute instead of runtime checks +- [MSTEST0080](mstest0080.md) - Use CICondition attribute instead of environment checks +- [MSTEST0081](mstest0081.md) - `[TestFilterProvider]` should reference a valid test filter type Related documentation: [Configure MSTest](../unit-testing-mstest-configure.md), [Running tests](../unit-testing-mstest-running-tests.md) @@ -299,6 +319,16 @@ Related documentation: [Configure MSTest](../unit-testing-mstest-configure.md), | [MSTEST0069](mstest0069.md) | Usage | Inherited `[TestClass]` is ignored by the MSTest source generator | Warning | | [MSTEST0070](mstest0070.md) | Usage | `[MemberCondition]` arguments should be valid | Warning | | [MSTEST0071](mstest0071.md) | Usage | Test method should not specify a display name equal to its name | Info | +| [MSTEST0072](mstest0072.md) | Usage | `[AssemblyFixtureProvider]` isn't supported with ahead-of-time compilation | Warning | +| [MSTEST0073](mstest0073.md) | Usage | Prefer a constant for the `[ResourceLock]` resource key | Info | +| [MSTEST0074](mstest0074.md) | Usage | Test mutating process-global state should declare a resource lock | Info | +| [MSTEST0075](mstest0075.md) | Usage | Avoid changing the current directory in a parallelized test | Info | +| [MSTEST0076](mstest0076.md) | Usage | Avoid mutating process-wide culture in a parallelized test | Info | +| [MSTEST0077](mstest0077.md) | Usage | Avoid hardcoded or shared filesystem paths in a parallelized test | Info | +| [MSTEST0078](mstest0078.md) | Usage | `[DependsOn]` arguments should be valid | Warning | +| [MSTEST0079](mstest0079.md) | Usage | Use ArchitectureCondition attribute instead of runtime checks | Info | +| [MSTEST0080](mstest0080.md) | Usage | Use CICondition attribute instead of environment checks | Info | +| [MSTEST0081](mstest0081.md) | Usage | `[TestFilterProvider]` should reference a valid test filter type | Warning | \* Escalated to Error in `Recommended` and `All` modes. diff --git a/docs/core/testing/mstest-analyzers/usage-rules.md b/docs/core/testing/mstest-analyzers/usage-rules.md index ad28a028a8ad6..04154cef03af1 100644 --- a/docs/core/testing/mstest-analyzers/usage-rules.md +++ b/docs/core/testing/mstest-analyzers/usage-rules.md @@ -3,7 +3,7 @@ title: MSTest Usage rules (code analysis) description: Learn about MSTest code analysis usage rules. author: evangelink ms.author: amauryleve -ms.date: 10/01/2025 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -40,7 +40,7 @@ Usage rules support proper usage of MSTest attributes, methods, and patterns. Th | [MSTEST0038](mstest0038.md) | Avoid Assert.AreSame with value types. | Info | Yes | | [MSTEST0039](mstest0039.md) | Use newer Assert.Throws methods. | Info | Yes | | [MSTEST0040](mstest0040.md) | Avoid using asserts in async void context. | Warning | No | -| [MSTEST0041](mstest0041.md) | Use condition-based attributes with test class. | Warning | No | +| [MSTEST0041](mstest0041.md) | Use condition-based attributes with test class. | Warning | Yes | | [MSTEST0042](mstest0042.md) | Duplicate DataRow. | Warning | No | | [MSTEST0043](mstest0043.md) | Use retry attribute on test method. | Warning → Error* | Yes | | [MSTEST0046](mstest0046.md) | Use Assert instead of StringAssert. | Info | Yes | @@ -67,6 +67,16 @@ Usage rules support proper usage of MSTest attributes, methods, and patterns. Th | [MSTEST0069](mstest0069.md) | Inherited `[TestClass]` is ignored by the MSTest source generator. | Warning | No | | [MSTEST0070](mstest0070.md) | `[MemberCondition]` arguments should be valid. | Warning | No | | [MSTEST0071](mstest0071.md) | Test method should not specify a display name equal to its name. | Info | Yes | +| [MSTEST0072](mstest0072.md) | `[AssemblyFixtureProvider]` isn't supported with ahead-of-time compilation. | Warning | No | +| [MSTEST0073](mstest0073.md) | Prefer a constant for the `[ResourceLock]` resource key. | Info | No | +| [MSTEST0074](mstest0074.md) | Test mutating process-global state should declare a resource lock. | Info | Yes | +| [MSTEST0075](mstest0075.md) | Avoid changing the current directory in a parallelized test. | Info | Yes | +| [MSTEST0076](mstest0076.md) | Avoid mutating process-wide culture in a parallelized test. | Info | No | +| [MSTEST0077](mstest0077.md) | Avoid hardcoded or shared filesystem paths in a parallelized test. | Info | No | +| [MSTEST0078](mstest0078.md) | `[DependsOn]` arguments should be valid. | Warning | No | +| [MSTEST0079](mstest0079.md) | Use ArchitectureCondition attribute instead of runtime checks. | Info | Yes | +| [MSTEST0080](mstest0080.md) | Use CICondition attribute instead of environment checks. | Info | Yes | +| [MSTEST0081](mstest0081.md) | `[TestFilterProvider]` should reference a valid test filter type. | Warning | No | \* Escalated to Error in `Recommended` and `All` modes. @@ -94,6 +104,7 @@ Validate initialization and cleanup methods: - **[MSTEST0013](mstest0013.md)**: AssemblyCleanup validation. - **[MSTEST0034](mstest0034.md)**: Set ClassCleanupBehavior.EndOfClass. - **[MSTEST0050](mstest0050.md)**: Global test fixture validation. +- **[MSTEST0072](mstest0072.md)**: Don't use shared assembly fixture providers with ahead-of-time compilation. ### Data-driven testing @@ -142,6 +153,16 @@ Rules for asynchronous test code: - **[MSTEST0040](mstest0040.md)**: Avoid asserts in async void methods. - **[MSTEST0067](mstest0067.md)**: Avoid `Thread.Sleep`, `Task.Wait`, and other synchronously blocking calls in test code (opt-in). +### Parallel test safety + +Rules for tests that run in parallel: + +- **[MSTEST0073](mstest0073.md)**: Use shared constants for resource lock keys. +- **[MSTEST0074](mstest0074.md)**: Lock mutations to process-global state. +- **[MSTEST0075](mstest0075.md)**: Lock current-directory changes. +- **[MSTEST0076](mstest0076.md)**: Avoid process-wide culture mutations. +- **[MSTEST0077](mstest0077.md)**: Avoid shared filesystem paths. + ### Test configuration - **[MSTEST0031](mstest0031.md)**: Use proper attributes (not System.ComponentModel.Description). @@ -156,6 +177,10 @@ Rules for asynchronous test code: - **[MSTEST0061](mstest0061.md)**: Use OSCondition attribute for platform checks. - **[MSTEST0070](mstest0070.md)**: `[MemberCondition]` arguments must reference valid members. - **[MSTEST0071](mstest0071.md)**: Don't set a display name equal to the test method name. +- **[MSTEST0078](mstest0078.md)**: Validate `[DependsOn]` targets and dependency graphs. +- **[MSTEST0079](mstest0079.md)**: Use `ArchitectureCondition` instead of runtime architecture checks. +- **[MSTEST0080](mstest0080.md)**: Use `CICondition` instead of environment checks. +- **[MSTEST0081](mstest0081.md)**: Validate test filter provider registrations. ## Related documentation diff --git a/docs/core/testing/unit-testing-mstest-configure.md b/docs/core/testing/unit-testing-mstest-configure.md index aa8ff713065ba..3e6cc21e5fa88 100644 --- a/docs/core/testing/unit-testing-mstest-configure.md +++ b/docs/core/testing/unit-testing-mstest-configure.md @@ -3,7 +3,7 @@ title: Configure MSTest description: Learn how to configure MSTest. author: Evangelink ms.author: amauryleve -ms.date: 06/19/2026 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -23,19 +23,21 @@ The following runsettings entries let you configure how MSTest behaves. | Configuration | Default | Values | |---------------|---------|--------| -|`AssemblyCleanupTimeout`|0|Specify globally the timeout to apply on each instance of assembly cleanup method. `[Timeout]` attribute specified on the assembly cleanup method overrides the global timeout .| -|`AssemblyInitializeTimeout`|0|Specify globally the timeout to apply on each instance of assembly initialize method. `[Timeout]` attribute specified on the assembly initialize method overrides the global timeout .| +|`AssemblyCleanupTimeout`|None|Specify globally the timeout to apply on each instance of assembly cleanup method. `[Timeout]` attribute specified on the assembly cleanup method overrides the global timeout.| +|`AssemblyInitializeTimeout`|None|Specify globally the timeout to apply on each instance of assembly initialize method. `[Timeout]` attribute specified on the assembly initialize method overrides the global timeout.| |`AssemblyResolution`|false|You can specify paths to extra assemblies when finding and running unit tests. For example, use these paths for dependency assemblies that aren't in the same directory as the test assembly. To specify a path, use a **Directory Path** element. Paths can include environment variables.

` `

This feature is only applied when using a .NET Framework target.| -|`CaptureTraceOutput`|true|Capture text messages coming from the `Console.Write*`, `Trace.Write*`, and `Debug.Write*` APIs that will be associated to the current running test.| +|`CaptureTraceOutput`|`Result`|Capture text from the `Console.Write*`, `Trace.Write*`, and `Debug.Write*` APIs and associate it with the current test. Starting with MSTest 4.4, use `None`, `Result`, or `Live`. `Live` also echoes `Console`, `Trace`, and `TestContext.Write*` output to the console while the test runs. The earlier Boolean values remain supported: `true` maps to `Result`, and `false` maps to `None`.| |`ClassCleanupLifecycle`|EndOfClass|If you want the class cleanup to occur at the end of assembly, set it to `EndOfAssembly`. (No longer supported starting from MSTest v4 as `EndOfClass` is the default and only [ClassCleanup]() behavior)| -|`ClassCleanupTimeout`|0|Specify globally the timeout to apply on each instance of class cleanup method. `[Timeout]` attribute specified on the class cleanup method overrides the global timeout.| -|`ClassInitializeTimeout`|0|Specify globally the timeout to apply on each instance of class initialize method. `[Timeout]` attribute specified on the class initialize method overrides the global timeout.| +|`ClassCleanupTimeout`|None|Specify globally the timeout to apply on each instance of class cleanup method. `[Timeout]` attribute specified on the class cleanup method overrides the global timeout.| +|`ClassInitializeTimeout`|None|Specify globally the timeout to apply on each instance of class initialize method. `[Timeout]` attribute specified on the class initialize method overrides the global timeout.| |`ConsiderFixturesAsSpecialTests`|false|To display `AssemblyInitialize`, `AssemblyCleanup`, `ClassInitialize`, `ClassCleanup` as individual entries in Visual Studio and Visual Studio Code `Test Explorer` and _.trx_ log, set this value to **true**| |`DeleteDeploymentDirectoryAfterTestRunIsComplete`|true|To retain the deployment directory after a test run, set this value to **false**.| |`DeploymentEnabled`|true|If you set the value to **false**, deployment items that you specify in your test method aren't copied to the deployment directory.| |`DeployTestSourceDependencies`|true|A value indicating whether the test source references are to be deployed.| |`EnableBaseClassTestMethodsFromOtherAssemblies`|true|A value indicating whether to enable discovery of test methods from base classes in a different assembly from the inheriting test class.| |`ForcedLegacyMode`|false|In older versions of Visual Studio, the MSTest adapter was optimized to make it faster and more scalable. Some behavior, such as the order in which tests are run, might not be exactly as it was in previous editions of Visual Studio. Set the value to **true** to use the older test adapter.

For example, you might use this setting if you have an *app.config* file specified for a unit test.

We recommend that you consider refactoring your tests to allow you to use the newer adapter.| +|`GlobalTestCleanupTimeout`|`TestCleanupTimeout`|Starting with MSTest 4.4, specify the timeout for each global test cleanup method. When you omit this entry, MSTest uses `TestCleanupTimeout`. A `[Timeout]` attribute on the method overrides both values.| +|`GlobalTestInitializeTimeout`|`TestInitializeTimeout`|Starting with MSTest 4.4, specify the timeout for each global test initialize method. When you omit this entry, MSTest uses `TestInitializeTimeout`. A `[Timeout]` attribute on the method overrides both values.| |`LaunchDebuggerOnTestFailure`|false|Starting with MSTest 4.2, when set to **true**, MSTest launches the debugger when a test fails.| |`MapInconclusiveToFailed`|false|If a test completes with an inconclusive status, it's mapped to the skipped status in **Test Explorer**. If you want inconclusive tests to be shown as failed, set the value to **true**.| |`MapNotRunnableToFailed`|true|A value indicating whether a not runnable result is mapped to failed test.| @@ -44,12 +46,14 @@ The following runsettings entries let you configure how MSTest behaves. |`RandomizeTestOrder`|false|Starting with MSTest 4.3, set this value to **true** to run tests in a random order, which helps surface hidden ordering dependencies between tests. This setting can't be combined with `OrderTestsByNameInClass`.| |`RandomTestOrderSeed`||Starting with MSTest 4.3, when `RandomizeTestOrder` is **true**, set an integer seed to make the random order reproducible across runs. When unset, a new seed is used for each run.| |`SettingsFile`||You can specify a test settings file to use with the MSTest adapter here. You can also specify a test settings file [from the settings menu](/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file#specify-a-run-settings-file-in-the-ide).

If you specify this value, you must also set the `ForcedLegacyMode` to **true**.

`true`| -|`TestCleanupTimeout`|0|Specify globally the timeout to apply on each instance of test cleanup method. `[Timeout]` attribute specified on the test cleanup method overrides the global timeout.| -|`TestInitializeTimeout`|0|Specify globally the timeout to apply on each instance of test initialize method. `[Timeout]` attribute specified on the test initialize method overrides the global timeout.| -|`TestTimeout`|0|Gets specified global test case timeout.| +|`TestCleanupTimeout`|None|Specify globally the timeout to apply on each instance of test cleanup method. `[Timeout]` attribute specified on the test cleanup method overrides the global timeout.| +|`TestInitializeTimeout`|None|Specify globally the timeout to apply on each instance of test initialize method. `[Timeout]` attribute specified on the test initialize method overrides the global timeout.| +|`TestTimeout`|None|Gets specified global test case timeout.| |`TreatClassAndAssemblyCleanupWarningsAsErrors`|false|To see your failures in class cleanups as errors, set this value to **true**.| |`TreatDiscoveryWarningsAsErrors`|false|To report test discovery warnings as errors, set this value to **true**.| +Timeout values must be positive integers in milliseconds. To run without a timeout, omit the entry instead of setting it to `0`. Global test fixture timeouts inherit the corresponding `TestInitializeTimeout` or `TestCleanupTimeout` value. + ### `TestRunParameter` element ```xml @@ -111,13 +115,14 @@ When running your tests with MSTest, you can use a `testconfig.json` file to con Starting with MSTest 3.7, you can also configure MSTest runs in the same configuration file. The following sections describe the settings that you can use in the `testconfig.json` file. +Starting with MSTest 4.3.3, .NET Framework runs also accept comments and trailing commas in `testconfig.json`. + ### MSTest element MSTest settings are grouped by functionality that are described in the sections that follow. | Entry | Default | Description | |-------|---------|-------------| -| orderTestsByNameInClass | false | If you want to run tests by test names both in Test Explorers and on the command line, set this value to **true**. | | enableBaseClassTestMethodsFromOtherAssemblies | true | A value indicating whether to enable discovery of test methods from base classes in a different assembly from the inheriting test class. | | classCleanupLifecycle | EndOfAssembly | If you want the class cleanup to occur at the end of the class, set it to **EndOfClass**. | @@ -171,7 +176,7 @@ All the settings in this section belong to the `output` element. | Entry | Default | Description | |-------|---------|-------------| -| captureTrace | true | Capture text messages coming from the `Console.Write*`, `Trace.Write*`, and `Debug.Write*` APIs that will be associated to the current running test. | +| captureTrace | `Result` | Capture `Console`, `Trace`, and `Debug` output and associate it with the current test. Starting with MSTest 4.4, use `None`, `Result`, or `Live`. `Live` also echoes output, including `TestContext.Write*` messages, while the test runs. The Boolean values remain supported: `true` maps to `Result`, and `false` maps to `None`. | Example: @@ -217,9 +222,11 @@ All the settings in this section belong to the `execution` element. |-------|---------|-------------| | considerEmptyDataSourceAsInconclusive | false | When set to `true`, an empty data source is considered as inconclusive. | | considerFixturesAsSpecialTests | false | To display `AssemblyInitialize`, `AssemblyCleanup`, `ClassInitialize`, `ClassCleanup` as individual entries in Visual Studio and Visual Studio Code `Test Explorer` and _.trx_ log, set this value to **true**. | +| dependencies | | Starting with MSTest 4.4, declare test dependency `chains` and `nodes`. This setting is available only with Microsoft.Testing.Platform. For more information, see [Test dependencies](unit-testing-mstest-writing-tests-controlling-execution.md#test-dependencies). | | mapInconclusiveToFailed | false | If a test completes with an inconclusive status, it's mapped to the skipped status in **Test Explorer**. If you want inconclusive tests to be shown as failed, set the value to **true**. | | launchDebuggerOnTestFailure | false | Starting with MSTest 4.2, when set to `true`, MSTest launches the debugger when a test fails. | | mapNotRunnableToFailed | true | A value indicating whether a not runnable result is mapped to failed test. | +| orderTestsByNameInClass | false | Run tests in alphabetical order within each class. Starting with MSTest 4.3, use `mstest.execution.orderTestsByNameInClass`. The earlier `mstest.orderTestsByNameInClass` key still works but produces a deprecation warning. | | randomizeTestOrder | false | Starting with MSTest 4.3, set this value to `true` to run tests in a random order, which helps surface hidden ordering dependencies between tests. This setting can't be combined with `orderTestsByNameInClass`. | | randomTestOrderSeed | | Starting with MSTest 4.3, when `randomizeTestOrder` is `true`, set an integer seed to make the random order reproducible across runs. When unset, a new seed is used for each run. | | treatClassAndAssemblyCleanupWarningsAsErrors | false | To see your failures in class cleanups as errors, set this value to **true**. | @@ -248,33 +255,26 @@ All the settings in this section belong to the `timeout` element. | Entry | Default | Description | |-------|---------|-------------| -| assemblyCleanup | 0 | Specify globally the timeout to apply on each instance of assembly cleanup method. | -| assemblyInitialize | 0 | Specify globally the timeout to apply on each instance of assembly initialize method. | -| classCleanup | 0 | Specify globally the timeout to apply on each instance of class cleanup method. | -| classInitialize | 0 | Specify globally the timeout to apply on each instance of class initialize method. | -| test | 0 | Specify globally the test timeout. | -| testCleanup | 0 | Specify globally the timeout to apply on each instance of test cleanup method. | -| testInitialize | 0 | Specify globally the timeout to apply on each instance of test initialize method. | +| assemblyCleanup | None | Specify globally the timeout to apply on each instance of assembly cleanup method. | +| assemblyInitialize | None | Specify globally the timeout to apply on each instance of assembly initialize method. | +| classCleanup | None | Specify globally the timeout to apply on each instance of class cleanup method. | +| classInitialize | None | Specify globally the timeout to apply on each instance of class initialize method. | +| globalTestCleanup | `testCleanup` | Starting with MSTest 4.4, specify the timeout for each global test cleanup method. When you omit this entry, MSTest uses `testCleanup`. | +| globalTestInitialize | `testInitialize` | Starting with MSTest 4.4, specify the timeout for each global test initialize method. When you omit this entry, MSTest uses `testInitialize`. | +| test | None | Specify globally the test timeout. | +| testCleanup | None | Specify globally the timeout to apply on each instance of test cleanup method. | +| testInitialize | None | Specify globally the timeout to apply on each instance of test initialize method. | | useCooperativeCancellation | false | When set to `true`, in case of timeout, MSTest will only trigger cancellation of the `CancellationToken` but will not stop observing the method. This behavior is more performant but relies on the user to correctly flow the token through all paths. | > [!NOTE] -> `[Timeout]` attribute specified on a method overrides the global timeout. For example, `[Timeout(1000)]` on a method marked with [AssemblyCleanup] will override the global `assemblyCleanup` timeout. +> Timeout values must be positive integers in milliseconds. To run without a timeout, omit the entry instead of setting it to `0`. Global test fixture timeouts inherit the corresponding `testInitialize` or `testCleanup` value, so omit both entries when you don't want a timeout on a global fixture. A `[Timeout]` attribute on a method overrides the configured timeout. Example: ```json { "mstest": { - "timeout": { - "assemblyCleanup": 0, - "assemblyInitialize": 0, - "classCleanup": 0, - "classInitialize": 0, - "test": 0, - "testCleanup": 0, - "testInitialize": 0, - "useCooperativeCancellation": false - } + "timeout": { "globalTestInitialize": 30000, "globalTestCleanup": 30000 } } } ``` @@ -316,6 +316,8 @@ Starting with MSTest 4.3, opt in to assembly-level parallelization from your pro | `MSTestParallelizeScope` | | The parallelization scope. Set it to `MethodLevel` or `ClassLevel` to emit `[assembly: Parallelize(Scope = ExecutionScope.MethodLevel)]` (or `ExecutionScope.ClassLevel`), or to `None` to emit `[assembly: DoNotParallelize]`. | | `MSTestParallelizeWorkers` | | The maximum number of worker threads, emitted as the `Workers` value of `[assembly: Parallelize]`. A value of `0` maps to the number of processors on the current machine. This property can't be set when `MSTestParallelizeScope` is `None`. | +MSTest validates both properties during the build. Invalid scope values, non-integer worker counts, and a worker count combined with a `None` scope fail the build. Don't also declare `[assembly: Parallelize]` or `[assembly: DoNotParallelize]` in source, because the generated attribute would duplicate it. When `GenerateAssemblyInfo` is `false`, declare the attribute in source instead. + The following example enables method-level parallelization with four workers for every test project that imports the `Directory.Build.props` file: ```xml diff --git a/docs/core/testing/unit-testing-mstest-intro.md b/docs/core/testing/unit-testing-mstest-intro.md index a76a53c0665b7..ddf88965e4890 100644 --- a/docs/core/testing/unit-testing-mstest-intro.md +++ b/docs/core/testing/unit-testing-mstest-intro.md @@ -3,7 +3,8 @@ title: MSTest overview description: Learn about MSTest, Microsoft's testing framework for .NET, including supported platforms, key features, and getting started. author: Evangelink ms.author: amauryleve -ms.date: 07/15/2025 +ms.date: 08/06/2026 +ai-usage: ai-assisted --- # MSTest overview @@ -34,6 +35,7 @@ MSTest supports a wide range of .NET platforms and target frameworks. The follow | **UWP** | UAP 10, .NET 9+ with UAP | UI thread | `UITestMethod` | Requires settings `true`; see [UWP sample](https://github.com/microsoft/testfx/tree/main/samples/public/BlankUwpNet9App) | | **WinUI 3** | .NET 8+ | UI thread | `UITestMethod` | Requires Windows App SDK; see [WinUI sample](https://github.com/microsoft/testfx/tree/main/samples/public/BlankWinUINet9App) | | **Native AOT** | .NET 8+ | Full parallelization | Most attributes | Limited feature set; see [Native AOT sample](https://github.com/microsoft/testfx/tree/main/samples/public/mstest-runner/NativeAotRunner) | +| **Browser WebAssembly** | .NET 10+ custom host | Single-threaded | Limited | Custom Microsoft.Testing.Platform host support starts with MSTest 4.4 | ### Platform-specific considerations @@ -77,10 +79,18 @@ public class WinUITests For WinUI setup, see the [BlankWinUINet9App sample](https://github.com/microsoft/testfx/tree/main/samples/public/BlankWinUINet9App) and [MSTestRunnerWinUI sample](https://github.com/microsoft/testfx/tree/main/samples/public/mstest-runner/MSTestRunnerWinUI). +Starting with MSTest 4.4, Microsoft.Testing.Platform also supports unpackaged WinUI test applications. VSTest doesn't support this scenario. For setup details, see the [unpackaged WinUI sample](https://github.com/microsoft/testfx/tree/main/samples/public/mstest-runner/MSTestRunnerWinUIUnpackaged). + #### Native AOT Native AOT compilation is supported with some limitations due to reduced reflection capabilities. Use source generators where possible and test your AOT scenarios with the [NativeAotRunner sample](https://github.com/microsoft/testfx/tree/main/samples/public/mstest-runner/NativeAotRunner). +#### Browser WebAssembly + +Starting with MSTest 4.4, a custom .NET 10 browser WebAssembly host can call `AddMSTest` to run tests from a referenced MSTest assembly. In the host project, set `EnableMSTestRunner` to `true` and `GenerateTestingPlatformEntryPoint` to `false` so the custom host owns the application entry point. Keep the MSTest and Microsoft.Testing.Platform package versions aligned. + +On a single-threaded WebAssembly runtime, MSTest can't interrupt timed-out tests, and debugger launch options aren't supported. For a complete host, see the [BrowserPlayground sample](https://github.com/microsoft/testfx/tree/main/samples/BrowserPlayground). + ### STA threading support For Windows COM interop scenarios, MSTest provides `STATestClass` and `STATestMethod` attributes to run tests in a single-threaded apartment. For details on STA threading, including async continuation support with `UseSTASynchronizationContext`, see [Threading attributes](unit-testing-mstest-writing-tests-controlling-execution.md#threading-attributes). @@ -109,6 +119,9 @@ MSTest has undergone significant evolution across major versions: - **MSTest v3**: Modern rewrite with improved architecture and features - **MSTest v4**: Current version with enhanced features +> [!NOTE] +> MSTest 4.4 is under development as of August 2026. Features marked as introduced in MSTest 4.4 require a preview build until version 4.4.0 is released. + For details on all releases, see the [MSTest changelog](https://github.com/microsoft/testfx/blob/main/docs/Changelog.md). If you're upgrading from an older version, see the migration guides: diff --git a/docs/core/testing/unit-testing-mstest-sdk.md b/docs/core/testing/unit-testing-mstest-sdk.md index 229513ad955d5..4de82c2381838 100644 --- a/docs/core/testing/unit-testing-mstest-sdk.md +++ b/docs/core/testing/unit-testing-mstest-sdk.md @@ -3,7 +3,7 @@ title: MSTest SDK configuration author: MarcoRossignoli description: Learn how to configure MSTest.Sdk profiles, extensions, and advanced features. ms.author: mrossignoli -ms.date: 02/13/2024 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -105,6 +105,8 @@ For example, to enable the crash dump extension (NuGet package [Microsoft.Testin For a list of all available extensions, see [MTP features](./microsoft-testing-platform-features.md). +Starting with MSTest.Sdk 4.3, enable the experimental JUnit report extension with `true`, then pass `--report-junit` when you run the test application. The extension is available only with Microsoft.Testing.Platform and isn't included in the `Default` or `AllMicrosoft` profiles. + > [!WARNING] > It's important to review the licensing terms for each extension as they might vary. @@ -241,6 +243,10 @@ The following MSTest 4.3 features are **experimental**. Their public APIs are su The MSTest reflection source generator discovers tests at compile time instead of relying on runtime reflection, which makes test projects compatible with trimming and Native AOT. Enable it by adding the [MSTest.SourceGeneration](https://www.nuget.org/packages/MSTest.SourceGeneration) package. When the source generator is active, test classes must declare `[TestClass]` directly rather than inherit it; the [MSTEST0069](mstest-analyzers/mstest0069.md) analyzer flags classes that rely on an inherited `[TestClass]`. +Starting with MSTest 4.3.2, `MSTestSourceGenMode` defaults to `ReflectionFree` for trimmed and Native AOT projects. + +Starting with MSTest 4.4, reflection-free generation materializes complete inherited attribute metadata, including `AttributeUsage` and `AllowMultiple`. When the generator can't materialize metadata statically, MSTest falls back to reflection where the runtime supports it. + ### Programmatic test filtering with `ITestFilter` > [!NOTE] @@ -248,6 +254,31 @@ The MSTest reflection source generator discovers tests at compile time instead o The experimental `ITestFilter` extension point, registered through `[TestFilterProviderAttribute]`, lets you decide programmatically whether each test runs, before any test class is loaded. This is useful for custom selection logic that can't be expressed with command-line filters. +Implement `ITestFilter.Filter(TestFilterContext)` to inspect metadata without loading the test class: + +```csharp +public sealed class MyFilter : ITestFilter +{ + public TestFilterResult Filter(TestFilterContext context) => + context.DisplayName.Contains("Nightly", StringComparison.Ordinal) + ? TestFilterResult.Run : TestFilterResult.Drop; +} +``` + +Return `TestFilterResult.Run` to run the test, `Drop` to omit it without a result, or `Skip(reason)` to report a skipped result. MSTest can call one filter instance concurrently, so implementations must be thread-safe. Command-line and test-explorer filters run before `ITestFilter`, while `[Ignore]` is evaluated afterward. + +Starting with MSTest 4.4, .NET projects can use the generic, type-safe registration form `[assembly: TestFilterProvider]`. The compiler then enforces that `MyFilter` implements `ITestFilter` and has a public parameterless constructor. The generic attribute isn't available for .NET Framework. For a multi-targeted project, select the generic or non-generic form with a target-framework preprocessor symbol. + +```csharp +#if NET +[assembly: TestFilterProvider] +#else +[assembly: TestFilterProvider(typeof(MyFilter))] +#endif +``` + +Starting with MSTest 4.4, the [MSTEST0081](mstest-analyzers/mstest0081.md) analyzer fully validates the non-generic registration form. For the generic form, it still reports generic filter types and assemblies that register more than one provider. + ### `TestRun.Current` and planned tests > [!NOTE] diff --git a/docs/core/testing/unit-testing-mstest-writing-tests-assertions.md b/docs/core/testing/unit-testing-mstest-writing-tests-assertions.md index 34b190e65b55d..035477b576cd7 100644 --- a/docs/core/testing/unit-testing-mstest-writing-tests-assertions.md +++ b/docs/core/testing/unit-testing-mstest-writing-tests-assertions.md @@ -3,7 +3,7 @@ title: MSTest assertions description: Learn about MSTest assertions including Assert, StringAssert, and CollectionAssert classes for validating test results. author: Evangelink ms.author: amauryleve -ms.date: 06/16/2026 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -144,10 +144,14 @@ When comparing collections, prefer these methods over `Assert.AreEqual`, which c MSTest 4.3 also adds: -- `Assert.AddValueFormatter` to customize how values are rendered in assertion failure messages. +- The experimental `Assert.AddValueFormatter` API to customize how values are rendered in assertion failure messages. - and overloads for `Assert.HasCount`. - Structured assertion failure messages for `Assert.IsTrue`, `Assert.IsFalse`, `Assert.IsNull`, and `Assert.IsNotNull` that include the evaluated expression. - Interpolated-string message overloads for the async `Assert.ThrowsAsync`/`Assert.ThrowsExactlyAsync` methods, and rejection of `ValueTask`-returning delegates that would otherwise not be awaited. +- Complete exception details, including stack traces and inner exceptions, in `Assert.Throws*` failure messages. +- Assertion failure stacks that hide MSTest implementation frames and render built-in numeric values at full precision. + +`Assert.AddValueFormatter` returns an registration. Dispose the registration to remove the formatter. The formatter applies only to the current asynchronous context, so parallel tests can use different formatters without changing each other's output. Because the API is experimental in MSTest 4.3, acknowledge or suppress the `MSTESTEXP` diagnostic before you use it. ### Soft assertions with `Assert.Scope()` diff --git a/docs/core/testing/unit-testing-mstest-writing-tests-controlling-execution.md b/docs/core/testing/unit-testing-mstest-writing-tests-controlling-execution.md index 0947a5ed5db3b..b695dfbf845e4 100644 --- a/docs/core/testing/unit-testing-mstest-writing-tests-controlling-execution.md +++ b/docs/core/testing/unit-testing-mstest-writing-tests-controlling-execution.md @@ -3,7 +3,7 @@ title: Test execution and control in MSTest description: Learn how to control test execution in MSTest with parallelization, threading, timeouts, retries, and conditional execution. author: Evangelink ms.author: amauryleve -ms.date: 06/19/2026 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -197,6 +197,74 @@ public class MixedTests > [!NOTE] > You only need `DoNotParallelize` when you've enabled parallel execution with the `Parallelize` attribute. +### `ResourceLockAttribute` + +> [!IMPORTANT] +> `ResourceLockAttribute` is planned for MSTest 4.4 and is available only in preview builds until MSTest 4.4.0 is released. + +Use `[ResourceLock]` to serialize only tests that access the same named resource. Unlike `[DoNotParallelize]`, a resource lock doesn't block tests that use unrelated resources. The default `ReadWrite` mode is exclusive, while multiple tests that request `ResourceAccessMode.Read` for the same resource can run together. + +```csharp +private const string Database = "integration-database"; + +[TestMethod] +[ResourceLock(Database, Mode = ResourceAccessMode.Read)] +public void ReadsSharedSchema() { } +``` + +For process-wide state, use the constants in `WellKnownResources`: `CurrentDirectory`, `EnvironmentVariables`, and `Console`. Lock names use ordinal, case-sensitive equality and coordinate tests only within one test source or assembly. They don't coordinate tests from separate assemblies, processes, or machines, even when those tests use the same key. + +Lock scope follows the configured parallelization scope: + +- With `ClassLevel`, MSTest combines every lock declared on the class and its methods, then holds the strongest mode for the entire class lifecycle. +- With `MethodLevel`, MSTest acquires locks for each test, including its test initialization and cleanup. +- When parallelization is disabled, resource locks have no effect. + +If a test also uses `[DoNotParallelize]`, `[DoNotParallelize]` takes precedence and MSTest ignores its resource locks. A test that waits for a contended lock occupies a worker, so heavy lock contention can still reduce throughput. + +> [!TIP] +> MSTest 4.4 adds parallel-safety analyzers [MSTEST0073](mstest-analyzers/mstest0073.md) through [MSTEST0077](mstest-analyzers/mstest0077.md) to help you declare stable lock keys and protect shared process state. + +## Test dependencies + +> [!IMPORTANT] +> Test dependencies are planned for MSTest 4.4 and are available only in preview builds until MSTest 4.4.0 is released. + +Use `[DependsOn]` for integration or end-to-end tests that must run after other tests. Dependencies form a directed acyclic graph, so independent branches can still run in parallel. + +```csharp +[TestMethod] +public void CreateCart() { } + +[TestMethod, DependsOn(nameof(CreateCart))] +public void PlaceOrder() { } +``` + +Apply multiple `[DependsOn]` attributes for fan-in, or apply the attribute to several tests for fan-out. You can reference a method in the same class, every test in another class, or one method in another class. Apply `[DependsOn]` to a test class to give every test in that class the dependency. When a dependency targets a data-driven test, MSTest waits for all its data rows. + +By default, a failed prerequisite skips its dependents, and the skip propagates. MSTest evaluates `ProceedOnFailure` per dependent test, not per edge. To run a dependent after failed prerequisites, set `ProceedOnFailure = true` on every `[DependsOn]` declaration for that test. One declaration left at the default causes MSTest to skip the dependent. MSTest reports dependency cycles before execution and fails the tests in the cycle. If a dependency isn't part of the selected run, MSTest warns and ignores the missing edge so filters and single-test runs still work. + +With Microsoft.Testing.Platform, you can also declare dependencies under `mstest.execution.dependencies` in `testconfig.json`: + +- Use `chains` for straight sequences. +- Use `nodes` for fan-in, fan-out, and `proceedOnFailure`. + +```json +{ + "mstest": { "execution": { "dependencies": { + "chains": [["Contoso.Setup.CreateDatabase", "Contoso.Tests.ImportData"]], + "nodes": [{ "test": "Contoso.Reports.*", "dependsOn": ["Contoso.Tests.ImportData"], "proceedOnFailure": true }] + }}} +} +``` + +Reference one test with its `Namespace.Class.Method` name, or reference every test in a class with `Namespace.Class.*`. MSTest merges configuration dependencies with dependencies declared through attributes. + +The `testconfig.json` form works only with Microsoft.Testing.Platform. The `[DependsOn]` attribute works with both Microsoft.Testing.Platform and VSTest. For build-time validation, enable [MSTEST0078](mstest-analyzers/mstest0078.md). + +> [!TIP] +> Prefer independent tests, fixtures, or per-test setup for unit tests. Dependencies make tests harder to run in isolation, so reserve them for suites where the sequence itself is part of the scenario. + ## Timeout attributes Timeout attributes prevent tests from running indefinitely and help identify performance issues. @@ -357,6 +425,8 @@ public class RetryTests > [!NOTE] > Starting with MSTest 4.3, `RetryAttribute` can also be applied at the test class level. When applied to a test class, it applies to every test method in the class. A `RetryAttribute` on a method takes precedence over one on the containing class. +> +> Starting with MSTest 4.4, Microsoft.Testing.Platform reports every retry attempt in terminal output and identifies flaky and retried tests in the run summary. CTRF reports include retry details. TRX and JUnit reports continue to contain one final result per test, and superseded attempts don't affect the process exit code. > [!TIP] > Related analyzers: @@ -367,6 +437,9 @@ public class RetryTests Starting with MSTest 3.8, create custom retry logic by inheriting from : +> [!IMPORTANT] +> The `RetryBaseAttribute.ExecuteAsync` API, and its `RetryContext` and `RetryResult` types, are experimental. Using them produces the `MSTESTEXP` diagnostic, which you must acknowledge before you use the API. + ```csharp public class CustomRetryAttribute : RetryBaseAttribute { @@ -463,19 +536,12 @@ The run public class CIAwareTests { [TestMethod] - [CICondition] // Default: runs only in CI + [CICondition(ConditionMode.Include)] public void CIOnlyTest() { // Runs only in CI environments } - [TestMethod] - [CICondition(ConditionMode.Include)] - public void ExplicitCIOnlyTest() - { - // Same as above, explicitly stated - } - [TestMethod] [CICondition(ConditionMode.Exclude)] public void LocalDevelopmentOnlyTest() diff --git a/docs/core/testing/unit-testing-mstest-writing-tests-lifecycle.md b/docs/core/testing/unit-testing-mstest-writing-tests-lifecycle.md index f45d7087980f9..42fb9785636e1 100644 --- a/docs/core/testing/unit-testing-mstest-writing-tests-lifecycle.md +++ b/docs/core/testing/unit-testing-mstest-writing-tests-lifecycle.md @@ -3,7 +3,7 @@ title: MSTest test lifecycle description: Learn about the creation and lifecycle of test classes and test methods in MSTest, including initialization and cleanup at assembly, class, and test levels. author: marcelwgn ms.author: marcelwagner -ms.date: 06/16/2026 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -105,6 +105,9 @@ public static class SharedAssemblyFixtures The attribute allows multiple providers per assembly, so you can compose fixtures from several shared types. +> [!WARNING] +> Starting with MSTest 4.4, MSTest skips `AssemblyFixtureProviderAttribute` discovery when dynamic code isn't supported, including Native AOT runs. Use fixture methods declared directly in the test assembly for Native AOT. The [MSTEST0072](mstest-analyzers/mstest0072.md) analyzer reports configurations that it can detect at build time. + ## Class-level lifecycle Class lifecycle methods run once per test class, before and after all test methods in that class. Use these for setup shared across tests in a class. @@ -230,7 +233,7 @@ public class GlobalTestLifecycleExample - Multiple methods with these attributes are allowed across the assembly > [!NOTE] -> When multiple `GlobalTestInitialize` or `GlobalTestCleanup` methods exist, the execution order isn't guaranteed. The `TimeoutAttribute` isn't supported on `GlobalTestInitialize` methods. +> When multiple `GlobalTestInitialize` or `GlobalTestCleanup` methods exist, the execution order isn't guaranteed. Starting with MSTest 4.4, global fixture methods support `[Timeout]` and the dedicated [global fixture timeout settings](unit-testing-mstest-configure.md#timeout-settings). > [!TIP] > Related analyzer: [MSTEST0050](mstest-analyzers/mstest0050.md) - validates global test fixture methods. diff --git a/docs/core/testing/unit-testing-mstest-writing-tests-testcontext.md b/docs/core/testing/unit-testing-mstest-writing-tests-testcontext.md index 297c723997a3e..0a2af6a216388 100644 --- a/docs/core/testing/unit-testing-mstest-writing-tests-testcontext.md +++ b/docs/core/testing/unit-testing-mstest-writing-tests-testcontext.md @@ -3,7 +3,7 @@ title: MSTest TestContext description: Learn about the TestContext class of MSTest. author: Evangelink ms.author: amauryleve -ms.date: 06/16/2026 +ms.date: 08/06/2026 ai-usage: ai-assisted --- @@ -44,6 +44,24 @@ The provides inf - - the directory where the test results are stored. Typically a subdirectory of the . - Starting with MSTest 3.9, - the number of times the current test has run, counting from 1. The value is greater than 1 when a test is retried with `[Retry]`. +### Per-test temporary directory + +> [!IMPORTANT] +> `TestContext.TestTempDirectory` is planned for MSTest 4.4 and is available only in preview builds until MSTest 4.4.0 is released. + +Use `TestContext.TestTempDirectory` as private scratch space for a test. MSTest creates the directory only when you first access the property, and each test execution receives a unique directory. Each data row also receives its own directory, so parallel tests don't share paths. + +```csharp +string path = Path.Combine(TestContext.TestTempDirectory!, "output.json"); +File.WriteAllText(path, json); +``` + +MSTest creates the directory under `TestResultsDirectory` when possible and falls back to the system temporary directory when the results path is unavailable, too long, or read-only. MSTest deletes the directory after a passing test and retains it after any non-passing outcome. Set the `MSTEST_TEST_TEMP_DIRECTORY_RETAIN` environment variable to `1` or `true` to retain directories for all outcomes. + +When a passing test registers a file from the directory with `AddResultFile`, MSTest retains the directory until the host collects the attachment. Cleanup is best effort and doesn't change the test outcome. + +`TestTempDirectory` is available for .NET and .NET Framework targets, but not for UWP or WinUI targets. The property doesn't change the process current directory. + In MSTest 3.7 and later, the class also provides new properties helpful for `TestInitialize` and `TestCleanup` methods: - - the data that will be provided to the parameterized test method, or `null` if the test is not parameterized. @@ -82,6 +100,8 @@ string value = TestContext.Properties["MyKey"]?.ToString(); > Starting with MSTest 4.2, test categories from `[TestCategory]` are included in . > > Starting with MSTest 4.3, custom properties added to `TestContext.Properties` in `[AssemblyInitialize]` flow to every class and test in the assembly, and properties added in `[ClassInitialize]` flow to every test in that class. This lets fixtures publish shared context that test methods can read. +> +> Starting with MSTest 4.3.3, `[TestProperty]` values, test categories, host-provided properties, and properties that a test adds remain scoped to that test and don't flow to sibling tests. ### Access `TestContext` from the current call stack @@ -93,7 +113,7 @@ The or methods to write custom messages directly to the test output. This is especially useful for debugging purposes, as it provides real-time logging information within your test execution context. +You can also use or methods to write custom messages directly to the test output. Starting with MSTest 4.4, the `Live` output capture mode echoes these messages while the test runs and still attaches them to the final test result. For more information, see [Configure MSTest output](unit-testing-mstest-configure.md#output-settings). ### Cancellation token diff --git a/docs/navigate/devops-testing/toc.yml b/docs/navigate/devops-testing/toc.yml index 6da6c8c909e45..620c217e4e7a1 100644 --- a/docs/navigate/devops-testing/toc.yml +++ b/docs/navigate/devops-testing/toc.yml @@ -232,6 +232,8 @@ items: href: ../../core/testing/mstest-analyzers/mstest0064.md - name: MSTEST0065 href: ../../core/testing/mstest-analyzers/mstest0065.md + - name: MSTEST0066 + href: ../../core/testing/mstest-analyzers/mstest0066.md - name: MSTEST0067 href: ../../core/testing/mstest-analyzers/mstest0067.md - name: MSTEST0068 @@ -242,6 +244,26 @@ items: href: ../../core/testing/mstest-analyzers/mstest0070.md - name: MSTEST0071 href: ../../core/testing/mstest-analyzers/mstest0071.md + - name: MSTEST0072 + href: ../../core/testing/mstest-analyzers/mstest0072.md + - name: MSTEST0073 + href: ../../core/testing/mstest-analyzers/mstest0073.md + - name: MSTEST0074 + href: ../../core/testing/mstest-analyzers/mstest0074.md + - name: MSTEST0075 + href: ../../core/testing/mstest-analyzers/mstest0075.md + - name: MSTEST0076 + href: ../../core/testing/mstest-analyzers/mstest0076.md + - name: MSTEST0077 + href: ../../core/testing/mstest-analyzers/mstest0077.md + - name: MSTEST0078 + href: ../../core/testing/mstest-analyzers/mstest0078.md + - name: MSTEST0079 + href: ../../core/testing/mstest-analyzers/mstest0079.md + - name: MSTEST0080 + href: ../../core/testing/mstest-analyzers/mstest0080.md + - name: MSTEST0081 + href: ../../core/testing/mstest-analyzers/mstest0081.md - name: Migration items: - name: Migrate from MSTest v1 to v3