diff --git a/CHANGELOG.md b/CHANGELOG.md index dc0be051..92f33fbf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,7 @@ # Changelog ## Unreleased - **Breaking:** Remove the `pytest-pants` runner. It had zero real usage over the last 3 months. Use the `custom` runner instead, which now supports selector-based test splitting (see `--selector-splitting` in 2.9.0). +- **Breaking:** Remove the `nunit` runner. It saw almost no real usage, and had known issues splitting tests by file (result path handling, and class name collisions across namespaces). ## 2.9.0 - 2026-07-06 - Add experimental selector-based test splitting for supported non-file runners (RSpec, custom runner, Go) via `--selector-splitting` and `--selector-file` (env: `BUILDKITE_TEST_ENGINE_SELECTOR_FILE`), splitting by runnable selector rather than by file. `--selector-file` must be used alongside `--selector-splitting`. diff --git a/README.md b/README.md index 9591302b..df454194 100644 --- a/README.md +++ b/README.md @@ -6,15 +6,15 @@ bktec supports multiple test runners and offers various features to enhance your -| Feature | RSpec | Jest | Playwright | Cypress | pytest | gotest | Cucumber | NUnit | Custom test runner | -| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | -| Split tests by file[^1] | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | -| [Split slow files by individual test example](https://github.com/buildkite/test-engine-client/blob/main/docs/rspec.md#split-slow-files-by-individual-test-example) | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | -| Filter test files | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | -| Filter tests by tag | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | -| Automatically retry failed test | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | -| Mute tests (ignore test failures) | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | -| Skip tests | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | +| Feature | RSpec | Jest | Playwright | Cypress | pytest | gotest | Cucumber | Custom test runner | +| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | +| Split tests by file[^1] | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | +| [Split slow files by individual test example](https://github.com/buildkite/test-engine-client/blob/main/docs/rspec.md#split-slow-files-by-individual-test-example) | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | +| Filter test files | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | +| Filter tests by tag | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | +| Automatically retry failed test | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | +| Mute tests (ignore test failures) | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | +| Skip tests | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ## Installation The latest version of bktec can be downloaded from https://github.com/buildkite/test-engine-client/releases @@ -254,7 +254,6 @@ To configure the test runner for bktec, please refer to the detailed guides prov - [go test](./docs/gotest.md) - [RSpec](./docs/rspec.md) - [Cucumber](./docs/cucumber.md) -- [NUnit](./docs/nunit.md) - [Custom Test Runner](./docs/custom-test-runner.md) diff --git a/docs/nunit.md b/docs/nunit.md deleted file mode 100644 index 5305c163..00000000 --- a/docs/nunit.md +++ /dev/null @@ -1,92 +0,0 @@ -# Using bktec with NUnit - -To integrate `bktec` with NUnit (.NET), you need the [JUnit XML test logger](https://github.com/spekt/junit.testlogger) NuGet package installed in your test project so that `dotnet test` can produce JUnit XML results. - -```sh -dotnet add package JUnitXml.TestLogger -``` - -## How it works - -`bktec` discovers `.cs` test files using a glob pattern, then maps each file to a class name (the filename without the `.cs` extension). It builds a `dotnet test --filter` expression using `FullyQualifiedName~.ClassName` predicates joined with `|` (OR), so only the test classes assigned to this node are executed. - -> [!IMPORTANT] -> Because test splitting is based on file/class name mapping, each `.cs` test file should contain a single test class whose name matches the filename (e.g. `CalculatorTests.cs` contains class `CalculatorTests`). This is the standard NUnit convention. - -## Quick start - -Build your solution first (so each split can use `--no-build`), then run `bktec`: - -```sh -# Build once upfront -dotnet build - -# Configure bktec -export BUILDKITE_TEST_ENGINE_TEST_RUNNER=nunit -export BUILDKITE_TEST_ENGINE_RESULT_PATH=test-results/results.xml -export BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN="tests/**/*Tests.cs" -export BUILDKITE_TEST_ENGINE_SUITE_SLUG=your-suite-slug -export BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN=your-token - -# Run -bktec run -``` - -## Configure test command - -By default, `bktec` runs tests with the following command: - -```sh -dotnet test --no-build --filter {{testFilter}} --logger junit;LogFilePath={{resultPath}} -``` - -In this command: -- `{{testFilter}}` is replaced by `bktec` with the filter expression (e.g. `FullyQualifiedName~.CalculatorTests|FullyQualifiedName~.StringUtilsTests`) -- `{{resultPath}}` is replaced with the `BUILDKITE_TEST_ENGINE_RESULT_PATH` environment variable - -You can customize this command using the `BUILDKITE_TEST_ENGINE_TEST_CMD` environment variable. For example, to target a specific project: - -```sh -export BUILDKITE_TEST_ENGINE_TEST_CMD="dotnet test MyProject.Tests --no-build --filter {{testFilter}} --logger junit;LogFilePath={{resultPath}}" -``` - -## Configure test file pattern - -By default, `bktec` discovers test files matching `**/*Tests.cs`. You can customize this with: - -```sh -export BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN="tests/**/*Tests.cs" -``` - -You can also exclude files: - -```sh -export BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN="tests/**/IntegrationTests.cs" -``` - -## Automatically retry failed tests - -You can configure `bktec` to automatically retry failed tests using the `BUILDKITE_TEST_ENGINE_RETRY_COUNT` environment variable: - -```sh -export BUILDKITE_TEST_ENGINE_RETRY_COUNT=1 -``` - -## Full Buildkite pipeline example - -```yaml -steps: - - name: "NUnit :dotnet:" - commands: - - dotnet build - - bktec run - env: - BUILDKITE_TEST_ENGINE_SUITE_SLUG: your-suite-slug - BUILDKITE_TEST_ENGINE_UPLOAD_RESULTS: "true" # This will upload test results to Test Engine - BUILDKITE_TEST_ENGINE_API_ACCESS_TOKEN: your-api-token - BUILDKITE_TEST_ENGINE_TEST_RUNNER: nunit - BUILDKITE_TEST_ENGINE_RESULT_PATH: test-results/results.xml - BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN: "tests/**/*Tests.cs" - BUILDKITE_TEST_ENGINE_RETRY_COUNT: 1 - parallelism: 4 -``` diff --git a/internal/config/validate.go b/internal/config/validate.go index bf3e8dd8..d399a372 100644 --- a/internal/config/validate.go +++ b/internal/config/validate.go @@ -119,7 +119,6 @@ func (c *Config) ValidateForRun() error { "playwright": true, "gotest": true, "cucumber": true, - "nunit": true, } if c.ResultPath == "" && runnersWithResultPath[c.TestRunner] { c.errs.appendFieldError("BUILDKITE_TEST_ENGINE_RESULT_PATH", "must not be blank") diff --git a/internal/runner/detector.go b/internal/runner/detector.go index 043b39d6..0b65b89a 100644 --- a/internal/runner/detector.go +++ b/internal/runner/detector.go @@ -37,12 +37,10 @@ func DetectRunner(cfg *config.Config) (TestRunner, error) { return NewGoTest(runnerConfig), nil case "cucumber": return NewCucumber(runnerConfig), nil - case "nunit": - return NewNUnit(runnerConfig), nil case "custom": return NewCustom(runnerConfig) default: // Update the error message to include the new runner - return nil, fmt.Errorf("runner value %q is invalid, possible values are 'rspec', 'jest', 'cypress', 'playwright', 'pytest', 'gotest', 'cucumber', 'nunit', or 'custom'", testRunner) + return nil, fmt.Errorf("runner value %q is invalid, possible values are 'rspec', 'jest', 'cypress', 'playwright', 'pytest', 'gotest', 'cucumber', or 'custom'", testRunner) } } diff --git a/internal/runner/doc.go b/internal/runner/doc.go index ffc96ea7..53f2461b 100644 --- a/internal/runner/doc.go +++ b/internal/runner/doc.go @@ -1,3 +1,3 @@ // Package runner provides the test runners that run sets of test cases. -// Supported runners: rspec, jest, cypress, playwright, pytest, gotest, cucumber, nunit. +// Supported runners: rspec, jest, cypress, playwright, pytest, gotest, cucumber. package runner diff --git a/internal/runner/nunit.go b/internal/runner/nunit.go deleted file mode 100644 index 66c1dc6c..00000000 --- a/internal/runner/nunit.go +++ /dev/null @@ -1,157 +0,0 @@ -package runner - -import ( - "fmt" - "path/filepath" - "strings" - - "github.com/buildkite/test-engine-client/v2/internal/debug" - "github.com/buildkite/test-engine-client/v2/internal/plan" - "github.com/kballard/go-shellquote" -) - -type NUnit struct { - RunnerConfig -} - -// Compile-time check that NUnit implements TestRunner -var _ TestRunner = (*NUnit)(nil) - -func (n NUnit) SupportedFeatures() SupportedFeatures { - return SupportedFeatures{ - SplitByFile: true, - SplitByExample: false, - FilterTestFiles: true, - FilterTestByTag: false, - AutoRetry: true, - Mute: true, - Skip: false, - } -} - -func NewNUnit(c RunnerConfig) NUnit { - if c.TestCommand == "" { - c.TestCommand = "dotnet test --no-build --filter {{testFilter}} --logger junit;LogFilePath={{resultPath}}" - } - - if c.TestFilePattern == "" { - c.TestFilePattern = "**/*Tests.cs" - } - - if c.RetryTestCommand == "" { - c.RetryTestCommand = c.TestCommand - } - - return NUnit{ - RunnerConfig: c, - } -} - -func (n NUnit) Name() string { - return "NUnit" -} - -func (n NUnit) ResultFormat() string { - return "junit" -} - -// GetFiles returns an array of .cs test file names using the discovery pattern. -func (n NUnit) GetFiles() ([]string, error) { - debug.Println("Discovering test files with include pattern:", n.TestFilePattern, "exclude pattern:", n.TestFileExcludePattern) - files, err := discoverTestFiles(n.TestFilePattern, n.TestFileExcludePattern) - debug.Println("Discovered", len(files), "files") - - if err != nil { - return nil, err - } - - if len(files) == 0 { - return nil, fmt.Errorf("no files found with pattern %q and exclude pattern %q", n.TestFilePattern, n.TestFileExcludePattern) - } - - return files, nil -} - -func (n NUnit) GetExamples(files []string) ([]plan.TestCase, error) { - return nil, fmt.Errorf("not supported in NUnit") -} - -// Run executes dotnet test with a --filter expression built from the test cases. -// Test cases are mapped from .cs file paths to class names, and joined into a -// FullyQualifiedName~ filter expression. -func (n NUnit) Run(result *RunResult, testCases []plan.TestCase, retry bool) error { - cmd, err := buildCommand(n, testCases, retry) - if err != nil { - return err - } - - cmdErr := runAndForwardSignal(cmd) - - testResults, parseErr := loadAndParseJUnitXML(n.ResultPath) - if parseErr != nil { - fmt.Printf("Buildkite Test Engine Client: Failed to read NUnit output, tests will not be retried: %v\n", parseErr) - return cmdErr - } - - for _, test := range testResults { - result.RecordTestResult(plan.TestCase{ - Scope: test.Classname, - Name: test.Name, - Path: test.Classname, - }, test.Result) - } - - return cmdErr -} - -// extractClassNames extracts unique class names from test case paths. -// Each path is expected to be a .cs file path like "tests/MyLib.Tests/CalculatorTests.cs". -// The class name is the filename without extension, e.g. "CalculatorTests". -func extractClassNames(testCases []plan.TestCase) []string { - seen := map[string]bool{} - var classNames []string - - for _, tc := range testCases { - className := strings.TrimSuffix(filepath.Base(tc.Path), ".cs") - if !seen[className] { - classNames = append(classNames, className) - seen[className] = true - } - } - - return classNames -} - -// buildTestFilter constructs a dotnet test --filter expression from class names. -// Each class name becomes a "FullyQualifiedName~.ClassName" predicate, -// joined with "|" (OR). -// The "~" operator is a "contains" match and the leading "." anchors to a -// namespace boundary, preventing false positives on partial name matches. -func buildTestFilter(classNames []string) string { - parts := make([]string, len(classNames)) - for i, name := range classNames { - parts[i] = fmt.Sprintf("FullyQualifiedName~.%s", name) - } - return strings.Join(parts, "|") -} - -func (n NUnit) CommandNameAndArgs(testCases []plan.TestCase, retry bool) (string, []string, error) { - classNames := extractClassNames(testCases) - - cmd := n.TestCommand - if retry { - cmd = n.RetryTestCommand - } - - filter := buildTestFilter(classNames) - - cmd = strings.Replace(cmd, "{{testFilter}}", filter, 1) - cmd = strings.Replace(cmd, "{{resultPath}}", n.ResultPath, 1) - - words, err := shellquote.Split(cmd) - if err != nil { - return "", []string{}, err - } - - return words[0], words[1:], nil -} diff --git a/internal/runner/nunit_test.go b/internal/runner/nunit_test.go deleted file mode 100644 index 4f243276..00000000 --- a/internal/runner/nunit_test.go +++ /dev/null @@ -1,171 +0,0 @@ -package runner - -import ( - "testing" - - "github.com/buildkite/test-engine-client/v2/internal/plan" - "github.com/google/go-cmp/cmp" -) - -func TestNewNUnit(t *testing.T) { - cases := []struct { - input RunnerConfig - want RunnerConfig - }{ - { - input: RunnerConfig{}, - want: RunnerConfig{ - TestCommand: "dotnet test --no-build --filter {{testFilter}} --logger junit;LogFilePath={{resultPath}}", - TestFilePattern: "**/*Tests.cs", - RetryTestCommand: "dotnet test --no-build --filter {{testFilter}} --logger junit;LogFilePath={{resultPath}}", - }, - }, - { - input: RunnerConfig{ - TestCommand: "dotnet test --filter {{testFilter}} --logger junit;LogFilePath={{resultPath}}", - TestFilePattern: "tests/**/*Tests.cs", - }, - want: RunnerConfig{ - TestCommand: "dotnet test --filter {{testFilter}} --logger junit;LogFilePath={{resultPath}}", - TestFilePattern: "tests/**/*Tests.cs", - RetryTestCommand: "dotnet test --filter {{testFilter}} --logger junit;LogFilePath={{resultPath}}", - }, - }, - } - - for _, c := range cases { - got := NewNUnit(c.input) - if diff := cmp.Diff(got.RunnerConfig, c.want, cmp.AllowUnexported(RunnerConfig{})); diff != "" { - t.Errorf("NewNUnit(%v) diff (-got +want):\n%s", c.input, diff) - } - } -} - -func TestNUnit_GetFiles(t *testing.T) { - changeCwd(t, "./testdata/nunit") - - nunit := NewNUnit(RunnerConfig{ - TestFilePattern: "tests/**/*Tests.cs", - }) - - got, err := nunit.GetFiles() - if err != nil { - t.Errorf("NUnit.GetFiles() error = %v", err) - } - - want := []string{ - "tests/MyLib.Tests/CalculatorTests.cs", - "tests/MyLib.Tests/SimpleStackTests.cs", - "tests/MyLib.Tests/StringUtilsTests.cs", - } - - if diff := cmp.Diff(got, want); diff != "" { - t.Errorf("NUnit.GetFiles() diff (-got +want):\n%s", diff) - } -} - -func TestNUnit_GetExamples(t *testing.T) { - nunit := NewNUnit(RunnerConfig{}) - _, err := nunit.GetExamples([]string{"tests/MyLib.Tests/CalculatorTests.cs"}) - if err == nil || err.Error() != "not supported in NUnit" { - t.Errorf("GetExamples() error = %v, want %q", err, "not supported in NUnit") - } -} - -func TestNUnit_ExtractClassNames(t *testing.T) { - testCases := []plan.TestCase{ - {Path: "tests/MyLib.Tests/CalculatorTests.cs"}, - {Path: "tests/MyLib.Tests/StringUtilsTests.cs"}, - {Path: "tests/MyLib.Tests/CalculatorTests.cs"}, // duplicate - } - - got := extractClassNames(testCases) - want := []string{"CalculatorTests", "StringUtilsTests"} - - if diff := cmp.Diff(got, want); diff != "" { - t.Errorf("extractClassNames() diff (-got +want):\n%s", diff) - } -} - -func TestNUnit_BuildTestFilter(t *testing.T) { - cases := []struct { - classNames []string - want string - }{ - { - classNames: []string{"CalculatorTests"}, - want: "FullyQualifiedName~.CalculatorTests", - }, - { - classNames: []string{"CalculatorTests", "StringUtilsTests"}, - want: "FullyQualifiedName~.CalculatorTests|FullyQualifiedName~.StringUtilsTests", - }, - } - - for _, c := range cases { - got := buildTestFilter(c.classNames) - if got != c.want { - t.Errorf("buildTestFilter(%v) = %q, want %q", c.classNames, got, c.want) - } - } -} - -func TestNUnit_CommandNameAndArgs(t *testing.T) { - nunit := NewNUnit(RunnerConfig{ - ResultPath: "test-results.xml", - }) - - classNames := []plan.TestCase{{Path: "CalculatorTests"}, {Path: "StringUtilsTests"}} - - gotName, gotArgs, err := nunit.CommandNameAndArgs(classNames, false) - if err != nil { - t.Errorf("commandNameAndArgs() error = %v", err) - } - - wantName := "dotnet" - wantArgs := []string{ - "test", - "--no-build", - "--filter", - "FullyQualifiedName~.CalculatorTests|FullyQualifiedName~.StringUtilsTests", - "--logger", - "junit;LogFilePath=test-results.xml", - } - - if gotName != wantName { - t.Errorf("commandNameAndArgs() name = %v, want %v", gotName, wantName) - } - - if diff := cmp.Diff(gotArgs, wantArgs); diff != "" { - t.Errorf("commandNameAndArgs() args diff (-got +want):\n%s", diff) - } -} - -func TestNUnit_ParseJUnitResults(t *testing.T) { - results, err := loadAndParseJUnitXML("./testdata/nunit/junit-results.xml") - if err != nil { - t.Fatalf("loadAndParseJUnitXmlResult() error = %v", err) - } - - if len(results) != 5 { - t.Fatalf("loadAndParseJUnitXmlResult() len = %d, want 5", len(results)) - } - - // Check passed test - if results[0].Name != "AddTwoNumbers" || results[0].Result != TestStatusPassed { - t.Errorf("results[0] = {Name: %q, Result: %q}, want {Name: \"AddTwoNumbers\", Result: \"passed\"}", results[0].Name, results[0].Result) - } - if results[0].Classname != "MyLib.Tests.CalculatorTests" { - t.Errorf("results[0].Classname = %q, want %q", results[0].Classname, "MyLib.Tests.CalculatorTests") - } - - // Check failed test - if results[2].Name != "DivideByZero" || results[2].Result != TestStatusFailed { - t.Errorf("results[2] = {Name: %q, Result: %q}, want {Name: \"DivideByZero\", Result: \"failed\"}", results[2].Name, results[2].Result) - } - - // Check skipped test - if results[4].Name != "SkippedTest" || results[4].Result != TestStatusSkipped { - t.Errorf("results[4] = {Name: %q, Result: %q}, want {Name: \"SkippedTest\", Result: \"skipped\"}", results[4].Name, results[4].Result) - } -} diff --git a/internal/runner/supported_features_test.go b/internal/runner/supported_features_test.go index 32d572aa..380f8f86 100644 --- a/internal/runner/supported_features_test.go +++ b/internal/runner/supported_features_test.go @@ -25,7 +25,6 @@ func TestSupportedFeatures_SplitBySelectorSupportedRunners(t *testing.T) { pytest := NewPytest(runnerConfig) gotest := NewGoTest(runnerConfig) cucumber := NewCucumber(runnerConfig) - nunit := NewNUnit(runnerConfig) runners := []TestRunner{ custom, @@ -36,7 +35,6 @@ func TestSupportedFeatures_SplitBySelectorSupportedRunners(t *testing.T) { pytest, gotest, cucumber, - nunit, } supportedRunners := []string{ diff --git a/internal/runner/testdata/nunit/junit-results.xml b/internal/runner/testdata/nunit/junit-results.xml deleted file mode 100644 index 296d61bc..00000000 --- a/internal/runner/testdata/nunit/junit-results.xml +++ /dev/null @@ -1,16 +0,0 @@ - - - - - - - at MyLib.Tests.CalculatorTests.DivideByZero() - - - - - - - - - diff --git a/internal/runner/testdata/nunit/tests/MyLib.Tests/CalculatorTests.cs b/internal/runner/testdata/nunit/tests/MyLib.Tests/CalculatorTests.cs deleted file mode 100644 index bd24091f..00000000 --- a/internal/runner/testdata/nunit/tests/MyLib.Tests/CalculatorTests.cs +++ /dev/null @@ -1 +0,0 @@ -// Test fixture placeholder for bktec test splitting discovery. diff --git a/internal/runner/testdata/nunit/tests/MyLib.Tests/SimpleStackTests.cs b/internal/runner/testdata/nunit/tests/MyLib.Tests/SimpleStackTests.cs deleted file mode 100644 index bd24091f..00000000 --- a/internal/runner/testdata/nunit/tests/MyLib.Tests/SimpleStackTests.cs +++ /dev/null @@ -1 +0,0 @@ -// Test fixture placeholder for bktec test splitting discovery. diff --git a/internal/runner/testdata/nunit/tests/MyLib.Tests/StringUtilsTests.cs b/internal/runner/testdata/nunit/tests/MyLib.Tests/StringUtilsTests.cs deleted file mode 100644 index bd24091f..00000000 --- a/internal/runner/testdata/nunit/tests/MyLib.Tests/StringUtilsTests.cs +++ /dev/null @@ -1 +0,0 @@ -// Test fixture placeholder for bktec test splitting discovery. diff --git a/util/supported_features/main.go b/util/supported_features/main.go index 1868632a..0861d4e4 100644 --- a/util/supported_features/main.go +++ b/util/supported_features/main.go @@ -47,7 +47,6 @@ func main() { runner.NewPytest(runnerConfig), runner.NewGoTest(runnerConfig), runner.NewCucumber(runnerConfig), - runner.NewNUnit(runnerConfig), custom, }