TE-6261 - Add a vitest runner

CHANGELOG.md

@@ -1,5 +1,6 @@
 # Changelog
 ## Unreleased
+- Add first-class support for the [Vitest](./docs/vitest.md) test runner (`BUILDKITE_TEST_ENGINE_TEST_RUNNER=vitest`), including test splitting by file, automatic retries, and muting. Vitest's JSON reporter is Jest-compatible, so results are parsed with the shared Jest report parser.
 - Add `--plan-out` to `bktec plan`. It writes the full test plan (tasks, per-node test breakdown, muted and skipped tests, and timing metadata) as the server's response, unmodified, without running the tests, so you can pull down a copy of the plan. Takes a destination: `-` for stdout, or a file path. `--json`, `--plan-out` and `--pipeline-upload` are mutually exclusive.
 - At parallelism 1 the split summary now prints only the case count without a known/unknown timing breakdown. The server no longer emits `known_timings_ratio` at parallelism 1 (it skips the timing fetch), so the previous breakdown was derived from a value the server no longer sends.
 - Add `--plan-identifier` (env: `BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER`) to `bktec plan`. The identifier sets the plan's server-side cache key, and supplying it lets `plan` run off-agent without `BUILDKITE_BUILD_ID`/`BUILDKITE_STEP_ID`. Previously this flag was only available on `bktec run`.

README.md

@@ -6,15 +6,15 @@ bktec supports multiple test runners and offers various features to enhance your
 
 <!-- DO NOT MANUALLY EDIT THE TABLE BELOW. The contents can be generate with `go run util/supported_features/main.go` -->
 
-| Feature | RSpec | Jest | Playwright | Cypress | pytest | pytest-pants | gotest | Cucumber | NUnit | Custom test runner |
-| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
-| Split tests by file[^1] | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
-| [Split slow files by individual test example](https://github.com/buildkite/test-engine-client/blob/main/docs/rspec.md#split-slow-files-by-individual-test-example) | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
-| Filter test files | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
-| Filter tests by tag | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
-| Automatically retry failed test | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
-| Mute tests (ignore test failures) | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
-| Skip tests | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
+| Feature | RSpec | Jest | Vitest | Playwright | Cypress | pytest | pytest-pants | gotest | Cucumber | NUnit | Custom test runner |
+| --- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| Split tests by file[^1] | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
+| [Split slow files by individual test example](https://github.com/buildkite/test-engine-client/blob/main/docs/rspec.md#split-slow-files-by-individual-test-example) | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
+| Filter test files | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
+| Filter tests by tag | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| Automatically retry failed test | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
+| Mute tests (ignore test failures) | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| Skip tests | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |
 
 ## Installation
 The latest version of bktec can be downloaded from https://github.com/buildkite/test-engine-client/releases
@@ -251,6 +251,7 @@ For detailed usage, flags, and configuration options, see the [Commit Metadata B
 ### Configure the test runner
 To configure the test runner for bktec, please refer to the detailed guides provided for each supported test runner. You can find the guides at the following links:
 - [Jest](./docs/jest.md)
+- [Vitest](./docs/vitest.md)
 - [Playwright](./docs/playwright.md)
 - [Cypress](./docs/cypress.md)
 - [pytest](./docs/pytest.md)

docs/vitest.md

@@ -0,0 +1,71 @@
+# Using bktec with Vitest
+To integrate bktec with Vitest, set the `BUILDKITE_TEST_ENGINE_TEST_RUNNER` environment variable to `vitest`. Then, specify the `BUILDKITE_TEST_ENGINE_RESULT_PATH` to define where the JSON result should be stored. bktec will instruct Vitest to output the JSON result to this path, which is necessary for bktec to read the test results for retries and verification purposes.
+
+```sh
+export BUILDKITE_TEST_ENGINE_TEST_RUNNER=vitest
+export BUILDKITE_TEST_ENGINE_RESULT_PATH=tmp/vitest-result.json
+```
+
+> [!NOTE]
+> Vitest's JSON reporter is Jest-compatible, so bktec reuses the same result parsing as the Jest runner.
+
+## Configure test command
+By default, bktec runs Vitest with the following command:
+
+```sh
+npx vitest run {{testExamples}} --reporter=json --outputFile {{resultPath}}
+```
+
+In this command, `{{testExamples}}` is replaced by bktec with the list of test files to run, and `{{resultPath}}` is replaced with the value set in `BUILDKITE_TEST_ENGINE_RESULT_PATH`. You can customize this command using the `BUILDKITE_TEST_ENGINE_TEST_CMD` environment variable.
+
+To customize the test command, set the following environment variable:
+```sh
+export BUILDKITE_TEST_ENGINE_TEST_CMD="yarn vitest run {{testExamples}} --reporter=json --outputFile {{resultPath}}"
+```
+
+> [!IMPORTANT]
+> Make sure to append `--reporter=json --outputFile {{resultPath}}` in your custom test command, as bktec requires this to read the test results for retries and verification purposes. Also ensure you use `vitest run` (not watch mode) so the process exits after the run completes.
+
+## Filter test files
+By default, bktec runs test files that match the `**/{__tests__/**/*,*.spec,*.test}.{ts,js,tsx,jsx}` pattern. You can customize this pattern using the `BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN` environment variable. For instance, to configure bktec to only run Vitest test files inside the `src/components` directory, use:
+
+```sh
+export BUILDKITE_TEST_ENGINE_TEST_FILE_PATTERN=src/components/**/*.test.{ts,tsx}
+```
+
+Additionally, you can exclude specific files or directories that match a certain pattern using the `BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN` environment variable. For example, to exclude test files inside the `src/utilities` directory, use:
+
+```sh
+export BUILDKITE_TEST_ENGINE_TEST_FILE_EXCLUDE_PATTERN=src/utilities
+```
+
+> [!TIP]
+> This option accepts the pattern syntax supported by the [zzglob](https://github.com/DrJosh9000/zzglob?tab=readme-ov-file#pattern-syntax) library.
+
+## Location prefix
+If you have configured the [Buildkite test collector](https://buildkite.com/docs/test-engine/test-collection) with a location prefix, you should set the same prefix for bktec so that test file paths match those reported by the collector. You can set this using the `--location-prefix` flag or the `BUILDKITE_TEST_ENGINE_LOCATION_PREFIX` environment variable.
+
+```sh
+export BUILDKITE_TEST_ENGINE_LOCATION_PREFIX=my/prefix/
+```
+
+## Automatically retry failed tests
+You can configure bktec to automatically retry failed tests using the `BUILDKITE_TEST_ENGINE_RETRY_COUNT` environment variable. When this variable is set to a number greater than `0`, bktec will retry each failed test up to the specified number of times, using the following command:
+
+```sh
+npx vitest run --testNamePattern '{{testNamePattern}}' --reporter=json --outputFile {{resultPath}}
+```
+
+In this command:
+- `{{testNamePattern}}` is replaced by bktec with the pattern matching the failed test names
+- `{{resultPath}}` is replaced with the value set in `BUILDKITE_TEST_ENGINE_RESULT_PATH`
+
+You can customize this command using the `BUILDKITE_TEST_ENGINE_RETRY_CMD` environment variable.
+
+```sh
+export BUILDKITE_TEST_ENGINE_RETRY_CMD="yarn vitest run --testNamePattern '{{testNamePattern}}' --reporter=json --outputFile {{resultPath}}"
+export BUILDKITE_TEST_ENGINE_RETRY_COUNT=2
+```
+
+> [!IMPORTANT]
+> Make sure to append `--testNamePattern '{{testNamePattern}}' --reporter=json --outputFile {{resultPath}}` in your custom retry command.

internal/runner/detector.go

@@ -27,6 +27,8 @@ func DetectRunner(cfg *config.Config) (TestRunner, error) {
 		return NewRspec(runnerConfig), nil
 	case "jest":
 		return NewJest(runnerConfig), nil
+	case "vitest":
+		return NewVitest(runnerConfig), nil
 	case "cypress":
 		return NewCypress(runnerConfig), nil
 	case "playwright":
@@ -45,6 +47,6 @@ func DetectRunner(cfg *config.Config) (TestRunner, error) {
 		return NewCustom(runnerConfig)
 	default:
 		// Update the error message to include the new runner
-		return nil, fmt.Errorf("runner value %q is invalid, possible values are 'rspec', 'jest', 'cypress', 'playwright', 'pytest', 'pytest-pants', 'gotest', 'cucumber', 'nunit', or 'custom'", testRunner)
+		return nil, fmt.Errorf("runner value %q is invalid, possible values are 'rspec', 'jest', 'vitest', 'cypress', 'playwright', 'pytest', 'pytest-pants', 'gotest', 'cucumber', 'nunit', or 'custom'", testRunner)
 	}
 }

internal/runner/doc.go

@@ -1,3 +1,3 @@
 // Package runner provides the test runners that run sets of test cases.
-// Supported runners: rspec, jest, cypress, playwright, pytest, gotest, cucumber, nunit.
+// Supported runners: rspec, jest, vitest, cypress, playwright, pytest, gotest, cucumber, nunit.
 package runner

internal/runner/jest.go

@@ -160,14 +160,20 @@ type JestReport struct {
 }
 
 func (j Jest) ParseReport(path string) (JestReport, error) {
+	return ParseJestReport(path)
+}
+
+// ParseJestReport parses a Jest-compatible JSON report file. It is shared by the
+// Jest and Vitest runners, since Vitest's JSON reporter emits the same shape.
+func ParseJestReport(path string) (JestReport, error) {
 	var report JestReport
 	data, err := os.ReadFile(path)
 	if err != nil {
-		return JestReport{}, fmt.Errorf("failed to read Jest output: %v", err)
+		return JestReport{}, fmt.Errorf("failed to read report output: %v", err)
 	}
 
 	if err := json.Unmarshal(data, &report); err != nil {
-		return JestReport{}, fmt.Errorf("failed to parse Jest output: %s", err)
+		return JestReport{}, fmt.Errorf("failed to parse report output: %s", err)
 	}
 
 	return report, nil

internal/runner/supported_features_test.go

@@ -27,6 +27,7 @@ func TestSupportedFeatures_SplitBySelectorSupportedRunners(t *testing.T) {
 	gotest := NewGoTest(runnerConfig)
 	cucumber := NewCucumber(runnerConfig)
 	nunit := NewNUnit(runnerConfig)
+	vitest := NewVitest(runnerConfig)
 
 	runners := []TestRunner{
 		custom,
@@ -39,6 +40,7 @@ func TestSupportedFeatures_SplitBySelectorSupportedRunners(t *testing.T) {
 		gotest,
 		cucumber,
 		nunit,
+		vitest,
 	}
 
 	supportedRunners := []string{gotest.Name(), rspec.Name(), custom.Name()}

internal/runner/testdata/.yarnrc.yml

@@ -0,0 +1,8 @@
+approvedGitRepositories:
+  - "**"
+
+enableScripts: true
+
+nodeLinker: node-modules
+
+npmMinimalAgeGate: 0