TE-6261 - Add a vitest runner by mcncl · Pull Request #580 · buildkite/test-engine-client

mcncl · 2026-07-03T00:53:45Z

Description

Currently folks need to use the Jest runner with Vitest, as Vitest supports the Jest shape of JSON, however Vitest is (arguably) becoming more popular and offers some benefits over Jest, like explicit skipped status.

Context

This was raised by a customer as an enhancement on the engine, rather than using Jest.

Changes

adds a Vitest test runner which can be used with BUILDKITE_TEST_ENGINE_RUNNER=vitest
- a lot of the runner is similar/the same as Jest, Vitest just handles things like skipping in a more convenient way
creates a ParseJestReport function for Jest and Vitest to share, with more generic erroring (removed Jest)
adds vitest to config locations

Testing

go test ./... with targeted testing passes, full test suite fails locally as it looks to use some CI specific function

Much of the logic can be re-used from the Jest runner, with the addition of `skipped` tests, which Jest incorrect reports as `pending`.

buildsworth-bk-app

I found one blocker in the Vitest retry path: top-level test failures can be converted into skipped results on retry.

Want to dig deeper? The full session log is attached to this Buildkite build. Download the session file and open a new pi session with it:
Download the buildsworth logs from build 2493, then answer my questions about the findings.

Co-authored-by: buildsworth-bk-app[bot] <265670805+buildsworth-bk-app[bot]@users.noreply.github.com>

socket-security · 2026-07-03T01:13:27Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/vitest@4.1.9

View full report

Previous concerns have been addressed.

buildsworth-bk-app

Previous blocker is resolved: the Vitest retry pattern now builds top-level test names without the leading separator. I didn’t find new code issues; leaving this as a comment rather than approval because this is a new runner integration (L2) and I don’t see an L2 approval grant.

Want to dig deeper? The full session log is attached to this Buildkite build. Download the session file and open a new pi session with it:
Download the buildsworth logs from build 2497, then answer my questions about the findings.

buildsworth-bk-app

Previous blocker is resolved: the Vitest retry pattern now builds top-level test names without the leading separator. I didn’t find new high-confidence code issues; leaving this as a comment rather than approval because this is a new runner integration (L2) and I don’t see an L2 approval grant.

Want to dig deeper? The full session log is attached to this Buildkite build. Download the session file and open a new pi session with it:
Download the buildsworth logs from build 2499, then answer my questions about the findings.

nprizal

Looks good to me. Can we update the default test command to include both default and json reporter?

Co-authored-by: Naufan P. Rizal <np.rizal@icloud.com>

mcncl · 2026-07-03T02:59:11Z

Thanks @nprizal! Applied the suggestions

buildsworth-bk-app

I found one blocker in the follow-up: the Vitest default command now includes the default reporter, but the new unit test expectations still assert the previous command. The CI check is currently blocked/pending, so this is from the diff rather than a completed build.

Want to dig deeper? The full session log is attached to this Buildkite build. Download the session file and open a new pi session with it:
Download the buildsworth logs from build 2541, then answer my questions about the findings.

buildsworth-bk-app · 2026-07-03T03:02:33Z

+		{
+			input: RunnerConfig{},
+			want: RunnerConfig{
+				TestCommand:            "npx vitest run {{testExamples}} --reporter=json --outputFile {{resultPath}}",


Blocking: NewVitest now sets the default command to include both --reporter=default and --reporter=json, but this expected value still only has the JSON reporter. The retry expectation on line 25 has the same mismatch, so TestNewVitest will fail until these defaults are updated to match the implementation.

buildsworth-bk-app · 2026-07-03T03:02:33Z

+By default, bktec runs Vitest with the following command:
+
+```sh
+npx vitest run {{testExamples}} --reporter=json --outputFile {{resultPath}}


Non-blocking: The implementation's default command now includes --reporter=default as well as --reporter=json, so this default-command snippet is out of sync. Users copying the current docs would get JSON-only output and lose the normal Vitest console reporter; could we update the default/retry snippets and custom-command guidance to include both reporters?

buildsworth-bk-app

I found one blocker in the default Vitest command follow-up: the constructor, tests, and docs are now out of sync on whether the default includes both reporters.

Want to dig deeper? The full session log is attached to this Buildkite build. Download the session file and open a new pi session with it:
Download the buildsworth logs from build 2542, then answer my questions about the findings.

buildsworth-bk-app · 2026-07-03T03:02:51Z

+
+func NewVitest(v RunnerConfig) Vitest {
+	if v.TestCommand == "" {
+		v.TestCommand = "npx vitest run {{testExamples}} --reporter=default --reporter=json --outputFile {{resultPath}}"


Blocking: This default is now --reporter=default --reporter=json, but TestNewVitest still expects the JSON-only command and docs/vitest.md documents the same JSON-only default/retry commands. As written, the default-case assertion fails and the user-facing docs don't describe the command this constructor actually uses. Could you update the expected defaults and docs alongside this change?

buildsworth-bk-app

I found two blockers: the retry path can overwrite first-attempt results with Vitest's filter-skipped entries, and the default-command test expectations are now stale. I also left one docs sync note.

Want to dig deeper? The full session log is attached to this Buildkite build. Download the session file and open a new pi session with it:
Download the buildsworth logs from build 2540, then answer my questions about the findings.

buildsworth-bk-app · 2026-07-03T03:03:40Z

+				Path:  testPath,
+			}
+
+			result.RecordTestResult(testCase, status)


Blocking: Vitest includes tests filtered out by --testNamePattern in the JSON report as status: "skipped". Because retries pass the same RunResult back through this method, this unconditional record overwrites first-attempt results for tests that were not retried. For example, retrying (multi fails) in a file with sibling multi passes emits multi passes as skipped, so the final bktec result changes that previously-passed test to skipped. Could we ignore Vitest's filter-skipped assertions on retry unless they correspond to one of the retry testCases?

buildsworth-bk-app · 2026-07-03T03:03:40Z

+				TestCommand:            "npx vitest run {{testExamples}} --reporter=json --outputFile {{resultPath}}",
+				TestFilePattern:        "**/{__tests__/**/*,*.spec,*.test}.{ts,js,tsx,jsx}",
+				TestFileExcludePattern: "",
+				RetryTestCommand:       "npx vitest run --testNamePattern '{{testNamePattern}}' --reporter=json --outputFile {{resultPath}}",


Blocking: NewVitest now defaults both commands to include --reporter=default, but this expectation still omits it, so TestNewVitest will fail once the Go tests run. Could we update the expected defaults here too?

Suggested change

TestCommand: "npx vitest run {{testExamples}} --reporter=json --outputFile {{resultPath}}",

TestFilePattern: "**/{__tests__/**/*,*.spec,*.test}.{ts,js,tsx,jsx}",

TestFileExcludePattern: "",

RetryTestCommand: "npx vitest run --testNamePattern '{{testNamePattern}}' --reporter=json --outputFile {{resultPath}}",

TestCommand: "npx vitest run {{testExamples}} --reporter=default --reporter=json --outputFile {{resultPath}}",

TestFilePattern: "**/{__tests__/**/*,*.spec,*.test}.{ts,js,tsx,jsx}",

TestFileExcludePattern: "",

RetryTestCommand: "npx vitest run --testNamePattern '{{testNamePattern}}' --reporter=default --reporter=json --outputFile {{resultPath}}",

buildsworth-bk-app · 2026-07-03T03:03:40Z

+By default, bktec runs Vitest with the following command:
+
+```sh
+npx vitest run {{testExamples}} --reporter=json --outputFile {{resultPath}}


Non-blocking: The code default now includes --reporter=default as well as --reporter=json, so this user-facing default is out of sync. Could we update this command and the retry command below to match NewVitest?

buildsworth-bk-app

Existing default-command/test/docs blocker from the active buildsworth reviews is still unresolved; I’m not duplicating it inline here. I found one additional blocker in the npm fallback install path for the Vitest fixture.

Want to dig deeper? The full session log is attached to this Buildkite build. Download the session file and open a new pi session with it:
Download the buildsworth logs from build 2539, then answer my questions about the findings.

buildsworth-bk-app · 2026-07-03T03:03:53Z

+        "node": ">=18"
+      }
+    },
+    "node_modules/@rolldown/binding-darwin-arm64": {


Blocking: This lockfile only includes the Darwin ARM Rolldown native binding (grep 'node_modules/@rolldown/binding' returns just this entry), but Vitest/Vite needs the platform binding at runtime. On Linux, the npm fallback path in bin/setup installs from this lockfile and then npx vitest run ... fails with Cannot find module '@rolldown/binding-linux-x64-gnu'. Could we regenerate the npm lockfile so the Linux binding is present too, or drop it if Yarn is the intended installer?

nprizal · 2026-07-03T03:13:38Z

The CI check is currently blocked/pending, so this is from the diff rather than a completed build.

Interesting that the last few commits were pushed by @buildkite-systems 🤔 therefore the build is blocked.

mcncl · 2026-07-03T03:14:28Z

@nprizal yeah, that's on me; I was setting something up in another tab and commented on your comments as buildkite-systems 😬

mcncl added 8 commits July 1, 2026 10:19

refactor(jest): vitest support in parser

c18f8d8

doc(vitest): note vitest support added

cd5ef52

feat(vitest): add a vitest runner

a4cf9f9

Much of the logic can be re-used from the Jest runner, with the addition of `skipped` tests, which Jest incorrect reports as `pending`.

chore(npm): add vitest to package.json

105f08e

chore(specs): add some vitest specs

3a483cb

chore(yarn): add yarn files and locks

0f92bb6

tests(vitest): add unit and integration tests

8fea0db

docs: update changelog and add vitest docs

ef9afb4

mcncl requested a review from a team as a code owner July 3, 2026 00:53

buildsworth-bk-app Bot previously requested changes Jul 3, 2026

View reviewed changes

Comment thread internal/runner/vitest.go Outdated

buildkite-systems and others added 2 commits July 3, 2026 11:11

Update internal/runner/vitest.go

b1d80bc

Co-authored-by: buildsworth-bk-app[bot] <265670805+buildsworth-bk-app[bot]@users.noreply.github.com>

Merge branch 'main' into feat/vitest_runner

cdd54df

buildsworth-bk-app Bot reviewed Jul 3, 2026

View reviewed changes

nprizal reviewed Jul 3, 2026

View reviewed changes

Comment thread internal/runner/supported_features_test.go Outdated

Comment thread internal/runner/supported_features_test.go Outdated

Comment thread internal/runner/vitest.go

Comment thread internal/runner/vitest.go Outdated

Comment thread internal/runner/vitest.go Outdated

buildkite-systems and others added 4 commits July 3, 2026 12:57

Update internal/runner/supported_features_test.go

7e2d758

Co-authored-by: Naufan P. Rizal <np.rizal@icloud.com>

Update internal/runner/supported_features_test.go

c192590

Co-authored-by: Naufan P. Rizal <np.rizal@icloud.com>

Update internal/runner/vitest.go

308cb64

Co-authored-by: Naufan P. Rizal <np.rizal@icloud.com>

Update internal/runner/vitest.go

619e84f

Co-authored-by: Naufan P. Rizal <np.rizal@icloud.com>

buildsworth-bk-app Bot requested changes Jul 3, 2026

View reviewed changes

Uh oh!

Conversation

mcncl commented Jul 3, 2026

Description

Context

Changes

Testing

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

socket-security Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

nprizal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcncl commented Jul 3, 2026

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot left a comment

Choose a reason for hiding this comment

Uh oh!

buildsworth-bk-app Bot Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

nprizal commented Jul 3, 2026

Uh oh!

mcncl commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

socket-security Bot commented Jul 3, 2026 •

edited

Loading