Tests Buildkite Plugin

A Buildkite plugin that sets up your pipeline to run tests with Buildkite Test Engine. The plugin downloads the Test Engine Client (bktec), requests an OIDC token, ensures your test suite exists, and exports the environment variables that bktec expects.

The plugin removes the need to manually install bktec, set up authentication, or create your test suite before running tests. The plugin works with every test runner that bktec supports, including RSpec, Jest, pytest, and Go test.

Example

steps:
  - label: "RSpec"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
    parallelism: 2

Each step must invoke bktec run (or bktec plan) in its command — the plugin sets up the environment but does not run the test command for you. The plugin downloads bktec by default; set install-client: false if bktec is already installed on the agent.

If the suite-slug attribute is not set, the plugin uses the pipeline slug as the test suite slug. If the test suite does not exist yet, the plugin creates it from the pipeline on the first run.

Examples by runner

Jest

steps:
  - label: "Jest"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: jest
          result-path: jest-results.json
    parallelism: 4

pytest

steps:
  - label: "pytest"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: pytest
    parallelism: 4

Go test

steps:
  - label: "Go test"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: gotest
          result-path: gotest-results.xml
    parallelism: 4

Dynamic parallelism

To let bktec choose the number of partitions for a target run time, run bktec plan in a planning step with the --pipeline-upload flag, pointing at a pipeline template you supply. bktec plan uploads a follow-up step that runs the test plan. --pipeline-upload is passed directly to bktec in command: because it is not exposed as a plugin option (see Unsupported bktec flags).

steps:
  - label: "Plan RSpec"
    key: rspec-plan
    command: bktec plan --pipeline-upload .buildkite/rspec-template.yml
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json
          max-parallelism: 10
          target-time: 4m30s

The pipeline template at .buildkite/rspec-template.yml contains the run step:

steps:
  - label: "RSpec"
    key: rspec-run
    depends_on: rspec-plan
    command: bktec run --plan-identifier ${BUILDKITE_TEST_ENGINE_PLAN_IDENTIFIER}
    parallelism: ${BUILDKITE_TEST_ENGINE_PARALLELISM}
    plugins:
      - tests#v1.0.0:
          test-runner: rspec
          result-path: tmp/rspec-result.json

Docker

When the test command runs inside a Docker container, the plugin downloads bktec on the host (the default) and sets BUILDKITE_TEST_ENGINE_CLIENT_PATH that you can mount into the container. Set propagate-environment: true on the Docker plugin so the container picks up the BUILDKITE_TEST_ENGINE_* variables that are required to run bktec. Set client-os and client-arch when the host operating system or architecture differs from the container.

steps:
  - label: "Jest"
    command: bktec run
    plugins:
      - tests#v1.0.0:
          test-runner: jest
          client-os: linux
          result-path: jest-results.json
      - docker#v5.13.0:
          image: node:24-slim
          expand-volume-vars: true
          volumes:
            - "$$BUILDKITE_TEST_ENGINE_CLIENT_PATH:/usr/local/bin/bktec"
          propagate-environment: true

Configuration

bktec options

These options map directly to bktec flags or environment variables. For the full reference, see the Test Engine Client repository.

test-runner (required, string)

Any runner supported by bktec, for example rspec, jest, pytest, or gotest.

test-cmd (required for custom and pytest-pants runners, string)

Command to run tests.

result-path (string)

Path to the test runner's output file. Required for all runners except cypress, pytest, pytest-pants, and custom.

suite-slug (optional, string)

Test suite slug. Defaults to the pipeline slug.

test-file-pattern (required for custom runner, string)

Glob pattern for test files.

test-file-exclude-pattern (optional, string)

Exclude test files matching this pattern.

tag-filters (optional, string)

Filter tests by tag. Currently only supported by pytest.

files (optional, string)

Path to a file that lists test files to run, one per line.

retry-count (optional, integer)

Number of times to retry failing tests.

retry-cmd (optional, string)

Command to run when retrying failed tests. Defaults to test-cmd.

disable-retry-for-muted-test (optional, boolean)

Disable retry for muted tests.

split-by-example (optional, boolean)

Enable example-level splitting. Not supported by every runner.

location-prefix (optional, string)

Prefix to prepend to test file paths when requesting a test plan.

max-parallelism (optional, integer)

Maximum parallelism for dynamic test plans. Used with bktec plan.

target-time (optional, string)

Desired target time for the test suite, for example 4m30s. Must be used with max-parallelism.

plan-identifier (optional, string)

Test plan identifier created by bktec plan. Set automatically in dynamic mode; can be set manually for static mode.

fail-on-no-tests (optional, boolean)

Exit with an error if no tests are assigned to this node.

upload-results (optional, boolean)

Upload test results to Test Engine. Default: true. Set to false if you use the test collectors for richer data collection, or if you want to handle the upload yourself.

Requires bktec 2.7.0 or later.

debug-enabled (optional, boolean)

Enable verbose output.

Plugin options

These options control plugin behavior and are not passed to bktec.

install-client (optional, boolean)

Download the bktec binary, add it to the PATH, and set BUILDKITE_TEST_ENGINE_CLIENT_PATH. Default: true. Set to false if bktec is already installed on the agent.

client-os (optional, string)

Target OS for the downloaded binary, for example linux or darwin. Defaults to the host OS. Useful when the host and a Docker container run different operating systems.

client-arch (optional, string)

Target architecture for the downloaded binary, for example amd64 or arm64. Defaults to the host architecture.

client-version (optional, string)

bktec version to download, for example 2.4.0. Defaults to the latest release.

oidc-lifetime (optional, integer)

Lifetime in seconds for the OIDC token. Default: 300.

Unsupported bktec flags

A small number of bktec flags cannot be set through environment variables, so the plugin intentionally does not expose them as options. To use any of these, pass them directly to bktec in your step's command:.

Flag	Command	Notes
--selection-param	run, plan	Preview selection
--metadata	run, plan	Preview selection
--json	plan	Print the plan as JSON to stdout
--pipeline-upload	plan	Upload a follow-up pipeline step that runs the plan
--output	backfill-commit-metadata	Write tarball locally
--upload	backfill-commit-metadata	Upload an existing tarball
--version	Global	No environment variable needed

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/buildkite-plugins/tests-buildkite-plugin.

License

MIT (see LICENSE.txt).