Skip to content

fix: handle benchmark-only go test output#571

Open
DevNewbie1826 wants to merge 1 commit intortk-ai:masterfrom
DevNewbie1826:fix/go-test-benchmark-output
Open

fix: handle benchmark-only go test output#571
DevNewbie1826 wants to merge 1 commit intortk-ai:masterfrom
DevNewbie1826:fix/go-test-benchmark-output

Conversation

@DevNewbie1826
Copy link

@DevNewbie1826 DevNewbie1826 commented Mar 13, 2026

Problem

rtk go test treated benchmark-only runs as if nothing had executed.

For a command like go test -json -run=^$ -bench=., Go emits benchmark result lines but does not emit a test-level pass event for the benchmark itself. The old parser only counted test-scoped pass/fail/skip events, so benchmark-only output could collapse to:

Go test: No tests found

What changed

  • detect benchmark result lines from go test -json output and store them separately from test pass counts
  • handle both observed output shapes:
    • benchmark result attached to Test
    • benchmark result emitted as package-level output
  • render a dedicated benchmark summary instead of falling back to Go test: No tests found
  • keep benchmark results out of the N passed test counter so the summary stays semantically accurate

Before / After

Given a benchmark-only run, the old behavior could report:

Go test: No tests found

With this change, rtk now produces a benchmark-aware summary like:

✓ Go test: benchmark results in 1 packages
═══════════════════════════════════════

📦 benchfixture [benchmarks]
  BenchmarkFib10-10           9558400       128.3 ns/op

Why this approach

Go's test2json benchmark event model is incomplete and does not consistently provide benchmark lifecycle events the same way it does for tests. This change takes the smallest safe approach:

  • preserve the existing test/failure accounting
  • use benchmark output lines as the source of truth for benchmark-only runs
  • avoid overcounting benchmarks as test passes

Tests

  • add regression coverage for benchmark-only output with test-scoped benchmark lines
  • add regression coverage for benchmark-only output with package-level benchmark lines

Verification

Run from the repository root.

  • docker run --rm -v "$PWD:/work" -w /work rust:1.85 cargo test benchmark_only -- --nocapture
  • docker run --rm -v "$PWD:/work" -w /work rust:1.85 cargo test go_cmd::tests -- --nocapture
  • docker run --rm -v "$PWD:/work" -w /work rust:1.85 cargo build

@DevNewbie1826
Copy link
Author

A small reviewer note: the important design choice here is that this PR does not count benchmarks as test passes. Instead, it detects benchmark result lines and uses them only to avoid collapsing benchmark-only runs into Go test: No tests found.

I also covered both output shapes I could verify:

  • benchmark result attached to Test
  • benchmark result emitted as package-level output

So the main review questions are whether that summary behavior feels right for rtk, and whether the benchmark-line detection is scoped tightly enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant