Skip to content

docs(eval): use conservative Terminal-Bench concurrency defaults - #2646

Draft
chadvoegele wants to merge 1 commit into
mainfrom
docs/terminal-bench-concurrency-eight
Draft

chadvoegele wants to merge 1 commit into
mainfrom
docs/terminal-bench-concurrency-eight

Conversation

@chadvoegele

Copy link
Copy Markdown
Contributor

What does this PR do?

Type of change: documentation.

Chad's Agent: split the concurrency changes out of #2475. Set Terminal-Bench 2.1's starting concurrency to 8 per shard, keeping max_concurrent and sandbox.concurrency equal in the recipe, example, and canary commands.

This is a conservative default, not a benchmark mandate or proven improvement. Capacity adjustments require latency/queueing evidence and matched comparison settings; benchmark limits and scoring remain unchanged. This PR is independent of the bounded-failure policy in #2475.

Usage

max_concurrent: 8
sandbox:
  concurrency: 8

Testing

  • Evaluation skill tests: 7 passed.
  • YAML parsing and recipe/example concurrency and repeat consistency: passed.
  • Changed-file pre-commit hooks: passed.
  • git diff --check: passed.
  • No live benchmark or Harbor rerun; timeout improvement is unverified.

Before your PR is "Ready for review"

Contributor and security guidance read; commit signed and signed off.

  • Is this change backward compatible?: ✅ — config schema unchanged; concurrency default reduced.
  • Copied code or new PIP dependency?: N/A
  • New necessary tests?: N/A — config/documentation only; consistency checked locally.
  • Changelog updated?: N/A
  • Claude approval?: ❌ — not requested; draft.

Additional Information

Concurrency-only split from #2475. No evaluation jobs submitted.

Signed-off-by: Chad Voegele <cvoegele@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 2, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Oct 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 68.76%. Comparing base (e4884ba) to head (d991a80).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2646   +/-   ##
=======================================
  Coverage   68.75%   68.76%           
=======================================
  Files         618      618           
  Lines       69455    69455           
=======================================
+ Hits        47752    47758    +6     
+ Misses      21703    21697    -6     
Flag Coverage Δ
unit 58.71% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant