testbot: target 3 files per run instead of 1 - #1289
Conversation
Every recent testbot PR covered exactly one source file (#1273, #1278, #1280, #1283, #1288 — the second changed file is just the BUILD entry). Raise max_targets 1 -> 3. max_turns goes 200 -> 400 to match: the generator runs the full read/write/verify/coverage loop per target, and a single target has already hit the limit once (runs/26536045087, 101/100). max_uncovered stays at 500. It is a per-target cap, and recent targets carried 35-231 uncovered lines, so it is not the binding constraint -- max_targets is. Schedule and timeout are unchanged. Also refresh the README table, which still documented max_turns=100 and timeout_minutes=30 against the workflow's 200 and 60. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughTestbot defaults now allow up to 3 targets and 400 agent turns. README guidance documents these values and a 60-minute workflow timeout. ChangesTestbot limits
Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/testbot.yaml:
- Line 214: Update the workflow step containing the max-targets argument so
inputs.max_targets is passed through the step’s env as MAX_TARGETS, validated as
a decimal integer before use, and supplied to the command as the quoted
"$MAX_TARGETS" value; remove direct shell interpolation of inputs.max_targets
while preserving the existing default of 3.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 464a01cc-c29f-47a9-a33f-ec7cafa8afae
📒 Files selected for processing (2)
.github/workflows/testbot.yamlsrc/scripts/testbot/README.md
| PYTHONPATH=. python src/scripts/testbot/select_targets_agent.py \ | ||
| --shortlist "$RUNNER_TEMP/shortlist.json" \ | ||
| --max-targets ${{ inputs.max_targets || '1' }} \ | ||
| --max-targets ${{ inputs.max_targets || '3' }} \ |
There was a problem hiding this comment.
🔒 Security & Privacy | 🔴 Critical | ⚡ Quick win
Do not interpolate inputs.max_targets into the shell command.
GitHub Actions expands this expression before the shell runs. A user who can dispatch the workflow can provide shell metacharacters and alter the generated command. The command can then execute arbitrary code in a step that has ANTHROPIC_API_KEY.
Pass the value through env, validate it as a decimal integer, and quote "$MAX_TARGETS".
Proposed fix
- name: Pick targets via subagent (Stage 2)
run: |
+ if ! [[ "$MAX_TARGETS" =~ ^[0-9]+$ ]]; then
+ echo "max_targets must be a non-negative integer, got: $MAX_TARGETS" >&2
+ exit 2
+ fi
PYTHONPATH=. python src/scripts/testbot/select_targets_agent.py \
--shortlist "$RUNNER_TEMP/shortlist.json" \
- --max-targets ${{ inputs.max_targets || '3' }} \
+ --max-targets "$MAX_TARGETS" \
--output "$RUNNER_TEMP/targets.txt" \
--meta-output "$RUNNER_TEMP/targets_meta.json"
env:
+ MAX_TARGETS: ${{ inputs.max_targets || '3' }}
ANTHROPIC_API_KEY: ${{ secrets.NVIDIA_NIM_KEY }}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| --max-targets ${{ inputs.max_targets || '3' }} \ | |
| - name: Pick targets via subagent (Stage 2) | |
| run: | | |
| if ! [[ "$MAX_TARGETS" =~ ^[0-9]+$ ]]; then | |
| echo "max_targets must be a non-negative integer, got: $MAX_TARGETS" >&2 | |
| exit 2 | |
| fi | |
| PYTHONPATH=. python src/scripts/testbot/select_targets_agent.py \ | |
| --shortlist "$RUNNER_TEMP/shortlist.json" \ | |
| --max-targets "$MAX_TARGETS" \ | |
| --output "$RUNNER_TEMP/targets.txt" \ | |
| --meta-output "$RUNNER_TEMP/targets_meta.json" | |
| env: | |
| MAX_TARGETS: ${{ inputs.max_targets || '3' }} | |
| ANTHROPIC_API_KEY: ${{ secrets.NVIDIA_NIM_KEY }} |
🧰 Tools
🪛 zizmor (1.29.0)
[error] 214-214: code injection via template expansion (template-injection): may expand into attacker-controllable code
(template-injection)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/testbot.yaml at line 214, Update the workflow step
containing the max-targets argument so inputs.max_targets is passed through the
step’s env as MAX_TARGETS, validated as a decimal integer before use, and
supplied to the command as the quoted "$MAX_TARGETS" value; remove direct shell
interpolation of inputs.max_targets while preserving the existing default of 3.
Source: Linters/SAST tools
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1289 +/- ##
==========================================
+ Coverage 70.82% 70.95% +0.12%
==========================================
Files 239 239
Lines 28501 28501
Branches 4298 4298
==========================================
+ Hits 20186 20222 +36
+ Misses 7491 7456 -35
+ Partials 824 823 -1
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
Superseded by #1290, which combines this with the other open testbot changes. |
1 participant
Description
Testbot PRs have been consistently small in scope — every recent one covers exactly one source file:
(The second changed file is the
BUILDentry, not a second target.)Changes
max_targets1 → 3, on both the dispatch input default and the schedule-path fallback. Nothing hardcodes the count —SELECT_TARGETS_PROMPT.mdalready renderspick at most {max_targets}and the generator prompt already loops per target — so this is a pure parameter change.max_turns200 → 400. The generator runs the full read/write/verify/coverage loop per target, and a single target has already exhausted the limit once (runs/26536045087, 101/100, after context auto-compaction).max_turns=100andtimeout_minutes=30against the workflow's actual 200 and 60.Deliberately unchanged
max_uncoveredstays 500. It is a per-target cap (_cap_rangesis applied inside the per-file loop inparse_coverage_entries, withremainingreset per file), and recent targets carried only 35–231 uncovered lines. Raising it would change nothing;max_targetsis the binding constraint. The per-PR ceiling still rises 500 → 1500 as a side effect of targeting 3 files.timeout_minutes. Note the workflow is hourly withconcurrency.cancel-in-progress: true, so a run is cancelled when the next hourly trigger fires — the effective wall-clock ceiling is ~60 min regardless oftimeout_minutes. If 3-target runs start getting cancelled mid-generation, the fix is to slow the cron rather than raise the timeout.Verification
bazel test //src/scripts/testbot:all //src/scripts/testbot/tests:all— 19 targets pass, including-pylintsiblings. Workflow YAML parses with the expected defaults.Issue - None
Checklist
🤖 Generated with Claude Code
Summary by CodeRabbit