testbot: include the test rules in the prompt, default to Opus 5 - #1276
Merged
Conversation
PR #1253 opened with 9 pylint errors (8x W0212 protected-access on _upload_worker_input_generator, 1x C2801 unnecessary-dunder-call) and failed ci-internal on //src/lib/data/storage/tests:test_uploading-pylint. It is not the first: #1184 needed a "fix pylint failures" follow-up and #1245 a lint-fix commit. The generator trace for run/30499270675 shows why. Of its 12 Bash calls, none contained "pylint" -- it ran `bazel test <single target>`, which runs the test but not the sibling `-pylint` target that osmo_py_test generates and that PR CI runs. Its own TodoWrite plan had no style step at all. And of its 23 Read calls, none opened TESTBOT_RULES.md, the only place that documents either the lint command or the "test PUBLIC behavior only" rule that the W0212s violate. Two fixes, both removing the need for the model to remember something: 1. TESTBOT_RULES.md now says to run `bazel test //<package>:all` instead of naming a single target. The lint sibling lives in the same package, so one command covers both and there is no second step to skip. This matches what the coverage step already tells the agent to do (`bazel coverage //src/<package>:all`). Also documents the two pylint failures generated tests actually hit. 2. The workflow now splices TESTBOT_RULES.md into the prompt instead of referencing it by path, the same way coverage targets are inlined. An instruction nothing verifies is followed only sometimes. Verified: `bazel query 'tests(//src/lib/data/storage/tests:all)'` lists test_uploading-pylint, so :all would have caught #1253 before the PR existed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughThe testbot workflows and scripts now default to Claude Opus 5. The generation workflow appends ChangesTestbot configuration and prompts
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Wording only, no behavior change. The rationale for the change lives in the previous commit message and PR #1276; the inline comment and prompt text only need to say what to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Revert the TESTBOT_RULES.md verification changes and trim the added comment and prompt wording. The remaining change is the minimum fix: the prompt carries the rules instead of referencing them by path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3 tasks
Switch the testbot generate and respond paths from bedrock-claude-opus-4-7 to bedrock-claude-opus-5. draft-release-notes still defaults to 4-7 and is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cypres
approved these changes
Aug 7, 2026
jiaenren
enabled auto-merge (squash)
August 7, 2026 23:55
tdewanNvidia
approved these changes
Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two changes to the testbot harness.
1. Put the test rules in the prompt. Testbot PRs keep failing
ci-internalon pylint — #1253 opened with 9 errors, #1184 needed a "fix pylint failures" follow-up, #1245 a lint-fix commit.TESTBOT_PROMPT.mdtold the generator to readTESTBOT_RULES.md, which is the only place documenting thebazel test <target>-pylintverification step and the "Test PUBLIC behavior only. Never call underscore-prefixed methods" rule. In the trace for run/30499270675 the generator made 23Readcalls and opened that file in none of them, so it followed neither.The workflow now passes the rules in with the prompt, the same way coverage targets already are. No change to the rules themselves.
2. Default to Opus 5.
bedrock-claude-opus-4-7→bedrock-claude-opus-5across the testbot generate and respond paths (workflow input default, bothANTHROPIC_MODELfallbacks,testbot-respond.yaml,select_targets_agent.py,respond.py, README).draft-release-notes.yamlstill defaults to 4-7 and is unchanged.Issue - None
Checklist
🤖 Generated with Claude Code
Summary by CodeRabbit
Tests
Documentation