Repository navigation
fix: accept a later simulated-user script revision in a lane, and disclose it - #25
Merged
Merged
Conversation
…close it The simulated-user protocol digest hashes the whole script, so an unrelated edit made every later report incompatible with a lane that ran before it. A later report is now accepted when the protocol and every setting match; the summary flags those reports and lists a limitation. The edit that triggered this added an optional parameter for visual-evidence grading and left the simulated user's own path unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A campaign lane accepts a later report whose simulated-user
protocol_digestdiffers from the original run's, provided the protocol name and every other setting match. Each such report is flaggedsimulated_user_revisedin the lane's members and the lane gains a limitation saying so. A different model, effort or protocol is still refused.Why
protocol_digestis a hash of the wholeSimulated_User.pyfile. Since the GPT-6 Astra lane ran in September, that file gained one optional parameter,present_visual_evidence=False, used by visual-evidence grading; the simulated user's own request path is unchanged (10 insertions, 4 deletions). The hash changed anyway, so no new report could join the lane:continuation report has incompatible conditions.This follows the existing rule for a corrected runtime adapter: disclosed, not hidden.
Testing
New test: a later revision is accepted and flagged; a different simulated-user model is still refused.
🤖 Generated with Claude Code