graph: stop the fix loop early when impl writes nothing - #156
Merged
Conversation
FixLoopDef.setup ran unconditionally: SequenceNode([impl, test])
always ran test after impl, even when impl never called write or edit
at all. Reproduced live: glm-5.3-flash spent its entire step budget
deliberating in plain assistant text (second-guessing Lex syntax,
drafting code in prose) without ever writing the target file, and the
pipeline still ran test (120 steps) and verify (84 steps) against a
file that was never created -- 298 steps burned before the final
`lex check` criterion in task_spec.lex caught it, the only thing that
ever did.
wrote_or_edited(events) scans a turn's trail for a cap.completed event
naming the write or edit tool -- same shape as verify_found_failure's
existing FAIL-substring scan, reading what the dispatcher recorded
rather than trusting the model's own closing text. run_setup runs
setup's first stage with event tracking when the shape allows it (an
AgentNode alone, or the head of a SequenceNode -- the exact shape
every current FixLoopDef.setup uses) and stops immediately, before
test or the mechanical verify_program ever run, if nothing was
written. Any other setup shape (ParallelNode, FixLoopNode, empty
SequenceNode) falls back to the previous unconditional run_node, so
this only ever adds coverage, never narrows it.
Verified two ways: wrote_or_edited's own examples{} block (lex check),
and a live integration test in a throwaway worktree -- a real task
answerable by reading alone ("how many functions does this file
define") correctly stopped after "impl done -- 16 steps" with no test
or verify stage following, versus the unguarded behavior that would
have run both anyway.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
FixLoopDef.setup(SequenceNode([impl, test])) ran unconditionally —testalways ran afterimpl, even whenimplnever calledwriteoreditat all.glm-5.3-flashspent its entire step budget deliberating in plain assistant text (second-guessing Lex syntax, drafting code in prose) without ever writing the target file, and the pipeline still rantest(120 steps) andverify(84 steps) against a file that was never created — 298 steps burned before the finallex checkcriterion intask_spec.lexcaught it, the only thing that ever did.wrote_or_edited(events)scans a turn's trail for acap.completedevent naming thewriteoredittool — same shape asverify_found_failure's existing FAIL-substring scan, reading what the dispatcher recorded rather than trusting the model's own closing text.run_setuprunssetup's first stage with event tracking when the shape allows it (anAgentNodealone, or the head of aSequenceNode— the exact shape every currentFixLoopDef.setupuses) and stops immediately, beforetestor the mechanicalverify_programever run, if nothing was written. Any other setup shape falls back to the previous unconditionalrun_node, so this only ever adds coverage, never narrows it.Test plan
lex check(full sweep),lex fmt --check,lex doc-sync --check,lex test(unit suite) all passwrote_or_edited's ownexamples {}block covers write/edit/read/invoked-not-completedimpl done — 16 stepswith notest/verifystage following, versus the unguarded behavior that would have run both anyway🤖 Generated with Claude Code