Skip to content

A fork's judge tells agents what really happened: failed build steps, broken neighbours, and code an #if leaves out - #178

Open
silvervest wants to merge 6 commits into
n-delic:mainfrom
silvervest:fix/fork-judge-verdicts
Open

silvervest wants to merge 6 commits into
n-delic:mainfrom
silvervest:fix/fork-judge-verdicts

Conversation

@silvervest

@silvervest silvervest commented Nov 2, 2026 •

Copy link
Copy Markdown

Merge order: first of a stack of three. #179 and #180 build on this PR (they carry its commits until it merges), and #185 needs it too.

Bug fix.

What a fork's judge and brief told agents that wasn't so, found on an x86 game started from scratch:

  • A unit built in steps counted as building when a step failed. failed_units matched the build's failed targets against each unit's object, but with ninja -k 0 a failed step means the object is never tried. An edit_file that broke a unit's stand-ins was kept as breaking nothing; the next compile found the unit broken, was told it had been broken before, and the agent lost its claim: 302 times over 156 functions in a week. failed_units now follows every target made from a failed one (ninja -t query) up to the unit's object.
  • A compile that breaks a matched neighbour lost its own score. Where a function's code changes how its neighbours compile (inlined into them, or whole-program optimisation), a draft short of exact breaks a matched neighbour and the compile is undone. The rejection now gives the function's own score and diff, says the neighbours match again once it matches exactly, and records the attempt (never as the best attempt, which would be put back in the file). Agents of 63 functions had written about this.
  • NOT IN YOUR BUILD named the row, not the symbol. A function whose row name was made unique with a trailing ' was reported as "its object doesn't define _f@4'"; agents took the quote for the defect and stopped looking.
  • Code an #if leaves out read as the function's. Agents parked drafts of 13 functions (about 18 KB) in #if defined(SOME_FLAG) blocks no build defines, often because, where the build sees them, they break a matched neighbour. The brief showed each as the function's current source with its score, and every compile replaced the definition inside the block, where the judge never saw it. csource.hidden_by follows a file's conditionals and says which branch leaves a position out, only when it can be sure (ForkProject.never_defined: macros nothing could define; only looked for when code would otherwise be hidden). The brief marks a hidden definition, and a compile puts the agent's version where the build sees it, by its place in the original, without the parked copy, carrying the copy's reccmp annotation and comment along.
  • (The rejection's list of broken functions no longer reuses the variable holding the function's names.)

Tests: test_a_unit_whose_build_step_fails_counts_as_not_building, test_not_in_your_build_names_the_symbol_not_the_rows_unique_name, test_which_if_branch_leaves_code_out_is_told_only_when_certain and test_a_macro_counts_as_never_defined_only_when_nothing_could_define_it (tests/test_forks.py); test_a_change_that_breaks_a_neighbour_still_says_how_close_the_function_came and test_a_definition_the_build_leaves_out_is_said_and_compiled_where_the_build_sees_it (tests/test_fork_judge.py). ruff check src tests projects is clean.

🤖 Generated with Claude Code

Nick Silvestro added 6 commits November 4, 2026 18:28
…nly its object

failed_units matched the build's failed targets against each unit's object. A unit built in steps (parts
compiled, then linked into its object) reports only the step that failed: with -k 0, ninja never tries the
object. On an x86 game, edit_file kept a new file of stand-ins that didn't compile as breaking nothing; the
agent's next compile then found the unit broken, was told it had been broken before the change, and lost its
claim. That happened 302 times over 156 functions in a week. failed_units now also counts every target
made from a failed one (ninja -t query, followed up to the unit's object), so the edit is undone with the
build's error instead.
…nction came

Where a function's code changes how its neighbours compile (inlined into them, or whole-program optimisation
choosing registers across functions), a draft short of exact breaks a matched neighbour, and the compile is
undone. The function's own score went with it: on an x86 game, agents of 63 functions wrote about this, and
parked drafts behind #if flags they rebuilt by hand just to measure them.

The rejection now gives the function's score and diff, says that such neighbours match again once it matches
exactly, and records the attempt with its score. It never becomes the best attempt (best=False in _record):
briefs offer the best to start from and a release puts it back in the file, where this code can't stay. Nothing
counts as a match that didn't before.
…ng the function's list of names

The message's list of broken functions was put in `names`, the variable that holds the ways the function may be spelled. Nothing read it after (the branch returns), but the next change to that branch could have.
…ion's unique name

A function whose row name was made unique with a trailing "'" (a re-sync after a split) was reported as
"its object doesn't define _f@4'". Agents took the quote for the defect, said no C makes such a name, and
stopped looking for why the function really wasn't in the object.
…never compiled into

On an x86 game, agents parked drafts of 13 functions (about 18 KB) in `#if defined(SOME_FLAG)` blocks no build
defines. The brief showed each as the function's current source, the score as its own, and every compile
replaced the definition inside the block, where the judge never saw it.

csource.hidden_by follows a file's conditionals and says which branch leaves a position out, only when it can
be sure: a test of a macro that may be defined (the compiler's own, one the build passes) hides nothing.
ForkProject.never_defined finds the macros nothing could define (no #define in the project's sources, no
mention in its build files, -D/ included); that search runs only when the code would be hidden with every such
macro undefined, so files with no parked code pay nothing for it. The brief now marks a hidden definition, and a
compile that would replace one is refused before anything is built, with how to bring it out.
… where the build sees it

Refusing to compile into a parked definition left an agent stuck: on an x86 game the drafts were parked because,
where the build sees them, they break a matched neighbour (whole-program optimisation ties their code together),
so edit_file refused to move them out too, and the agent could do nothing with the function at all.

A compile now puts the agent's version where the build sees it, by its place in the original, without the parked
copy, and carries the copy's reccmp annotation (and its comment, unless the agent sent one) along with it. Kept, the
function is visible from then on; broken neighbours undo it, and say how close the function came. The brief says
so. csource.marker_above finds the annotation above a definition's comment.
@silvervest
silvervest force-pushed the fix/fork-judge-verdicts branch from 8332118 to 376087d Compare November 4, 2026 08:18

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant