Repository navigation
stop the agent from observing the code it just replaced - #61
Merged
Merged
Conversation
Found by running the tool-calling agent end to end on a real broken repository
rather than through the test suite. The agent read the source, applied a correct
one-character fix through edit_file, re-ran the tests, and observed the
pre-edit behaviour:
cat m.py -> return 1 - 1 (the file IS patched)
python3 -c "import m; print(m.f())" -> 2 (the OLD code runs)
CPython validates a cached .pyc against the source mtime truncated to whole
seconds plus its size. An agent edits and re-runs milliseconds later, and a
same-length replacement is the common shape of a bug fix -- + to -, < to >,
== to !=. Both edit_file and write_file reproduce it; nothing about this is
specific to the newer tool.
The consequence is the failure mode this project exists to prevent, produced by
its own tooling: a correct fix reports as broken, so the agent reverts it or
thrashes, and a reward-stall rollback can then discard the one good change.
LocalEnvironment already builds a deliberate environment around every command,
so PYTHONDONTWRITEBYTECODE joins it and no cache is ever written. Both residual
gaps are stated beside the existing not-a-sandbox note rather than left implied:
a workspace that already ships .pyc files can still execute stale bytecode, and
the remote/Harbor path does not route through this class at all.
Scoped to local.py on purpose. The archived 170-trial run went through the
Harbor remote environment, and nothing here touches it.
A side effect worth naming: __pycache__ entries no longer appear in
changed_paths, where they were being counted as work the agent had done. Four
.pytest_cache entries still are, which is a measurement-semantics question
rather than a defect, and is left alone here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by actually running the tool-calling agent end to end on a real broken repository — real workspace, real subprocesses through
LocalEnvironment, the real runner — rather than through the test suite.The agent read the source, applied a correct one-character fix, re-ran the tests, and observed the pre-edit behaviour. It concluded its own correct patch had failed and gave up.
CPython validates a cached
.pycagainst the source mtime truncated to whole seconds plus its size. An agent edits and re-runs milliseconds later, and a same-length replacement is the ordinary shape of a bug fix —+→-,<→>,==→!=. Bothedit_fileandwrite_filereproduce it; nothing here is specific to the newer tool.The consequence is the failure mode this project exists to prevent, produced by its own tooling: a correct fix reports as broken, so the agent reverts it or thrashes, and a reward-stall rollback can then discard the one good change it made.
The fix
LocalEnvironmentalready builds a deliberate environment around every command (HOME,LANG,LC_ALL,PATH).PYTHONDONTWRITEBYTECODE=1joins it, so the cache is never written and every import compiles from source.Both residual gaps are stated beside the existing not-a-sandbox note rather than left implied: a workspace that already ships
.pycfiles can still execute stale bytecode, and the remote/Harbor path does not route through this class at all.Scoped to
local.pyon purpose. The archived 170-trial run went through the Harbor remote environment; nothing here touches it, so its replayability is unaffected.Before and after, same scenario
FAILED test_subtract - assert 14 == 63 passedchanged_pathsafter onepytestA second finding, deliberately not fixed here
LocalWorkspaceDeltaObserverexcludes only.git, so tool caches are hashed into the snapshot. A step that authored nothing was recorded as having changed six files:That pollutes
changed_pathsand thegit_viewdiff the fine judge reads, and suppressesno_file_changeon the step that most often accomplishes nothing — degrading the signal behind "a flat segment is a stretch of steps that provably bought nothing".This change removes the
__pycache__half as a side effect. The four.pytest_cacheentries remain. Separating tool-cache churn from agent-authored change alters what the judge sees, which is a measurement-semantics decision rather than a defect, and it should be made deliberately — and by reporting the two classes separately rather than silently dropping one, per this repository's own rule about unobservable versus empty.Gates
pytest1281 passed, 3 skipped (baseline 1278 / 3).ruff check .andruff format --check .exit 0. Two files, +84 lines, no deletions; no existing test modified.The new tests drive real subprocesses at the
LocalEnvironment.execseam. Falsifiability verified by removing the flag in a scratch copy withPYTHONPATHforced and the import path confirmed to resolve to the copy: the regression test exits 1, observingoldinstead ofnew.