Skip to content

stop the agent from observing the code it just replaced - #61

Merged
hyj28 merged 1 commit into
mainfrom
pycache-staleness
Sep 25, 2026
Merged

hyj28 merged 1 commit into
mainfrom
pycache-staleness

Conversation

@hyj28

@hyj28 hyj28 commented Sep 25, 2026

Copy link
Copy Markdown
Owner

Found by actually running the tool-calling agent end to end on a real broken repository — real workspace, real subprocesses through LocalEnvironment, the real runner — rather than through the test suite.

The agent read the source, applied a correct one-character fix, re-ran the tests, and observed the pre-edit behaviour. It concluded its own correct patch had failed and gave up.

before edit:  m.py mtime_ns=1790374821453574050  size=26
after  edit:  m.py mtime_ns=1790374821684801818  size=26     (+231ms, same second, same size)

cat m.py                              ->  return 1 - 1       the file IS patched
python3 -c "import m; print(m.f())"   ->  2                  the OLD code executes

CPython validates a cached .pyc against the source mtime truncated to whole seconds plus its size. An agent edits and re-runs milliseconds later, and a same-length replacement is the ordinary shape of a bug fix — +→-, <→>, ==→!=. Both edit_file and write_file reproduce it; nothing here is specific to the newer tool.

The consequence is the failure mode this project exists to prevent, produced by its own tooling: a correct fix reports as broken, so the agent reverts it or thrashes, and a reward-stall rollback can then discard the one good change it made.

The fix

LocalEnvironment already builds a deliberate environment around every command (HOME, LANG, LC_ALL, PATH). PYTHONDONTWRITEBYTECODE=1 joins it, so the cache is never written and every import compiles from source.

Both residual gaps are stated beside the existing not-a-sandbox note rather than left implied: a workspace that already ships .pyc files can still execute stale bytecode, and the remote/Harbor path does not route through this class at all.

Scoped to local.py on purpose. The archived 170-trial run went through the Harbor remote environment; nothing here touches it, so its replayability is unaffected.

Before and after, same scenario

before after
agent's own verification FAILED test_subtract - assert 14 == 6 3 passed
steps to finish 7, ending in give-up 6, completed
changed_paths after one pytest 6 cache files 4

A second finding, deliberately not fixed here

LocalWorkspaceDeltaObserver excludes only .git, so tool caches are hashed into the snapshot. A step that authored nothing was recorded as having changed six files:

changed_paths=('.pytest_cache/.gitignore', '.pytest_cache/CACHEDIR.TAG', '.pytest_cache/README.md',
               '.pytest_cache/v/cache/nodeids', '__pycache__/m.cpython-314.pyc',
               '__pycache__/test_m.cpython-314-pytest-9.1.1.pyc')

That pollutes changed_paths and the git_view diff the fine judge reads, and suppresses no_file_change on the step that most often accomplishes nothing — degrading the signal behind "a flat segment is a stretch of steps that provably bought nothing".

This change removes the __pycache__ half as a side effect. The four .pytest_cache entries remain. Separating tool-cache churn from agent-authored change alters what the judge sees, which is a measurement-semantics decision rather than a defect, and it should be made deliberately — and by reporting the two classes separately rather than silently dropping one, per this repository's own rule about unobservable versus empty.

Gates

pytest 1281 passed, 3 skipped (baseline 1278 / 3). ruff check . and ruff format --check . exit 0. Two files, +84 lines, no deletions; no existing test modified.

The new tests drive real subprocesses at the LocalEnvironment.exec seam. Falsifiability verified by removing the flag in a scratch copy with PYTHONPATH forced and the import path confirmed to resolve to the copy: the regression test exits 1, observing old instead of new.

Found by running the tool-calling agent end to end on a real broken repository
rather than through the test suite. The agent read the source, applied a correct
one-character fix through edit_file, re-ran the tests, and observed the
pre-edit behaviour:

    cat m.py                             ->  return 1 - 1   (the file IS patched)
    python3 -c "import m; print(m.f())"  ->  2              (the OLD code runs)

CPython validates a cached .pyc against the source mtime truncated to whole
seconds plus its size. An agent edits and re-runs milliseconds later, and a
same-length replacement is the common shape of a bug fix -- + to -, < to >,
== to !=. Both edit_file and write_file reproduce it; nothing about this is
specific to the newer tool.

The consequence is the failure mode this project exists to prevent, produced by
its own tooling: a correct fix reports as broken, so the agent reverts it or
thrashes, and a reward-stall rollback can then discard the one good change.

LocalEnvironment already builds a deliberate environment around every command,
so PYTHONDONTWRITEBYTECODE joins it and no cache is ever written. Both residual
gaps are stated beside the existing not-a-sandbox note rather than left implied:
a workspace that already ships .pyc files can still execute stale bytecode, and
the remote/Harbor path does not route through this class at all.

Scoped to local.py on purpose. The archived 170-trial run went through the
Harbor remote environment, and nothing here touches it.

A side effect worth naming: __pycache__ entries no longer appear in
changed_paths, where they were being counted as work the agent had done. Four
.pytest_cache entries still are, which is a measurement-semantics question
rather than a defect, and is left alone here.
@hyj28
hyj28 merged commit d471fff into main Sep 25, 2026
2 checks passed
@hyj28
hyj28 deleted the pycache-staleness branch September 25, 2026 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant