Repository navigation
tell tool-cache churn apart from work the agent did - #64
Merged
Merged
Conversation
Four consecutive steps, each `python3 -m pytest -q`, authoring nothing, with the
coarse detector set to fire after four unchanged steps -- it stayed silent,
because pytest wrote four `.pytest_cache` files on step one. Repeatedly
re-running the suite while getting nowhere is the most ordinary way an agent
stalls, and that is exactly where the signal went blind. The same paths travel
into the fine judge's evidence, so the judge read cache churn as work.
This is the claim the README rests on: a flat segment is a stretch of steps that
provably bought nothing. A cache write filled the flat segment in.
Classify, do not filter. `changed_paths` keeps every path, in the same order,
byte-identical in both observers; a new `tool_cache_paths` names the classified
subset and travels through WorkspaceDelta, StepOutcome and the Terminus
boundary; `no_file_change` decides on the authored complement; and the judge
evidence carries both, with the preamble saying which is a subset of which.
Nothing is discarded, so the raw observation stays recoverable -- which is the
whole reason "just filter the cache out" was the wrong fix.
The rule is a named set of directories a tool writes as a side effect of
running: __pycache__, .pytest_cache, .ruff_cache, .mypy_cache, .tox. Dependency
directories are deliberately excluded, because creating or deleting node_modules
is real work. Classification is a heuristic and says so where it lives.
Review caught that the first attempt left the defect unfixed on Harbor, the
backend the experiments actually run on. That observer emits an entry per
directory including the workspace root, and a directory's hash includes its
mtime, so creating a cache directory bumps its parent and the parent is not
itself a cache path -- leaving authored=('.', 'tests') and the detector still
silent. The local observer was immune only by accident: it skips non-file
entries, and LocalEnvironment sets PYTHONDONTWRITEBYTECODE so __pycache__ never
appears there. A directory whose every changed descendant is cache is now
classified with them, the root included.
The parity test that should have caught that could not: it hand-built identical
snapshots for both observers, so it was structurally blind to the one way they
differ. It now drives the real manifest parser.
Also from review: the subset validation was quadratic and ran three times per
step (1.3s at 20,000 changed paths, which one `tox -r` reaches), now set-based;
the ordered-subset property the docstring claimed is now enforced rather than
described; and the classifier is exported alongside the observers it belongs to.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four consecutive steps, each
python3 -m pytest -q, authoring nothing, with the coarse detector set to fire after four unchanged steps:Four wasted steps and the detector stayed silent, because pytest wrote four cache files on the first one. Repeatedly re-running the suite while getting nowhere is the most ordinary way an agent stalls, and that is precisely where the signal went blind. The same paths travel into the fine judge's evidence (
judges.py:132), so the judge read cache churn as work the agent did.This is the claim the README rests on — a flat segment is a stretch of steps that provably bought nothing. A cache write filled the flat segment in.
Classify, do not filter
changed_pathskeeps every path, in the same order, byte-identical in both observers. A newtool_cache_pathsnames the classified subset and travels throughWorkspaceDelta→StepOutcome→ the Terminus boundary.no_file_changedecides on the authored complement. The judge evidence carries both, with one preamble clause saying which is a subset of which.Nothing is discarded, so the raw observation stays recoverable — which is the whole reason "just filter the cache out" was the wrong fix, and why this was put to a decision rather than patched quietly.
The rule is a named set of directories a tool writes as a side effect of running:
__pycache__,.pytest_cache,.ruff_cache,.mypy_cache,.tox. Dependency directories are deliberately excluded — creating or deletingnode_modulesis real work. Classification is a heuristic and says so where it lives.Review caught that the first attempt left Harbor broken
Harbor is the observer the experiments actually run on. It emits an entry per directory including the workspace root, and a directory's hash includes its mtime — so creating a cache directory bumps its parent, and the parent is not itself a cache path:
PurePosixPath(".").parts == (), so the root could never be classified by any segment rule at all. The local observer was immune only by accident: it skips non-file entries, andLocalEnvironmentsetsPYTHONDONTWRITEBYTECODEso__pycache__never appears there. Neither accident transfers.A directory whose every changed descendant is cache is now classified with them, the root included. Verified through the real manifest parser:
()('.', 'tests', 'tests/test_a.py')The parity test that should have caught this could not. It hand-built identical snapshots for both observers, so it was structurally blind to the one way they differ. It now drives the real manifest parser and fails without the fix.
Also from review
tox -rreaches. Now set-based at all three sites.tool_cache_pathswas accepted. Now rejected.classify_tool_cache_pathsandTOOL_CACHE_DIRECTORY_NAMESare exported alongside the observers they belong to.Gates
pytest1295 passed, 3 skipped (baseline 1286).ruff check .andruff format --check .exit 0. Tests: 241 additions, 0 deletions across five files — no existing test modified, skipped or mocked.Gate 4, both observers:
no_file_changefires;changed_pathsstill lists all four cache entriesecho idlestallno_file_changefires, unchangedScope
harbor_agent.py,harbor_native_agent.pyandnative_lhtb.pyare untouched.changed_pathsand the diff string are unchanged in both observers, so the only measurement change is the added evidence key and the detector's input — which is the change that was approved.