Skip to content

tell tool-cache churn apart from work the agent did - #64

Merged
hyj28 merged 1 commit into
mainfrom
tool-cache-churn
Sep 28, 2026
Merged

hyj28 merged 1 commit into
mainfrom
tool-cache-churn

Conversation

@hyj28

@hyj28 hyj28 commented Sep 28, 2026

Copy link
Copy Markdown
Owner

Four consecutive steps, each python3 -m pytest -q, authoring nothing, with the coarse detector set to fire after four unchanged steps:

step 1: changed_paths=('.pytest_cache/.gitignore', '.pytest_cache/CACHEDIR.TAG',
                       '.pytest_cache/README.md', '.pytest_cache/v/cache/nodeids')
step 2: changed_paths=()
step 3: changed_paths=()
step 4: changed_paths=()

no_file_change fired: False

Four wasted steps and the detector stayed silent, because pytest wrote four cache files on the first one. Repeatedly re-running the suite while getting nowhere is the most ordinary way an agent stalls, and that is precisely where the signal went blind. The same paths travel into the fine judge's evidence (judges.py:132), so the judge read cache churn as work the agent did.

This is the claim the README rests on — a flat segment is a stretch of steps that provably bought nothing. A cache write filled the flat segment in.

Classify, do not filter

changed_paths keeps every path, in the same order, byte-identical in both observers. A new tool_cache_paths names the classified subset and travels through WorkspaceDelta → StepOutcome → the Terminus boundary. no_file_change decides on the authored complement. The judge evidence carries both, with one preamble clause saying which is a subset of which.

Nothing is discarded, so the raw observation stays recoverable — which is the whole reason "just filter the cache out" was the wrong fix, and why this was put to a decision rather than patched quietly.

The rule is a named set of directories a tool writes as a side effect of running: __pycache__, .pytest_cache, .ruff_cache, .mypy_cache, .tox. Dependency directories are deliberately excluded — creating or deleting node_modules is real work. Classification is a heuristic and says so where it lives.

Review caught that the first attempt left Harbor broken

Harbor is the observer the experiments actually run on. It emits an entry per directory including the workspace root, and a directory's hash includes its mtime — so creating a cache directory bumps its parent, and the parent is not itself a cache path:

changed:  ('.', '.pytest_cache', '.pytest_cache/CACHEDIR.TAG', 'tests', 'tests/__pycache__', ...)
authored: ('.', 'tests')          ← non-empty, detector still silent

PurePosixPath(".").parts == (), so the root could never be classified by any segment rule at all. The local observer was immune only by accident: it skips non-file entries, and LocalEnvironment sets PYTHONDONTWRITEBYTECODE so __pycache__ never appears there. Neither accident transfers.

A directory whose every changed descendant is cache is now classified with them, the root included. Verified through the real manifest parser:

scenario authored
pure cache creation ()
same, plus one real edit ('.', 'tests', 'tests/test_a.py')

The parity test that should have caught this could not. It hand-built identical snapshots for both observers, so it was structurally blind to the one way they differ. It now drives the real manifest parser and fails without the fix.

Also from review

  • The subset validation was quadratic and ran three times per step — 1.3 s at 20,000 changed paths, which a single tox -r reaches. Now set-based at all three sites.
  • The ordered-subset property the docstring claimed was not enforced; a reordered tool_cache_paths was accepted. Now rejected.
  • classify_tool_cache_paths and TOOL_CACHE_DIRECTORY_NAMES are exported alongside the observers they belong to.

Gates

pytest 1295 passed, 3 skipped (baseline 1286). ruff check . and ruff format --check . exit 0. Tests: 241 additions, 0 deletions across five files — no existing test modified, skipped or mocked.

Gate 4, both observers:

local, cache-only churn no_file_change fires; changed_paths still lists all four cache entries
local, real file written no signal
local, genuine echo idle stall no_file_change fires, unchanged
Harbor, pure cache creation authored empty
Harbor, cache plus a real edit the edited file stays authored

Scope

harbor_agent.py, harbor_native_agent.py and native_lhtb.py are untouched. changed_paths and the diff string are unchanged in both observers, so the only measurement change is the added evidence key and the detector's input — which is the change that was approved.

Four consecutive steps, each `python3 -m pytest -q`, authoring nothing, with the
coarse detector set to fire after four unchanged steps -- it stayed silent,
because pytest wrote four `.pytest_cache` files on step one. Repeatedly
re-running the suite while getting nowhere is the most ordinary way an agent
stalls, and that is exactly where the signal went blind. The same paths travel
into the fine judge's evidence, so the judge read cache churn as work.

This is the claim the README rests on: a flat segment is a stretch of steps that
provably bought nothing. A cache write filled the flat segment in.

Classify, do not filter. `changed_paths` keeps every path, in the same order,
byte-identical in both observers; a new `tool_cache_paths` names the classified
subset and travels through WorkspaceDelta, StepOutcome and the Terminus
boundary; `no_file_change` decides on the authored complement; and the judge
evidence carries both, with the preamble saying which is a subset of which.
Nothing is discarded, so the raw observation stays recoverable -- which is the
whole reason "just filter the cache out" was the wrong fix.

The rule is a named set of directories a tool writes as a side effect of
running: __pycache__, .pytest_cache, .ruff_cache, .mypy_cache, .tox. Dependency
directories are deliberately excluded, because creating or deleting node_modules
is real work. Classification is a heuristic and says so where it lives.

Review caught that the first attempt left the defect unfixed on Harbor, the
backend the experiments actually run on. That observer emits an entry per
directory including the workspace root, and a directory's hash includes its
mtime, so creating a cache directory bumps its parent and the parent is not
itself a cache path -- leaving authored=('.', 'tests') and the detector still
silent. The local observer was immune only by accident: it skips non-file
entries, and LocalEnvironment sets PYTHONDONTWRITEBYTECODE so __pycache__ never
appears there. A directory whose every changed descendant is cache is now
classified with them, the root included.

The parity test that should have caught that could not: it hand-built identical
snapshots for both observers, so it was structurally blind to the one way they
differ. It now drives the real manifest parser.

Also from review: the subset validation was quadratic and ran three times per
step (1.3s at 20,000 changed paths, which one `tox -r` reaches), now set-based;
the ordered-subset property the docstring claimed is now enforced rather than
described; and the classifier is exported alongside the observers it belongs to.
@hyj28
hyj28 merged commit 0d730a7 into main Sep 28, 2026
2 checks passed
@hyj28
hyj28 deleted the tool-cache-churn branch September 28, 2026 01:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant