Skip to content

feat(decisions): read test outcomes, script edits and subagents (0.23.0) - #58

Merged
taylorg009 merged 3 commits into
mainfrom
fix/subagent-decisions
Oct 21, 2026
Merged

taylorg009 merged 3 commits into
mainfrom
fix/subagent-decisions

Conversation

@taylorg009

@taylorg009 taylorg009 commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Why

"What the agent decided" (0.20.0) showed a line for 1 of 96 checkpointed commits on a real web-app repository since 2026-09-20. The rules assumed sessions that rarely happen:

  • Test outcome by exit status, bare runs only. Agents pipe runs (vitest … 2>&1 | tail), so the status is the pipe's and nothing counted. The status was also wrong both ways: a denied permission read as a failure, and a run whose output said Tests 4 failed exited 0.
  • Repo root from Edit/Write paths only. Agents that edit through python3 - <<EOF or cat > file had no root, so nothing was read.
  • Two roots meant nothing. Long sessions work in several worktrees, and some of those carry the same repo paths.
  • Subagents invisible. Entire 0.11.5 keeps their transcripts under tasks/<tool_use_id>/, and nothing read them.

What changes

  • A test run's outcome comes from the runner's own summary line (vitest/jest Tests, pytest ===, cargo test result:, go ok/FAIL), read and never shown. A bare run with no summary still uses its status, unless it never ran.
  • One run per command. A run followed by a step that puts files back (cp, mv, git apply/checkout/restore/stash, patch) is a probe and is skipped. T="path"; vitest run "$T" names path.
  • A run names test files only when it targets nothing but files the change carries. "Failed on its last run" is dropped when the same runner passed later in the window.
  • Root from Bash. A Bash call run in a known directory that names a changed file by its repo path gives that directory as the root. The directory is followed across calls; cd "$PWD" stays put. Temp folders are never a root.
  • Two roots. When there are two roots, the one the last git commit ran in wins, if it's among them.
  • Subagents. Their tool calls are read inside the parent's turn, right after the Agent call, for decisions only.

Evidence

The same 96 commits were replayed through checkpointFor with each commit's own files:

build commits with a line
0.22.0 1 (a design note)
this branch 20: 13 with failed, then passed, 6 failed on its last run, 4 with a design note (a commit can have several)

I spot-checked lines by hand against the transcripts:

  • Red check: a red check against swapped-in code no longer reads as "failed on its last run".
  • Earlier false claim: a run followed by echo/tsc steps now counts, which fixed a "failed on its last run" claim that turned out false during this work.
  • Directory runs: a run over a whole __tests__ folder no longer names one file.

Subagent transcripts are real. The repository's entire/checkpoints/v1 holds 50 tasks/<tool_use_id>/agent-<id>.jsonl files across 7 checkpoints written by Entire 0.11.5. So far all of them are read-only helper agents, so none adds a decision yet.

An independent review requested changes and then approved at 5540b0c. The fixes, each with a test that fails on the earlier code:

  • A bare run that exits non-zero stays failed. Vitest Test Files/Errors and jest Test Suites: lines count as failures.
  • go and cargo summary lines only ever mark a run as failed.
  • The summary patterns no longer backtrack. pytest took 55 s on a 30k-character line before.
  • cd/pushd are read with redirects stripped and with variables assigned earlier in the same command substituted, in order. popd, a bare cd, or a step that can't be read makes the directory unknown.
  • Only test files are named, so a --config file isn't.
  • A temp-folder root, such as an agent's scratchpad, is never the repo.

Tests: 422 passed (16 new cases in decisions.test.ts and checkpoint.test.ts, run against real git checkpoints).

🤖 Generated with Claude Code

The "What the agent decided" section almost never fired on real
sessions: agents pipe test runs, edit through scripts, work across
several worktrees, and delegate to subagents.

- A run's outcome comes from the runner's own summary in its output
  (vitest/jest, pytest, cargo, go), read and never shown; one run per
  command; probes against swapped-in code are skipped; variables the
  command assigns resolve to the files they name.
- Files are named only when the run targets nothing else, and a later
  pass of the same runner outranks an earlier failure.
- The repo root also comes from Bash calls run in a known directory
  (the shell's directory followed across calls) that name a changed
  file; two roots resolve to the one the last git commit ran in.
- Subagent transcripts Entire keeps under tasks/ are read inside the
  turn that started them, for decisions only.

Replayed on 96 checkpointed commits: 1 line before, 13 after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

PR brief: 3 agent commits · 8 of 25 test files reach this diff

Run at least these. Selection is a safe superset; it never marks a test safe to skip.

Summary PR #58 at be36328
Changed 7 files: 4 mapped to 20 symbols, 3 ignored by policy
Tests reached 8 of 25 ▮▮▮▮▮▮▯▯▯▯▯▯▯▯▯▯▯▯▯▯ 32% of the suite
Downstream code 8 files, 23 dependents · review effort high
Intent 3 of 3 commits attributed by trailer (claude-code)
Reviewed by no reviewer other than the author (taylorg009) so far · the 12 changed and reached files were last changed by Taylor Gagne (59 commits in 10 files), Taylor (15 commits in 10 files) · Taylor has no prior commit in 2 of the 12 files this change touches or reaches: src/checkpoint-write.ts, src/owners.ts
Symbols 12 changed, 10 added, 1 removed within the diff
Since push 5540b0c 1 new commit, 0 changed files, 0 tests reached
Compared against base 64f5992

What each commit did

Commit Intent Files Reaches Ran before push
fefd69d feat(decisions): read test outcomes, script edits and subag… claude-code by Co-authored-by 7 files 8 files, 8 tests
5540b0c fix(decisions): never pass a failed run; lose the directory… claude-code by Co-authored-by 5 files 8 files, 8 tests
be36328 fix(decisions): substitute a variable only from assignments… claude-code by Co-authored-by 2 files 8 files, 8 tests

Claims checked

  • refuted: PR body names echo. no changed symbol bears it and no changed line contains it (phantom change).
  • refuted: PR body names tsc. no changed symbol bears it and no changed line contains it (phantom change).
  • consistent: bash removed from src/decisions.test.ts. 1 static caller in the base graph, all in files this diff changes: src/decisions.test.ts.
  • consistent: the narrative names the changed code. all 3 files with symbol changes are named.

What each changed file reaches

Read Changed file Change Reaches Tests
1 decisions.ts Block changed, result changed, decisionsIn changed, +4 8 files 8
with decisions.ts checkpoint.ts checkpointFor changed, leaves changed, repoRoot changed, +11 8 files 7
2 checkpoint.test.ts test code, selected directly 1
3 decisions.test.ts test code, selected directly 1
3 files under the repo root ignored by policy 0

Paths above are under src/ unless shown in full.

Tests to run (8)

Blast radius by changed file: 23 dependents

blastline 0.23.0. Selection is advisory unless your workflow gates on it. How selection works. Graph 992ac5e. change-context budget 43432: omitted 0 impacts, 751 context entries (truncated); symbols are classified within the diff only. Intent: 0 of 3 commits, 3 attributed by trailer.

taylorg009 and others added 2 commits October 20, 2026 18:16
…it cannot follow

Review of #58:
- A bare run that exits non-zero stays failed whatever its summary says;
  vitest Test Files/Errors and jest Test Suites lines count as failures.
- go and cargo lines only ever say a run failed (a pipe can cut the
  failing package off).
- Summary patterns stay on one line; the pytest one backtracked
  cubically on long = lines (55 s on 30k characters).
- cd/pushd are read with redirects stripped and assigned variables
  substituted (WT=/x; cd "$WT"); popd, a bare cd or one that cannot be
  read makes the directory unknown, so a run in another repo is not
  this one's.
- Only test files are named (not a --config file); subagent ids may
  hold dashes; a temp-folder root is never the repo, pinned by a test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s use

A name assigned again later in the command no longer reaches back to an
earlier cd. Adds a test for cd to an assigned variable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@taylorg009
taylorg009 marked this pull request as ready for review October 21, 2026 01:21
@taylorg009
taylorg009 merged commit ced9903 into main Oct 21, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant