Version: 0.9.1, dev @ 715b454 · harness claude-code · Weeknight fixture @ b6d269c
Filed together because each is small; split if preferred.
1. teardown-guard reports "nothing to remove" while env guards are armed
finalize ended with:
⚠ Guard still armed — run `eval-magic teardown` to disarm before editing source.
Running the narrower command instead gave a false all-clear:
$ eval-magic teardown-guard --iteration 1
No write guard was installed — nothing to remove.
Both task environments still held live guards at that moment:
env-g1-with_skill/.claude/settings.local.json → 1 PreToolUse hook
env-g1-with_skill/.claude/skills/.slow-powers-eval-guard.json
→ {"active": true, "expiresAt": "2026-08-28T09:04:26.324Z", …}
env-g1-without_skill/… → same
The full eval-magic teardown then removed both correctly, so the capability
exists — teardown-guard just doesn't reach the per-(group, condition) env
guards. run --help already describes teardown as disarming "the cwd guard and
every per-(group, condition) Cli env's guard", so the narrower command appears
to cover only the first.
The message is the real problem: teardown-guard --help says it is for use
"mid-run, before hand-editing files the guard would block", which is exactly when
a false "nothing to remove" is most costly. Either widen it to the env guards, or
report honestly — "removed 0 cwd guards; 2 env guards remain, run
eval-magic teardown".
2. The scratch directory the framework designates is counted in diff scope
Every dispatch prompt says:
Task-local scratch directory: <eval-root>/tmp
- Keep temporary and scratch files in the task-local scratch directory, not in a
host temp directory.
The agent complies, and everything it writes there is then measured as part of
its change. In our control arm, tmp/ accounted for 3 of 11 files touched and
228 of 296 lines added:
tmp/IMPLEMENTATION_SUMMARY.md +63
tmp/integration-pantry-test.test.ts +76
tmp/test-pantry-staples.test.ts +89
Consequences:
- A
diff_scope assertion can fail on scratch files alone. Ours (max_files_touched: 14,
max_lines_changed: 700) passed, but a tighter, reasonable bound would have been
tripped by throwaway notes.
diff.patch is the judge's primary evidence, so judges read scratch files as if
they were the deliverable.- The two arms are not symmetric here in practice: whether scratch files count
depends on whether that agent happened to use the directory it was told to use.
docs codebase states that "Framework artifacts under .eval-magic-outputs/
never count". The framework-designated scratch directory reasonably belongs in
the same category. Simplest fix: put the scratch directory under
.eval-magic-outputs/tmp (already excluded), or exclude <eval-root>/tmp from
diff measurement the same way.
Judging this genuinely useful as a signal is a defensible alternative position —
but then it should be documented, because the prompt currently reads as an
instruction to put files somewhere harmless.
3. Teardown's "delete manually" path predates the workspace relocation
or delete .eval-magic/working-with-tdd/ manually to discard.
No such path exists. The workspace was
~/.local/share/eval-magic/skills-c61a1930/working-with-tdd/, and every other
command in the same message carried the correct absolute --workspace-dir.
src/cli/commands/workspace.rs:185 builds it from a hardcoded literal:
Path::new(".eval-magic").join(&ctx.skill_name).display()That was right before the eval home moved out of the skill repo. It should be
ctx.workspace_root.join(&ctx.skill_name) — the value the same function already
has in scope and already prints via command_target_args.
Found during a pre-release shakedown of the #244 work: a four-skill treatment (hardening-plans, working-in-isolation, working-with-tdd, verifying-development-work) implementing a real feature in the Weeknight fixture, driven end to end through run → dispatch → ingest → compare → judges → finalize on claude-code with a haiku-4-5 agent and sonnet-5 judges.
Version: 0.9.1,
dev@715b454· harnessclaude-code· Weeknight fixture @b6d269cFiled together because each is small; split if preferred.
1.
teardown-guardreports "nothing to remove" while env guards are armedfinalizeended with:Running the narrower command instead gave a false all-clear:
Both task environments still held live guards at that moment:
The full
eval-magic teardownthen removed both correctly, so the capabilityexists —
teardown-guardjust doesn't reach the per-(group, condition)envguards.
run --helpalready describes teardown as disarming "the cwd guard andevery per-
(group, condition)Cli env's guard", so the narrower command appearsto cover only the first.
The message is the real problem:
teardown-guard --helpsays it is for use"mid-run, before hand-editing files the guard would block", which is exactly when
a false "nothing to remove" is most costly. Either widen it to the env guards, or
report honestly — "removed 0 cwd guards; 2 env guards remain, run
eval-magic teardown".2. The scratch directory the framework designates is counted in diff scope
Every dispatch prompt says:
The agent complies, and everything it writes there is then measured as part of
its change. In our control arm,
tmp/accounted for 3 of 11 files touched and228 of 296 lines added:
Consequences:
diff_scopeassertion can fail on scratch files alone. Ours (max_files_touched: 14,max_lines_changed: 700) passed, but a tighter, reasonable bound would have beentripped by throwaway notes.
diff.patchis the judge's primary evidence, so judges read scratch files as ifthey were the deliverable.
depends on whether that agent happened to use the directory it was told to use.
docs codebasestates that "Framework artifacts under.eval-magic-outputs/never count". The framework-designated scratch directory reasonably belongs in
the same category. Simplest fix: put the scratch directory under
.eval-magic-outputs/tmp(already excluded), or exclude<eval-root>/tmpfromdiff measurement the same way.
Judging this genuinely useful as a signal is a defensible alternative position —
but then it should be documented, because the prompt currently reads as an
instruction to put files somewhere harmless.
3. Teardown's "delete manually" path predates the workspace relocation
No such path exists. The workspace was
~/.local/share/eval-magic/skills-c61a1930/working-with-tdd/, and every othercommand in the same message carried the correct absolute
--workspace-dir.src/cli/commands/workspace.rs:185builds it from a hardcoded literal:That was right before the eval home moved out of the skill repo. It should be
ctx.workspace_root.join(&ctx.skill_name)— the value the same function alreadyhas in scope and already prints via
command_target_args.Found during a pre-release shakedown of the #244 work: a four-skill treatment (
hardening-plans,working-in-isolation,working-with-tdd,verifying-development-work) implementing a real feature in the Weeknight fixture, driven end to end through run → dispatch → ingest → compare → judges → finalize onclaude-codewith a haiku-4-5 agent and sonnet-5 judges.