fix(run): hide framework-staged files from the codebase's own tooling (#296)

[slowdini]

Closes #296.

The problem

Staged skills live inside the task repository, under the harness's skills dir. A codebase whose lint or format step globs the whole tree therefore reported eval-magic's own artifacts as project failures — and only in the arm that stages a skill, because the control arm has nothing to find.

That is asymmetric by construction: any command_check running the project's checks was biased against with_skill. Worse, the agent under test runs those same checks, so it saw a permanently red step listing files it did not create and must not touch. A run meant to measure honest self-verification was measuring the framework instead.

The change

run writes a delimited block into the project's own ignore files, naming the paths the runner placed:

# >>> eval-magic framework files >>>
# Staged by `eval-magic run` so this project's own tooling does not report them.
# See `eval-magic docs codebase`.
/.eval-magic-outputs/
/.claude/skills/
/.claude/settings.local.json
# <<< eval-magic framework files <<<

• Paths are descriptor-derived — skills_dir plus the file the guard stages (hooks_file, or plugin_file for a plugin engine), plus the framework outputs dir. A BYOH harness contributes its own without configuring anything.
• Written into every environment — both arms, every repetition, --no-stage and --dry-run alike. An entry present in one arm only would trade one asymmetry for another. It lands before the baseline commit, so it never appears in a run's diff-scope.
• Which files receive it is detected from the codebase's tooling through a new packaged ignore-profiles/ family, embedded by build.rs the way guard-profiles/ already is. Each profile declares its markers and whether its file may be created when the project has none — ESLint's may not, since ESLint 9's flat config no longer reads .eslintignore.
• Both profile families now share one walk, extracted to src/core/tree_profiles.rs.
• codebase.ignore_files names the files outright when detection cannot find them; [] opts out. Paths may not escape the environment.

.gitignore is deliberately never a target: the baseline force-adds harness config dirs, so an entry there would hide nothing from Git — but it would hide the staged skills from every .gitignore-aware tool the agent uses, such as rg, damaging the treatment arm instead of protecting it.

No artifact records the effective list. The written file is committed into each environment's baseline, so git show refs/eval-magic/baseline:.prettierignore is the evidence, per arm and per run. run also prints the files it wrote.

Before / after

The shipped Weeknight fixture (.prettierrc.json, no .prettierignore of its own), npm run lint in env-g1-with_skill:

$ npm run lint            # before
[warn] .claude/skills/slow-powers-eval-1-with_skill__demo/SKILL.md
[warn] Code style issues found in the above file. Run Prettier with --write to fix.
exit 1

$ npm run lint            # after
Checking formatting...
All matched files use Prettier code style!
exit 0

Both arms carry a byte-identical .prettierignore, and the control arm still passes.

Schema / CLI / docs

• schema/evals.schema.json — ignore_files on both codebase shapes.
• eval-magic run --help — a paragraph beside the exclude_skill_sources note.
• eval-magic docs codebase — new "Framework files stay out of the project's own tooling" section: the detected set, the override, the .gitignore rationale, and where the evidence lives.
• eval-magic docs byoh and harnesses/template.toml — one line telling descriptor authors that skills_dir and the guard's staged file are what the entries derive from.
• docs/developer_overview.md — repository-map entry for ignore-profiles/.

Verification

cargo fmt --check                        clean
cargo clippy --all-targets -D warnings   clean
cargo test                               1363 passed, 0 failed
git diff --check                         clean

Plus the end-to-end run above against the real fixture, driven through run on claude-code.

New coverage: unit tests for the shared detection walk and for the writer (creation, idempotent rewrite, create_if_missing = false, declared-list override, opt-out, a directory in the way, symlinks inside and escaping the environment, packaged-profile integrity); validation tests for ignore_files containment; tests/run/ignore_files.rs for arm symmetry, baseline membership, the declared list, the opt-out, and a codebase with no matching tooling; and a docs contract test.

🤖 Generated with Claude Code

https://claude.ai/code/session_018H2oD4ry5huonXVyrpDGFj