perf(core): ShadowSnapshot burns minutes of CPU when cwd contains a large ignored tree (target/, node_modules/)

[BunsDev]

Summary

ShadowSnapshot (crates/core/src/snapshot/shadow.rs) can stall a session for minutes and burn tens of CPU-minutes when the working directory contains a large ignored tree (e.g. a Rust target/ dir). Observed: coven-code --print "..." run from src-rust/ in this repo produced no output for 4+ minutes while consuming ~39 CPU-minutes across the tokio worker pool; a sample trace put all the time under ShadowSnapshot in String/slice equality and tokio::fs::metadata.

Root cause (two compounding defects)

1. The repo-root .gitignore is invisible when cwd is a subdirectory.
The shadow repo's --work-tree is the session cwd. When that cwd is a subdirectory of the real repository (here src-rust/, while the src-rust/target/ rule lives in the repo root .gitignore), ls-files --others --exclude-standard in stage() never sees the rule — gitignore chains above the worktree root are not consulted, and sync_excludes() only copies the user-level exclude file. Result: stage() enumerates the entire target/ tree (hundreds of thousands of files).

2. Quadratic ignore intersection.
check_ignore_user() then pipes that list to git check-ignore --stdin -z (which correctly flags all of it against the real repo) but intersects the result with:

let raw: Vec<&str> = r.text.split('\0').filter(|s| !s.is_empty()).collect();
files.iter().copied().filter(|f| raw.contains(f)).collect()

raw.contains(f) is a linear scan per file — O(n*m) string comparisons. With n ~ m ~ 10^5+, that is ~10^10 compares per snapshot, which matches the hot frames in the sample (slice_contains, String::eq, Vec::partial_eq).

Impact

• Any session whose cwd has a large ignored tree (Rust target/, node_modules/, build output) pays this on every turn: first token latency of minutes and sustained multi-core spin.
• Correctness is unaffected (ignored files do not leak into patches) — this is purely a performance defect.

Suggested fixes

1. One-liner for the quadratic term: collect raw into a HashSet<&str> before the intersection in check_ignore_user().
2. Stop enumerating ignored trees at all: make the shadow repo honor the ignore rules the real repository would apply to the worktree — e.g. seed the shadow's info/exclude from the real repo's effective ignore decisions for the worktree (or run the untracked-file enumeration against the real repo instead of the shadow when the cwd is inside one).

Fix 1 removes the CPU blowup; fix 2 removes the pointless tokio::fs::metadata crawl of ~10^5 files. Both are worth doing.

Repro

cd <coven-code checkout>/src-rust   # large target/ present, ignored only by the parent .gitignore
env -u ANTHROPIC_API_KEY ./target/debug/coven-code --print "Reply with exactly one word: pong"
# stalls for minutes before the first token; sample(1) shows ShadowSnapshot hot

Found while smoke-testing fb989ab (claude CLI runtime); unrelated to that change.

[covencat]

Cody dogfood result

Status: failure

Cody could not complete the task.

Cody

Could not complete the requested change.

Files changed

No files changed.

Evidence

• No PR review evidence was captured for this run.

Structured review

• Mode: none
• Evidence status: not_applicable
• Reviewed files: 0
• Supporting files inspected: 0
• Findings: 0
• Tests reported by runtime: 0
• Limitations: 0

Files changed: 0
Commits: 0

Task fe67c83e-7b9c-11f1-92c8-d3221cdd377d. Publication is enabled on the hosted test adapter only.

[BunsDev]

Defect 2 (the quadratic intersections) is fixed in 10bf3ef: both check_ignore_user() and the untracked-membership test in stage() now use HashSet lookups instead of Vec::contains.

Measured on this repo from src-rust/ (coven-code --print, no API key):

• before: 4+ minutes with no output, ~39 CPU-minutes across the worker pool; sampled runs never finished
• after: completes in 159s wall / 67 CPU-s ("pong" round-trips); sample shows zero slice_contains/String::eq frames

The remaining 159s is defect 1 — the shadow worktree rooted at src-rust/ cannot see the repo-root .gitignore, so stage() still enumerates and hashes the whole target/ tree. Keeping this issue open for that.

[BunsDev]

Correction and additional data on the numbers above, after re-measuring without a stale shadow index.lock:

• The "completes in 159s" figure was measured while stage()/write-tree were aborting early on a leftover index.lock (from earlier killed runs), so it covers enumeration + the metadata crawl but not full staging. The CPU claim stands: quadratic frames are gone from samples and CPU dropped from ~39 CPU-min to ~67 CPU-s for that phase.
• With the lock cleared, the first snapshot from src-rust/ proceeds to git add the enumerated tree — and target/ here is 202 GB. The run was still I/O-bound hashing after 13 minutes (~9 MB written to the shadow object store); projected completion is an hour-plus, and it would copy build artifacts into ~/Library/Application Support/coven-code/snapshot/.

So defect 1 is worse than a slowdown: when the cwd sits below the repo-root .gitignore, the shadow snapshot attempts to ingest the entire ignored build tree (time + disk). The HashSet fix (10bf3ef) removes the CPU blowup but cannot help with that; the enumeration/ignore-visibility fix is the real remedy.

Workaround until then: run coven-code from the repository root (where .gitignore is visible to the shadow worktree), or add a local ignore for build dirs in the subdirectory.

[BunsDev]

Both defects are now fixed:

• Defect 2 (quadratic intersections): 10bf3ef
• Defect 1 (ignore-rule visibility): b7700cf — check_ignore_user() discovers the enclosing repo with -C so the full gitignore chain applies, and stage() seeds the shadow's info/exclude from git ls-files --others --ignored --exclude-standard --directory so ignored trees are never enumerated.

Final measurement from src-rust/ (202 GB ignored target/), coven-code --print with no API key:

state	wall	CPU	behavior
before	4+ min, never completed	~39 CPU-min	spun in quadratic ignore filtering, then attempted a 202 GB ingest
after 10bf3ef	unbounded	~67 CPU-s (filter phase)	still attempted the 202 GB ingest
after b7700cf	10.7 s	1.6 CPU-s	claude-CLI roundtrip time; target/ invisible to the snapshot

Regression test parent_gitignore_respected_in_subdirectory_worktree covers the subdirectory-worktree case for both patch contents and staged state.