[P1][Performance] Excluir worktrees internos do escopo de mapeamento do projeto

[wesleysimplicio]

Contexto

O scan do simplicio-agent real inclui worktrees internos do próprio agente em .claude/worktrees/. Isso inflaciona o universo de arquivos e impede que o gate de orientação conclua dentro de um limite operacional razoável.

Evidência reproduzível

Ambiente: Windows, 2026-07-17.

1. Target: C:\Users\Z0059V7A\m\repos\simplicio-agent.
2. O macro map encontrou 8.482 arquivos e 81.972 símbolos; o módulo .claude sozinho aparece com 8.820 arquivos no inventário agregado por causa dos worktrees internos.
3. O scan síncrono via Runtime/source corrigido foi executado com --timeout 300.
4. Após exatamente 300s o envelope terminalizou com phase=timeout, failure_reason=scan_timeout, exit_code=1, lock recuperado e fresh=false.
5. O parser possui SKIP_DIRS para .git, node_modules, caches e artefatos, mas não exclui .claude/worktrees nem outros worktrees gerenciados.

Impacto

• O loop não consegue obter survey/handoff fresco do checkout de integração.
• O custo de tokens, memória e tempo inclui código duplicado de workers que não pertence ao produto avaliado.
• O timeout bounded evita hang, mas deixa o backlog bloqueado em mapping.
• Excluir indiscriminadamente .claude pode esconder instruções legítimas; a correção precisa distinguir configuração raiz de worktrees aninhados.

Plano passo a passo

1. Instrumentar o contador de arquivos por diretório raiz para confirmar o peso de .claude/worktrees em um checkout representativo.
2. Definir a política de escopo: preservar arquivos de configuração de .claude na raiz, ignorar apenas .claude/worktrees e worktrees equivalentes identificados por manifesto.
3. Implementar a filtragem na camada de walk, mantendo follow_symlinks=False e o comportamento de SKIP_DIRS existente.
4. Adicionar uma opção explícita de inclusão/override somente se consumidores legítimos precisarem mapear worktrees; não introduzir variável de ambiente nova para configuração comportamental.
5. Adicionar testes unitários de descoberta e um teste real contra uma fixture com .claude/settings.json e .claude/worktrees/worker/src.py.
6. Reexecutar o scan completo do Agent, comparando arquivos/símbolos, tempo, memória e freshness antes/depois.
7. Validar status --json, inspect --json e handoff --json com artefatos frescos.
8. Atualizar ADR/documentação e changelog se a política de escopo for considerada decisão arquitetural ou release-relevant.
9. Publicar pacote e verificar que o operador global consome a versão com a correção; relacionar com [P1][Release] Publicar e instalar versão com a correção de timeout do PR #232 #233.

Matriz de testes obrigatória

• fixture sem worktree: nenhuma regressão de contagem;
• .claude/settings.json e .claude/skills/*.md: preservados;
• .claude/worktrees/<name>/...: excluído;
• .git/worktrees e worktree file: não seguir nem duplicar conteúdo;
• symlink/junction para worktree: não seguir;
• múltiplos worktrees e nomes contendo espaços;
• scan sync normal, timeout, lock recovery e async status;
• contrato de macro map e handoff permanecem estáveis;
• runtime passthrough e executável instalado usam a mesma política.

Critérios de aceite

• A política de escopo está documentada e não remove configuração raiz legítima.
• Worktrees gerenciados não aparecem nos artefatos do projeto principal.
• O scan do checkout Agent conclui dentro do orçamento acordado ou produz timeout terminal reproduzível com causa explícita.
• fresh=true, terminal=true, lock=false e handoff.ready=true após scan bem-sucedido.
• Contagens, símbolos e relações fora do worktree continuam corretos.
• Testes unitários, integração/CLI e a execução real do Agent passam.
• A release publicada e a versão instalada são verificadas independentemente.
• Não há falso COMPLETE enquanto a issue de release [P1][Release] Publicar e instalar versão com a correção de timeout do PR #232 #233 permanecer sem prova.

[wesleysimplicio]

Fresh validation after minimal scope filter

Added a TDD regression test and a minimal filter that preserves .claude/settings.json but excludes .claude/worktrees in _should_skip_dir.

Evidence:

• Regression test before production change: failed because .claude/worktrees/worker/src/app.py was discovered.
• After change: focused test passed (1 passed in 0.08s).
• Focused parser/CLI/lock suite: 95 passed, 3 failed; the 3 failures are the existing Windows WinError 6 harness problem tracked in [P1][Windows] Background/index worker falha com WinError 6 ao herdar stdin inválido #231.
• ruff check on changed Python files: pass.
• py_compile and git diff --check: pass (with only a warning that ignored .simplicio/map-job.json has CRLF normalization).
• Real Agent scan using the patched source, --sync --timeout 300: still terminalized with phase=failed, failure_reason=scan_timeout, fresh=false, lock=false after 300 seconds.

Conclusion: the filter is correct and tested, but it is not sufficient to make the full Agent scan finish within 300s. The issue remains open for broader scope/performance investigation; no completion claim is made.

[wesleysimplicio]

Additional evidence — long-running worker after scope filter

The 900s real-Agent scan using commit 2e0a5b2 (worktree filter + TDD test) is still deep_running after 416s:

• PID 10972, live owner token/process-start identity;
• fresh=false, old artifacts unchanged;
• no child process visible;
• process is responsive but only ~11.6s CPU accumulated and ~404 MB resident memory;
• the existing background-index.log has an old timestamp from 2026-07-16, so its repeated index skipped lines are stale evidence and must not be used as current root cause.

This narrows the remaining investigation: the worker is not currently proven deadlocked by the lock layer, but it is spending several minutes in a low-CPU phase with no progress receipt. Add phase/progress instrumentation around git status, inventory, artifact emission and graph generation before changing behavior. The run remains bounded at 900s and must not be treated as a fresh handoff until terminal success.

[wesleysimplicio]

Final bounded-run evidence — 900s

The real Agent scan with the worktree filter and --sync --timeout 900 reached the bound and returned a terminal envelope:

• phase=timeout / status phase=failed;
• failure_reason=scan_timeout;
• exit_code=1;
• fresh=false;
• lock=false, no orphan lock;
• no fresh project-map/symbol-index/call-graph artifacts.

The filter itself is correct (the collector no longer traverses the 14,467 files under .claude/worktrees), but the remaining Agent deep path still cannot finish within 15 minutes and exposes no phase progress receipt. This issue is therefore not resolved by the minimal filter; further instrumentation/optimization is required before a PR can claim the acceptance criteria.

[wesleysimplicio]

Fresh phase-boundary diagnosis (2026-07-17)

This is a new bounded measurement after the worktree-scope filter. The issue remains OPEN and no closure or PR claim is made.

Reproduction and environment

• Repository under test: simplicio-agent
• Mapper source: simplicio-mapper branch codex/issue-234-scope-worktrees
• Installed public binary remains simplicio-mapper 0.23.1
• Runtime-first invocation was attempted through simplicio shell compact; Python phase instrumentation required native read-only fallback because the Runtime passthrough rejected Python and exposed the command as simplicio compat. This fallback is explicitly UNVERIFIED as Runtime parity.
• The corrected collector excludes all files below .claude/worktrees: 14,467 files are present there, while _collect_text_files(agent_root) returned 8,486 text files and 0 nested-worktree files.

Measured phase evidence

1. git status --porcelain --untracked-files=all: 0.459s, 0 entries, no degraded flag.
2. _collect_text_files: 0.134s, 8,486 files.
3. _tree_signature: 0.548s.
4. The full inventory, with the disk cache replaced by a no-op in-memory adapter for diagnosis only, processed 8,486 files / 95,768,640 bytes in 45.631s.
5. In the same run: _detect_changed_files 0.018s; architecture signals 0.019s; symbol index 5.564s.
6. _build_call_graph did not emit its END receipt after more than 150s. At the last sample the process was responsive, CPU was about 61s, and working set about 371MB. The controlled diagnostic process was then stopped; no mapping artifacts were produced.
7. A no-cache inventory-only run completed in 121.772s, confirming that parsing itself is material but not sufficient to explain the 900s scan bound.

Code-level hypothesis to verify

The call graph currently scans every file, extracts call expressions, then _nearest_symbol filters and sorts the complete symbol list for each call. This is a measured hot-path candidate, not yet a proven complexity claim. The previous scope filter only changes _collect_text_files; it does not add phase budgets, progress receipts, or a graph-specific bound.

Required implementation plan

1. Add deterministic phase receipts/counters around inventory, symbol index, call graph, architecture inventory, retrieval index, and artifact writes; receipts must identify root, file count, byte count, elapsed time, and terminal reason.
2. Add a focused regression fixture with repeated symbols/calls that proves the optimized graph has the same edge set and ordering as the current semantics.
3. Implement the smallest indexed lookup that removes repeated full-symbol filtering/sorting without dropping legitimate call edges; preserve the 1,000-edge output cap and confidence fields.
4. Add benchmark coverage for the fixture and a representative Agent subset, recording elapsed time, peak memory where available, file count, symbol count, call count, and edge count.
5. Re-run the real simplicio-agent scan through the public operator with --sync and bounded timeouts (30s, 300s, 900s), requiring a fresh terminal receipt and fresh artifacts before considering the issue resolved.
6. Only then publish a PR, run the complete relevant suite, and update the linked Agent E2E issue.

Acceptance criteria

• Every deep phase emits a machine-readable start/end or terminal-failure receipt; a timeout identifies the active phase.
• Unit tests prove edge-set/order equivalence for the optimized graph and cover zero-symbol, duplicate-symbol, recursive, and imported-call cases.
• Integration test runs the real mapper command against a temporary git repository and verifies fresh project-map, symbol-index, call-graph, and status artifacts.
• System test runs the public CLI/operator on the real Agent checkout with a fresh terminal envelope.
• Regression suite is green, including existing Windows process/handle tests; unrelated WinError 6 failures remain tracked separately in [P1][Windows] Background/index worker falha com WinError 6 ao herdar stdin inválido #231.
• Benchmark records a before/after number and demonstrates completion within the accepted bound without silently truncating required graph data.
• Touched-file line/branch coverage is at least 85% (90% branch target where reported).
• No generated .simplicio artifacts, stale lock, or unverified native fallback is presented as completion evidence.

Status: diagnosis only; keep #234 open until all criteria above have receipts.