diff --git a/.recursive/evaluations/0020.md b/.recursive/evaluations/0020.md new file mode 100644 index 0000000..d7e03e7 --- /dev/null +++ b/.recursive/evaluations/0020.md @@ -0,0 +1,39 @@ +# Evaluation #0020 +**Date**: 2026-04-09 +**Target**: Phractal (github.com/fazxes/Phractal) +**Agent**: claude +**Cycles**: 2 +**After task**: archived #0243 (Phractal eval rerun) +**Eval gate status**: BLOCKED (score < 80) + +## Scores + +| Dimension | Score | Notes | +|-----------|-------|-------| +| Startup | 0/10 | Default run cloned and ran baseline verification, but both cycles failed before any agent work because the spawned `claude` subprocess could not launch inside the current Claude Code session. | +| Discovery | 0/10 | No fixes or logged issues were produced. | +| Fix quality | 0/10 | No candidate fixes were committed. | +| Shift log | 3/10 | Shift log exists, but it is still the template stub with no real fix entries, commit hashes, or recommendations. | +| State file | 8/10 | Valid JSON with the expected keys. The file clearly records the failure state (`agent_failures=2`, `cycles[*].status=agent_failed`, `halt_reason` populated), even though no fixes were produced. | +| Verification | 5/10 | Baseline verification ran successfully, but neither cycle reached post-cycle verification. | +| Guard rails | 9/10 | No blocked paths were touched and the run stayed within the declared limits. | +| Clean state | 10/10 | The target repo clone is clean at the end of the run. | +| Breadth | 0/10 | No cycle completed, so no breadth was demonstrated. | +| Usefulness | 0/10 | The run produced no actionable fixes; the only usable artifact is the failure evidence itself. | +| **Total** | **35/100** | Eval gate BLOCKED. | + +## Tasks Created + +- #0277: Make Claude eval reruns scorable from Claude Code sessions (Startup 0/10, Discovery 0/10, Fix quality 0/10, Verification 5/10, Breadth 0/10, Usefulness 0/10) + +## Raw Evidence + +- Default attempt command: `PYTHONPATH=$(pwd) python3 -m nightshift test --agent claude --cycles 2 --cycle-minutes 5 --repo-dir /tmp/nightshift-eval-0020` +- Retry attempt command: `env -u CLAUDECODE PYTHONPATH=$(pwd) python3 -m nightshift test --agent claude --cycles 2 --cycle-minutes 5 --repo-dir /tmp/nightshift-eval-0020` +- Baseline verification: `python3 -m compileall apps/api/app` passed both attempts. +- Shift log: `/var/folders/dm/48s5j0qs7yq4nvq9s61jcbqw0000gn/T/nightshift-test-runs/nightshift-eval-0020-47e0cf34a31c/worktree-2026-04-09/Docs/Nightshift/2026-04-09.md` +- State file: `/var/folders/dm/48s5j0qs7yq4nvq9s61jcbqw0000gn/T/nightshift-test-runs/nightshift-eval-0020-47e0cf34a31c/2026-04-09.state.json` +- Runner log: `/var/folders/dm/48s5j0qs7yq4nvq9s61jcbqw0000gn/T/nightshift-test-runs/nightshift-eval-0020-47e0cf34a31c/2026-04-09.runner.log` +- State observations: `agent_failures=2`, `fixes=0`, `issues_logged=0`, `files_touched=0`, `tests_written=0`, `cycles[0].status=agent_failed`, `cycles[1].status=agent_failed`, `halt_reason="Agent command failed twice in a row."` +- Runner failure: the nested Claude invocation error repeated in both attempts, so no scorable cycle output was produced. + diff --git a/.recursive/tasks/.next-id b/.recursive/tasks/.next-id index 2681747..3d242f5 100644 --- a/.recursive/tasks/.next-id +++ b/.recursive/tasks/.next-id @@ -1 +1 @@ -277 +278 diff --git a/.recursive/tasks/0277.md b/.recursive/tasks/0277.md new file mode 100644 index 0000000..ec0f716 --- /dev/null +++ b/.recursive/tasks/0277.md @@ -0,0 +1,27 @@ +--- +status: pending +priority: urgent +target: v0.0.9 +vision_section: self-maintaining +created: 2026-04-09 +source: evaluation-0020 +completed: +--- + +# Make Claude eval reruns scorable from Claude Code sessions + +## Problem + +The fresh Phractal eval rerun for `#0020` could not complete in this environment because the spawned `claude` subprocess failed inside an existing Claude Code session. Unsetting `CLAUDECODE` was not enough to produce a scorable 2-cycle run. + +## Fix + +Make the eval runner or launch path resilient to Claude-in-Claude execution so a default `nightshift test --agent claude --cycles 2 --cycle-minutes 5` rerun can complete and produce a report without manual shell workarounds. + +## Acceptance Criteria + +- [ ] `nightshift test --agent claude --cycles 2 --cycle-minutes 5` can complete from a Claude Code shell without the nested-session failure +- [ ] If Claude-in-Claude remains unsupported, the runner automatically falls back to a supported agent or emits a clear, actionable failure before starting cycles +- [ ] Regression coverage exists for the nested-session path +- [ ] A fresh eval rerun produces a scorable report instead of halting after two agent failures +