Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .recursive/evaluations/0020.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Evaluation #0020
**Date**: 2026-04-09
**Target**: Phractal (github.com/fazxes/Phractal)
**Agent**: claude
**Cycles**: 2
**After task**: archived #0243 (Phractal eval rerun)
**Eval gate status**: BLOCKED (score < 80)

## Scores

| Dimension | Score | Notes |
|-----------|-------|-------|
| Startup | 0/10 | Default run cloned and ran baseline verification, but both cycles failed before any agent work because the spawned `claude` subprocess could not launch inside the current Claude Code session. |
| Discovery | 0/10 | No fixes or logged issues were produced. |
| Fix quality | 0/10 | No candidate fixes were committed. |
| Shift log | 3/10 | Shift log exists, but it is still the template stub with no real fix entries, commit hashes, or recommendations. |
| State file | 8/10 | Valid JSON with the expected keys. The file clearly records the failure state (`agent_failures=2`, `cycles[*].status=agent_failed`, `halt_reason` populated), even though no fixes were produced. |
| Verification | 5/10 | Baseline verification ran successfully, but neither cycle reached post-cycle verification. |
| Guard rails | 9/10 | No blocked paths were touched and the run stayed within the declared limits. |
| Clean state | 10/10 | The target repo clone is clean at the end of the run. |
| Breadth | 0/10 | No cycle completed, so no breadth was demonstrated. |
| Usefulness | 0/10 | The run produced no actionable fixes; the only usable artifact is the failure evidence itself. |
| **Total** | **35/100** | Eval gate BLOCKED. |

## Tasks Created

- #0277: Make Claude eval reruns scorable from Claude Code sessions (Startup 0/10, Discovery 0/10, Fix quality 0/10, Verification 5/10, Breadth 0/10, Usefulness 0/10)

## Raw Evidence

- Default attempt command: `PYTHONPATH=$(pwd) python3 -m nightshift test --agent claude --cycles 2 --cycle-minutes 5 --repo-dir /tmp/nightshift-eval-0020`
- Retry attempt command: `env -u CLAUDECODE PYTHONPATH=$(pwd) python3 -m nightshift test --agent claude --cycles 2 --cycle-minutes 5 --repo-dir /tmp/nightshift-eval-0020`
- Baseline verification: `python3 -m compileall apps/api/app` passed both attempts.
- Shift log: `/var/folders/dm/48s5j0qs7yq4nvq9s61jcbqw0000gn/T/nightshift-test-runs/nightshift-eval-0020-47e0cf34a31c/worktree-2026-04-09/Docs/Nightshift/2026-04-09.md`
- State file: `/var/folders/dm/48s5j0qs7yq4nvq9s61jcbqw0000gn/T/nightshift-test-runs/nightshift-eval-0020-47e0cf34a31c/2026-04-09.state.json`
- Runner log: `/var/folders/dm/48s5j0qs7yq4nvq9s61jcbqw0000gn/T/nightshift-test-runs/nightshift-eval-0020-47e0cf34a31c/2026-04-09.runner.log`
- State observations: `agent_failures=2`, `fixes=0`, `issues_logged=0`, `files_touched=0`, `tests_written=0`, `cycles[0].status=agent_failed`, `cycles[1].status=agent_failed`, `halt_reason="Agent command failed twice in a row."`
- Runner failure: the nested Claude invocation error repeated in both attempts, so no scorable cycle output was produced.

2 changes: 1 addition & 1 deletion .recursive/tasks/.next-id
Original file line number Diff line number Diff line change
@@ -1 +1 @@
277
278
27 changes: 27 additions & 0 deletions .recursive/tasks/0277.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
status: pending
priority: urgent
target: v0.0.9
vision_section: self-maintaining
created: 2026-04-09
source: evaluation-0020
completed:
---

# Make Claude eval reruns scorable from Claude Code sessions

## Problem

The fresh Phractal eval rerun for `#0020` could not complete in this environment because the spawned `claude` subprocess failed inside an existing Claude Code session. Unsetting `CLAUDECODE` was not enough to produce a scorable 2-cycle run.

## Fix

Make the eval runner or launch path resilient to Claude-in-Claude execution so a default `nightshift test --agent claude --cycles 2 --cycle-minutes 5` rerun can complete and produce a report without manual shell workarounds.

## Acceptance Criteria

- [ ] `nightshift test --agent claude --cycles 2 --cycle-minutes 5` can complete from a Claude Code shell without the nested-session failure
- [ ] If Claude-in-Claude remains unsupported, the runner automatically falls back to a supported agent or emits a clear, actionable failure before starting cycles
- [ ] Regression coverage exists for the nested-session path
- [ ] A fresh eval rerun produces a scorable report instead of halting after two agent failures

Loading