Pluggable harnesses: benchmark + run plans on any {provider, model, harness} - #36
Draft
richardbenson wants to merge 68 commits into
Draft
richardbenson wants to merge 68 commits into
richardbenson wants to merge 68 commits into
Conversation
richardbenson
marked this pull request as draft
June 2, 2026 16:24
…istry Add provider/model/harness as persisted per-run RunMeta fields and a Harness adapter contract (src/harness/types.ts) with a registry and the claude-code adapter (wraps runSession, structured mode). Add --provider/--model/--harness options to plan/queue/prompt that persist into RunMeta; unknown --harness fails fast. No runtime dispatch change yet (Phase 02) — claude still runs via the existing path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hase-loop) Route phase and single-prompt execution through the Harness registry instead of calling runSession directly. Both runners resolve the adapter via registry.get(meta.harness ?? harness_for_phases ?? 'claude-code') and dispatch through adapter.run(); an unknown harness fails fast down the existing error path. The claude-code adapter forwards byte-for-byte identical args to runSession (provider env + modelArgs + dangerouslySkipPermissions), so default claude behaviour is unchanged. finalise.ts gates summarise to structured adapters — opaque harnesses skip it with an informational event. Extend HarnessContext with modelArgs + dangerouslySkipPermissions to preserve provider-driven model selection and permission handling. resumeOrRestart left claude-specific. Regression gate: full suite (incl. envelope/jsonl-tail) passes unchanged; build/typecheck/lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ct, no proxy) Enable claude-code to run gemma4-cpe:31b locally at zero Anthropic API cost. Modern Ollama (verified 0.30.2) natively serves the Anthropic /v1/messages API including faithful tool use, so NO translation proxy is needed — cpe points ANTHROPIC_BASE_URL straight at Ollama (the mechanism src/runner/provider.ts already has). Adds a `cpe provider add --preset desktop-ollama` shortcut pointing directly at Ollama (overridable via CPE_OLLAMA_* env, no hardcoded secrets; health_check_url=/api/tags). Validated: claude -p direct and a full cpe single-prompt run both edited+committed files on the local model with valid structured output and zero api.anthropic.com traffic. docs/harness-bench/proxy.md documents the direct path; a LiteLLM proxy (docker-compose.yml + litellm-config.yaml, verified working) is kept only as an optional appendix for backends that are not Anthropic-native (older Ollama, OpenAI/other providers). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ bail + pause Add full-clone-per-run isolation for bench runs, post-run result capture with optional harnesstests/* branch push, an activity-based timeout that kills the whole process tree when output goes silent or merely repeats, manual-bail plumbing, and a configurable inter-run pause. - src/git/clone.ts: clone the baseline (CWD repo at current branch, overridable) into a fresh per-run dir; agent-git-safe; cleanup guarded to the cpe state dir. - src/runner/proc-tree.ts: setsid process-group spawn wrap + killTree(-pid) so forked harness children can't orphan (we saw multi-minute runaways). - src/runner/run-guard.ts: activity-based timeout (inactivity + optional max runtime) with identical-line repeat suppression, plus a bail registry the Phase 06 TUI keybind will drive. - src/runner/capture.ts: diff/--stat against the baseline + transcript + a structured meta.json -> results/<harness>__<model>/; pushes harnesstests/<combo> when the baseline has a remote (best-effort, never errors). - single-prompt.ts: bench (isolation 'clone') runs take a distinct path — clone cwd, guarded adapter.run, capture — with no retry/PR ceremony. Default worktree runs are unchanged. Activity is fed from raw JSONL line growth via a new startJsonlTail onActivity hook (the same reader the live tail uses) so a slow model mid-turn isn't killed. - runSession gains an optional abort signal (process-group spawn + tree kill), gated so the default path is byte-for-byte unchanged. - start.ts: interruptible inter-run pause. Added 'timeout'/'bailed' RunStatus and the bench/isolation/timeout fields on RunMeta + AppConfig. Validated: 0-orphan process-tree kill; silent run -> 'timeout', manual bail -> 'bailed' (both capture, 0 orphans); real non-empty diff capture + baseline; harnesstests push to a bare remote; real gemma4-cpe:31b clone run made+committed a change in the clone with the baseline untouched; pause unit-tested. 55 tests pass, lint/build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `cpe bench` to enqueue the harness×model cross-product of one prompt as clone-isolated single-prompt runs, and `cpe bench summary` to tabulate captured results. - src/commands/bench.ts: prompt via args/--prompt-file/stdin; --harness and --model as comma lists; --provider/--repo/--branch overrides (baseline defaults to the CWD repo at its current branch — nothing hardcoded). Validates every harness against the registry up front (enqueues nothing on unknown); requires ≥1 model; de-dupes combos; skips combos with existing results unless --force; prints the planned matrix before enqueueing. `bench summary` reads every results/<combo>/meta.json and prints harness/model/outcome/duration/files/ lines/tokens/cost/branch, tolerating missing/partial/unreadable meta. - src/cli.ts: register `bench` + `bench summary`. - src/commands/start.ts: skip the worktree reconcile for clone-isolation runs (they create their clone lazily) so the queue processor runs bench runs instead of failing them as "missing worktree". Validated: fail-fast enqueues nothing; matrix enqueue names runs <harness>__<model>; skip/--force; summary table (incl. partial rows); and the queue processor runs two clone runs sequentially with the configured pause. 55 tests pass, lint/build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generic output-tail (output-tail.ts) emits per-line `output` events for opaque harnesses and feeds the Phase 04 activity-timeout via the bus; the bench dispatch selects jsonl-tail (structured/claude) vs output-tail (opaque) by adapter.completionMode. New Bench TUI view (Bench.tsx + useBenchState): matrix progress (running→pending→finished, state glyphs + counts), a NOW line with elapsed + last-output age (quiet-aware, 1s clock), and a bounded live pane (height-clamped, overflow hidden). `b` in the bench view confirms then fires requestBail → guard.bail → process-tree kill → 'bailed' → capture still runs → matrix continues. `v` now cycles watch→manage→bench (watch→manage unchanged). Build/lint/typecheck clean, 63 tests pass (8 new). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
runBenchSinglePrompt passed only the provider's modelArgs (empty for a model-less provider like desktop-ollama) and resolved the provider from provider_for_phases — so meta.model and meta.provider set by `cpe bench` were both dropped. claude then fell back to its built-in default model (claude-opus-4-8), which a local Ollama endpoint 404s, failing every combo in ~4s with no output. Now the matrix model is passed as --model (and as ctx.model for opaque adapters) and overrides any provider-bundled model, and the provider resolves from meta.provider first. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dels Two defects made the bench live pane show a permanent "waiting for output…": - Bench didn't seed from the bus ring buffer on mount, so navigating into the view mid-run missed the phase-start event that anchors elapsed/last-output age. Now it replays the active run's buffered events on mount + combo change. - claude writes nothing renderable to its JSONL during a turn (the assistant entry lands only when the turn completes), so on a slow first turn there were no classified events at all. The structured bench path now emits a compact `· <type>` output breadcrumb for structural JSONL lines (queue-operation, attachment, ai-title, …) — the same raw-line signal the activity-timeout watches — so the pane shows live proof-of-life and the age ticks, per the DESIGN spec. assistant/user lines are skipped (the classifier renders those). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First non-claude harness adapter. opencode runs headless via `opencode run --dir <cwd> --model <provider>/<model> --format json --dangerously-skip-permissions`; completionMode is OPAQUE (it streams newline-delimited event objects, not a result envelope), so outcome is derived from exit code + git diff (exit0+diff=completed, exit0+no-diff=no-op, nonzero=error). Provider env (ANTHROPIC_BASE_URL) is translated into a temp OPENCODE_CONFIG (@ai-sdk/openai-compatible at <base>/v1) kept out of the clone diff; ANTHROPIC_* is stripped from opencode's env; step_finish tokens/cost parsed best-effort; ctx.signal drives a process-tree kill. Also fixes two issues this first opaque run surfaced in the shared bench path: - the prompt was always wrapped in the claude structured-output template; now opaque harnesses get the raw prompt. - isolation: opencode ignores the spawn cwd and follows the clone's local git origin, editing the ORIGINAL repo — fixed by passing --dir <clone>. - 'no-op' now maps to status 'complete' (clean non-error), not 'failed'. Validated end-to-end on gemma4-cpe:31b: captured results + a pushed harnesstests/opencode__gemma4-cpe-31b branch whose diff matches opencode's change in the clone (real repo untouched). 69 tests pass; lint/build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A ProviderEntry was a fixed provider/model pair, so one endpoint serving several models needed duplicate entries, and a model-less provider silently let the harness fall back to claude's default model (404 on Ollama). Now a provider is an *endpoint*; model selection is orthogonal: - ProviderEntry gains models[] (catalogue) + default_model; legacy `model` is still read as the default (backwards compatible, no migration). - resolveProvider takes the per-run requested model and applies one precedence everywhere (request > default_model > legacy model), returns the resolved model, and FAILS FAST with a clear error if a healthy custom endpoint has no resolvable model — instead of the silent claude-opus-4-8 fallback + 404. - All call sites pass the requested model (meta.model / --model): phase-loop (run + resume, wrapped so the throw fails the phase, not the processor), single-prompt (non-bench now honors --model too; bench simplified), finalise, plan. This also fixes `--model` being silently dropped on non-bench runs. - provider CLI: `add` prompts for a model list + default; `list` shows the default with a +N catalogue hint; desktop-ollama preset ships models[] + default_model (CPE_OLLAMA_MODELS/MODEL). 75 tests pass (6 new provider tests: precedence, args, fail-fast, anthropic passthrough); lint/build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the homelab-specific `desktop-ollama` preset and the --preset flag —
nothing host/model-specific belongs in a general-purpose tool. `cpe provider
add` is now fully generic and, after the user enters the endpoint URL + key,
offers to fetch the model catalogue from the provider:
- new resolveProvider-adjacent helpers: modelListEndpoints() probes the common
shapes (OpenAI-compatible/Anthropic /v1/models, native Ollama /api/tags),
parseModelList() handles {data:[{id}]} and {models:[{name}]}, and
fetchProviderModels() returns the first non-empty catalogue + the endpoint
that served it (reused as a sensible health-check default).
- `provider add` reorders to URL/key first, then "Fetch model list? [Y/n]";
on success it fills models[]/default_model, else falls back to manual entry.
Verified against a live Ollama endpoint (16 models via /v1/models). 77 tests
pass (4 new: endpoints, parser); lint/build clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`cpe provider refresh [--provider <name>]` re-queries the model list for one or all configured providers (reusing fetchProviderModels) and updates each provider's models[], summarising added/removed models. Providers with no endpoint or key are skipped; a default model that vanished from the catalogue is flagged (left as-is for the user to fix). Lets you pick up newly pulled models without re-adding the provider. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The model-list endpoint that responds during `provider add` is, by definition, a reachable health check — so derive health_check_url from it (stored as a base-relative path like /v1/models when under the base URL) instead of asking the user. The prompt becomes a one-key confirm (Enter to keep, or override / 'none'). Also make checkProvider send the provider's auth (Bearer + x-api-key + anthropic-version) so an authenticated endpoint reports healthy rather than 401 — necessary now that the guessed health URL may require a key. Shared authHeaders() between checkProvider and fetchProviderModels. 78 tests pass (1 new: healthCheckPath); verified live (guessed /v1/models, auth-aware checkProvider returns healthy). Lint/build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Prompt with "Base URL" / "API key" / "Auth token" instead of ANTHROPIC_BASE_URL/_API_KEY/_AUTH_TOKEN — the env var names are an implementation detail. The stored config keys are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the aider adapter following the Phase 07 template. completionMode=opaque (aider prints human text + auto-commits; no result envelope) — outcome from exit code + whether the run produced a change (HEAD advanced or dirty tree). - Model wiring: OpenAI-compatible (openai/<model> + OPENAI_API_BASE=<base>/v1 + OPENAI_API_KEY) translated from cpe's provider env; ANTHROPIC_* stripped. - Edit format whole|diff|udiff via CPE_AIDER_EDIT_FORMAT (default whole). - Keep aider's .aider* dotfiles out of the captured diff via the clone's .git/info/exclude + --no-gitignore, instead of letting aider auto-commit a .gitignore line — that housekeeping commit otherwise moved HEAD and falsely reported 'completed' when the real edit failed. Now completed/no-op is honest. - Auto-commit reconciliation: capture diffs index vs base ref (no double-commit). Validated end-to-end on gemma4-cpe:31b via the bench path: completed, captured diff matches aider's commit, branch pushed, clone isolation held. 88 tests pass, build/lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Leave aider streaming on so its response is written incrementally to the run log; the bench output tail surfaces it line-by-line to the live TUI and feeds the activity-based timeout, instead of one silent burst at the end. Keep --no-pretty so the stream stays plain, tailable text (no ANSI/markdown repaint). Verified on gemma4-cpe:31b: response lines arrive with progressing timestamps; only the pre-first-token prefill is silent (inherent to the model). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the goose adapter following the Phase 07 template. completionMode=opaque (goose prints progress + tool traces, no envelope; does not auto-commit) — outcome from exit code + git diff in the clone. - Headless: goose run --no-session --max-turns <n> -t <prompt> in ctx.cwd. File editing works via goose's built-in developer tools (no MCP setup). - Provider/model + isolation: drive provider purely from env (GOOSE_PROVIDER= ollama, GOOSE_MODEL=<model>, OLLAMA_HOST=<base url from ANTHROPIC_BASE_URL>) and point all four XDG base dirs (CONFIG/DATA/STATE/CACHE) at a per-run temp dir so the user's real ~/.config/goose is never touched and goose state never lands in the captured diff. ANTHROPIC_* stripped. Temp dir removed after. - max-turns loop backstop via CPE_GOOSE_MAX_TURNS (default 50). - Tokens parsed best-effort from goose's per-request usage logs. Validated end-to-end on gemma4-cpe:31b via the bench path: completed, captured diff matches goose's change, branch pushed, real repo + real goose config untouched. 94 tests pass, build/lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the openhands adapter following the Phase 07 template. completionMode=opaque (--json event stream, no envelope; does not auto-commit) — outcome from exit code + git diff in the clone. - Headless: openhands --headless --json --override-with-envs --exit-without-confirmation -t <prompt>, spawned in ctx.cwd. - Runtime: the SDK v1 CLI uses a LOCAL runtime (no Docker) and edits the spawn cwd directly, so the clone IS the workspace — verified no container/image was created and edits landed in the clone. - LLM via env (--override-with-envs): LLM_MODEL=ollama/<model>, LLM_BASE_URL= <ANTHROPIC_BASE_URL>, LLM_API_KEY=<token|ollama>. ANTHROPIC_* stripped. - Isolation: openhands keys its ~/.openhands state off HOME, so point HOME at a per-run temp dir (verified the real ~/.openhands gets no new conversation) and remove it after. It writes nothing into the cwd, so the captured diff is clean. - Tokens parsed best-effort from the conversation base_state.json metrics. Validated end-to-end on gemma4-cpe:31b via the bench path: completed, captured diff matches openhands' change, branch pushed, real repo + ~/.openhands untouched. 100 tests pass, build/lint clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on deferred) Add the plandex adapter following the Phase 07 template, plus the bonus orchestrator write-up. completionMode=opaque (applies changes to the working tree; outcome from exit + git diff). - Adapter (src/harness/plandex.ts, registered): plandex new --no-auto then plandex tell <prompt> --apply --skip-commit --no-exec --skip-menu --stop in ctx.cwd; ANTHROPIC_* stripped; stdin:null so a missing server fails fast. - Key finding: plandex is client/server and the SERVER is the engine (model providers, agent loop, plan state all server-side). The client can't do anything without a reachable, authenticated server, and the local model is configured server-side rather than via cpe's provider env. - Server PARKED by decision, so the end-to-end bench run is DEFERRED. Installed the CLI (v2.2.1, from the GitHub release tarball — docs site was down), implemented + unit-tested the adapter from the documented client interface. - Bonus: docs/harness-bench/orchestrator-notes.md — keep plandex as just another adapter (low priority); evolve cpe's own bench pipeline into the orchestrator rather than adopting plandex's server. 105 tests pass, build/lint clean. PROGRESS marks Phase 11 complete with the live-validation-deferred caveat. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opaque-mode adapter for the pi coding agent (v0.78.0). Runs `pi -p --provider ollama --model <model> --mode json --no-session -t <tools>` in the clone; ANTHROPIC_* stripped; provider/model wired via a per-run temp PI_CODING_AGENT_DIR/models.json (ollama, OpenAI- completions, <base>/v1) so the user's ~/.pi and the captured diff stay clean. Outcome from exit + git diff (pi edits the tree, no auto-commit); tokens parsed from the JSON stream deduped by responseId. Validated end-to-end on gemma4-cpe:31b (completed, diff/tokens/branch captured). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opaque-mode adapter for the crush coding agent (v0.75.0). crush ships a headless `crush run` subcommand, so it's a normal adapter (not parked). Runs `crush --data-dir <dd> --cwd <clone> run --quiet -m ollama/<model>` in the clone; ANTHROPIC_* stripped; provider/model + headless tool permissions wired via a per-run temp CRUSH_GLOBAL_CONFIG dir (crush.json: openai-compat ollama provider at <base>/v1/, permissions.allowed_tools) and --data-dir, keeping ~/.config/crush and the captured diff clean. Outcome from exit + git diff (crush edits the tree, no auto-commit); tokens read from crush.db sessions table via bun:sqlite. Validated end-to-end on gemma4-cpe:31b (completed, diff/tokens/branch captured). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opaque adapter running `codex exec --json` against the per-run clone, with the local model wired via a custom model_provider using wire_api="responses" (codex 0.137.0 removed wire_api="chat"; our Ollama serves the Responses API natively, so no proxy is needed). Isolated via a temp CODEX_HOME; tokens parsed from the turn.completed usage event. Validated end-to-end on gemma4-cpe:31b. 128 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Opaque adapter running mini-swe-agent's non-interactive DefaultAgent (`mini --agent-class default`) against the per-run clone, with the local model wired via LiteLLM/Ollama (OLLAMA_API_BASE) and step/wall-time backstops. Isolated via a temp MSWEA_GLOBAL_CONFIG_DIR; tokens parsed from the trajectory JSON. Validated end-to-end on gemma4-cpe:31b. 135 tests pass. Completes adapter phases 08-15. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace docs/harness-bench/ (README, PROGRESS, 15 phase files, backlog, orchestrator/proxy write-ups, proxy config) with docs/harness-bench.md capturing requirements, per-phase work, and lessons learned. All adapter phases (08-15) complete; full harness x model matrix unblocked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each adapter declares an install descriptor (bin + version + url). New `cpe harness check`/`list` probe every harness's CLI on PATH, record version, and cache results in config; detection auto-runs on first launch of a harness-selecting command. Selecting an uninstalled harness is hard-blocked (explicit --harness at selection time; the resolved default at dispatch) with a clear install link. Scanning shows live per-harness progress on stderr. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the {provider, model, harness} triple to the intro; a Harnesses section
with the supported-harness list + install detection (`cpe harness check/list`)
+ selection; a Benchmarking section (`cpe bench`); a Local models note; and the
new commands and global config fields (harness_for_*, isolation, bench timeouts).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the reference adapter up to the same standard as the opaque adapters: a header comment block (default/structured adapter, delegates to runSession, model wiring, envelope-based outcome) and src/harness/claude-code.test.ts. Export outcomeFromEnvelope + a new buildHarnessResult helper so the envelope→result mapping is unit-testable without spawning claude. Run logic unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Headless bench runs have no human to answer questions, and a question-style / open-ended task otherwise risks the agent replying in chat and making no file changes (a no-op, since bench scores the git diff). buildBenchTask() now frames every bench task — structured AND opaque — with a preamble: don't ask/await input, make reasonable assumptions, and deliver concrete committed changes. The structured harness still gets the commit/no-PR template on top. 163 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalise BENCH_AUTONOMY_PREAMBLE -> AUTONOMY_PREAMBLE + withAutonomy(), and prepend it everywhere a prompt is built for a headless run: bench (structured + opaque), non-bench single-prompt (structured + opaque), and phases (via the effective prompt file, which both the claude and opaque phase paths read). All headless runs now instruct: no human to answer, don't ask/await input, make assumptions, deliver concrete file changes. 163 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eamble
From the paused homelab matrix logs:
- claude-code on a verbose local model hard-errored ("response exceeded the 32000
output token maximum") -> spurious 'error'. Set CLAUDE_CODE_MAX_OUTPUT_TOKENS
(default 64000, env-overridable) in the claude-code adapter.
- a harness (goose) replied with a markdown to-do checklist instead of editing
files -> no-op. Harden AUTONOMY_PREAMBLE: must implement by editing files now;
explicitly forbid replying with a plan/checklist/description; text-only = failure.
Other no-ops were infra (Ollama stream-decode / OOM on the desktop) or 12b model
limits, not CPE issues. 163 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Paused harness x model run (9 harnesses x gemma4-cpe 12b/26b/31b) against the homelab repo, kept under docs/logs/ for later analysis. Per-combo meta.json + diff + transcript (pi's 11-33MB transcripts gzipped), plus the 4 runs that never reached capture, and a README triaging each no-op/error into infra (desktop OOM) / cpe-fixable / model-limitation. ~2.1MB total. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
crush.db only stores the LAST turn's snapshot (completion_tokens: 6/13 for a multi-turn run) and its messages table has no token columns, so reported output tokens were badly under-counted. Run crush with --debug (logs full provider round-trips to <data-dir>/logs/crush.log) and sum the per-request OpenAI `usage` across all requests instead. Verified on a real run: output 193 (40+140+13) vs the old 13. Drops the bun:sqlite dependency from the adapter. 163 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… tracking
A provider with type:"litellm" routes any harness through a self-hosted LiteLLM
proxy and replaces adapter-parsed token counts with the gateway's spend logs
(docs/litellm-integration-spec.md; validated live against the homelab gateway).
- src/runner/litellm.ts: gateway client — mint an ephemeral virtual key per run
(2x max_runtime lifetime), poll /spend/logs?api_key=<key> for the flush
(~3 min budget), sum split prompt/completion tokens across all turns, revoke
the key. Fail-safe throughout: collection failure is never a run failure.
- provider.ts: litellm-aware resolveProvider — default /health/readiness gate,
per-run key mint (admin-key degraded routing when minting fails; auxiliary
calls never mint), env reuses the existing ANTHROPIC_* shape every adapter
already consumes, plus a CPE_GATEWAY hint.
- goose.ts: switch to goose's openai provider behind the gateway (LiteLLM does
not serve the Ollama-native /api/* API goose defaults to).
- dispatch (single-prompt structured/opaque/bench, phase structured/opaque):
settle on completion paths; bench settles on any outcome. tokens +
token_source ('litellm' | 'adapter') recorded in run meta / phase entries /
bench capture.
- 15 new tests against a fake gateway (Bun.serve) with response shapes copied
from the live validation. 178 pass; admin key only ever read from env
($CPE_LITELLM_KEY); README providers section documents the setup.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pure-observer proxy design (R0-R11) that preceded the LiteLLM direction. Kept as the fallback for the niche LiteLLM+Langfuse don't serve well (live per-turn TUI, wire-level activity liveness); marked SHELVED in the header. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… check" checkProvider already defaults litellm entries to /health/readiness; the test command's own no-health-url branch hid that. Now shows the probed URL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… dirty tree Agents that commit their work (the bench prompt asks them to) leave a clean working tree, so the porcelain-only check mislabelled real completions as 'no-op' (7 of 12 successful runs in the 2026-06-10 matrix). Extract aider's HEAD-moved-or-dirty-tree rule into a shared git-changes module and use it in all nine opaque adapters. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
aider's repo-map protocol tells the model to name the files needing changes and stop so they can be added to the chat; the preamble's 'do not stop partway' made a 12b model spend a 6k-token reasoning spiral reconciling the contradiction before naming the file anyway. Saying the tool's own workflow doesn't count as stopping cut that turn's reasoning by ~40% on re-test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Spend rows flush in batches, so a short run's first non-empty read can be partial — crush's title-generation row landed a poll ahead of its agent rows and 10.9k tokens were recorded as 375. Require two consecutive agreeing polls; on budget exhaustion return the best snapshot seen. Verified on re-run: crush now records its full 135.9k/2.7k. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
codex refuses to create its helpers (incl. its bundled rg) under a /tmp CODEX_HOME, leaving exec shells with 'command not found: rg' while /usr/bin/rg existed — which derailed model discovery. Use ~/.cache instead; verified on re-run (no warning, rg searches succeed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… fixes - CHANGELOG: Unreleased section covering the branch (harness platform, bench, LiteLLM tokens, the matrix-run fixes) - litellm spec: stable-totals polling (a0b0270) + §4.8 gateway prerequisites (drop_params, the #20585 streaming finish_reason patch and when to drop it) - harness-bench: deferred list updated — matrices run + archived, opaque normal runs shipped; plandex still parked - README: openhands/mini route via openai/ not ollama/; opaque outcome wording covers committed work Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…odels Nothing to enable gateway-side (LiteLLM only passes through cloud-provider cache accounting); Ollama's KV/prefix cache silently provides the latency benefit with no usage fields. Spend-log tokens compare harnesses honestly but overstate local compute for big constant prefixes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 'Structured output: final message must be JSON' standing instruction was claude-code-shaped: the runner already supplies the schema for structured runs, and opaque runs get the .cpe/result.json contract appended at run time — so a phase prompt carrying it contradicts the opaque contract, the exact instruction-conflict shape that stalls weaker models. Tell planbot to leave result reporting to the executor's injected contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In a worktree .git is a file, so the hardcoded <cwd>/.git/info/exclude write failed silently; aider's .aider* droppings then counted as a dirty tree and a zero-edit run was reported 'completed' (observed on the first plan-vs-vague arm-A run). rev-parse --git-path returns the right path for clones and worktrees alike. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The summarise step routed through an opaque adapter wrote only to finalise.log, so the TUI went silent for the whole finalise while the harness worked. Wire the same output tail phases use (phaseNumber -1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nbuffer aider Three findings from the 31b plan-vs-vague overnighter: - A max-runtime kill can race a finished phase (twice now: agent committed and wrote .cpe/result.json seconds before the cap). When the guard fires but a completed self-report + commit exist, record the phase complete instead of discarding hours of local-model work. - markPhaseFailed only emitted the reason to the bus; persist it on the phase entry so post-mortems don't need log archaeology. - aider's stdout block-buffers through a pipe, making the live tail look dead for minutes while it works; PYTHONUNBUFFERED=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LiteLLM gateways require auth on /v1/models, but refresh only passed the
inline anthropic_* fields — a litellm entry keeps its admin key in an env
var (admin_key_env), so the fetch went out unauthenticated and got a 401
('could not fetch models'). Resolve the key the same way the runner does.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tion gap persists Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h it, gemma4:26b out Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ound) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two misclassifications surfaced by the 2026-06-14 bake-off: - a context-length 400 (LiteLLM 'exceeds the available context size') was bucketed as 'auth error: 400' — 400 is a bad request, not auth (only 401/403 are). Now content-matched to a terminal 'context-overflow' outcome (a retry hits the same wall; fix is a bigger context window). - a request timeout (api_error_status null) was 'unknown api error: null'; now content-matched to transient-error and retried. 400 without an overflow signature is a clear 'bad request (HTTP 400)'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The earlier finalise-tail fix only covered the opaque path; the structured claude-code finalise still wrote straight to the log fd with no bus tail, so the TUI showed nothing for the whole finalise (looked idle while claude -p worked). Tail it the same way. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A queue failure after createWorktree (bootstrap error, missing plan dir) left the worktree AND its feature branch behind. That debris later poisoned unrelated runs: opencode's project resolver followed git to the orphaned worktree, and a resume checked out the orphaned branch — both derailing a self-update run (2026-06-15). queuePlan now throws instead of process.exit on bootstrap failure, and both callers (queue, plan) remove the worktree and delete the branch on any post-worktree error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…de-code the only working feature Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes cpe harness-agnostic: every run carries a selectable
{ provider, model, harness }triple, and any harness/model can now do cpe's core job — interactive planning, headless phases, single-prompt tasks, the summarise→push→PR finalise — not just benchmark comparisons. Motivation: theclaude -ppricing change, plus local models now being good enough to run real work at zero API cost.The branch then grew (in a good way): real matrix runs surfaced a token-accuracy problem, which became a LiteLLM gateway integration with wire-accurate per-run token accounting, which in turn flushed out and fixed two gateway-level bugs and four cpe bugs.
Status: complete & validated (184 tests, build + lint clean; three real harness×model matrices run and archived). claude-code's structured path is preserved byte-for-byte (regression gate).
What's here
1. Platform + benchmarking (phases 01–15)
{ provider, model, harness }per run;Harnessadapter contract + registry; dispatch refactor (default claude-code unchanged).harnesstests/*push;cpe benchharness×model matrix + summary; bounded live TUI + manual bail.2. Harness install detection + gating
cpe harness check/cpe harness listprobe each harness's CLI + version, cache in config (auto-run on first use, live scan progress). Selecting an uninstalled harness is hard-blocked with the install link.3. Harness-agnostic phase execution
PhaseReportcontract (src/runner/report.ts). claude-code maps its envelope to it; opaque harnesses produce it via a hybrid: agent writes.cpe/result.json→ else cpe asks the model to summarise the diff+transcript → else git-derived. Commit truth from git; commit/PR mechanics routed through the harness; finalise generalised to the selected harness.--harness): a 2-phase plan committed both phases via self-report, phase-1 notes flowed into phase-2, finalise wrotedocs/<plan>.md, removed the plan folder, skipped PR (no remote). Opaque single-prompt validated likewise.4. Activity guard for non-bench runs
RunGuard(activity-based inactivity timeout + absolute max-runtime cap + manual bail) now protects opaque worktree runs too, fed by the output tail. Off by default; a guard-fired abort is terminal. Note: most harnesses stream output, but mini-swe-agent /crush --quietare sparse between turns, so set the inactivity window above per-turn latency (or rely onmax_runtime_seconds).5. LiteLLM gateway integration — wire-accurate tokens (
docs/litellm-integration-spec.md)type: "litellm"routes every harness through one self-hosted gateway (OpenAI-compatible/v1for most, Anthropic/v1/messagesfor claude-code; goose auto-switches to its OpenAI provider via aCPE_GATEWAYhint)./spend/logsfiltered by that key (every request, every turn, regardless of what the harness self-reports) → revoke. Totals carrytoken_source: "litellm" | "adapter"; collection failures fall back to adapter numbers and never fail a run. Admin key lives in$CPE_LITELLM_KEY, never in a file.drop_params: true(codex's extra params 400 onollama_chat) and the streamingfinish_reasonfix (open upstream PR fix(ollama): track tool_calls state across streaming chunks for correct finish_reason BerriAI/litellm#20585; without it strict clients like crush silently no-op on tool calls).6. Correctness fixes from the live matrices
no-op; "changed" is now HEAD-moved-or-dirty-tree, shared across all 9 opaque adapters (src/harness/git-changes.ts).--debugHTTP log.rgthere, derailing discovery).Validation: three real matrices (archived under
docs/logs/with findings)2026-06-07-homelab-matrix— 9 harnesses × 3 gemma4-cpe models on a real repo (paused early, kept for triage reference).2026-06-10-litellm-synthetic— 9 harnesses × {base, cpe} 12b models through the gateway: first live validation oftoken_source: "litellm"end-to-end; the cpe model build beat base 7/9 vs 5/9.2026-06-10-vague-prompt— the same matrix under a deliberately ambiguous prompt, run before and after the fixes above (7/9 both runs; crush went from silent no-op to completing; failure causes now model-level, not infrastructure).Docs
README.md: harnesses run the core job, supported-harness table, install detection, benchmarking, local models, LiteLLM gateway + token accounting, new config fields.docs/litellm-integration-spec.md(as-built, incl. gateway prerequisites),docs/observability-proxy-spec.md(own-proxy alternative, shelved),docs/harness-bench.md(consolidated plan + lessons),CHANGELOG.mdUnreleased section.Deferred / follow-up
🤖 Generated with Claude Code