Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ All notable changes to cc-autodream. Format loosely follows Keep a Changelog.
- **`AUTODREAM_L1_MODEL_<NAME>` is keyed by the engine that makes the call, not by the harness that holds the session.** A model id belongs to an engine, so `AUTODREAM_L1_MODEL_OMP` is the model of every L1 call on the omp engine, whichever harness the session came from, and never reaches the claude CLI. A host whose harnesses each review their own sessions sees no change. One that moves a harness to another engine pins that engine's model, the same keying `AUTODREAM_L2_MODEL_<NAME>` already had.
- **A session with no usable L1 engine is retried once a later run gives it one.** Its error record counted as done, which held while the only cause was an adapter missing on disk. An engine choice naming no accepted adapter, a missing model and a failed login check are config, so the record now carries `no_engine: true`, and a later run of the date (a catch-up or `AUTODREAM_FORCE=1`) that resolves an engine and a model for the session's source drops it and triages the session.

### Fixed
- **`AUTODREAM_L1_REQUIRE_SERVED=1` no longer trusts a findings JSON to say who wrote it (#167).** It kept any record with `error` or `skipped` and no findings as the runner's own, so a substituted model that wrote `{"error":"unreadable","findings":[]}` left its session untriaged. The runner now marks each record it writes itself with `.served` reading `runner`, and that marker is the only thing that exempts a record from the verdict. Runner records written before this change are triaged once more.
- **A findings JSON holding two JSON values is no longer a finished triage.** Every check used `jq -e`, which judges only the last value, so a forged object followed by a valid one passed and L2 read the first. One definition, `L1_FINDINGS_OK` under `jq -s`, now requires exactly one value at every site.

### Added
- **`AUTODREAM_SESSION_ROOTS_<NAME>`** lists the session roots of one adapter other than claude, colon-separated, in place of its manifest's defaults, for a harness that keeps one store per profile.
- **Who each engine runs as, and which model answered each L1 call.** Each engine the run uses reports its login once through a new adapter verb, `whoami` (claude: `claude auth status --json`, no model call), logged and written to run-stats as `login_<engine>`. The claude L1 worker runs with `--output-format json`, and the new `served` verb reads the models that answered, the CLI's cost estimate at API prices and a verdict, `match` or `mismatch`, that knows the CLI's naming (`haiku` answers as `claude-haiku-5-5`). Every call goes to `findings/<date>/l1-served.tsv`. run-stats gains `l1_engine_<source>`, `l1_served_models`, `l1_served_mismatch`, `l1_served_unmeasured` (calls whose engine reports no model, such as omp), `l1_cost_usd_estimate` and `l1_cost_unmeasured` (calls left out of that sum, so an all-omp night does not read as free).
Expand Down
6 changes: 3 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,8 +109,8 @@ plus env `CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 DISABLE_TELEMETRY=1 DISABLE_ERROR_REP
- **The source map is held in the environment, not read from the findings dir.** `AUTODREAM_SOURCE_MAP` (hash and adapter per session) and `AUTODREAM_L1_MODELS` (model per adapter) are built before the first model call. An L1 worker holds the Write tool and could rewrite `sessions-source.txt`, and the source decides which binary the next worker executes. `test_l1_engine_cannot_be_redirected_by_a_worker` has a worker do exactly that and checks the next session still runs on the real engine; reading the file instead fails it.
- **Haiku 5.5, effort and a replaced system prompt (2026-10-07).** The claude manifest's `l1_model` is `claude-haiku-5-5` (verified: `--model claude-haiku-5-5` and the `haiku` alias both answer as it; an id the CLI does not know falls back silently, so re-check `modelUsage` in `--output-format json` when changing it). `l1-argv` and `warmup-argv` add `--effort` from `AUTODREAM_L1_EFFORT_CLAUDE`, then `AUTODREAM_L1_EFFORT`, then `high`; `off` omits it and an unknown level exits 2. They also pass `--system-prompt` instead of `--append-system-prompt`, which drops Claude Code's default prompt: one-word call 6.9k -> 5.0k input tokens, and a real triage run on a 250 KB slice cost $0.159 against $0.215 with the same number of findings. L2 is untouched.
- **The engine, then the model.** A source's L1 engine is `AUTODREAM_L1_ENGINE_<SOURCE>`, then `AUTODREAM_L1_ENGINE`, then the run's reviewer (the L2 engine) (`adapter_l1_engine`). The model is keyed by that engine, not by the source: `AUTODREAM_L1_MODEL_<ENGINE>`, then `AUTODREAM_L1_MODEL`, then the engine's manifest `l1_model` (`adapter_l1_model`). A model id belongs to one engine, so a source-keyed pin sent omp sessions to the claude CLI with `--model deepseek/...` once a claude reviewer read them (PR #165 review). The engine choice is per source and the model pin per engine, the same as `AUTODREAM_L2_MODEL_<NAME>`.
- **What answered, and whether it counts.** Every L1 call's stdout goes through the engine adapter's `served` verb with the requested model: it returns the models that answered, the cost estimate and a verdict, `match` or `mismatch`, judged by the adapter because only it knows its CLI's naming (claude: `haiku` answers as `claude-haiku-5-5`). An engine with no `served` verb (omp) is `unmeasured`. The verdict is the sixth column of `l1-served.tsv` (run-stats `l1_served_mismatch`, `l1_served_unmeasured`) and is stored beside every accepted answer, findings JSON or chunk answer, as `<answer>.served`. With `AUTODREAM_L1_REQUIRE_SERVED=1` a call whose verdict is not `match` loses its answer, and before L1 `drop_unacceptable_l1_cache` holds the cache to the same rule: a findings JSON or chunk answer from an earlier run with no verdict or a non-`match` one is dropped and triaged again (`l1_cache_dropped`). A record with `error` or `skipped` and `findings: []` is kept, because the runner writes those itself and the post-L1 passes add fields to them, so their shape alone cannot tell them from a worker's. The empty array is the test, not the key: a worker can write either key, but every record the runner writes has no findings, so one with findings is model output and is held to its verdict.
- **No engine is an error for this run's config, not forever.** A session whose engine is not executable, resolves no model, names no accepted adapter or fails `AUTODREAM_REQUIRE_LOGIN` gets a findings record with `no_engine: true` and `error: no L1 engine for this session`, counted by `l1_findings_with_error` and skipped by the retry rounds, since retrying in the same run cannot change the answer, and the run still reports. Every one of those causes is config, which can change between runs, so `drop_unacceptable_l1_cache` drops the record in a later run (a catch-up or `AUTODREAM_FORCE=1`) once that run resolves an engine and a model for the session's source, read from `AUTODREAM_SOURCE_MAP` and never from the record. Records written before the marker are recognised by their error text. Either way the record must have `findings: []`: one with findings is model output, and with `AUTODREAM_L1_REQUIRE_SERVED=1` it is held to its verdict like any other.
- **What answered, and whether it counts.** Every L1 call's stdout goes through the engine adapter's `served` verb with the requested model: it returns the models that answered, the cost estimate and a verdict, `match` or `mismatch`, judged by the adapter because only it knows its CLI's naming (claude: `haiku` answers as `claude-haiku-5-5`). An engine with no `served` verb (omp) is `unmeasured`. The verdict is the sixth column of `l1-served.tsv` (run-stats `l1_served_mismatch`, `l1_served_unmeasured`) and is stored beside every accepted answer, findings JSON or chunk answer, as `<answer>.served`. With `AUTODREAM_L1_REQUIRE_SERVED=1` a call whose verdict is not `match` loses its answer, and before L1 `drop_unacceptable_l1_cache` holds the cache to the same rule: a findings JSON or chunk answer from an earlier run with no verdict or a non-`match` one is dropped and triaged again (`l1_cache_dropped`). A record the runner wrote with no model behind it (noise-gate skip, unreadable session, normalize failure, no-engine record, final-round stub) is kept, and the only thing that says so is its `.served` reading `runner`, which the dispatcher writes beside each one. Keys and shape prove nothing: a worker can write `error`, `skipped` or `no_engine` with `findings: []` as easily as the runner can, and that record would have skipped the verdict and left the session untriaged (#167). The empty findings array is kept as a second guard. A runner record from before the marker is triaged once more and comes back marked.
- **No engine is an error for this run's config, not forever.** A session whose engine is not executable, resolves no model, names no accepted adapter or fails `AUTODREAM_REQUIRE_LOGIN` gets a findings record with `no_engine: true` and `error: no L1 engine for this session`, counted by `l1_findings_with_error` and skipped by the retry rounds, since retrying in the same run cannot change the answer, and the run still reports. Every one of those causes is config, which can change between runs, so `drop_unacceptable_l1_cache` drops the record in a later run (a catch-up or `AUTODREAM_FORCE=1`) once that run resolves an engine and a model for the session's source, read from `AUTODREAM_SOURCE_MAP` and never from the record. Records written before the marker are recognised by their error text. Either way the record must have `findings: []`. Its shape is enough to drop it, since a forged one only buys a retriage; keeping it under `AUTODREAM_L1_REQUIRE_SERVED=1` takes the runner marker above.

## Self-pollution (the eating-its-own-tail bug)

Expand Down Expand Up @@ -293,7 +293,7 @@ A run killed during L2 leaves a complete findings dir and no report, and every s

The worker's stdout used to go to `/dev/null`, so every failure looked the same: an `.err` holding the one line `Working...` plus a hand-written sentence. On 2026-09-04 that hid a dead network behind three fine transcripts, and the report blamed their size (`oversized_errored / oversized_total` read 3/4, the number that opens issue #12). Re-run by hand against a live network, those transcripts triaged in 73 to 78 seconds. Now a failed worker's `.err` records its exit code and elapsed time and the last 40 lines of its stdout, or says plainly that stdout was empty. The `.err` and `.out` are deleted only when the worker succeeds.

- **Non-empty is not valid.** A worker that writes malformed JSON, or JSON whose `.findings` is not an array, is a failure with its evidence kept, not a success. `jq -e .findings` is truthy for a string, so the check is `.findings | arrays`, on the way in (idempotency, `l1_missing_count`) and on the way out. The malformed file is removed so it cannot reach L2.
- **Non-empty is not valid.** A worker that writes malformed JSON, or JSON whose `.findings` is not an array, is a failure with its evidence kept, not a success. `jq -e .findings` is truthy for a string and judges only the last value in a file, so a forged object followed by a good one passed and L2 read the first (#167). The check is `L1_FINDINGS_OK` under `jq -s` (one value, an object, a `findings` array), the same rule `merge-chunks.sh --check` applies to chunks, used on the way in (idempotency, `l1_missing_count`), on the way out and by the skill-field pass. The malformed file is removed so it cannot reach L2.
- `bin/failure-class.sh` is the single failure predicate sourced by both `run.sh` and
`bin/oversized-gate.sh`. It assigns every error stub to one of four classes from its
`.err`: `unclassified` when the file is missing, empty, or predates the
Expand Down
Loading
Loading