diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index f292f95..56e8223 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -65,10 +65,16 @@ jobs: with: filter: blob:none fetch-depth: 0 + - name: Set up Node.js + uses: actions/setup-node@v6 + with: + node-version: 22 - name: Install uv uses: astral-sh/setup-uv@v7 with: python-version: ${{ matrix.env.python }} + - name: Install agent CLIs + run: npm install --global @anthropic-ai/claude-code @openai/codex - name: create hatch environment run: uvx hatch env create ${{ matrix.env.name }} - name: list all all installed package versions diff --git a/.gitignore b/.gitignore index eea4877..fc17263 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,10 @@ __pycache__/ # Distribution / packaging /dist/ +# uv's resolved dependency set. acumen is a library — consumers resolve their own tree, and +# CI resolves fresh through hatch — so a committed lock would pin nothing that is checked. +/uv.lock + # Tests and coverage /data/ /node_modules/ diff --git a/.vscode/settings.json b/.vscode/settings.json index e034b91..7c09f6d 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -15,4 +15,5 @@ "python.analysis.typeCheckingMode": "basic", "python.testing.pytestEnabled": true, "python.testing.pytestArgs": ["-vv", "--color=yes"], + "cursorpyright.analysis.typeCheckingMode": "basic", } diff --git a/CHANGELOG.md b/CHANGELOG.md index 6700dde..e686949 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,3 +9,196 @@ and this project adheres to [Semantic Versioning][]. [semantic versioning]: https://semver.org/ ## [0.0.1dev] + +### Added + +- Review each task for internal consistency in `acumen check`, in a second phase after the + reproducers run: one agent reads every split's prompt, recorded answer and reproducer + together and adds an `ok`/`mismatch` column, with one line naming the contradiction and one + naming the fix. Reproducing an answer only proves the code and the answer agree; a prompt + asking for something else — ordering, statistic, group, count, or answer format — fails every + agent that reads it correctly, and costs a whole pass to discover. The reviewer reads a staged + copy of the task set with no path back to `tasks.yaml`, never edits a task, and picks its model + from the new `check_model` config key. `--no-review` runs the reproducers alone and spends + nothing. +- Add `acumen check`, which verifies every task's ground truth by rerunning the script that + produced it. Each task keeps a reproducer at `tasks/-.py` that redoes the + analysis in the target venv and writes its answer to `answer.md`, graded by the same + comparison a benchmark run gets. The command reports a row per task and split, the summary + statistics (share of the task set with a reproducer, share that reproduces, tasks that + reproduce on both splits), and exits non-zero if anything did not, so a wrong answer or a + broken pipeline is caught before a pass pays to discover it. It probes that the package + imports at all before running anything, since that failure would otherwise be reported once + per task. +- Keep the scripts `acumen tasks` runs to obtain each answer instead of discarding them: the + generator now writes one reproducer per split into `--scripts` (default `tasks/`), and + reports any split it left without one. An answer nothing can recompute is an answer nobody + can check. +- Add an optional task-level `needs_script` key to `tasks.yaml`, defaulting to true. A task + that needs no code to answer sets it false and is reported as not applicable by + `acumen check` rather than as a missing reproducer. + +- Run Claude and Codex models side by side in benchmark matrices and use either + provider for drafting, improving, task generation, and shipping. +- Compute `cost_usd` from each run's token breakdown rather than the provider's own + figure, so both providers are priced by one arithmetic path and cached input is billed + at its own rate. The rates used are frozen into `result.json`. Where a backend reports + dollars of its own, that figure is recorded beside it as `provider_cost_usd` (the report's + sidecar CSV calls it `recorded_cost_usd`) together with its distance from the inferred + value, but nothing plotted, printed or summed reads it: a Claude SDK total covers nested + subagents its own usage block does not, so a console tally on that basis would disagree + with the report it summarises. A model no layer prices stays unpriced even when the + provider reported dollars, and the report warns at the top, naming the models that need a + `prices:` entry. +- Add `acumen prices` to show the rate table and `acumen prices --refresh` to diff it + against the providers' published pricing, plus a `prices:` config key to override it. +- Ship no rate table at all: rates are read from the providers' pricing pages on every + command that prices something, since a table compiled into a release is wrong from + whatever date prices next move, and each run's cost is frozen when written rather than + corrected later. `prices:` in `config.yaml` still overrides, and still wins. +- Fail `acumen bench` when the pricing pages cannot be read, before anything is spent: + cost is a headline metric of the report, so a pass that cannot establish rates should + not run. `draft`, `improve`, `tasks`, and `ship` degrade to unpriced instead, warning + that Codex's `max_usd` cannot be enforced without rates. +- Record `price_source` and `price_rates_as_of` next to `price_rates` in every + `result.json`, so runs benchmarked months apart remain individually attributable and a + single report can mix them without restating either. +- Flag arms in a report that were priced on different dates: the cost difference between + them includes any change in provider pricing, not only the skill's effect. +- Render an HTML transcript for Codex runs too, from the `codex exec` event stream. +- Run the whole comparison from one `acumen bench`: with no arm selected it now covers + every arm the project has — the baseline plus each version in `skills/` — benching them + one after another against a single prepared target, with per-arm counts and tallies and a + combined total. `--dry-run` plans the same set for free, and `--no-skill` / `--skill vN` + still restrict the pass to one arm. A version in `skills/` that fails to load stops the + pass at planning rather than being dropped from the comparison. + +### Changed + +- Order the report's cost-vs-success figure with arrows instead of marker shapes. Skill + versions are a sequence, so each model's own marks are now joined baseline to v1 to v2 and + on, in that model's colour and never crossing to another model, with the pooled grey marks + carrying their own chain. Shape is left to say only whether a mark is a skill or the + baseline, which frees the version labels off the panel entirely: nothing is named in place, + and the key holds two marks and an arrow however many versions ran. Every hop is drawn and + every hop is straight, leaving one mark's rim and landing on the next; two versions that + landed on the same result simply hide their arrow under the overlap, which is the reading. + The pooled mark is now the size of every other one, set apart by its colour and its error + bars alone, since drawn larger it read as a bigger measurement rather than a summary. +- Run the Pareto staircase out to the right edge of the cost-vs-success panel, and dash it. It + stopped at the dearest frontier mark, which left the stretch beyond it looking like open + ground when paying more than the best mark cannot buy less than it did; dashing separates a + line no run lies along from the arrows, which are drawn between marks that do. + +- Make both backends optional, so a Claude-only and a Codex-only install are each complete: + the Claude Agent SDK moves to the `claude` extra (`pip install acumen[claude]`, or + `acumen[all]`) and Codex needs only its CLI on `PATH`. Selecting a model whose backend is + not installed fails preflight with the command that installs it. +- Enforce `max_turns` and `max_usd` for Codex, which has no cap of its own, from its event + stream. Turns are counted in completed model actions rather than `codex exec` invocations — + one invocation is a single Codex turn, so the old count was always 1 — and the run is + stopped at the cap. `max_usd` can only mark the outcome: Codex reports usage when a turn + ends, so an over-budget run is recorded as a `budget` failure after the spend. + +- Allow `acumen bench --auth {auto,session,api}`, defaulting to the provider's subscription + like every other command. The old API-only rule existed because a per-run `cost_usd` needed + metered billing; cost is now derived from token counts, which a subscription run reports + just as fully. `result.json` records the run's `auth_mode`, since under `session` the figure + is what the run would have cost at API rates rather than metered spend. +- Treat exhausted provider subscription usage and API credit as benchmark-invalid + infrastructure failures: print the provider error, cancel only that provider's remaining + cells while other providers finish, exit non-zero, and refuse to report or improve from the + invalid evidence. Resuming automatically retries the invalid and cancelled cells after the + credential is replenished. + +### Fixed + +- Remove the target package's own agent guidance from the venv before any agent runs, so the + baseline arm is really skill-free. A package can ship a first-party skill inside itself + (`site-packages//_skills/data/SKILL.md` plus a `references/` tree, the shape `acumen ship` + produces), and although nothing registers it and no prompt mentions it, an agent that greps the + venv it was handed finds it: measured across four passes and 4608 runs, 18.2% of baseline runs + read it and 12.2% of skill-arm runs read both it and the skill under test, concentrated in + exactly the tasks that discriminate between arms and varying fiftyfold by model. So the + comparison the whole benchmark rests on was partly against a skill nobody chose. `prepare_target` + now scrubs the finished venv of skill directories, `SKILL.md`/`CLAUDE.md`/`AGENTS.md`/Copilot + instructions, `.claude`/`.agents`/`.codex`/`.cursor`/`.claude-plugin` trees, and any console + script left pointing into what it removed; only agent-facing data goes, never code, so the + package imports and behaves exactly as installed. A `skills/` directory holding code is kept — + a directory only counts as guidance when a `SKILL.md` sits somewhere beneath it. The scrub is + idempotent and also runs on a cache hit, so a venv built by an earlier version is cleaned in + place rather than needing `--refresh-target`. The source checkout is deliberately untouched: + `ship` commits from it, and for a local target it is the user's own working tree. `draft` instead + joins `tasks` in reading a filtered copy of the checkout, since a skill drafted from the + maintainer's own skill is not the independent artifact the report presents. The host user's skill + catalog is left alone — it is a realistic environment and identical across arms. + +- Close a Claude run's session when the run is over, and record the result that run produced. A + terminal result was not the end of a session: a Bash command the agent left running keeps the + CLI alive, and when it finishes the CLI queues the notification as a *new* prompt and re-enters + the model. The old loop kept assigning whatever result arrived last, so one measured run of + 1117 seconds over 41 turns that hit its cap was recorded as a 4-second, two-turn success, with + inferred cost 96% low and the answer graded `no_answer_file`. The re-entry was also past the + turn cap, where the CLI answers every tool call with a cancelled-permission denial — including + `echo "test"` and reads inside the run's own working directory — so the agent sat waiting for + an operator who does not exist while the harness had already moved on to grading. Two of 44 + background-task runs lost their result to this; it was a race on whether a notification landed + before the process was torn down. The Claude backend now runs through `ClaudeSDKClient` and + stops reading at the first result, so the turns, duration, usage and cost a run is judged on + are the graded prompt's by construction. Teardown then stops every outstanding background task, + waits briefly for the CLI to confirm each is gone so its output file is flushed before the + artifacts are collected, interrupts the turn, and disconnects — on every exit path, including a + crash. Work the agent abandoned after declaring itself done is not resumed. The sandbox-path + denials are unchanged: those are the containment guard doing its job, not this. +- Give a sandboxed Bash command 600 seconds before the CLI moves it to the background, up from + the default 120, with a 1800-second ceiling an agent can still ask for. A command that outruns + its timeout is not failed but backgrounded, and a benchmark target downloads its own datasets + and priors — one such fetch measured over 300 seconds — so almost every one of them was being + backgrounded, which is what created the session-lifecycle problem above in the first place. +- Record what a capped or crashed Codex run actually spent. `codex exec --json` reports usage + once, in `turn.completed`, which a run acumen stops at its turn cap never reaches, so every + turn-capped run recorded zero tokens and a cost of `$0.00` after minutes of real work. acumen + now follows the running total Codex writes to its rollout session file, which means dropping + `--ephemeral` so that file exists; it lands inside the run-local `CODEX_HOME` and is discarded + with the sandbox. A shipper run given no config directory will now leave a session in the + operator's own `~/.codex`, as a plain `codex exec` would. A capped run is also stopped with an + interrupt rather than killed outright, which is what lets Codex finish writing the record for + the response that has just landed; a hard kill loses it. The recovered figure is still a lower + bound, since the very last response can be cut off before it is recorded, but a measured + turn-capped run went from `$0.00` on nothing to a real cost on 30k real tokens. + `turn.completed` stays authoritative whenever it arrives, and a rollout that cannot be read or + parsed leaves the run exactly as it was before. The same total is what `max_usd` is now checked + against, once per model response rather than once per turn, so a Codex budget cap stops the run + partway through the turn instead of only labelling the overspend after the fact. A single + response can still overshoot. +- Grade a capped run on the answer it managed to write. A run stopped at its turn or budget cap + was failed on the cap alone, so an agent that had already written a correct `answer.md` before + being cut off was recorded as a failure with its answer sitting unscored in `result.json`. The + cap now defers to the grade whenever there is an answer to grade; with no `answer.md`, or an + empty one, the run still fails as `max_turns` or `budget`. The cap remains visible in the run's + `subtype` and error list. Applies to Claude and Codex alike, and does not change how a crash, a + failed sandbox or an exhausted account is classified: those still override the grade. +- Give isolated agents unrestricted internet again. Sandboxing every run also put an egress + policy in front of it, and the policy denied the hosts a target actually needs: EBI, Zenodo, + figshare, OmniPath, NCBI and cellxgene were all unreachable while GitHub and PyPI were not. + A refused host does not stop an agent — it improvises from memory and returns a confident + wrong answer — so the runs scored as evidence that the models had got worse. Codex now runs + its proxy in `full` rather than `limited` mode, which had also been rejecting every request + that was not GET, HEAD or OPTIONS. Claude runs without an OS sandbox, whose proxy cannot be + opened from the settings file the SDK passes, and is confined to its sandbox directory by a + tool-layer guard instead: it can use system paths and the target venv, and cannot list or + read anywhere else on the host. +- Record a run whose sandbox could not execute anything as an infrastructure failure rather + than a wrong answer. bubblewrap reports its own startup failure as the command's output, not + on the agent CLI's stderr, so the existing check never saw it and a run where every single + command died still entered the report as a graded result. +- Let `acumen tasks` generate over the untouched `tasks.yaml` placeholder that `acumen init` + writes, instead of demanding `--force` — the two documented first steps of the loop + contradicted each other. A file the user has edited is still protected. +- Stop feeding Codex transcripts to `claude-code-log`, which reads the SDK-native format + only: it skipped every line, exited 0, and wrote an empty page that was then recorded as a + successfully rendered transcript. +- Drop `Claude` from the drafting and improving prompts, which described the artifact as a + "Claude Skill" even when a Codex agent was writing it for a non-Claude skills directory. +- Remove `check_auth`/`auth_available`, which only ever looked for Claude credentials and + would report a Codex-only setup as unauthenticated. `resolve_auth_mode` replaced them. diff --git a/README.md b/README.md index cecf89e..4f0cdf8 100644 --- a/README.md +++ b/README.md @@ -21,9 +21,13 @@ that loop: point it at a Python package and a few tasks, and it drafts a skill, against a no-skill baseline, and improves it across a train/test split so the gains are real generalization, not memorized answers. +- **`acumen check`** — rerun the script behind each task's answer, then have an agent judge + whether the prompt actually asks for what that script and answer produce. Both before a + benchmark pass pays to find out. - **`acumen draft`** — write `skills/v1` from the package's own source. - **`acumen bench`** — score a skill against a no-skill baseline, in a scrubbed sandbox where - the skill is the only difference between arms. + the skill is the only difference between arms. Agent guidance the target ships itself is + removed from the venv first, so the baseline really is skill-free. - **`acumen improve`** — refine the skill from its train results, then benchmark again. - **`acumen report`** — aggregate every run into one self-contained `report.html`: success rate per version, train vs. test. Bars are coloured by model, with a grey bar pooling all @@ -40,7 +44,8 @@ rather than genuinely helping. acumen init # 2. Fill in config.yaml (repo). Write tasks.yaml by hand, or generate it: -acumen tasks # mine the package for real analyses -> tasks.yaml +acumen tasks # mine the package for real analyses -> tasks.yaml + tasks/ +acumen check # is the ground truth right, and does each prompt ask for it? # 3. Then run the loop: acumen bench --no-skill # the baseline arm @@ -48,6 +53,7 @@ acumen draft # generate skills/v1 from the package source, o acumen bench --skill v1 # benchmark the skill against the baseline acumen improve # generate skills/v2 from v1's train results, or write by hand acumen bench --skill v2 +acumen bench # or: every arm at once (baseline + each skills/vN) acumen report # aggregate every run into report.html # 4. Once a version proves out, ship it into the package itself: @@ -60,6 +66,68 @@ agent the user names — `--agent {claude,codex,agents,claude-science}`, or an e so the package's own users get the guidance with one command, wherever they run their agent. The same bundle installs verbatim into every framework. +## Checking the ground truth + +A task is only worth benchmarking if its recorded answer is actually correct. A wrong answer makes +every model fail that task: real money spent, and the failure reads in the report as the model's +fault rather than the task's. `acumen check` catches the two ways that happens. + +**Does the answer still come out of running the code?** Each task keeps a **reproducer** at +`tasks/-.py`, a self-contained script that redoes the analysis in the target venv and +writes its answer to `answer.md` — the same contract a benchmark run has, graded the same way. +`acumen tasks` writes them as it generates the tasks; `acumen check` reruns them: + +```bash +acumen check # every task, both splits +acumen check --task bulk --split train # one cell, while you fix it +acumen check --jobs 8 --timeout 600 # or: --keep to inspect what a script wrote +``` + +You get one row per task and split — reproduced, wrong answer, script error, timed out, or no +script at all — then the summary statistics: how much of the task set has a reproducer, how much +of it reproduces, and how many tasks reproduce on both splits. Before running anything it checks +that the package imports in the venv at all, since that one failure would otherwise be reported +once per task. + +**Does the prompt actually ask for what the script and answer produce?** Reproducing an answer +proves the code and the answer agree. It says nothing about the prompt, and a prompt describing +something else fails every agent that reads it correctly. A real example: + +> Find the 3 most deactivated PROGENy pathways in Megakaryocytes … Report only the pathway names +> sorted by score **(ascending)**. + +The script sorted descending, the recorded answer was descending, the reproducer check said `ok` — +and every agent that honoured the prompt produced the reverse order and was graded wrong. So after +the scripts run, one agent reads every split's prompt, recorded answer and reproducer together and +adds a `review` column of `ok` or `mismatch`, with one line naming the contradiction and one naming +the fix. It never edits `tasks.yaml`: which of the three artifacts to repair is your call. + +``` +task split status review detail +scell train ok ok MAPK;Estrogen;TGFb +scell test ok mismatch Trail;JAK-STAT;Estrogen + +1 split the review flagged + scell/test prompt says ascending; script and answer are descending + fix: say descending in the prompt, or reverse the answer +``` + +The review is on by default and picks its model from `check_model`; it is the one phase that costs +money, so `acumen check --no-review` runs the reproducers alone and spends nothing — what you want +while iterating on a script. `check` takes the same `--auth`, `--stream` and `--log-dir` flags as +the other agentic commands, and `--max-turns`/`--max-usd` bound the reviewer. + +Either phase failing exits non-zero, so `acumen check` works as a gate before a pass. + +A task that needs no code to answer (a licence, a supported species, a documented default) sets +`needs_script: false`; its reproducer column reads `n/a` rather than counting as a gap, and its +prompt and answer are still reviewed. + +The reproducers hold the answers to the held-out test split, so nothing must feed them to an agent +under test. They are safe where they are: `bench`, `draft`, and `improve` confine their agents to +explicit read roots that never include your project directory, and the reviewer reads a staged copy +with no path back to `tasks.yaml`. + `acumen tasks`, `acumen draft`, and `acumen improve` each accept `--feedback "…"` to steer the agent with context it can't infer — which functionality to skip when generating tasks, what a skill should emphasise or fix. The guidance is added to the prompt without overriding the @@ -67,10 +135,103 @@ train/test isolation, and for `draft`/`improve` it is recorded in the version's shown in the report. (Don't paste held-out test answers into `improve` feedback — that would defeat the split.) -`draft`, `improve`, `tasks`, and `ship` each drive a long autonomous agent. Every run writes a -live `logs/acumen--.jsonl` (one event per step, flushed as it goes — so you -can watch progress by reading the file) and a rendered `.html` transcript. Add `--stream` to -mirror the conversation to the terminal, or `--log-dir` to change where the logs land. +Claude and Codex can run side by side. Put both model families in `models` to compare them +in one matrix; model IDs beginning with `claude` use Claude Code, while `gpt-*`, `o1`, +`o3`, `o4`, and `codex-*` use Codex: + +```yaml +models: + - claude-opus-5 + - claude-sonnet-5 + - claude-haiku-4-5-20251001 + - gpt-5.6-sol + - gpt-5.6-terra + - gpt-5.6-luna +``` + +This spans each provider's quality/cost range; it is not a claim that the tiers are +one-to-one equivalents. + +Neither backend is required. Claude is an optional dependency and Codex is an external CLI, +so install only the one you run — `pip install acumen[claude]`, or plain `acumen` plus the +`codex` CLI on `PATH`. Selecting a model whose backend is missing fails immediately, with the +install command, before acumen prepares a target or spends anything. + +Claude API runs use `ANTHROPIC_API_KEY`; Codex API runs use `CODEX_API_KEY` (or +`OPENAI_API_KEY`). The meta-agent commands also accept a Codex model through their +`*_model` config keys or `--model`. + +Every agentic command — `bench` included — takes `--auth {auto,session,api}` and defaults to +the provider's logged-in subscription, falling back to its API key. Both billing modes report +tokens, so Acumen can calculate the same API-rate estimate for either. Under `session`, that +estimate is what the run *would* have cost at API rates, not money billed — so each run records +its `auth_mode` alongside the figure. + +If the selected subscription runs out of usage or the API account runs out of credit, Acumen +invalidates the pass instead of scoring that as an agent failure: it prints the provider error, +cancels remaining cells for that provider, lets other providers finish all running and queued +cells, and exits non-zero. Replenish the credential and rerun the same command; automatic resume +retries the invalid and cancelled cells. Reports and `improve` refuse invalid quota/credit +evidence. + +`max_turns` and `max_usd` apply to both providers, but they are not equally strict for Codex, +which has no cap of its own — acumen enforces both against its event stream: + +- **`max_turns` bounds the run.** One `codex exec` is a single Codex turn however much work + happens inside it, so turns are counted in completed model actions (a message, a command, a + file change, a tool or search call) and the agent is stopped at the cap. +- **`max_usd` cannot.** Codex reports usage once, when the turn ends, so a breach is only + visible after the money is spent. The run is recorded as a budget failure — the same outcome + Claude gives it — but bound Codex spend with `max_turns`. acumen prints this before the pass. + +**Every cost acumen shows is inferred from tokens.** Each run records its breakdown (fresh +input, cache reads, cache writes, and output) and Acumen prices it with the rate table stored +in `result.json`. That gives Claude and Codex one comparable basis and prevents an old +benchmark from being silently re-priced, so it is what `cost_usd` holds and what every figure, +table, CSV column and console line reports. Where a backend supplies a dollar figure of its own +it is recorded beside it as `provider_cost_usd` (`recorded_cost_usd` in the report's sidecar +CSV), with the gap between the two, but nothing is plotted or tallied from it: Claude's SDK +total covers nested subagents that the run's own usage block does not, so a console reading it +would disagree with the report it summarises. A model no layer prices stays unpriced even when +the provider reported dollars, since one run on a basis the rest of the pass is not on is worse +than a visible gap. + +**Rates are read from the providers' pricing pages, never shipped with the package.** Prices +move, and each run's cost is frozen into its `result.json` and never recomputed, so a table +compiled into a release would store numbers that were already wrong. `bench` resolves rates +before it spends anything and **fails the pass** if the pages cannot be read: cost is a headline +metric, and a benchmark that cannot establish rates has not earned the numbers it would print. +`draft`, `improve`, `tasks`, `check`, and `ship` fetch too but degrade to unpriced instead — their +cost line is progress reporting, not stored evidence. + +Alongside the rates themselves each run records `price_source` (`config` or `fetched`) and +`price_rates_as_of`, so a pass run in August and another in October stay individually +attributable and one report can cover both without restating either. When arms in a report were +priced on different dates, the report says so: the cost gap between them includes the price +change, not only the skill's effect. + +```bash +acumen prices # the rates in use today, and where each came from +acumen prices --refresh # check pinned rates against what the providers publish +``` + +Pin rates with a `prices:` block in `config.yaml` to price a model the providers don't publish, +to price a gateway, or to record negotiated rates — pins outrank a live fetch, since only you +know what you are billed. They are also the only rates that can drift unnoticed, which is what +`--refresh` checks; it prints a diff for you to accept and never rewrites anything, because +picking the wrong tier or context band would silently misprice future runs. A model no layer +prices records its tokens and leaves report cost unavailable — never zero, which would read as +free. + +> One consequence worth knowing: Codex's `max_usd` cap is enforced from these same rates, so an +> unpriced model under Codex has no enforceable budget cap. Bound those runs with `max_turns`, +> or pin the rates. + +`draft`, `improve`, `tasks`, `ship`, and `check`'s review phase each drive an autonomous agent. +Every run writes a live `logs/acumen--.jsonl` (one event per step, flushed as it +goes — so you can watch progress by reading the file) and a rendered `.html` transcript. Add +`--stream` to mirror the conversation to the terminal, or `--log-dir` to change where the logs +land. ## Getting started @@ -82,6 +243,15 @@ in particular, the [API documentation][]. You need to have Python 3.12 or newer installed on your system. If you don't have Python installed, we recommend installing [uv][]. +Install the backend you actually run — both are optional, and either alone is a complete +install: + +| you run | install | also needs | +|---|---|---| +| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login | +| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key | +| both | `pip install acumen[all]` | both of the above | +