diff --git a/README.md b/README.md index 122081b..8527ae8 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,13 @@ Because the eval goes through the full harness — system prompt, skill discover - **Negative controls**: `should_trigger: false` inverts assertions to catch accidental skill triggering - **Honest grading**: assertions the grader can't check are *skipped*, not failed; a missing API key warns upfront instead of silently zeroing your pass rate - **Pinned models**: `--agent-model claude-code=haiku --agent-model codex=gpt-5-mini` makes runs reproducible across machines +- **Recorded effective config**: every run's `run_meta.json` records model, reasoning effort, base URL, agent CLI version, and harness version — so "codex passed 97%" is attributable to an exact configuration, not whatever local config the CLI silently inherited +- **Reasoning control and telemetry**: `--reasoning-effort` pins the reasoning setting for agents that support it (codex), and reasoning output tokens are persisted in `timing.json`/`summary.json` when the agent reports them +- **Targeted reruns**: `--eval-id` runs/validates a single case instead of the whole suite — the cheap inner loop after a one-line skill edit +- **Budget guards**: per-case token/cost/duration limits with `warn`, `fail`, or `stop-suite` actions, so a runaway recovery loop can't silently burn quota +- **Live progress**: per-case console lines plus a tailable `progress.jsonl` — long runs are no longer opaque +- **Lifecycle hooks**: `--pre-run-command`, `--post-grade-command`, `--post-run-command` plug external setup, side-effect graders, and teardown into the run with structured `ASE_*` metadata +- **One-file summaries**: every run writes `summary.json` (pass rates, failures, token/cost totals, cleanup state); `agent-skill-eval status` reads it back without a model call - **Scoped cleanup**: only removes artifacts recorded in `cleanup.json`; never closes unrelated PRs or deletes unrelated branches - **Re-grading**: `agent-skill-eval grade` re-grades saved outputs without re-running agents - **Markdown reports**: paste `agent-skill-eval report --format markdown` straight into a PR or blog post @@ -151,10 +158,17 @@ agent-skill-eval run \ --skill --evals \ --agent opencode --agent claude-code --agent codex \ [--agent-model claude-code=haiku] [--agent-model codex=gpt-5-mini] \ + [--reasoning-effort medium] \ [--harness-base-url https://openrouter.ai/api/v1] \ [--runs 3] [--concurrency 2] [--iteration 1] \ [--baseline/--no-baseline] [--cleanup] \ [--timeout 600] [--retries 1] \ + [--eval-id wif-too-long-stop] \ + [--max-total-tokens-per-case 200000] [--budget-action fail] \ + [--pre-run-command "python scripts/setup.py"] \ + [--post-grade-command "python scripts/grade_external.py"] \ + [--post-run-command "python scripts/teardown.py"] \ + [--pricing-config pricing.json] \ [--grader-model deepseek/deepseek-v4-flash] [--grader-base-url URL] \ [--source-repo https://github.com/foo/bar.git] \ [--workspace ./eval-workspace] @@ -163,17 +177,32 @@ agent-skill-eval run \ Key options: - `--agent-model, -m`: model per agent as `agent=model`; a bare value applies to all agents. Repeatable. +- `--reasoning-effort`: reasoning setting passed to agents that support it (codex: `model_reasoning_effort`, e.g. `minimal`/`low`/`medium`/`high`/`xhigh`). Without it, agents silently inherit their own local config (e.g. `~/.codex/config.toml`) — two machines can then benchmark different reasoning settings under the same command line. Agents without a pass-through flag print a warning and record `null`. The effective value lands in `run_meta.json`. - `--harness-base-url`: injected as `ANTHROPIC_BASE_URL` (claude-code) / `OPENAI_BASE_URL` (codex, opencode) into the agent process. - `--runs, -n`: repeat each (eval, agent, config) N times; enables pass@k stats. Results land in `run-1/`, `run-2/`, ... subdirectories. - `--timeout` / `--retries`: per-run agent timeout (default 600s) and retries on timeout or non-zero exit (default 1). Also settable via `ASE_AGENT_TIMEOUT` / `ASE_AGENT_RETRIES`. +- `--eval-id`: run only the named case(s); repeatable. Unknown ids fail fast with the list of available ids, before any model call. The applied filter is recorded in `evals_meta.json` (`selected_eval_ids`) and `summary.json` (`eval_ids`), and all artifacts naturally describe only the selected cases. The cheap inner loop after a small skill edit is `validate --eval-id X` then `run --eval-id X`. +- Budget guards (`--max-input-tokens-per-case`, `--max-non-cached-input-tokens-per-case`, `--max-output-tokens-per-case`, `--max-total-tokens-per-case`, `--max-reasoning-tokens-per-case`, `--max-cost-per-case`, `--max-duration-per-case`, `--budget-action warn|fail|stop-suite`): per-case limits checked after each agent run (mid-run kills remain `--timeout`'s job — budgets exist because a case can finish within the timeout and still be far too expensive). A violation is recorded in `timing.json` (`budget_exceeded`, `budget_reason`); with `fail` (the default) it also appends a failed `method: "budget"` assertion to the run's `grading.json`, and with `stop-suite` it additionally skips every run that hasn't started yet — useful to stop a live side-effect suite before more remote branches/PRs get created. `--max-cost-per-case` only fires when the run has a cost source (see Cost reporting); likewise `--max-reasoning-tokens-per-case` and `--max-non-cached-input-tokens-per-case` only fire when the run reported those values — an agent exposing less telemetry is never failed over an unknown quantity. +- Lifecycle hooks (`--pre-run-command`, `--post-grade-command`, `--post-run-command`): see [Lifecycle hooks](#lifecycle-hooks). +- `--pricing-config`: JSON file with USD-per-token rates for agents whose CLI reports no cost (see Cost reporting). + +While the suite runs, each case prints start/finish lines (elapsed, tokens in/cached/out, cost), and `progress.jsonl` in the iteration directory gets one JSON event per line (`suite_started`, `run_started`, `agent_started`, `agent_finished`, `run_finished`, `budget_exceeded`, ...) — `tail -f` it from another shell to watch a long run. ### `report` ```bash -agent-skill-eval report --workspace [--iteration N] [--format table|markdown] [--show-evidence] +agent-skill-eval report --workspace [--iteration N] [--format table|markdown] [--show-evidence] [--failures-only] +``` + +`--show-evidence` prints the evidence string for every failed or skipped assertion — the state diff for deterministic checks, the judge's reasoning for LLM checks. `--failures-only` limits the per-eval detail to runs that had at least one failed assertion. `--format markdown` emits a paste-ready table. + +### `status` + +```bash +agent-skill-eval status --workspace [--iteration N] ``` -`--show-evidence` prints the evidence string for every failed or skipped assertion — the state diff for deterministic checks, the judge's reasoning for LLM checks. `--format markdown` emits a paste-ready table. +Answers "what happened in the last run?" without a model call: pass rates per configuration, failed/skipped/errored runs, token totals with the cached share, total cost (or `unavailable`), budget violations, hook outcomes, and whether `cleanup.json` recorded remote side effects. Reads the iteration's `summary.json`; for runs that predate it, a reduced summary is rebuilt from the saved artifacts. ### `compare` @@ -181,15 +210,15 @@ agent-skill-eval report --workspace [--iteration N] [--format agent-skill-eval compare --workspace 1 2 ``` -Side-by-side pass rates of two iterations, per configuration, with the change — the feedback loop for iterating on a SKILL.md. +Side-by-side pass rates of two iterations, per configuration, with pass-rate, time, and token deltas — the feedback loop for iterating on a SKILL.md. ### `validate` ```bash -agent-skill-eval validate path/to/evals.json +agent-skill-eval validate path/to/evals.json [--eval-id ] ``` -Schema check plus referenced-fixture existence and duplicate-id detection. Exit code 1 on any problem (CI-friendly). +Schema check plus referenced-fixture existence and duplicate-id detection. Exit code 1 on any problem (CI-friendly). `--eval-id` validates only the selected case(s) (duplicate-id detection still covers the whole file); pair it with `run --eval-id` to confirm a targeted case is well-formed before spending a model call on it. ### `list` @@ -221,6 +250,24 @@ Only closes PRs and deletes remote branches recorded in `cleanup.json`. Never to agent-skill-eval init my-skill [--output ./examples] ``` +## Lifecycle hooks + +Skills with external side effects (push, PR/MR creation, provider APIs) need setup, provider-specific grading, and teardown that the harness can't know about. Instead of hard-coding providers, `run` exposes three shell-command hook points; each receives run metadata as `ASE_*` environment variables on top of the parent environment: + +- `--pre-run-command` (repeatable): runs once before any agent case — seed or reset a scratch repo, run static checks on the skill text. A non-zero exit **aborts the suite before a single model call**. Env: `ASE_SKILL_NAME`, `ASE_SKILL_DIR`, `ASE_EVALS_PATH`, `ASE_ITERATION`, `ASE_ITERATION_DIR`, `ASE_WORKSPACE_BASE`, `ASE_SOURCE_REPO`, `ASE_RUN_ID`, `ASE_EVAL_IDS` (comma-separated). +- `--post-grade-command` (repeatable): runs once per (eval, agent, config) run, after `grading.json` is written and **before the workspace is deleted** — verify remote state, check side-effect files. On top of the suite-level vars it gets `ASE_EVAL_ID`, `ASE_AGENT`, `ASE_WITH_SKILL` (`1`/`0`), `ASE_RUN_INDEX`, `ASE_WORKSPACE_PATH`, `ASE_OUTPUT_DIR`, `ASE_PRE_STATE_PATH`, `ASE_POST_STATE_PATH`, `ASE_TIMING_PATH`, `ASE_RUN_META_PATH`, `ASE_GRADING_PATH`. If the hook prints a JSON array of `{"text": ..., "passed": ..., "evidence": ...}` objects to stdout, they are appended to the run's `grading.json` as `method: "hook"` assertion results and the summary is recomputed — external checks land in the same artifact `report` reads. A non-zero exit appends one failed hook check. Raw hook output is saved to `outputs/post_grade_hooks.json`. +- `--post-run-command` (repeatable): runs once after the suite (teardown, cleanup verification). Failures are recorded in `summary.json` and printed, but don't fail the run — the results already exist. + +Hook commands run through the shell from the invocation directory, with a timeout of `ASE_HOOK_TIMEOUT` seconds (default 600). + +```bash +agent-skill-eval run --skill ./skills/my-skill --evals ./evals/evals.json --agent codex \ + --eval-id feature-branch \ + --pre-run-command "python scripts/setup_source_repo.py" \ + --post-grade-command "python scripts/grade_remote_state.py" \ + --post-run-command "python scripts/teardown_remote.py" +``` + ## Example skills Five example skills ship with the repo, chosen to exercise different grading surfaces: @@ -243,18 +290,21 @@ Each has a matching eval suite under [`examples/`](examples/). `skills/` holds t eval-workspace/ └── -workspace/ └── iteration-1/ - ├── evals_meta.json # eval definitions (used by `grade`) + ├── evals_meta.json # eval definitions + selected_eval_ids (used by `grade`) ├── cleanup.json # manifest of artifacts created by this run ├── benchmark.json # aggregate stats per (agent, config) + ├── summary.json # one-file run summary (used by `status`) + ├── progress.jsonl # live per-run events; tail -f during the run └── eval-/// # config = with_skill | without_skill ├── run-N/ # only when --runs > 1 ├── outputs/ │ ├── output.txt # final agent output │ ├── stdout.log / stderr.log + │ ├── post_grade_hooks.json # only with --post-grade-command │ └── pre_state.json / post_state.json - ├── timing.json # tokens, duration, exit_code, timed_out, retries + ├── timing.json # tokens (incl. non-cached/reasoning splits), duration, exit_code, timed_out, retries, budget verdict ├── grading.json # per-assertion results - └── run_meta.json # agent, with_skill, run_index, ... + └── run_meta.json # agent, with_skill, run_index, model, reasoning_effort, base_url, agent_cli_version, harness_version, ... ``` ### grading.json @@ -316,6 +366,7 @@ How to read it: the **delta rows** are the skill's measured value per agent — - `OPENROUTER_API_KEY` / `OPENAI_API_KEY`: API key for LLM rubric grading - `OPENAI_BASE_URL`: custom grader endpoint (defaults to OpenRouter) - `ASE_AGENT_TIMEOUT` / `ASE_AGENT_RETRIES`: harness timeout/retry defaults +- `ASE_HOOK_TIMEOUT`: lifecycle hook command timeout in seconds (default 600) - `ASE_KEEP_WORKSPACE`: keep per-eval workspaces for debugging ## Agent-specific details @@ -328,9 +379,25 @@ How to read it: the **delta rows** are the skill's measured value per agent — ### Cost reporting -`timing.json` records `cost_usd` per run, taken from the agent CLI itself (claude: `total_cost_usd`, opencode: per-step `cost`; codex reports nothing). One sharp edge: the claude CLI prices runs at **Anthropic list prices** regardless of the endpoint it talks to. When you route claude-code through OpenRouter (`--harness-base-url` or `ANTHROPIC_BASE_URL`), agent-skill-eval reconciles the cost by recomputing it from the run's token counts and OpenRouter's published per-model rates. `timing.json` then shows: +`timing.json` records `cost_usd` per run, taken from the agent CLI itself (claude: `total_cost_usd`, opencode: per-step `cost`). When no cost is available, `cost_usd` is `null` — **unavailable, not $0.00** — and reports/`benchmark.json` show `n/a` instead of a misleading zero. Cost stats are computed only over the runs that reported a cost (`cost_runs` in `benchmark.json`, `runs_with_cost` in `summary.json`). + +Codex's CLI reports token counts but no cost. To get cost numbers anyway, pass `--pricing-config pricing.json` with your provider's USD-per-token rates, keyed by the exact `--agent-model` value (fields mirror OpenRouter's pricing schema): + +```json +{ + "gpt-5.4-mini": { + "prompt": 2.5e-07, + "completion": 2e-06, + "input_cache_read": 2.5e-08 + } +} +``` + +The computed value lands in `cost_usd` with `cost_usd_source: "pricing-config"`. Cached input is billed at `input_cache_read` (falling back to `prompt`); note that codex and opencode report `input_tokens` *including* cache reads while claude-code reports them separately — the harness accounts for this per agent. `summary.json` splits the totals (`tokens.cached`, `tokens.non_cached_input`, `tokens.cached_pct`), because on long agent sessions most input tokens are cache reads and the raw `input_tokens` number wildly overstates billable spend. Each run's `timing.json` also persists `non_cached_input_tokens` (computed with that run's CLI semantics) and `reasoning_output_tokens` when the agent reports reasoning telemetry (codex `turn.completed`, opencode `step_finish`) — `null` means unreported, not zero. + +One sharp edge: the claude CLI prices runs at **Anthropic list prices** regardless of the endpoint it talks to. When you route claude-code through OpenRouter (`--harness-base-url` or `ANTHROPIC_BASE_URL`), agent-skill-eval reconciles the cost by recomputing it from the run's token counts and OpenRouter's published per-model rates. `timing.json` then shows: -- `cost_usd_source`: `"cli"` (the CLI's own number), `"openrouter-pricing"` (reconciled), or `"cli-unreconciled"` (reconciliation failed — e.g. a short model alias like `haiku` that can't be mapped to an OpenRouter slug — so `cost_usd` is the CLI's list-price estimate and actual billing differs) +- `cost_usd_source`: `"cli"` (the CLI's own number), `"openrouter-pricing"` (reconciled), `"pricing-config"` (computed from `--pricing-config`), or `"cli-unreconciled"` (reconciliation failed — e.g. a short model alias like `haiku` that can't be mapped to an OpenRouter slug — so `cost_usd` is the CLI's list-price estimate and actual billing differs) - `cost_usd_cli`: the CLI's original estimate, kept alongside the reconciled value Pin full model IDs (`claude-haiku-4-5-20251001`, not `haiku`) to keep reconciliation working. diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..da293b9 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,33 @@ +# Roadmap + +Planned harness improvements, distilled from running real eval suites against side-effect-heavy skills. Ordered roughly by value-to-effort; items are independent unless noted. + +## 1. Record a skill content hash in run artifacts + +**What:** Hash the skill's content (SKILL.md plus bundled resources) at run time and record it in `run_meta.json` and `evals_meta.json`. `compare` warns when the hash differs between the iterations being compared. + +**Why:** Iteration-vs-iteration deltas are only meaningful when exactly one thing changed. In practice, skill wording and run configuration (model, reasoning effort) often change between iterations, and the comparison silently mixes both effects — e.g. attributing an accuracy gain to a cheaper reasoning setting when it actually came from a skill edit. The model configuration is already recorded; the skill content is the missing half of reproducibility. + +## 2. Side-effect tags on eval cases + +**What:** An optional per-case field such as `side_effect_level: static | local-only | live-safe | live-external`, plus a way to run by tag (e.g. `--side-effect-level live-safe`). + +**Why:** For skills with external side effects, the cheap and safe iteration order is: negative/no-side-effect cases first, one happy-path smoke next, full live suite last. Today that ordering is user discipline plus hand-maintained `--eval-id` lists. Tagging makes it a structural property of the suite: targeted safe runs become the default path, and a full live suite becomes a deliberate choice. Also gives reports a place to show which cases can create remote state. + +## 3. "How to iterate cheaply" documentation + +**What:** A README section laying out the recommended loop: `validate --eval-id` → targeted `run --eval-id` with budget guards → `status` → full suite only as a final gate. Include the decision tree: accuracy low → inspect evidence and patch the skill, rerun the one failing case; time high → lower reasoning effort or model; tokens high → targeted reruns and pre-run static checks; side-effect risk high → run safe cases first, verify cleanup between runs. + +**Why:** The features for cheap iteration exist, but nothing teaches the workflow they were built for. Users who default to full-suite reruns either overspend or stop running evals. + +## 4. Provider-neutral side-effect manifest and cleanup + +**What:** A manifest format describing expected and recorded external side effects (branches, pull/merge requests) independent of the hosting provider, with cleanup, cleanup dry-run, and read-only cleanup-verify operating from the manifest. Provider adapters (GitHub, GitLab) implement the operations. + +**Why:** Cleanup is currently GitHub-oriented; evals against other providers need hand-built teardown and verification scripts. Side-effect setup, grading, cleanup, and reporting should share one source of truth, and users need a read-only proof that remote state is clean before the next run. + +## 5. Agent configuration variants as the comparison unit + +**What:** Allow one run to evaluate the same agent under multiple configurations (model, reasoning effort), e.g. a repeatable `--agent-config` flag, with benchmark rows and deltas keyed by configuration rather than agent alone. A cross-run matrix view (case × configuration × pass rate × runtime × tokens × cost) aggregating over saved summaries belongs to the same theme. + +**Why:** The real optimization space is agent × model × reasoning setting, not agent alone. Comparing two reasoning settings today takes two full invocations plus manual artifact diffing. This is the largest item here — it touches benchmark keys and artifact layout — so it should come after the items above. diff --git a/src/agent_skill_eval/cli.py b/src/agent_skill_eval/cli.py index ecac095..6efb02f 100644 --- a/src/agent_skill_eval/cli.py +++ b/src/agent_skill_eval/cli.py @@ -15,8 +15,11 @@ from agent_skill_eval import __version__ # noqa: E402 from agent_skill_eval.graders import LLMGrader, grade_assertions # noqa: E402 -from agent_skill_eval.models import AgentType, CleanupManifest, EvalSuite # noqa: E402 +from agent_skill_eval.hooks import HookError # noqa: E402 +from agent_skill_eval.models import AgentType, BudgetConfig, CleanupManifest, EvalSuite # noqa: E402 from agent_skill_eval.runner import EvalRunner # noqa: E402 +from agent_skill_eval.summary import build_summary_from_artifacts, load_summary # noqa: E402 +from agent_skill_eval.summary import leaf_config_dirs as _leaf_config_dirs # noqa: E402 def _version_callback(value: bool) -> None: @@ -84,12 +87,85 @@ def run( "(ANTHROPIC_BASE_URL for claude-code, OPENAI_BASE_URL for codex/opencode)", ), runs: int = typer.Option(1, "--runs", "-n", help="Number of runs per (eval, agent, config) for pass@k stats"), + reasoning_effort: Optional[str] = typer.Option( + None, + "--reasoning-effort", + help="Reasoning effort passed to agents that support it (codex: " + "model_reasoning_effort, e.g. minimal/low/medium/high/xhigh). Without it, " + "agents inherit their own local config. Recorded in run_meta.json.", + ), agent_timeout: Optional[int] = typer.Option( None, "--timeout", help="Per-run agent timeout in seconds (default 600, env ASE_AGENT_TIMEOUT)" ), agent_retries: Optional[int] = typer.Option( None, "--retries", help="Retries on agent timeout/non-zero exit (default 1, env ASE_AGENT_RETRIES)" ), + eval_ids: list[str] = typer.Option( + [], + "--eval-id", + help="Only run the eval case with this id. Repeatable for multiple cases.", + ), + max_input_tokens_per_case: Optional[int] = typer.Option( + None, "--max-input-tokens-per-case", help="Budget: max input tokens per run (checked after the run)" + ), + max_non_cached_input_tokens_per_case: Optional[int] = typer.Option( + None, + "--max-non-cached-input-tokens-per-case", + help="Budget: max non-cached (billable-rate) input tokens per run (checked after the run)", + ), + max_output_tokens_per_case: Optional[int] = typer.Option( + None, "--max-output-tokens-per-case", help="Budget: max output tokens per run (checked after the run)" + ), + max_reasoning_tokens_per_case: Optional[int] = typer.Option( + None, + "--max-reasoning-tokens-per-case", + help="Budget: max reasoning output tokens per run; only enforced for " + "agents that report reasoning telemetry (checked after the run)", + ), + max_total_tokens_per_case: Optional[int] = typer.Option( + None, "--max-total-tokens-per-case", help="Budget: max total tokens per run (checked after the run)" + ), + max_cost_per_case: Optional[float] = typer.Option( + None, "--max-cost-per-case", help="Budget: max USD cost per run (needs a cost source; checked after the run)" + ), + max_duration_per_case: Optional[float] = typer.Option( + None, + "--max-duration-per-case", + help="Budget: max seconds per run including retries (post-run check; --timeout still kills mid-run)", + ), + budget_action: str = typer.Option( + "fail", + "--budget-action", + help="What a budget violation does: warn (print only), fail (failed 'budget' " + "assertion in grading), stop-suite (fail + skip runs not yet started)", + ), + pre_run_commands: list[str] = typer.Option( + [], + "--pre-run-command", + help="Shell command run once before any agent case (setup/preflight). " + "Non-zero exit aborts the suite before any model call. Repeatable. " + "Receives ASE_* run metadata env vars.", + ), + post_grade_commands: list[str] = typer.Option( + [], + "--post-grade-command", + help="Shell command run per case after grading (external side-effect grader). " + "stdout JSON [{text, passed, evidence}] is merged into grading.json; " + "non-zero exit adds a failed check. Repeatable. Receives ASE_* env vars.", + ), + post_run_commands: list[str] = typer.Option( + [], + "--post-run-command", + help="Shell command run once after the suite (teardown). Failures are " + "recorded in summary.json but don't fail the run. Repeatable.", + ), + pricing_config: Optional[Path] = typer.Option( + None, + "--pricing-config", + help="JSON file mapping model id -> {prompt, completion, input_cache_read, " + "input_cache_write} USD-per-token rates, used to compute cost_usd for " + "agents whose CLI reports no cost (e.g. codex)", + ), ): """Run skill evaluations.""" if not skill.exists(): @@ -124,24 +200,57 @@ def run( "Assertions that need LLM rubric grading will be SKIPPED (not failed).[/yellow]" ) - runner = EvalRunner( - skill_path=skill, - evals_path=evals, - workspace_base=workspace, - agents=agent_types, - concurrency=concurrency, - with_baseline=baseline, - grader_model=grader_model, - grader_base_url=grader_base_url, - source_repo=source_repo, - agent_models=agent_model_map, - harness_base_url=harness_base_url, - agent_timeout=agent_timeout, - agent_max_retries=agent_retries, - runs=runs, + if budget_action not in ("warn", "fail", "stop-suite"): + console.print(f"[red]Unknown --budget-action: {budget_action}. Choose warn, fail, or stop-suite.[/red]") + raise typer.Exit(1) + budget = BudgetConfig( + max_input_tokens=max_input_tokens_per_case, + max_non_cached_input_tokens=max_non_cached_input_tokens_per_case, + max_output_tokens=max_output_tokens_per_case, + max_total_tokens=max_total_tokens_per_case, + max_reasoning_tokens=max_reasoning_tokens_per_case, + max_cost_usd=max_cost_per_case, + max_duration_seconds=max_duration_per_case, + action=budget_action, ) - result_dir = runner.run(iteration) + pricing = _load_pricing_config(pricing_config) + + try: + runner = EvalRunner( + skill_path=skill, + evals_path=evals, + workspace_base=workspace, + agents=agent_types, + concurrency=concurrency, + with_baseline=baseline, + grader_model=grader_model, + grader_base_url=grader_base_url, + source_repo=source_repo, + agent_models=agent_model_map, + harness_base_url=harness_base_url, + agent_timeout=agent_timeout, + agent_max_retries=agent_retries, + runs=runs, + eval_ids=eval_ids, + budget=budget, + pricing=pricing, + pre_run_commands=pre_run_commands, + post_grade_commands=post_grade_commands, + post_run_commands=post_run_commands, + reasoning_effort=reasoning_effort, + ) + except ValueError as e: + # Unknown --eval-id, invalid evals.json, etc.: fail before any model call. + console.print(f"[red]{e}[/red]") + raise typer.Exit(1) + + try: + result_dir = runner.run(iteration) + except HookError as e: + console.print(f"[red]{e}[/red]") + console.print("[red]Suite aborted by pre-run hook; no agent was run.[/red]") + raise typer.Exit(1) console.print(f"\n[bold green]Done! Results in: {result_dir}[/bold green]") if auto_cleanup: @@ -179,6 +288,31 @@ def _parse_agent_models(specs: list[str], agent_types: list[AgentType]) -> dict[ return mapping +def _load_pricing_config(path: Optional[Path]) -> dict[str, dict]: + """Load a ``--pricing-config`` file: ``{model_id: {rate_field: usd_per_token}}``. + + Rate fields mirror OpenRouter's pricing schema: ``prompt``, ``completion``, + ``input_cache_read``, ``input_cache_write``. + """ + if path is None: + return {} + if not path.exists(): + console.print(f"[red]Pricing config not found: {path}[/red]") + raise typer.Exit(1) + if not path.is_file(): + console.print(f"[red]Pricing config {path} is not a file[/red]") + raise typer.Exit(1) + try: + data = json.loads(path.read_text()) + except json.JSONDecodeError as e: + console.print(f"[red]Invalid JSON in pricing config {path}: {e}[/red]") + raise typer.Exit(1) + if not isinstance(data, dict) or not all(isinstance(v, dict) for v in data.values()): + console.print(f"[red]Pricing config {path} must map model ids to rate objects[/red]") + raise typer.Exit(1) + return data + + def _cleanup_iteration(iteration_dir: Path, workspace_base: Path) -> None: """Clean up only artifacts recorded for this iteration's cleanup manifest. @@ -419,24 +553,6 @@ def grade( console.print(f"[green]Re-grading complete! Updated {updated} grading file(s).[/green]") -def _leaf_config_dirs(agent_dir: Path) -> list[Path]: - """Yield config dirs that directly contain run artifacts. - - Layouts: ``agent/with_skill/outputs`` (single run) or - ``agent/with_skill/run-N/outputs`` (``--runs N`` > 1). - """ - leaves: list[Path] = [] - for config_dir in sorted(agent_dir.iterdir()): - if not config_dir.is_dir(): - continue - run_dirs = sorted(p for p in config_dir.glob("run-*") if p.is_dir()) - if run_dirs: - leaves.extend(run_dirs) - else: - leaves.append(config_dir) - return leaves - - def _collect_results(workspace: Path) -> dict[str, dict]: """Reconstruct per-run results for benchmark recompute. @@ -498,8 +614,10 @@ def _benchmark_rows(benchmark: dict) -> list[tuple[str, str, str, str, str, str] pass_rate = stats.get("pass_rate", {}) time_s = stats.get("time_seconds", {}) tokens = stats.get("tokens", {}) - # benchmark.json from runs before cost stats existed lacks the field. - cost = stats.get("cost_usd") or {} + # None/absent means cost is unavailable for these runs (e.g. codex + # without --pricing-config), which is different from a real $0. + cost = stats.get("cost_usd") + cost_str = f"{cost.get('mean', 0):.4f} +/- {cost.get('stddev', 0):.4f}" if cost else "n/a" k = stats.get("k", 1) pass_at_k = f"{stats.get('pass_at_k', 0):.0%} (k={k})" if k > 1 else f"{stats.get('full_pass_rate', 0):.0%}" rows.append( @@ -509,12 +627,22 @@ def _benchmark_rows(benchmark: dict) -> list[tuple[str, str, str, str, str, str] pass_at_k, f"{time_s.get('mean', 0):.1f} +/- {time_s.get('stddev', 0):.1f}", f"{tokens.get('mean', 0):.0f} +/- {tokens.get('stddev', 0):.0f}", - f"{cost.get('mean', 0):.4f} +/- {cost.get('stddev', 0):.4f}", + cost_str, ) ) return rows +def _delta_line(delta: dict) -> str: + cost = delta.get("cost_usd") + cost_str = f"{cost:+.4f} USD" if cost is not None else "n/a" + return ( + f"pass rate {delta.get('pass_rate', 0):+.1%}, " + f"time {delta.get('time_seconds', 0):+.1f}s, tokens {delta.get('tokens', 0):+.0f}, " + f"cost {cost_str}" + ) + + def _benchmark_deltas(benchmark: dict) -> dict[str, dict]: deltas = benchmark.get("deltas") or {} if not deltas and benchmark.get("delta"): @@ -535,11 +663,7 @@ def _print_markdown_report(iter_name: str, benchmark: dict) -> None: if deltas: print("\n**Delta (with_skill - without_skill):**\n") for agent, delta in deltas.items(): - print( - f"- `{agent}`: pass rate {delta.get('pass_rate', 0):+.1%}, " - f"time {delta.get('time_seconds', 0):+.1f}s, tokens {delta.get('tokens', 0):+.0f}, " - f"cost {delta.get('cost_usd', 0):+.4f} USD" - ) + print(f"- `{agent}`: {_delta_line(delta)}") def _print_table_report(iter_name: str, benchmark: dict) -> None: @@ -557,19 +681,12 @@ def _print_table_report(iter_name: str, benchmark: dict) -> None: if deltas: console.print("\n[bold]Delta (with_skill - without_skill):[/bold]") for agent, delta in deltas.items(): - console.print( - f" {agent}: pass rate {delta.get('pass_rate', 0):+.1%}, " - f"time {delta.get('time_seconds', 0):+.1f}s, tokens {delta.get('tokens', 0):+.0f}, " - f"cost {delta.get('cost_usd', 0):+.4f} USD" - ) + console.print(f" {agent}: {_delta_line(delta)}") -def _print_eval_details(iter_dir: Path, show_evidence: bool, markdown: bool) -> None: +def _print_eval_details(iter_dir: Path, show_evidence: bool, markdown: bool, failures_only: bool = False) -> None: for eval_dir in sorted(iter_dir.glob("eval-*")): - if markdown: - print(f"\n#### {eval_dir.name}\n") - else: - console.print(f"\n[bold]{eval_dir.name}[/bold]") + header_printed = False for agent_dir in sorted(eval_dir.iterdir()): if not agent_dir.is_dir(): continue @@ -580,6 +697,14 @@ def _print_eval_details(iter_dir: Path, show_evidence: bool, markdown: bool) -> with open(grading_path, encoding="utf-8") as f: grading = json.load(f) summary = grading.get("summary", {}) + if failures_only and summary.get("failed", 0) == 0: + continue + if not header_printed: + if markdown: + print(f"\n#### {eval_dir.name}\n") + else: + console.print(f"\n[bold]{eval_dir.name}[/bold]") + header_printed = True label = f"{agent_dir.name}/{config_dir.relative_to(agent_dir)}" skipped = summary.get("skipped", 0) skipped_note = f", {skipped} skipped" if skipped else "" @@ -612,6 +737,9 @@ def report( show_evidence: bool = typer.Option( False, "--show-evidence", help="Show evidence for failed/skipped assertions from grading.json" ), + failures_only: bool = typer.Option( + False, "--failures-only", help="Only list runs that had at least one failed assertion" + ), ): """Display a summary report of eval results.""" if not workspace.exists(): @@ -645,7 +773,7 @@ def report( else: _print_table_report(iter_dir.name, benchmark) - _print_eval_details(iter_dir, show_evidence, markdown) + _print_eval_details(iter_dir, show_evidence, markdown, failures_only=failures_only) @app.command() @@ -668,26 +796,160 @@ def compare( summary_b = benchmarks[iteration_b].get("run_summary", {}) configs = sorted(set(summary_a) | set(summary_b)) + def _mean(summary: dict, config: str, field: str): + return (summary.get(config, {}).get(field) or {}).get("mean") + + def _change(a, b, fmt: str) -> str: + return format(b - a, fmt) if a is not None and b is not None else "-" + table = Table(title=f"iteration-{iteration_a} vs iteration-{iteration_b}") table.add_column("Configuration", style="cyan") table.add_column(f"Pass Rate (it-{iteration_a})", style="green") table.add_column(f"Pass Rate (it-{iteration_b})", style="green") table.add_column("Change", style="magenta") + table.add_column("Time Δ (s)", style="yellow") + table.add_column("Tokens Δ", style="blue") for config in configs: - a = summary_a.get(config, {}).get("pass_rate", {}).get("mean") - b = summary_b.get(config, {}).get("pass_rate", {}).get("mean") + a = _mean(summary_a, config, "pass_rate") + b = _mean(summary_b, config, "pass_rate") a_str = f"{a:.1%}" if a is not None else "-" b_str = f"{b:.1%}" if b is not None else "-" - change = f"{b - a:+.1%}" if a is not None and b is not None else "-" - table.add_row(config, a_str, b_str, change) + table.add_row( + config, + a_str, + b_str, + _change(a, b, "+.1%"), + _change(_mean(summary_a, config, "time_seconds"), _mean(summary_b, config, "time_seconds"), "+.1f"), + _change(_mean(summary_a, config, "tokens"), _mean(summary_b, config, "tokens"), "+.0f"), + ) console.print(table) +@app.command() +def status( + workspace: Path = typer.Option(..., "--workspace", "-w", help="Path to workspace containing iteration dirs"), + iteration: Optional[int] = typer.Option(None, "--iteration", "-i", help="Iteration to show (default: latest)"), +): + """Show the latest run's summary (pass rate, failures, tokens, cost, cleanup) without any model call. + + Reads the iteration's ``summary.json``; for runs that predate it, a + reduced summary is rebuilt from the saved artifacts. + """ + if not workspace.exists(): + console.print(f"[red]Workspace not found: {workspace}[/red]") + raise typer.Exit(1) + + if iteration is not None: + iter_dir = workspace / f"iteration-{iteration}" + if not iter_dir.exists(): + console.print(f"[red]Iteration directory not found: {iter_dir}[/red]") + raise typer.Exit(1) + else: + iteration_dirs = sorted( + (p for p in workspace.glob("iteration-*") if p.is_dir()), + key=lambda p: int(p.name[len("iteration-") :]) if p.name[len("iteration-") :].isdigit() else -1, + ) + if not iteration_dirs: + console.print(f"[yellow]No iterations found under {workspace}[/yellow]") + raise typer.Exit(1) + iter_dir = iteration_dirs[-1] + + summary = load_summary(iter_dir) + if summary is None: + summary = build_summary_from_artifacts(iter_dir) + if summary is None: + console.print(f"[red]No summary.json and no readable artifacts in {iter_dir}[/red]") + raise typer.Exit(1) + console.print("[yellow]No summary.json (run predates it); rebuilt a reduced summary from artifacts.[/yellow]") + + console.print(f"\n[bold]{iter_dir}[/bold]") + # `or`-defaults everywhere below: summary.json fields can be + # present-but-null (hand-edited or partially written files). + console.print( + f"skill={summary.get('skill_name')} iteration={summary.get('iteration')} " + f"agents={','.join(summary.get('agents') or [])} " + f"eval_ids={','.join(str(e) for e in summary.get('eval_ids') or [])}" + ) + console.print( + f"runs: {summary.get('runs_completed') or 0} completed, " + f"{summary.get('runs_failed_grading') or 0} with failed assertions, " + f"{len(summary.get('errors') or [])} errored, " + f"{len(summary.get('skipped_runs') or [])} skipped; " + f"duration {summary.get('duration_seconds') or 0:.0f}s" + ) + + pass_rates = summary.get("pass_rates") or {} + if pass_rates: + table = Table(title="Pass rates") + table.add_column("Configuration", style="cyan") + table.add_column("Pass rate", style="green") + for config, rate in pass_rates.items(): + table.add_row(config, f"{rate:.1%}" if rate is not None else "-") + console.print(table) + + tokens = summary.get("tokens") or {} + if tokens: + reasoning = tokens.get("reasoning_output") + reasoning_str = f" (reasoning={reasoning:,})" if reasoning is not None else "" + console.print( + f"tokens: input={tokens.get('input', 0):,} " + f"(cached={tokens.get('cached', 0):,}, {tokens.get('cached_pct', 0):.0%}; " + f"non-cached={tokens.get('non_cached_input', 0):,}) " + f"output={tokens.get('output', 0):,}{reasoning_str} total={tokens.get('total', 0):,}" + ) + cost = summary.get("cost") or {} + total_cost = cost.get("total_usd") + cost_str = f"{total_cost:.4f} USD" if total_cost is not None else "unavailable" + console.print(f"cost: {cost_str} ({cost.get('runs_with_cost', 0)}/{cost.get('runs_total', 0)} runs reported cost)") + + for failed in summary.get("failed_runs") or []: + config = "with_skill" if failed.get("with_skill") else "without_skill" + console.print( + f" [red]FAILED {failed.get('eval_id')}/{failed.get('agent')}/{config}: " + f"{', '.join(failed.get('failed_assertions') or []) or 'see grading.json'}[/red]" + ) + for skipped in summary.get("skipped_runs") or []: + console.print(f" [yellow]SKIPPED {skipped.get('run')}: {skipped.get('reason')}[/yellow]") + for error in summary.get("errors") or []: + console.print(f" [red]ERROR {error.get('run')}: {error.get('error')}[/red]") + + budget = summary.get("budget") or {} + for exceeded in budget.get("exceeded_runs") or []: + config = "with_skill" if exceeded.get("with_skill") else "without_skill" + console.print( + f" [yellow]BUDGET {exceeded.get('eval_id')}/{exceeded.get('agent')}/{config}: " + f"{exceeded.get('reason')}[/yellow]" + ) + + cleanup_info = summary.get("cleanup") or {} + branches = cleanup_info.get("remote_branches") or [] + prs = cleanup_info.get("pr_numbers") or [] + if branches or prs: + console.print( + f"[yellow]side effects recorded: {len(branches)} remote branch(es), {len(prs)} PR(s) — " + f"run 'agent-skill-eval cleanup' to remove them[/yellow]" + ) + else: + console.print("side effects recorded: none") + + hooks_info = summary.get("hooks") or {} + for stage, records in hooks_info.items(): + failures = [r for r in records if r.get("exit_code") != 0] + state = f"[red]{len(failures)} failed[/red]" if failures else "ok" + console.print(f"hooks ({stage}): {len(records)} ran, {state}") + + @app.command() def validate( evals: Path = typer.Argument(..., help="Path to an evals.json file to validate"), + eval_ids: list[str] = typer.Option( + [], + "--eval-id", + help="Only validate the eval case with this id. Repeatable. " + "Pairs with 'run --eval-id' for a cheap targeted inner loop.", + ), ): """Validate an evals.json file against the eval suite schema.""" if not evals.exists(): @@ -706,6 +968,21 @@ def validate( console.print(f"[red]Schema validation failed:[/red]\n{e}") raise typer.Exit(1) + # Duplicate ids are checked over the FULL suite (filtering by a + # duplicated id must not hide the duplication), then the file/assertion + # checks run on the selected subset only. + ids = [str(e.id) for e in suite.evals] + duplicates = sorted({i for i in ids if ids.count(i) > 1}) + if duplicates: + console.print(f"[red]Duplicate eval ids: {duplicates}[/red]") + raise typer.Exit(1) + + try: + suite = suite.filtered_by_ids(eval_ids) + except ValueError as e: + console.print(f"[red]{e}[/red]") + raise typer.Exit(1) + missing_files = [] for eval_case in suite.evals: for file_path in eval_case.files: @@ -717,12 +994,6 @@ def validate( console.print(f" - {m}") raise typer.Exit(1) - ids = [str(e.id) for e in suite.evals] - duplicates = sorted({i for i in ids if ids.count(i) > 1}) - if duplicates: - console.print(f"[red]Duplicate eval ids: {duplicates}[/red]") - raise typer.Exit(1) - from agent_skill_eval.graders import classify_assertion llm_assertions: list[tuple[str, str]] = [] @@ -733,7 +1004,8 @@ def validate( if classify_assertion(assertion) == "llm": llm_assertions.append((str(eval_case.id), assertion)) - console.print(f"[green]Valid: {len(suite.evals)} eval(s) for skill '{suite.skill_name}'[/green]") + selected = " (selected by --eval-id)" if eval_ids else "" + console.print(f"[green]Valid: {len(suite.evals)} eval(s) for skill '{suite.skill_name}'{selected}[/green]") if llm_assertions: console.print( f"[yellow]{len(llm_assertions)}/{total_assertions} assertion(s) match no deterministic " diff --git a/src/agent_skill_eval/graders/__init__.py b/src/agent_skill_eval/graders/__init__.py index 9abf4fb..f55f9b3 100644 --- a/src/agent_skill_eval/graders/__init__.py +++ b/src/agent_skill_eval/graders/__init__.py @@ -874,19 +874,29 @@ def grade_assertions( else: all_results = det_results - passed = sum(1 for r in all_results if r.passed) - skipped = sum(1 for r in all_results if r.skipped) - failed = sum(1 for r in all_results if not r.passed and not r.skipped) - total = len(all_results) - graded = total - skipped - return GradingResult( assertion_results=all_results, - summary=GradingSummary( - passed=passed, - failed=failed, - total=total, - pass_rate=passed / graded if graded > 0 else 0.0, - skipped=skipped, - ), + summary=summarize_assertion_results(all_results), + ) + + +def summarize_assertion_results(results: list[AssertionResult]) -> GradingSummary: + """Compute a GradingSummary over assertion results. + + Skipped assertions count toward ``total`` but not toward the pass rate. + Shared by ``grade_assertions`` and the post-grade hook path, which + appends hook-produced results and must recompute the summary. + """ + passed = sum(1 for r in results if r.passed) + skipped = sum(1 for r in results if r.skipped) + failed = sum(1 for r in results if not r.passed and not r.skipped) + total = len(results) + graded = total - skipped + + return GradingSummary( + passed=passed, + failed=failed, + total=total, + pass_rate=passed / graded if graded > 0 else 0.0, + skipped=skipped, ) diff --git a/src/agent_skill_eval/harnesses/__init__.py b/src/agent_skill_eval/harnesses/__init__.py index dc9298c..9862701 100644 --- a/src/agent_skill_eval/harnesses/__init__.py +++ b/src/agent_skill_eval/harnesses/__init__.py @@ -15,10 +15,21 @@ RETRY_BACKOFF_SECONDS = 2.0 +def _rate(entry: dict, field: str) -> float: + try: + return float(entry.get(field) or 0.0) + except (TypeError, ValueError): + return 0.0 + + class AgentHarness(ABC): agent_type: AgentType # Environment variable each agent CLI reads its API base URL from. base_url_env: Optional[str] = None + # Command that prints the agent CLI's version (None = no probe). + version_command: Optional[list[str]] = None + # Whether build_command can pass --reasoning-effort through to the CLI. + supports_reasoning_effort: bool = False def __init__( self, @@ -27,14 +38,22 @@ def __init__( base_url: Optional[str] = None, timeout: Optional[int] = None, max_retries: Optional[int] = None, + pricing: Optional[dict[str, dict]] = None, + reasoning_effort: Optional[str] = None, ): self.workspace = workspace self.model = model self.base_url = base_url + # Only meaningful when supports_reasoning_effort; callers should + # treat the effective value as None for harnesses that ignore it. + self.reasoning_effort = reasoning_effort if self.supports_reasoning_effort else None self.timeout = timeout or int(os.environ.get("ASE_AGENT_TIMEOUT", DEFAULT_TIMEOUT_SECONDS)) if max_retries is None: max_retries = int(os.environ.get("ASE_AGENT_RETRIES", DEFAULT_MAX_RETRIES)) self.max_retries = max_retries + # User-provided model -> rate map (--pricing-config), used to compute + # cost_usd when the CLI itself reports nothing. + self.pricing = pricing or {} @abstractmethod def build_command(self, prompt: str, output_dir: Path) -> list[str]: @@ -44,10 +63,48 @@ def build_command(self, prompt: str, output_dir: Path) -> list[str]: def parse_output(self, stdout: str, stderr: str) -> tuple[str, TimingData]: pass + def non_cached_input_tokens(self, timing: TimingData) -> int: + """Input tokens billed at the non-cached rate. + + Most CLIs (codex, opencode) report input_tokens INCLUDING cache + reads; claude-code reports them separately and overrides this. + """ + return max(timing.input_tokens - timing.cached_tokens, 0) + + def _cost_from_pricing_config(self, timing: TimingData) -> Optional[float]: + """Compute cost from a user-provided --pricing-config, or None. + + Rates are USD per token (same semantics as OpenRouter's pricing + fields): prompt, completion, input_cache_read, input_cache_write. + Matched by exact model string. + """ + if not self.model or self.model not in self.pricing: + return None + if timing.total_tokens <= 0 and timing.cached_tokens <= 0 and timing.cache_creation_tokens <= 0: + return None + entry = self.pricing[self.model] + prompt = _rate(entry, "prompt") + completion = _rate(entry, "completion") + if prompt <= 0 and completion <= 0: + return None + cache_read = _rate(entry, "input_cache_read") or prompt + return ( + self.non_cached_input_tokens(timing) * prompt + + timing.cached_tokens * cache_read + + timing.output_tokens * completion + + timing.cache_creation_tokens * _rate(entry, "input_cache_write") + ) + def finalize_timing(self, timing: TimingData) -> None: """Post-process timing after a run (e.g. cost reconciliation).""" - if timing.cost_usd and timing.cost_usd_source is None: + if timing.cost_usd is not None and timing.cost_usd_source is None: timing.cost_usd_source = "cli" + return + if timing.cost_usd is None: + computed = self._cost_from_pricing_config(timing) + if computed is not None: + timing.cost_usd = computed + timing.cost_usd_source = "pricing-config" def _build_env(self) -> Optional[dict[str, str]]: if self.base_url and self.base_url_env: @@ -114,6 +171,7 @@ def run(self, prompt: str, output_dir: Path) -> tuple[str, TimingData, str, str] capture_output=True, text=True, encoding="utf-8", + errors="replace", timeout=self.timeout, env=env, ) @@ -124,8 +182,16 @@ def run(self, prompt: str, output_dir: Path) -> tuple[str, TimingData, str, str] if result.returncode == 0: break except subprocess.TimeoutExpired as e: - stdout = (e.stdout or b"").decode() if isinstance(e.stdout, bytes) else (e.stdout or "") - stderr = (e.stderr or b"").decode() if isinstance(e.stderr, bytes) else (e.stderr or "") + stdout = ( + (e.stdout or b"").decode("utf-8", errors="replace") + if isinstance(e.stdout, bytes) + else (e.stdout or "") + ) + stderr = ( + (e.stderr or b"").decode("utf-8", errors="replace") + if isinstance(e.stderr, bytes) + else (e.stderr or "") + ) stderr += f"\n[agent-skill-eval] timed out after {self.timeout}s (attempt {attempt + 1}/{attempts})" exit_code = None timed_out = True @@ -146,6 +212,10 @@ def run(self, prompt: str, output_dir: Path) -> tuple[str, TimingData, str, str] timing.exit_code = exit_code timing.timed_out = timed_out timing.retries = retries_used + # Persist the billable input split using this harness's semantics; + # downstream readers must not re-derive it (codex/opencode include + # cache reads in input_tokens, claude-code does not). + timing.non_cached_input_tokens = self.non_cached_input_tokens(timing) self.finalize_timing(timing) with open(output_dir / "stdout.log", "w", encoding="utf-8") as f: @@ -159,6 +229,7 @@ def run(self, prompt: str, output_dir: Path) -> tuple[str, TimingData, str, str] class OpenCodeHarness(AgentHarness): agent_type = AgentType.OPENCODE base_url_env = "OPENAI_BASE_URL" + version_command = ["opencode", "--version"] def build_command(self, prompt: str, output_dir: Path) -> list[str]: cmd = [ @@ -178,6 +249,9 @@ def build_command(self, prompt: str, output_dir: Path) -> list[str]: def parse_output(self, stdout: str, stderr: str) -> tuple[str, TimingData]: timing = TimingData() messages = [] + # None until at least one step reports a cost field: a run whose + # steps never report cost is "cost unavailable", not a free run. + cost: Optional[float] = None for line in stdout.strip().split("\n"): if not line.strip(): @@ -199,11 +273,20 @@ def parse_output(self, stdout: str, stderr: str) -> tuple[str, TimingData]: timing.output_tokens += tokens.get("output", 0) cache = tokens.get("cache", {}) timing.cached_tokens += cache.get("read", 0) - timing.cost_usd += part.get("cost", 0) or 0 + # Only set when a step actually reports the field: a run + # with no reasoning telemetry stays None (unknown), which + # is not the same as reasoning=0. + if "reasoning" in tokens: + timing.reasoning_output_tokens = (timing.reasoning_output_tokens or 0) + ( + tokens.get("reasoning") or 0 + ) + if "cost" in part: + cost = (cost or 0.0) + (part.get("cost") or 0) except json.JSONDecodeError: continue + timing.cost_usd = cost timing.total_tokens = timing.input_tokens + timing.output_tokens return "\n".join(messages), timing @@ -211,6 +294,7 @@ def parse_output(self, stdout: str, stderr: str) -> tuple[str, TimingData]: class ClaudeCodeHarness(AgentHarness): agent_type = AgentType.CLAUDE_CODE base_url_env = "ANTHROPIC_BASE_URL" + version_command = ["claude", "--version"] def build_command(self, prompt: str, output_dir: Path) -> list[str]: cmd = [ @@ -238,7 +322,8 @@ def parse_output(self, stdout: str, stderr: str) -> tuple[str, TimingData]: timing.output_tokens = usage.get("output_tokens", 0) timing.cached_tokens = usage.get("cache_read_input_tokens", 0) timing.cache_creation_tokens = usage.get("cache_creation_input_tokens", 0) - timing.cost_usd = data.get("total_cost_usd", 0.0) or 0.0 + raw_cost = data.get("total_cost_usd") + timing.cost_usd = float(raw_cost) if raw_cost is not None else None timing.total_tokens = timing.input_tokens + timing.output_tokens duration_s = data.get("duration_ms", 0) if duration_s: @@ -267,16 +352,27 @@ def finalize_timing(self, timing: TimingData) -> None: reconciled = reconcile_claude_cost(timing, self.model) if reconciled is None: - timing.cost_usd_source = "cli-unreconciled" + if timing.cost_usd is None: + # The CLI reported no cost either: fall back to the generic + # path (pricing config, or genuinely unavailable). + super().finalize_timing(timing) + else: + timing.cost_usd_source = "cli-unreconciled" return timing.cost_usd_cli = timing.cost_usd timing.cost_usd = reconciled timing.cost_usd_source = "openrouter-pricing" + def non_cached_input_tokens(self, timing: TimingData) -> int: + # Anthropic usage reports input_tokens EXCLUDING cache reads. + return timing.input_tokens + class CodexHarness(AgentHarness): agent_type = AgentType.CODEX base_url_env = "OPENAI_BASE_URL" + version_command = ["codex", "--version"] + supports_reasoning_effort = True def build_command(self, prompt: str, output_dir: Path) -> list[str]: cmd = [ @@ -291,6 +387,11 @@ def build_command(self, prompt: str, output_dir: Path) -> list[str]: ] if self.model: cmd.extend(["--model", self.model]) + if self.reasoning_effort: + # Without this, codex silently inherits model_reasoning_effort + # from ~/.codex/config.toml — two users running the "same" eval + # can benchmark different reasoning settings. + cmd.extend(["-c", f'model_reasoning_effort="{self.reasoning_effort}"']) cmd.append(prompt) return cmd @@ -315,6 +416,12 @@ def parse_output(self, stdout: str, stderr: str) -> tuple[str, TimingData]: timing.input_tokens += usage.get("input_tokens", 0) timing.output_tokens += usage.get("output_tokens", 0) timing.cached_tokens += usage.get("cached_input_tokens", 0) + # Only set when the turn reports the field; absent + # telemetry stays None (unknown), not 0. + if "reasoning_output_tokens" in usage: + timing.reasoning_output_tokens = (timing.reasoning_output_tokens or 0) + ( + usage.get("reasoning_output_tokens") or 0 + ) except json.JSONDecodeError: continue @@ -330,7 +437,9 @@ def build_command(self, prompt: str, output_dir: Path) -> list[str]: return ["fake-agent", prompt] def parse_output(self, stdout: str, stderr: str) -> tuple[str, TimingData]: - timing = TimingData(total_tokens=1, input_tokens=1, duration_ms=1) + # The fake agent makes no API calls, so its cost is a real 0.0 + # (known-free), not an unavailable None. + timing = TimingData(total_tokens=1, input_tokens=1, duration_ms=1, cost_usd=0.0) return stdout, timing def run(self, prompt: str, output_dir: Path) -> tuple[str, TimingData, str, str]: @@ -344,8 +453,9 @@ def run(self, prompt: str, output_dir: Path) -> tuple[str, TimingData, str, str] stdout = final_output stderr = "" timing = TimingData( - total_tokens=1, input_tokens=1, output_tokens=0, cached_tokens=0, duration_ms=1, exit_code=0 + total_tokens=1, input_tokens=1, output_tokens=0, cached_tokens=0, duration_ms=1, exit_code=0, cost_usd=0.0 ) + timing.non_cached_input_tokens = self.non_cached_input_tokens(timing) with open(output_dir / "stdout.log", "w", encoding="utf-8") as f: f.write(stdout) @@ -370,6 +480,38 @@ def get_harness( base_url: Optional[str] = None, timeout: Optional[int] = None, max_retries: Optional[int] = None, + pricing: Optional[dict[str, dict]] = None, + reasoning_effort: Optional[str] = None, ) -> AgentHarness: cls = HARNESSES[agent_type] - return cls(workspace, model, base_url=base_url, timeout=timeout, max_retries=max_retries) + return cls( + workspace, + model, + base_url=base_url, + timeout=timeout, + max_retries=max_retries, + pricing=pricing, + reasoning_effort=reasoning_effort, + ) + + +def get_cli_version(agent_type: AgentType) -> Optional[str]: + """Best-effort agent CLI version (first line of `` --version``). + + Recorded in run_meta.json so a result can be tied to the exact agent + binary that produced it. None when the agent has no version command or + the probe fails — never raises. + """ + command = HARNESSES[agent_type].version_command + if not command: + return None + try: + result = subprocess.run( + command, capture_output=True, text=True, encoding="utf-8", errors="replace", timeout=15, check=False + ) + except (subprocess.SubprocessError, OSError): + return None + if result.returncode != 0: + return None + first_line = (result.stdout or "").strip().splitlines() + return first_line[0].strip() if first_line else None diff --git a/src/agent_skill_eval/hooks.py b/src/agent_skill_eval/hooks.py new file mode 100644 index 0000000..11ef1f6 --- /dev/null +++ b/src/agent_skill_eval/hooks.py @@ -0,0 +1,181 @@ +"""Lifecycle hook commands for eval runs. + +Three hook points, all plain shell commands so eval suites can keep +provider- or skill-specific logic (GitLab setup, custom graders, teardown) +outside the harness: + +- pre-run: once before any agent case starts. Non-zero exit aborts the + suite before a single model call. +- post-grade: once per run, after ``grading.json`` is written and before + the workspace is removed. Can append extra assertion results to the + run's grading (see ``apply_post_grade_hooks``). +- post-run: once after the suite finishes (teardown). Failures are + recorded but do not fail the run — results already exist. + +Every hook receives run metadata through ``ASE_*`` environment variables +on top of the parent environment. +""" + +from __future__ import annotations + +import json +import os +import subprocess +from typing import Optional + +from agent_skill_eval.graders import summarize_assertion_results +from agent_skill_eval.models import AssertionResult, GradingResult + +DEFAULT_HOOK_TIMEOUT_SECONDS = 600 + + +class HookError(Exception): + """A suite-level hook failed in a way that must abort the run.""" + + +def _hook_timeout() -> int: + return int(os.environ.get("ASE_HOOK_TIMEOUT", DEFAULT_HOOK_TIMEOUT_SECONDS)) + + +def run_hook_command(command: str, env_extra: dict[str, str]) -> dict: + """Run one hook command through the shell and capture its outcome. + + Returns a JSON-serializable record: command, exit_code, stdout, stderr, + timed_out. Never raises; callers decide what a failure means. + """ + env = dict(os.environ) + env.update({k: v for k, v in env_extra.items() if v is not None}) + timeout = _hook_timeout() + try: + result = subprocess.run( + command, + shell=True, + capture_output=True, + text=True, + encoding="utf-8", + errors="replace", + timeout=timeout, + env=env, + ) + return { + "command": command, + "exit_code": result.returncode, + "stdout": result.stdout or "", + "stderr": result.stderr or "", + "timed_out": False, + } + except subprocess.TimeoutExpired as e: + stdout = (e.stdout or b"").decode("utf-8", "replace") if isinstance(e.stdout, bytes) else (e.stdout or "") + stderr = (e.stderr or b"").decode("utf-8", "replace") if isinstance(e.stderr, bytes) else (e.stderr or "") + return { + "command": command, + "exit_code": None, + "stdout": stdout, + "stderr": stderr + f"\n[agent-skill-eval] hook timed out after {timeout}s", + "timed_out": True, + } + + +def run_suite_hooks( + commands: list[str], + env_extra: dict[str, str], + label: str, + fail_fast: bool = True, +) -> list[dict]: + """Run suite-level hooks (pre-run / post-run) in order. + + With ``fail_fast`` a failing hook raises HookError immediately (pre-run: + abort before model calls). Without it, all hooks run and failures are + only recorded (post-run teardown). + """ + records: list[dict] = [] + for command in commands: + record = run_hook_command(command, env_extra) + records.append(record) + if record["exit_code"] != 0 and fail_fast: + detail = (record["stderr"] or record["stdout"] or "").strip() + raise HookError( + f"{label} hook failed (exit {record['exit_code']}): {command}" + (f"\n{detail}" if detail else "") + ) + return records + + +def _parse_hook_assertions(stdout: str) -> Optional[list[AssertionResult]]: + """Parse a post-grade hook's stdout as a list of assertion results. + + Expected shape: a JSON array of {"text": ..., "passed": bool, + "evidence": ...} objects. Returns None when stdout is not that shape — + hooks are free to print anything; only valid JSON results are merged. + """ + try: + data = json.loads(stdout) + except (json.JSONDecodeError, TypeError): + return None + if not isinstance(data, list): + return None + + results: list[AssertionResult] = [] + for item in data: + if not isinstance(item, dict) or "text" not in item or "passed" not in item: + return None + results.append( + AssertionResult( + text=str(item["text"]), + passed=bool(item["passed"]), + evidence=str(item.get("evidence", "")), + method="hook", + ) + ) + return results + + +def apply_post_grade_hooks( + commands: list[str], + env_extra: dict[str, str], + grading: GradingResult, +) -> tuple[GradingResult, list[dict]]: + """Run post-grade hooks and merge their checks into the grading result. + + Contract: a hook that prints a JSON array of + ``{"text", "passed", "evidence"}`` objects gets those appended as + ``method="hook"`` assertion results. A hook that exits non-zero adds one + failed result so external grading failures show up in the same + ``grading.json`` that ``report`` reads. The summary is recomputed. + """ + records: list[dict] = [] + extra_results: list[AssertionResult] = [] + + for command in commands: + record = run_hook_command(command, env_extra) + records.append(record) + + parsed = _parse_hook_assertions(record["stdout"]) + if parsed: + extra_results.extend(parsed) + + if record["exit_code"] != 0: + detail = (record["stderr"] or record["stdout"] or "").strip() + extra_results.append( + AssertionResult( + text=f"post-grade hook succeeded: {command}", + passed=False, + evidence=( + f"hook exited {record['exit_code']}" + + (" (timed out)" if record["timed_out"] else "") + + (f": {detail[:500]}" if detail else "") + ), + method="hook", + ) + ) + + if not extra_results: + return grading, records + + all_results = list(grading.assertion_results) + extra_results + return ( + GradingResult( + assertion_results=all_results, + summary=summarize_assertion_results(all_results), + ), + records, + ) diff --git a/src/agent_skill_eval/models.py b/src/agent_skill_eval/models.py index ad97b63..8b4561f 100644 --- a/src/agent_skill_eval/models.py +++ b/src/agent_skill_eval/models.py @@ -2,7 +2,7 @@ from enum import Enum from pathlib import Path -from typing import Optional +from typing import Literal, Optional from pydantic import BaseModel, Field, model_validator @@ -35,6 +35,28 @@ class EvalSuite(BaseModel): skill_name: str evals: list[EvalCase] + def filtered_by_ids(self, eval_ids: Optional[list[str]] = None) -> EvalSuite: + """Return a copy containing only the requested eval ids. + + With no ids the suite is returned unchanged. Unknown ids raise a + ValueError listing the available ids, so a typo fails fast before + any workspace or model work starts. + """ + if not eval_ids: + return self + + requested = [str(eval_id) for eval_id in eval_ids] + requested_set = set(requested) + available = {str(eval_case.id) for eval_case in self.evals} + missing = [eval_id for eval_id in requested if eval_id not in available] + if missing: + raise ValueError( + "Unknown eval id(s): " + ", ".join(missing) + ". Available eval ids: " + ", ".join(sorted(available)) + ) + + selected = [eval_case for eval_case in self.evals if str(eval_case.id) in requested_set] + return self.model_copy(update={"evals": selected}) + class GitStateSnapshot(BaseModel): local_branches: list[str] = Field(default_factory=list) @@ -59,6 +81,22 @@ class RunMeta(BaseModel): source_repo: Optional[str] = None run_id: str = "" run_index: int = 1 + # Effective model configuration, recorded so results are reproducible: + # two users running "codex" can otherwise unknowingly benchmark different + # models or reasoning settings inherited from local user config. + model: Optional[str] = None + # Reasoning effort the harness itself passed to the agent CLI + # (--reasoning-effort). None means the harness did not set one — the + # agent then uses whatever its own user/default config says, which this + # harness cannot see. + reasoning_effort: Optional[str] = None + # API base URL injected into the agent CLI's environment + # (--harness-base-url). None = the CLI's own default endpoint. + base_url: Optional[str] = None + harness_version: str = "" + # Output of the agent CLI's --version, best effort (None if the probe + # failed or the agent has no version command). + agent_cli_version: Optional[str] = None class CleanupManifest(BaseModel): @@ -75,15 +113,27 @@ class TimingData(BaseModel): output_tokens: int = 0 cached_tokens: int = 0 cache_creation_tokens: int = 0 + # Input tokens billed at the non-cached rate, filled in after the run by + # the harness that knows its CLI's semantics (codex/opencode report + # input_tokens INCLUDING cache reads, claude-code excludes them). None + # only for timing.json files written before this field existed. + non_cached_input_tokens: Optional[int] = None + # Reasoning/thinking output tokens when the agent CLI exposes them + # (codex turn.completed usage, opencode step_finish tokens). None means + # the agent reported none — unknown, not zero. + reasoning_output_tokens: Optional[int] = None # USD cost as reported by the agent CLI itself (claude: total_cost_usd, - # opencode: per-step cost). 0.0 when the CLI reports nothing (codex, fake). - # For claude-code routed through OpenRouter, this is reconciled against - # OpenRouter pricing (see cost_usd_source / cost_usd_cli). - cost_usd: float = 0.0 + # opencode: per-step cost). None when no cost is available (codex reports + # nothing and no --pricing-config entry matched) — unavailable cost must + # not be confused with a real $0.00 run. For claude-code routed through + # OpenRouter, this is reconciled against OpenRouter pricing (see + # cost_usd_source / cost_usd_cli). + cost_usd: Optional[float] = None # Where cost_usd came from: "cli" (the agent CLI's own number), # "openrouter-pricing" (recomputed from token counts and OpenRouter's - # published per-model rates), or "cli-unreconciled" (claude-code ran via - # OpenRouter but reconciliation failed, so this is the CLI's + # published per-model rates), "pricing-config" (recomputed from a + # user-provided --pricing-config file), or "cli-unreconciled" (claude-code + # ran via OpenRouter but reconciliation failed, so this is the CLI's # Anthropic-list-price estimate and actual billing differs). cost_usd_source: Optional[str] = None # The CLI's original estimate, kept when cost_usd was reconciled. @@ -92,6 +142,76 @@ class TimingData(BaseModel): exit_code: Optional[int] = None timed_out: bool = False retries: int = 0 + # Budget guard verdict (see BudgetConfig). Filled in by the runner after + # the agent process exits; enforcement is post-run, mid-run kills remain + # the timeout's job. + budget_exceeded: bool = False + budget_reason: Optional[str] = None + + +class BudgetConfig(BaseModel): + """Per-case budget limits checked after each agent run. + + Any limit left at None is not enforced. ``action`` decides what a + violation does: "warn" only prints, "fail" appends a failed + ``budget`` assertion to the run's grading, "stop-suite" additionally + skips all runs that have not started yet. + """ + + max_input_tokens: Optional[int] = None + max_non_cached_input_tokens: Optional[int] = None + max_output_tokens: Optional[int] = None + max_total_tokens: Optional[int] = None + max_reasoning_tokens: Optional[int] = None + max_cost_usd: Optional[float] = None + max_duration_seconds: Optional[float] = None + action: Literal["warn", "fail", "stop-suite"] = "fail" + + @property + def enabled(self) -> bool: + return any( + limit is not None + for limit in ( + self.max_input_tokens, + self.max_non_cached_input_tokens, + self.max_output_tokens, + self.max_total_tokens, + self.max_reasoning_tokens, + self.max_cost_usd, + self.max_duration_seconds, + ) + ) + + def violations(self, timing: TimingData) -> list[str]: + problems: list[str] = [] + if self.max_input_tokens is not None and timing.input_tokens > self.max_input_tokens: + problems.append(f"input tokens {timing.input_tokens} > {self.max_input_tokens}") + # Like cost: only enforced when the run actually reported the value. + # A None is "unavailable", and failing a run over an unknown quantity + # would punish agents for exposing less telemetry. + if ( + self.max_non_cached_input_tokens is not None + and timing.non_cached_input_tokens is not None + and timing.non_cached_input_tokens > self.max_non_cached_input_tokens + ): + problems.append( + f"non-cached input tokens {timing.non_cached_input_tokens} > {self.max_non_cached_input_tokens}" + ) + if self.max_output_tokens is not None and timing.output_tokens > self.max_output_tokens: + problems.append(f"output tokens {timing.output_tokens} > {self.max_output_tokens}") + if ( + self.max_reasoning_tokens is not None + and timing.reasoning_output_tokens is not None + and timing.reasoning_output_tokens > self.max_reasoning_tokens + ): + problems.append(f"reasoning tokens {timing.reasoning_output_tokens} > {self.max_reasoning_tokens}") + if self.max_total_tokens is not None and timing.total_tokens > self.max_total_tokens: + problems.append(f"total tokens {timing.total_tokens} > {self.max_total_tokens}") + if self.max_cost_usd is not None and timing.cost_usd is not None and timing.cost_usd > self.max_cost_usd: + problems.append(f"cost {timing.cost_usd:.4f} USD > {self.max_cost_usd:.4f} USD") + if self.max_duration_seconds is not None and timing.duration_ms / 1000.0 > self.max_duration_seconds: + problems.append(f"duration {timing.duration_ms / 1000.0:.1f}s > {self.max_duration_seconds:.1f}s") + return problems class AssertionResult(BaseModel): @@ -133,8 +253,12 @@ class BenchmarkStats(BaseModel): pass_rate: StatsPair time_seconds: StatsPair tokens: StatsPair - # USD cost per run as reported by the agent CLI (see TimingData.cost_usd). - cost_usd: StatsPair = Field(default_factory=lambda: StatsPair(mean=0.0, stddev=0.0)) + # USD cost per run (see TimingData.cost_usd), computed over the runs + # that reported a cost. None when no run had a usable cost — cost is + # unavailable, not zero. + cost_usd: Optional[StatsPair] = None + # How many of the runs had a usable cost_usd. + cost_runs: int = 0 # Fraction of runs where ALL assertions passed (pass@1 estimate). full_pass_rate: float = 0.0 # Fraction of evals where at least one of k runs fully passed. @@ -160,4 +284,5 @@ class DeltaStats(BaseModel): pass_rate: float time_seconds: float tokens: float - cost_usd: float = 0.0 + # None when either side of the comparison had no usable cost. + cost_usd: Optional[float] = None diff --git a/src/agent_skill_eval/progress.py b/src/agent_skill_eval/progress.py new file mode 100644 index 0000000..6df57ec --- /dev/null +++ b/src/agent_skill_eval/progress.py @@ -0,0 +1,57 @@ +"""Live per-run progress events for long eval runs. + +The runner appends one JSON object per line to ``progress.jsonl`` in the +iteration directory while the suite runs, so a separate shell can +``tail -f`` it to see which case is active, how long it has been running, +and what it consumed — without digging through workspace directories. +""" + +from __future__ import annotations + +import json +import threading +import time +from datetime import datetime, timezone +from pathlib import Path +from typing import Optional + + +class ProgressLog: + """Thread-safe JSONL event log (the runner grades runs concurrently).""" + + def __init__(self, path: Path): + self.path = path + self._lock = threading.Lock() + self._start = time.time() + # Start each run with a fresh file: progress is per-invocation + # state, unlike the cumulative result artifacts. + self.path.write_text("") + + def emit(self, event: str, **fields) -> None: + record = { + "event": event, + "ts": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "elapsed_s": round(time.time() - self._start, 1), + **{k: v for k, v in fields.items() if v is not None}, + } + line = json.dumps(record) + with self._lock: + with open(self.path, "a", encoding="utf-8") as f: + f.write(line + "\n") + + +def run_label(eval_id, agent: str, with_skill: bool, run_index: int, runs: int) -> str: + """Human-readable identity of one run for console progress lines.""" + config = "with_skill" if with_skill else "without_skill" + label = f"eval={eval_id} agent={agent} config={config}" + if runs > 1: + label += f" run={run_index}" + return label + + +def format_tokens(count: Optional[int]) -> str: + if count is None: + return "?" + if count >= 1000: + return f"{count / 1000:.0f}k" + return str(count) diff --git a/src/agent_skill_eval/runner.py b/src/agent_skill_eval/runner.py index f0efee6..b5b74d9 100644 --- a/src/agent_skill_eval/runner.py +++ b/src/agent_skill_eval/runner.py @@ -4,29 +4,40 @@ import os import statistics import subprocess +import threading +import time import uuid from concurrent.futures import ThreadPoolExecutor, as_completed from pathlib import Path from typing import Optional from rich.console import Console +from rich.markup import escape from rich.progress import BarColumn, Progress, SpinnerColumn, TaskProgressColumn, TextColumn +from agent_skill_eval import __version__ from agent_skill_eval.git_state import capture_git_state, github_repo_slug -from agent_skill_eval.graders import LLMGrader, grade_assertions -from agent_skill_eval.harnesses import get_harness +from agent_skill_eval.graders import LLMGrader, grade_assertions, summarize_assertion_results +from agent_skill_eval.harnesses import HARNESSES, get_cli_version, get_harness +from agent_skill_eval.hooks import apply_post_grade_hooks, run_suite_hooks from agent_skill_eval.models import ( AgentType, + AssertionResult, BenchmarkResult, BenchmarkStats, + BudgetConfig, CleanupManifest, DeltaStats, EvalCase, EvalSuite, + GradingResult, RunMeta, StatsPair, + TimingData, ) +from agent_skill_eval.progress import ProgressLog, format_tokens, run_label from agent_skill_eval.skills import SkillInstaller +from agent_skill_eval.summary import build_run_summary, write_run_summary from agent_skill_eval.workspace import WorkspaceManager console = Console() @@ -49,6 +60,13 @@ def __init__( agent_timeout: Optional[int] = None, agent_max_retries: Optional[int] = None, runs: int = 1, + eval_ids: Optional[list[str]] = None, + budget: Optional[BudgetConfig] = None, + pricing: Optional[dict[str, dict]] = None, + pre_run_commands: Optional[list[str]] = None, + post_grade_commands: Optional[list[str]] = None, + post_run_commands: Optional[list[str]] = None, + reasoning_effort: Optional[str] = None, ): self.skill_path = skill_path self.evals_path = evals_path @@ -64,23 +82,60 @@ def __init__( self.agent_timeout = agent_timeout self.agent_max_retries = agent_max_retries self.runs = max(1, runs) + self.eval_ids = [str(e) for e in eval_ids] if eval_ids else [] + self.budget = budget if budget and budget.enabled else None + self.pricing = pricing or {} + self.pre_run_commands = pre_run_commands or [] + self.post_grade_commands = post_grade_commands or [] + self.post_run_commands = post_run_commands or [] + self.reasoning_effort = reasoning_effort self.run_id = uuid.uuid4().hex[:8] + # One version probe per agent for the whole suite; recorded in every + # run_meta.json so results are attributable to an exact CLI build. + self._cli_versions: dict[AgentType, Optional[str]] = {a: get_cli_version(a) for a in self.agents} self.suite = self._load_suite() self.skill_installer = SkillInstaller(skill_path) self.workspace_mgr = WorkspaceManager(workspace_base, source_repo=source_repo) self.llm_grader = LLMGrader(model=grader_model, base_url=grader_base_url) if grader_model else None + # Set when a budget violation with action=stop-suite fires; runs + # that have not started yet short-circuit to "skipped". + self._stop_suite = threading.Event() + self._stop_reason: Optional[str] = None + self._progress: Optional[ProgressLog] = None + def _load_suite(self) -> EvalSuite: with open(self.evals_path, encoding="utf-8") as f: data = json.load(f) - return EvalSuite(**data) + # Filter once, before tasks or metadata exist, so every downstream + # artifact (evals_meta.json, workspaces, benchmark) naturally + # describes only the selected cases. + return EvalSuite(**data).filtered_by_ids(self.eval_ids) def run(self, iteration: int = 1) -> Path: + started = time.time() iteration_dir = self.workspace_base / f"{self.suite.skill_name}-workspace" / f"iteration-{iteration}" iteration_dir.mkdir(parents=True, exist_ok=True) self._save_eval_metadata(iteration_dir) + self._progress = ProgressLog(iteration_dir / "progress.jsonl") + + if self.reasoning_effort: + unsupported = [a.value for a in self.agents if not HARNESSES[a].supports_reasoning_effort] + if unsupported: + console.print( + f"[yellow]Warning: --reasoning-effort is not supported by {', '.join(unsupported)}; " + f"those agents run on their own local defaults (run_meta.json records null)[/yellow]" + ) + + hook_env = self._suite_hook_env(iteration_dir, iteration) + hook_records: dict[str, list[dict]] = {"pre_run": [], "post_run": []} + if self.pre_run_commands: + console.print(f"[cyan]Running {len(self.pre_run_commands)} pre-run hook(s)...[/cyan]") + # A failing pre-run hook raises HookError here, before any + # model call or workspace mutation. + hook_records["pre_run"] = run_suite_hooks(self.pre_run_commands, hook_env, label="pre-run", fail_fast=True) tasks = [] for eval_case in self.suite.evals: @@ -90,6 +145,16 @@ def run(self, iteration: int = 1) -> Path: if self.with_baseline: tasks.append((eval_case, agent_type, False, run_index)) + self._progress.emit( + "suite_started", + skill=self.suite.skill_name, + iteration=iteration, + run_id=self.run_id, + total_runs=len(tasks), + eval_ids=[str(e.id) for e in self.suite.evals], + agents=[a.value for a in self.agents], + ) + results: dict[str, dict] = {} cleanup_entries: list[CleanupManifest] = [] @@ -116,18 +181,29 @@ def run(self, iteration: int = 1) -> Path: ) futures[future] = (eval_case, agent_type, with_skill, run_index) - for future in as_completed(futures): - eval_case, agent_type, with_skill, run_index = futures[future] - key = f"{eval_case.id}-{agent_type.value}-{'with' if with_skill else 'without'}-r{run_index}" - try: - result = future.result() - results[key] = result["summary"] - if result.get("cleanup_entry"): - cleanup_entries.append(result["cleanup_entry"]) - except Exception as e: - console.print(f"[red]Error running {key}: {e}[/red]") - results[key] = {"error": str(e)} - progress.advance(task_id) + # Inside the executor block on purpose: collecting results + # as they complete keeps the progress bar and progress.jsonl + # live instead of only updating after every run finished. + for future in as_completed(futures): + eval_case, agent_type, with_skill, run_index = futures[future] + key = f"{eval_case.id}-{agent_type.value}-{'with' if with_skill else 'without'}-r{run_index}" + try: + result = future.result() + results[key] = result["summary"] + if result.get("cleanup_entry"): + cleanup_entries.append(result["cleanup_entry"]) + except Exception as e: + console.print(f"[red]Error running {key}: {e}[/red]") + results[key] = {"error": str(e)} + self._progress.emit( + "run_error", + eval_id=str(eval_case.id), + agent=agent_type.value, + with_skill=with_skill, + run_index=run_index, + error=str(e), + ) + progress.advance(task_id) benchmark = self._compute_benchmark(results, iteration_dir) @@ -146,6 +222,46 @@ def run(self, iteration: int = 1) -> Path: with open(iteration_dir / "cleanup.json", "w", encoding="utf-8") as f: json.dump(manifest.model_dump(exclude_none=True), f, indent=2) + if self.post_run_commands: + console.print(f"[cyan]Running {len(self.post_run_commands)} post-run hook(s)...[/cyan]") + hook_records["post_run"] = run_suite_hooks( + self.post_run_commands, hook_env, label="post-run", fail_fast=False + ) + for record in hook_records["post_run"]: + if record["exit_code"] != 0: + console.print( + f"[yellow]post-run hook failed (exit {record['exit_code']}): {record['command']}[/yellow]" + ) + + summary = build_run_summary( + skill_name=self.suite.skill_name, + iteration=iteration, + run_id=self.run_id, + agents=[a.value for a in self.agents], + eval_ids=[str(e.id) for e in self.suite.evals], + runs=self.runs, + with_baseline=self.with_baseline, + duration_seconds=time.time() - started, + results=results, + benchmark=benchmark.model_dump(), + cleanup=manifest.model_dump(exclude_none=True), + budget=self.budget.model_dump() if self.budget else None, + hooks=hook_records, + ) + write_run_summary(iteration_dir, summary) + + self._progress.emit( + "suite_finished", + total_runs=len(tasks), + completed=summary["runs_completed"], + failed_grading=summary["runs_failed_grading"], + errors=len(summary["errors"]), + skipped=len(summary["skipped_runs"]), + ) + + if self._stop_suite.is_set(): + console.print(f"[yellow]Suite stopped early by budget: {self._stop_reason}[/yellow]") + console.print(f"\n[green]Results saved to {iteration_dir}[/green]") return iteration_dir @@ -154,10 +270,27 @@ def _save_eval_metadata(self, iteration_dir: Path) -> None: "skill_name": self.suite.skill_name, "evals": [eval_case.model_dump() for eval_case in self.suite.evals], "source_repo": self.source_repo, + # The case filter actually applied (--eval-id); empty means the + # full suite ran. report/compare readers can tell filtered runs + # apart without diffing the evals list. + "selected_eval_ids": self.eval_ids, } with open(iteration_dir / "evals_meta.json", "w", encoding="utf-8") as f: json.dump(meta, f, indent=2) + def _suite_hook_env(self, iteration_dir: Path, iteration: int) -> dict[str, str]: + return { + "ASE_SKILL_NAME": self.suite.skill_name, + "ASE_SKILL_DIR": str(self.skill_path), + "ASE_EVALS_PATH": str(self.evals_path), + "ASE_ITERATION": str(iteration), + "ASE_ITERATION_DIR": str(iteration_dir), + "ASE_WORKSPACE_BASE": str(self.workspace_base), + "ASE_SOURCE_REPO": self.source_repo or "", + "ASE_RUN_ID": self.run_id, + "ASE_EVAL_IDS": ",".join(str(e.id) for e in self.suite.evals), + } + def _build_cleanup_manifest(self, entries: list[CleanupManifest], iteration_dir: Path) -> CleanupManifest: all_branches: list[str] = [] all_prs: list[int] = [] @@ -177,6 +310,10 @@ def _build_cleanup_manifest(self, entries: list[CleanupManifest], iteration_dir: workspaces=[str(p) for p in self.workspace_mgr.workspaces], ) + def _emit(self, event: str, **fields) -> None: + if self._progress is not None: + self._progress.emit(event, **fields) + def _run_single( self, eval_case: EvalCase, @@ -186,6 +323,34 @@ def _run_single( iteration: int = 1, run_index: int = 1, ) -> dict: + label = run_label(eval_case.id, agent_type.value, with_skill, run_index, self.runs) + identity = { + "eval_id": str(eval_case.id), + "agent": agent_type.value, + "with_skill": with_skill, + "run_index": run_index, + } + + if self._stop_suite.is_set(): + reason = f"skipped: {self._stop_reason or 'suite stopped by budget'}" + console.print(f" [yellow]{escape(f'[{label}]')} {reason}[/yellow]") + self._emit("run_skipped", **identity, reason=reason) + return { + "summary": { + "eval_id": eval_case.id, + "agent": agent_type.value, + "with_skill": with_skill, + "run_index": run_index, + "error": reason, + "skipped": True, + }, + "cleanup_entry": None, + } + + run_started = time.time() + console.print(f" [dim]{escape(f'[{label}]')} started[/dim]") + self._emit("run_started", **identity) + eval_slug = f"eval-{eval_case.id}" config_slug = "with_skill" if with_skill else "without_skill" agent_slug = agent_type.value @@ -226,16 +391,37 @@ def _run_single( base_url=self.harness_base_url, timeout=self.agent_timeout, max_retries=self.agent_max_retries, + pricing=self.pricing, + reasoning_effort=self.reasoning_effort, ) prompt = self._build_prompt(eval_case, with_skill) + self._emit("snapshot_started", **identity, phase="pre") pre_state = capture_git_state(workspace, source_repo=self.source_repo) with open(output_dir / "pre_state.json", "w", encoding="utf-8") as f: json.dump(pre_state.model_dump(), f, indent=2) + self._emit("agent_started", **identity) agent_output, timing, stdout, stderr = harness.run(prompt, output_dir) + self._emit( + "agent_finished", + **identity, + duration_s=round(timing.duration_ms / 1000.0, 1), + input_tokens=timing.input_tokens, + cached_tokens=timing.cached_tokens, + non_cached_input_tokens=timing.non_cached_input_tokens, + output_tokens=timing.output_tokens, + reasoning_output_tokens=timing.reasoning_output_tokens, + total_tokens=timing.total_tokens, + cost_usd=timing.cost_usd, + exit_code=timing.exit_code, + timed_out=timing.timed_out, + ) + + budget_violations = self._check_budget(timing, label) + self._emit("snapshot_started", **identity, phase="post") post_state = capture_git_state(workspace, source_repo=self.source_repo) with open(output_dir / "post_state.json", "w", encoding="utf-8") as f: json.dump(post_state.model_dump(), f, indent=2) @@ -256,6 +442,13 @@ def _run_single( source_repo=self.source_repo, run_id=self.run_id, run_index=run_index, + model=model, + # The value the harness actually applied: None for agents that + # cannot pass it through (they run on their own local defaults). + reasoning_effort=(self.reasoning_effort if HARNESSES[agent_type].supports_reasoning_effort else None), + base_url=self.harness_base_url, + harness_version=__version__, + agent_cli_version=self._cli_versions.get(agent_type), ) meta_path = output_dir.parent / "run_meta.json" with open(meta_path, "w", encoding="utf-8") as f: @@ -277,6 +470,33 @@ def _run_single( with open(grading_path, "w", encoding="utf-8") as f: json.dump(grading.model_dump(), f, indent=2) + if self.post_grade_commands: + hook_env = self._suite_hook_env(iteration_dir, iteration) + hook_env.update( + { + "ASE_EVAL_ID": str(eval_case.id), + "ASE_AGENT": agent_type.value, + "ASE_WITH_SKILL": "1" if with_skill else "0", + "ASE_RUN_INDEX": str(run_index), + "ASE_WORKSPACE_PATH": str(workspace), + "ASE_OUTPUT_DIR": str(output_dir), + "ASE_PRE_STATE_PATH": str(output_dir / "pre_state.json"), + "ASE_POST_STATE_PATH": str(output_dir / "post_state.json"), + "ASE_TIMING_PATH": str(timing_path), + "ASE_RUN_META_PATH": str(meta_path), + "ASE_GRADING_PATH": str(grading_path), + } + ) + grading, hook_records = apply_post_grade_hooks(self.post_grade_commands, hook_env, grading) + with open(output_dir / "post_grade_hooks.json", "w", encoding="utf-8") as f: + json.dump(hook_records, f, indent=2) + + grading = self._apply_budget_verdict(grading, budget_violations) + + # Rewritten when hooks or budget added results above; cheap either way. + with open(grading_path, "w", encoding="utf-8") as f: + json.dump(grading.model_dump(), f, indent=2) + cleanup_entry = self._build_cleanup_entry(pre_state, post_state) if os.environ.get("ASE_KEEP_WORKSPACE"): @@ -284,6 +504,30 @@ def _run_single( else: self.workspace_mgr.cleanup(workspace) + cost_str = f"{timing.cost_usd:.4f} USD" if timing.cost_usd is not None else "n/a" + reasoning_str = ( + f" reasoning={format_tokens(timing.reasoning_output_tokens)}" + if timing.reasoning_output_tokens is not None + else "" + ) + console.print( + f" [dim]{escape(f'[{label}]')} finished in {time.time() - run_started:.0f}s: " + f"{grading.summary.passed}/{grading.summary.total} passed, " + f"tokens_in={format_tokens(timing.input_tokens)} " + f"cached={format_tokens(timing.cached_tokens)} " + f"out={format_tokens(timing.output_tokens)}{reasoning_str} cost={cost_str}[/dim]" + ) + self._emit( + "run_finished", + **identity, + duration_s=round(time.time() - run_started, 1), + pass_rate=grading.summary.pass_rate, + passed=grading.summary.passed, + total=grading.summary.total, + total_tokens=timing.total_tokens, + budget_exceeded=timing.budget_exceeded, + ) + return { "summary": { "eval_id": eval_case.id, @@ -297,6 +541,48 @@ def _run_single( "cleanup_entry": cleanup_entry, } + def _check_budget(self, timing: TimingData, label: str) -> list[str]: + """Evaluate per-case budgets against a finished run's timing. + + Enforcement is post-run: the agent process has already exited (a + case that must be killed mid-run is the timeout's job). Records the + verdict on the timing object; "stop-suite" additionally prevents + any not-yet-started run from launching. + """ + if not self.budget: + return [] + violations = self.budget.violations(timing) + if not violations: + return [] + + reason = "; ".join(violations) + timing.budget_exceeded = True + timing.budget_reason = reason + console.print(f" [yellow]{escape(f'[{label}]')} budget exceeded ({self.budget.action}): {reason}[/yellow]") + self._emit("budget_exceeded", label=label, reason=reason, action=self.budget.action) + + if self.budget.action == "stop-suite" and not self._stop_suite.is_set(): + self._stop_reason = f"budget exceeded by [{label}]: {reason}" + self._stop_suite.set() + return violations + + def _apply_budget_verdict(self, grading: GradingResult, violations: list[str]) -> GradingResult: + """Fail the run's grading when its budget was exceeded (action != warn).""" + if not violations or not self.budget or self.budget.action == "warn": + return grading + all_results = list(grading.assertion_results) + [ + AssertionResult( + text="The run stayed within its budget", + passed=False, + evidence="budget exceeded: " + "; ".join(violations), + method="budget", + ) + ] + return GradingResult( + assertion_results=all_results, + summary=summarize_assertion_results(all_results), + ) + def _build_prompt(self, eval_case: EvalCase, with_skill: bool) -> str: """Build the prompt sent to the agent. @@ -362,14 +648,15 @@ def compute_stats(results: list[dict]) -> BenchmarkStats: pass_rate=StatsPair(mean=0.0, stddev=0.0), time_seconds=StatsPair(mean=0.0, stddev=0.0), tokens=StatsPair(mean=0.0, stddev=0.0), - cost_usd=StatsPair(mean=0.0, stddev=0.0), ) pass_rates = [r["grading"]["summary"]["pass_rate"] for r in results] times = [r["timing"]["duration_ms"] / 1000.0 for r in results] tokens = [r["timing"]["total_tokens"] for r in results] + # Only runs that reported a cost contribute; None means unavailable + # (codex without --pricing-config), which must not average in as $0. # .get: timing.json from runs before cost_usd existed lacks the field. - costs = [r["timing"].get("cost_usd", 0.0) or 0.0 for r in results] + costs = [c for r in results if (c := r["timing"].get("cost_usd")) is not None] full_passes = [_is_full_pass(r) for r in results] @@ -395,9 +682,12 @@ def compute_stats(results: list[dict]) -> BenchmarkStats: stddev=statistics.stdev(tokens) if len(tokens) > 1 else 0.0, ), cost_usd=StatsPair( - mean=statistics.mean(costs) if costs else 0.0, + mean=statistics.mean(costs), stddev=statistics.stdev(costs) if len(costs) > 1 else 0.0, - ), + ) + if costs + else None, + cost_runs=len(costs), full_pass_rate=sum(full_passes) / len(full_passes), pass_at_k=pass_at_k, k=k, @@ -434,11 +724,14 @@ def compute_benchmark(results: dict[str, dict], agents: list[AgentType], with_ba if with_baseline: run_summary[f"{agent_type.value}_without_skill"] = without_stats if with_results and without_results: + cost_delta = None + if with_stats.cost_usd is not None and without_stats.cost_usd is not None: + cost_delta = with_stats.cost_usd.mean - without_stats.cost_usd.mean deltas[agent_type.value] = DeltaStats( pass_rate=with_stats.pass_rate.mean - without_stats.pass_rate.mean, time_seconds=with_stats.time_seconds.mean - without_stats.time_seconds.mean, tokens=with_stats.tokens.mean - without_stats.tokens.mean, - cost_usd=with_stats.cost_usd.mean - without_stats.cost_usd.mean, + cost_usd=cost_delta, ) delta = None diff --git a/src/agent_skill_eval/summary.py b/src/agent_skill_eval/summary.py new file mode 100644 index 0000000..dc1c6e4 --- /dev/null +++ b/src/agent_skill_eval/summary.py @@ -0,0 +1,267 @@ +"""Iteration-level summary artifacts. + +``run`` writes a ``summary.json`` next to ``benchmark.json`` so pass rate, +failures, token totals (cached vs non-cached), cost, budget verdicts, and +cleanup state are available from one file — no shell aggregation over +``timing.json``/``grading.json`` trees. ``agent-skill-eval status`` reads it +back (and can rebuild a reduced version from artifacts for runs that +predate it). +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Optional + +SUMMARY_FILENAME = "summary.json" + + +def leaf_config_dirs(agent_dir: Path) -> list[Path]: + """Yield config dirs that directly contain run artifacts. + + Layouts: ``agent/with_skill/outputs`` (single run) or + ``agent/with_skill/run-N/outputs`` (``--runs N`` > 1). + """ + leaves: list[Path] = [] + for config_dir in sorted(agent_dir.iterdir()): + if not config_dir.is_dir(): + continue + run_dirs = sorted(p for p in config_dir.glob("run-*") if p.is_dir()) + if run_dirs: + leaves.extend(run_dirs) + else: + leaves.append(config_dir) + return leaves + + +def _token_totals(timings: list[dict]) -> dict: + # `or 0` (not a .get default): timing.json fields can be present-but-null. + input_tokens = sum(t.get("input_tokens") or 0 for t in timings) + cached = sum(t.get("cached_tokens") or 0 for t in timings) + output = sum(t.get("output_tokens") or 0 for t in timings) + total = sum(t.get("total_tokens") or 0 for t in timings) + reported_input = max(input_tokens, cached) + # Prefer the per-run value the harness recorded (it knows whether its + # CLI's input_tokens include cache reads); fall back to the + # subtraction lower bound only for timing.json files predating it. + non_cached = 0 + for t in timings: + nc = t.get("non_cached_input_tokens") + if nc is None: + nc = max((t.get("input_tokens") or 0) - (t.get("cached_tokens") or 0), 0) + non_cached += nc + # None when no run reported reasoning telemetry: unknown, not zero. + reasoning_known = [r for t in timings if (r := t.get("reasoning_output_tokens")) is not None] + return { + "input": input_tokens, + "cached": cached, + "non_cached_input": non_cached, + "output": output, + "reasoning_output": sum(reasoning_known) if reasoning_known else None, + "total": total, + "cached_pct": round(cached / reported_input, 4) if reported_input else 0.0, + } + + +def _cost_totals(timings: list[dict]) -> dict: + costs = [t.get("cost_usd") for t in timings] + known = [c for c in costs if c is not None] + return { + # null means unavailable (no run reported a cost), NOT free. + "total_usd": round(sum(known), 6) if known else None, + "runs_with_cost": len(known), + "runs_total": len(costs), + } + + +def build_run_summary( + *, + skill_name: str, + iteration: int, + run_id: str, + agents: list[str], + eval_ids: list[str], + runs: int, + with_baseline: bool, + duration_seconds: float, + results: dict[str, dict], + benchmark: dict, + cleanup: dict, + budget: Optional[dict] = None, + hooks: Optional[dict] = None, +) -> dict: + """Build the iteration summary dict written to ``summary.json``.""" + completed = [r for r in results.values() if "timing" in r and "grading" in r] + errors = [ + {"run": key, "error": r["error"]} for key, r in sorted(results.items()) if "error" in r and not r.get("skipped") + ] + skipped = [{"run": key, "reason": r["error"]} for key, r in sorted(results.items()) if r.get("skipped")] + + failed_runs = [] + budget_exceeded = [] + for r in completed: + grading_summary = r["grading"]["summary"] + identity = { + "eval_id": r.get("eval_id"), + "agent": r.get("agent"), + "with_skill": r.get("with_skill"), + "run_index": r.get("run_index", 1), + } + if grading_summary.get("failed", 0) > 0: + failed_runs.append( + { + **identity, + "pass_rate": grading_summary.get("pass_rate", 0.0), + "failed_assertions": [ + a["text"] + for a in r["grading"].get("assertion_results", []) + if not a.get("passed") and not a.get("skipped") + ], + } + ) + if r["timing"].get("budget_exceeded"): + budget_exceeded.append({**identity, "reason": r["timing"].get("budget_reason")}) + + timings = [r["timing"] for r in completed] + pass_rates = { + config: stats.get("pass_rate", {}).get("mean") for config, stats in benchmark.get("run_summary", {}).items() + } + + summary = { + "skill_name": skill_name, + "iteration": iteration, + "run_id": run_id, + "agents": agents, + "eval_ids": eval_ids, + "runs_per_case": runs, + "with_baseline": with_baseline, + "duration_seconds": round(duration_seconds, 1), + "pass_rates": pass_rates, + "runs_completed": len(completed), + "runs_failed_grading": len(failed_runs), + "failed_runs": failed_runs, + "errors": errors, + "skipped_runs": skipped, + "tokens": _token_totals(timings), + "cost": _cost_totals(timings), + "cleanup": { + "remote_branches": cleanup.get("remote_branches", []), + "pr_numbers": cleanup.get("pr_numbers", []), + "workspaces": len(cleanup.get("workspaces", [])), + }, + } + if budget: + summary["budget"] = {**budget, "exceeded_runs": budget_exceeded} + if hooks: + summary["hooks"] = { + stage: [ + {"command": rec["command"], "exit_code": rec["exit_code"], "timed_out": rec["timed_out"]} + for rec in records + ] + for stage, records in hooks.items() + if records + } + return summary + + +def write_run_summary(iteration_dir: Path, summary: dict) -> Path: + path = iteration_dir / SUMMARY_FILENAME + with open(path, "w", encoding="utf-8") as f: + json.dump(summary, f, indent=2) + return path + + +def load_summary(iteration_dir: Path) -> Optional[dict]: + path = iteration_dir / SUMMARY_FILENAME + if not path.exists(): + return None + try: + return json.loads(path.read_text()) + except (json.JSONDecodeError, OSError): + return None + + +def build_summary_from_artifacts(iteration_dir: Path) -> Optional[dict]: + """Rebuild a reduced summary from saved artifacts (pre-summary runs). + + Walks ``eval-*//[/run-N]`` for grading/timing files. Pass + rates come from ``benchmark.json`` when present. Budget/hook/cleanup + detail beyond cleanup.json is not reconstructable. + """ + eval_dirs = sorted(iteration_dir.glob("eval-*")) + if not eval_dirs: + return None + + results: dict[str, dict] = {} + for eval_dir in eval_dirs: + for agent_dir in sorted(p for p in eval_dir.iterdir() if p.is_dir()): + for config_dir in leaf_config_dirs(agent_dir): + grading_path = config_dir / "grading.json" + timing_path = config_dir / "timing.json" + if not grading_path.exists() or not timing_path.exists(): + continue + try: + grading = json.loads(grading_path.read_text()) + timing = json.loads(timing_path.read_text()) + except (json.JSONDecodeError, OSError): + continue + + meta = {} + meta_path = config_dir / "run_meta.json" + if meta_path.exists(): + try: + meta = json.loads(meta_path.read_text()) + except (json.JSONDecodeError, OSError): + meta = {} + + key = f"{eval_dir.name}/{agent_dir.name}/{config_dir.relative_to(agent_dir)}" + results[key] = { + "eval_id": meta.get("eval_id", eval_dir.name[len("eval-") :]), + "agent": meta.get("agent", agent_dir.name), + "with_skill": meta.get("with_skill", "with_skill" in config_dir.parts), + "run_index": meta.get("run_index", 1), + "timing": timing, + "grading": grading, + } + + if not results: + return None + + benchmark = {} + benchmark_path = iteration_dir / "benchmark.json" + if benchmark_path.exists(): + try: + benchmark = json.loads(benchmark_path.read_text()) + except (json.JSONDecodeError, OSError): + benchmark = {} + + cleanup = {} + cleanup_path = iteration_dir / "cleanup.json" + if cleanup_path.exists(): + try: + cleanup = json.loads(cleanup_path.read_text()) + except (json.JSONDecodeError, OSError): + cleanup = {} + + first = next(iter(results.values())) + iteration_num = 0 + if iteration_dir.name.startswith("iteration-"): + try: + iteration_num = int(iteration_dir.name[len("iteration-") :]) + except ValueError: + iteration_num = 0 + + return build_run_summary( + skill_name=iteration_dir.parent.name.removesuffix("-workspace"), + iteration=iteration_num, + run_id=first.get("run_id", ""), + agents=sorted({r["agent"] for r in results.values()}), + eval_ids=sorted({str(r["eval_id"]) for r in results.values()}), + runs=max(r.get("run_index", 1) for r in results.values()), + with_baseline=any(not r["with_skill"] for r in results.values()), + duration_seconds=sum(r["timing"].get("duration_ms", 0) for r in results.values()) / 1000.0, + results=results, + benchmark=benchmark, + cleanup=cleanup, + ) diff --git a/tests/test_feedback_features.py b/tests/test_feedback_features.py new file mode 100644 index 0000000..8b4486f --- /dev/null +++ b/tests/test_feedback_features.py @@ -0,0 +1,549 @@ +"""Tests for the 2026-06-11 harness feedback batch: --eval-id case +filtering, per-case budget guards, live progress + progress.jsonl, +lifecycle hooks (pre-run / post-grade / post-run), summary.json + status, +unavailable-vs-zero cost semantics with --pricing-config, reasoning token +persistence, --reasoning-effort, and effective-model-config recording in +run_meta.json.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from agent_skill_eval import __version__ +from agent_skill_eval.cli import app +from agent_skill_eval.harnesses import ClaudeCodeHarness, CodexHarness, OpenCodeHarness +from agent_skill_eval.models import AgentType, BudgetConfig, EvalSuite, TimingData +from agent_skill_eval.runner import compute_benchmark, compute_stats + +FIXTURES = Path(__file__).parent / "fixtures" +SKILL = FIXTURES / "skills" / "format-json" +EVALS = FIXTURES / "evals" / "format-json.json" +ALL_EVAL_IDS = ["explicit-invoke", "negative-control", "content-contains"] + + +def _run_cli(workspace: Path, *extra: str): + runner = CliRunner() + return runner.invoke( + app, + [ + "run", + "--skill", + str(SKILL), + "--evals", + str(EVALS), + "--agent", + "fake", + "--workspace", + str(workspace), + "--grader-model", + "", + "--no-baseline", + *extra, + ], + ) + + +def _iteration_dir(workspace: Path) -> Path: + return workspace / "format-json-workspace" / "iteration-1" + + +def _suite(ids: list[str]) -> EvalSuite: + return EvalSuite( + skill_name="demo", + evals=[{"id": i, "prompt": "p", "expected_output": "ok"} for i in ids], + ) + + +class TestEvalIdFiltering: + def test_no_filter_returns_same_suite(self): + suite = _suite(["a", "b"]) + assert suite.filtered_by_ids([]) is suite + assert suite.filtered_by_ids(None) is suite + + def test_filter_selects_subset_preserving_order(self): + suite = _suite(["a", "b", "c"]) + filtered = suite.filtered_by_ids(["c", "a"]) + assert [e.id for e in filtered.evals] == ["a", "c"] + # The original suite is untouched. + assert [e.id for e in suite.evals] == ["a", "b", "c"] + + def test_unknown_id_lists_available_ids(self): + with pytest.raises(ValueError, match=r"Unknown eval id\(s\): nope.*Available eval ids: a, b"): + _suite(["a", "b"]).filtered_by_ids(["nope"]) + + def test_int_ids_match_string_filters(self): + suite = _suite([1, 2]) + assert [e.id for e in suite.filtered_by_ids(["2"]).evals] == [2] + + def test_run_with_eval_id_only_runs_selected_cases(self, tmp_path): + result = _run_cli(tmp_path, "--eval-id", "explicit-invoke") + assert result.exit_code == 0, result.output + + iteration_dir = _iteration_dir(tmp_path) + eval_dirs = sorted(p.name for p in iteration_dir.glob("eval-*")) + assert eval_dirs == ["eval-explicit-invoke"] + + meta = json.loads((iteration_dir / "evals_meta.json").read_text()) + assert meta["selected_eval_ids"] == ["explicit-invoke"] + assert [e["id"] for e in meta["evals"]] == ["explicit-invoke"] + + summary = json.loads((iteration_dir / "summary.json").read_text()) + assert summary["eval_ids"] == ["explicit-invoke"] + + def test_run_with_unknown_eval_id_fails_before_running(self, tmp_path): + result = _run_cli(tmp_path, "--eval-id", "missing-case") + assert result.exit_code == 1 + assert "Unknown eval id(s): missing-case" in result.output + assert not _iteration_dir(tmp_path).exists() + + def test_validate_with_eval_id(self): + runner = CliRunner() + ok = runner.invoke(app, ["validate", str(EVALS), "--eval-id", "explicit-invoke"]) + assert ok.exit_code == 0, ok.output + assert "1 eval(s)" in ok.output + assert "selected by --eval-id" in ok.output + + bad = runner.invoke(app, ["validate", str(EVALS), "--eval-id", "missing-case"]) + assert bad.exit_code == 1 + assert "Unknown eval id(s): missing-case" in bad.output + + +class TestBudgetGuards: + def test_violations(self): + budget = BudgetConfig(max_total_tokens=100, max_duration_seconds=1.0, max_cost_usd=0.5) + timing = TimingData(total_tokens=150, input_tokens=150, duration_ms=2000, cost_usd=1.0) + problems = budget.violations(timing) + assert len(problems) == 3 + + def test_cost_limit_skipped_when_cost_unavailable(self): + budget = BudgetConfig(max_cost_usd=0.5) + assert budget.violations(TimingData(total_tokens=10, cost_usd=None)) == [] + + def test_disabled_when_no_limits(self): + assert not BudgetConfig().enabled + assert BudgetConfig(max_total_tokens=1).enabled + + def test_budget_fail_adds_failed_assertion_and_marks_timing(self, tmp_path): + # The fake harness reports 1 total token, so a 0-token budget trips. + result = _run_cli(tmp_path, "--eval-id", "explicit-invoke", "--max-total-tokens-per-case", "0") + assert result.exit_code == 0, result.output + + config_dir = _iteration_dir(tmp_path) / "eval-explicit-invoke" / "fake" / "with_skill" + timing = json.loads((config_dir / "timing.json").read_text()) + assert timing["budget_exceeded"] is True + assert "total tokens 1 > 0" in timing["budget_reason"] + + grading = json.loads((config_dir / "grading.json").read_text()) + budget_results = [a for a in grading["assertion_results"] if a["method"] == "budget"] + assert len(budget_results) == 1 + assert budget_results[0]["passed"] is False + assert grading["summary"]["failed"] >= 1 + + def test_budget_warn_does_not_touch_grading(self, tmp_path): + result = _run_cli( + tmp_path, + "--eval-id", + "explicit-invoke", + "--max-total-tokens-per-case", + "0", + "--budget-action", + "warn", + ) + assert result.exit_code == 0, result.output + + config_dir = _iteration_dir(tmp_path) / "eval-explicit-invoke" / "fake" / "with_skill" + timing = json.loads((config_dir / "timing.json").read_text()) + assert timing["budget_exceeded"] is True + grading = json.loads((config_dir / "grading.json").read_text()) + assert not [a for a in grading["assertion_results"] if a["method"] == "budget"] + + def test_stop_suite_skips_unstarted_runs(self, tmp_path): + result = _run_cli( + tmp_path, + "--max-total-tokens-per-case", + "0", + "--budget-action", + "stop-suite", + "--concurrency", + "1", + ) + assert result.exit_code == 0, result.output + + summary = json.loads((_iteration_dir(tmp_path) / "summary.json").read_text()) + # First run completes (and trips the budget), the rest are skipped. + assert summary["runs_completed"] == 1 + assert len(summary["skipped_runs"]) == len(ALL_EVAL_IDS) - 1 + assert summary["budget"]["exceeded_runs"] + assert "stopped early by budget" in result.output + + def test_unknown_budget_action_rejected(self, tmp_path): + result = _run_cli(tmp_path, "--budget-action", "explode") + assert result.exit_code == 1 + assert "Unknown --budget-action" in result.output + + +class TestLifecycleHooks: + def test_failing_pre_run_hook_aborts_before_any_case(self, tmp_path): + result = _run_cli(tmp_path, "--pre-run-command", "echo setup broken >&2; exit 3") + assert result.exit_code == 1 + assert "pre-run hook failed (exit 3)" in result.output + assert "setup broken" in result.output + assert not list(_iteration_dir(tmp_path).glob("eval-*")) + + def test_pre_run_hook_receives_ase_env(self, tmp_path): + out_file = tmp_path / "hook-env.txt" + result = _run_cli( + tmp_path, + "--eval-id", + "explicit-invoke", + "--pre-run-command", + f'echo "$ASE_SKILL_NAME|$ASE_ITERATION|$ASE_EVAL_IDS" > {out_file}', + ) + assert result.exit_code == 0, result.output + assert out_file.read_text().strip() == "format-json|1|explicit-invoke" + + def test_post_grade_hook_merges_json_results_into_grading(self, tmp_path): + hook_json = '[{"text": "remote MR exists", "passed": false, "evidence": "no MR found"}]' + result = _run_cli( + tmp_path, + "--eval-id", + "explicit-invoke", + "--post-grade-command", + f"echo '{hook_json}'", + ) + assert result.exit_code == 0, result.output + + config_dir = _iteration_dir(tmp_path) / "eval-explicit-invoke" / "fake" / "with_skill" + grading = json.loads((config_dir / "grading.json").read_text()) + hook_results = [a for a in grading["assertion_results"] if a["method"] == "hook"] + assert len(hook_results) == 1 + assert hook_results[0]["text"] == "remote MR exists" + assert hook_results[0]["passed"] is False + assert grading["summary"]["total"] == len(grading["assertion_results"]) + assert (config_dir / "outputs" / "post_grade_hooks.json").exists() + + def test_failing_post_grade_hook_adds_failed_check(self, tmp_path): + result = _run_cli( + tmp_path, + "--eval-id", + "explicit-invoke", + "--post-grade-command", + "echo external grader crashed >&2; exit 7", + ) + assert result.exit_code == 0, result.output + + config_dir = _iteration_dir(tmp_path) / "eval-explicit-invoke" / "fake" / "with_skill" + grading = json.loads((config_dir / "grading.json").read_text()) + hook_results = [a for a in grading["assertion_results"] if a["method"] == "hook"] + assert len(hook_results) == 1 + assert hook_results[0]["passed"] is False + assert "exited 7" in hook_results[0]["evidence"] + + def test_post_grade_hook_can_read_run_artifacts(self, tmp_path): + out_file = tmp_path / "post-grade-env.txt" + result = _run_cli( + tmp_path, + "--eval-id", + "explicit-invoke", + "--post-grade-command", + f'echo "$ASE_EVAL_ID|$ASE_AGENT|$ASE_WITH_SKILL" > {out_file}; test -f "$ASE_TIMING_PATH"', + ) + assert result.exit_code == 0, result.output + assert out_file.read_text().strip() == "explicit-invoke|fake|1" + + def test_failing_post_run_hook_does_not_fail_the_run(self, tmp_path): + result = _run_cli(tmp_path, "--eval-id", "explicit-invoke", "--post-run-command", "exit 5") + assert result.exit_code == 0, result.output + + summary = json.loads((_iteration_dir(tmp_path) / "summary.json").read_text()) + assert summary["hooks"]["post_run"][0]["exit_code"] == 5 + assert "post-run hook failed" in result.output + + +class TestProgressLog: + def test_progress_jsonl_records_lifecycle_events(self, tmp_path): + result = _run_cli(tmp_path, "--eval-id", "explicit-invoke") + assert result.exit_code == 0, result.output + + progress_path = _iteration_dir(tmp_path) / "progress.jsonl" + events = [json.loads(line) for line in progress_path.read_text().splitlines()] + names = [e["event"] for e in events] + for expected in ("suite_started", "run_started", "agent_started", "agent_finished", "run_finished"): + assert expected in names, f"missing {expected} in {names}" + assert names[-1] == "suite_finished" + + finished = next(e for e in events if e["event"] == "run_finished") + assert finished["eval_id"] == "explicit-invoke" + assert "pass_rate" in finished and "duration_s" in finished + assert all("ts" in e and "elapsed_s" in e for e in events) + + +class TestSummaryAndStatus: + def test_summary_json_contents(self, tmp_path): + result = _run_cli(tmp_path) + assert result.exit_code == 0, result.output + + summary = json.loads((_iteration_dir(tmp_path) / "summary.json").read_text()) + assert summary["skill_name"] == "format-json" + assert summary["agents"] == ["fake"] + assert sorted(summary["eval_ids"]) == sorted(ALL_EVAL_IDS) + assert summary["runs_completed"] == len(ALL_EVAL_IDS) + assert summary["tokens"]["total"] == len(ALL_EVAL_IDS) # fake reports 1 token per run + # The fake harness is known-free: cost is a real 0.0, not unavailable. + assert summary["cost"]["total_usd"] == 0.0 + assert summary["cost"]["runs_with_cost"] == len(ALL_EVAL_IDS) + assert "fake_with_skill" in summary["pass_rates"] + + def test_status_command_reads_summary(self, tmp_path): + run_result = _run_cli(tmp_path) + assert run_result.exit_code == 0, run_result.output + + runner = CliRunner() + result = runner.invoke(app, ["status", "--workspace", str(tmp_path / "format-json-workspace")]) + assert result.exit_code == 0, result.output + assert "iteration-1" in result.output + assert "Pass rates" in result.output + assert "cost:" in result.output + + def test_status_rebuilds_summary_for_old_runs(self, tmp_path): + run_result = _run_cli(tmp_path) + assert run_result.exit_code == 0, run_result.output + (_iteration_dir(tmp_path) / "summary.json").unlink() + + runner = CliRunner() + result = runner.invoke(app, ["status", "--workspace", str(tmp_path / "format-json-workspace")]) + assert result.exit_code == 0, result.output + assert "rebuilt a reduced summary" in result.output + assert "Pass rates" in result.output + + def test_status_without_iterations_fails_cleanly(self, tmp_path): + runner = CliRunner() + result = runner.invoke(app, ["status", "--workspace", str(tmp_path)]) + assert result.exit_code == 1 + assert "No iterations found" in result.output + + +class TestCostSemantics: + def test_codex_cost_is_unavailable_not_zero(self, tmp_path): + harness = CodexHarness(tmp_path) + stdout = json.dumps( + {"type": "turn.completed", "usage": {"input_tokens": 100, "output_tokens": 10, "cached_input_tokens": 60}} + ) + _, timing = harness.parse_output(stdout, "") + harness.finalize_timing(timing) + assert timing.cost_usd is None + assert timing.cost_usd_source is None + + def test_codex_pricing_config_computes_cost(self, tmp_path): + pricing = {"gpt-test": {"prompt": 1e-6, "completion": 2e-6, "input_cache_read": 1e-7}} + harness = CodexHarness(tmp_path, model="gpt-test", pricing=pricing) + stdout = json.dumps( + {"type": "turn.completed", "usage": {"input_tokens": 100, "output_tokens": 10, "cached_input_tokens": 60}} + ) + _, timing = harness.parse_output(stdout, "") + harness.finalize_timing(timing) + # codex input_tokens INCLUDE cache reads: 40 non-cached + 60 cached. + assert timing.cost_usd == pytest.approx(40 * 1e-6 + 60 * 1e-7 + 10 * 2e-6) + assert timing.cost_usd_source == "pricing-config" + + def test_pricing_config_ignored_for_unknown_model(self, tmp_path): + harness = CodexHarness(tmp_path, model="other-model", pricing={"gpt-test": {"prompt": 1e-6}}) + timing = TimingData(input_tokens=100, total_tokens=100) + harness.finalize_timing(timing) + assert timing.cost_usd is None + + def test_opencode_without_cost_fields_is_unavailable(self, tmp_path): + harness = OpenCodeHarness(tmp_path) + stdout = json.dumps({"type": "step_finish", "part": {"tokens": {"input": 10, "output": 2}}}) + _, timing = harness.parse_output(stdout, "") + assert timing.cost_usd is None + + def test_opencode_zero_cost_step_is_real_zero(self, tmp_path): + harness = OpenCodeHarness(tmp_path) + stdout = json.dumps({"type": "step_finish", "part": {"cost": 0, "tokens": {"input": 10, "output": 2}}}) + _, timing = harness.parse_output(stdout, "") + assert timing.cost_usd == 0.0 + + def test_claude_missing_total_cost_is_unavailable(self, tmp_path): + harness = ClaudeCodeHarness(tmp_path) + stdout = json.dumps({"result": "done", "usage": {"input_tokens": 5}}) + _, timing = harness.parse_output(stdout, "") + assert timing.cost_usd is None + + def test_claude_non_cached_input_excludes_nothing(self, tmp_path): + # Anthropic usage reports input_tokens already excluding cache reads. + harness = ClaudeCodeHarness(tmp_path) + timing = TimingData(input_tokens=100, cached_tokens=60) + assert harness.non_cached_input_tokens(timing) == 100 + assert CodexHarness(tmp_path).non_cached_input_tokens(timing) == 40 + + def test_compute_stats_cost_unavailable_when_no_run_reported(self): + results = [ + { + "eval_id": "e1", + "agent": "codex", + "with_skill": True, + "grading": {"summary": {"pass_rate": 1.0, "total": 1}}, + "timing": {"duration_ms": 1000, "total_tokens": 100, "cost_usd": None}, + } + ] + stats = compute_stats(results) + assert stats.cost_usd is None + assert stats.cost_runs == 0 + + def test_compute_stats_cost_means_only_reported_runs(self): + def r(cost): + return { + "eval_id": "e1", + "agent": "a", + "with_skill": True, + "grading": {"summary": {"pass_rate": 1.0, "total": 1}}, + "timing": {"duration_ms": 1000, "total_tokens": 100, "cost_usd": cost}, + } + + stats = compute_stats([r(0.5), r(None), r(1.5)]) + assert stats.cost_usd is not None + assert stats.cost_usd.mean == pytest.approx(1.0) + assert stats.cost_runs == 2 + + def test_delta_cost_none_when_one_side_unavailable(self): + def r(agent, with_skill, cost): + return { + "eval_id": "e1", + "agent": agent, + "with_skill": with_skill, + "grading": {"summary": {"pass_rate": 1.0, "total": 1}}, + "timing": {"duration_ms": 1000, "total_tokens": 100, "cost_usd": cost}, + } + + results = {"a": r("codex", True, None), "b": r("codex", False, 0.5)} + benchmark = compute_benchmark(results, [AgentType.CODEX], with_baseline=True) + assert benchmark.deltas["codex"].cost_usd is None + + def test_report_renders_unavailable_cost_as_na(self, tmp_path): + from agent_skill_eval.cli import _benchmark_rows + + benchmark = { + "run_summary": { + "codex_with_skill": { + "pass_rate": {"mean": 1.0, "stddev": 0.0}, + "time_seconds": {"mean": 1.0, "stddev": 0.0}, + "tokens": {"mean": 100.0, "stddev": 0.0}, + "cost_usd": None, + } + } + } + rows = _benchmark_rows(benchmark) + assert rows[0][-1] == "n/a" + + +class TestReasoningTokens: + def test_codex_persists_reasoning_tokens_across_turns(self, tmp_path): + harness = CodexHarness(tmp_path) + events = [ + json.dumps( + { + "type": "turn.completed", + "usage": {"input_tokens": 100, "output_tokens": 10, "reasoning_output_tokens": 200}, + } + ), + json.dumps( + { + "type": "turn.completed", + "usage": {"input_tokens": 50, "output_tokens": 5, "reasoning_output_tokens": 47}, + } + ), + ] + _, timing = harness.parse_output("\n".join(events), "") + assert timing.reasoning_output_tokens == 247 + + def test_codex_without_reasoning_telemetry_is_none_not_zero(self, tmp_path): + harness = CodexHarness(tmp_path) + stdout = json.dumps({"type": "turn.completed", "usage": {"input_tokens": 100, "output_tokens": 10}}) + _, timing = harness.parse_output(stdout, "") + assert timing.reasoning_output_tokens is None + + def test_opencode_persists_reasoning_tokens(self, tmp_path): + harness = OpenCodeHarness(tmp_path) + stdout = json.dumps({"type": "step_finish", "part": {"tokens": {"input": 10, "output": 2, "reasoning": 30}}}) + _, timing = harness.parse_output(stdout, "") + assert timing.reasoning_output_tokens == 30 + + def test_opencode_without_reasoning_telemetry_is_none(self, tmp_path): + harness = OpenCodeHarness(tmp_path) + stdout = json.dumps({"type": "step_finish", "part": {"tokens": {"input": 10, "output": 2}}}) + _, timing = harness.parse_output(stdout, "") + assert timing.reasoning_output_tokens is None + + def test_reasoning_budget_violation(self): + budget = BudgetConfig(max_reasoning_tokens=100) + timing = TimingData(reasoning_output_tokens=150) + assert budget.violations(timing) == ["reasoning tokens 150 > 100"] + + def test_reasoning_budget_skipped_when_unreported(self): + # Like cost: an agent that exposes no reasoning telemetry must not + # fail a reasoning budget over an unknown quantity. + budget = BudgetConfig(max_reasoning_tokens=100) + assert budget.violations(TimingData(reasoning_output_tokens=None)) == [] + + def test_non_cached_budget_violation_and_skip(self): + budget = BudgetConfig(max_non_cached_input_tokens=10) + assert budget.violations(TimingData(non_cached_input_tokens=25)) == ["non-cached input tokens 25 > 10"] + assert budget.violations(TimingData(non_cached_input_tokens=None)) == [] + + def test_summary_reasoning_total_none_when_unreported(self, tmp_path): + result = _run_cli(tmp_path, "--eval-id", "explicit-invoke") + assert result.exit_code == 0, result.output + summary = json.loads((_iteration_dir(tmp_path) / "summary.json").read_text()) + # The fake agent reports no reasoning telemetry: unknown, not 0. + assert summary["tokens"]["reasoning_output"] is None + # The fake harness records its billable input split per run. + assert summary["tokens"]["non_cached_input"] == 1 + + +class TestModelConfigRecording: + def test_run_meta_records_effective_config(self, tmp_path): + result = _run_cli(tmp_path, "--eval-id", "explicit-invoke") + assert result.exit_code == 0, result.output + + config_dir = _iteration_dir(tmp_path) / "eval-explicit-invoke" / "fake" / "with_skill" + meta = json.loads((config_dir / "run_meta.json").read_text()) + assert meta["harness_version"] == __version__ + assert meta["model"] is None + assert meta["reasoning_effort"] is None + assert meta["base_url"] is None + # The fake agent has no version command. + assert meta["agent_cli_version"] is None + + timing = json.loads((config_dir / "timing.json").read_text()) + assert timing["non_cached_input_tokens"] == 1 + + def test_codex_command_includes_reasoning_effort(self, tmp_path): + harness = CodexHarness(tmp_path, reasoning_effort="medium") + cmd = harness.build_command("fix it", tmp_path) + assert "-c" in cmd + assert 'model_reasoning_effort="medium"' in cmd + + def test_codex_command_omits_reasoning_effort_when_unset(self, tmp_path): + cmd = CodexHarness(tmp_path).build_command("fix it", tmp_path) + assert "-c" not in cmd + + def test_unsupported_harness_ignores_reasoning_effort(self, tmp_path): + # claude-code/opencode have no pass-through flag: the effective value + # is None and run_meta must not claim an effort that was never set. + assert ClaudeCodeHarness(tmp_path, reasoning_effort="medium").reasoning_effort is None + assert OpenCodeHarness(tmp_path, reasoning_effort="medium").reasoning_effort is None + assert CodexHarness(tmp_path, reasoning_effort="medium").reasoning_effort == "medium" + + def test_run_warns_when_agent_ignores_reasoning_effort(self, tmp_path): + result = _run_cli(tmp_path, "--eval-id", "explicit-invoke", "--reasoning-effort", "medium") + assert result.exit_code == 0, result.output + assert "--reasoning-effort is not supported by fake" in result.output + + config_dir = _iteration_dir(tmp_path) / "eval-explicit-invoke" / "fake" / "with_skill" + meta = json.loads((config_dir / "run_meta.json").read_text()) + assert meta["reasoning_effort"] is None diff --git a/tests/test_runner.py b/tests/test_runner.py index 2f59418..42b4566 100644 --- a/tests/test_runner.py +++ b/tests/test_runner.py @@ -5,7 +5,7 @@ from pathlib import Path from unittest.mock import patch -from agent_skill_eval.models import AgentType, EvalCase +from agent_skill_eval.models import AgentType, EvalCase, TimingData from agent_skill_eval.runner import EvalRunner @@ -27,7 +27,7 @@ def __init__(self, agent_type=None, workspace=None, model=None, **kwargs): def run(self, prompt, output_dir): captured_prompts.append(prompt) - return "", type("T", (), {"model_dump": lambda self: {}})(), "", "" + return "", TimingData(), "", "" with patch("agent_skill_eval.runner.get_harness", side_effect=FakeHarness): runner = EvalRunner( @@ -63,7 +63,7 @@ def __init__(self, agent_type=None, workspace=None, model=None, **kwargs): def run(self, prompt, output_dir): captured_prompts.append(prompt) - return "", type("T", (), {"model_dump": lambda self: {}})(), "", "" + return "", TimingData(), "", "" with patch("agent_skill_eval.runner.get_harness", side_effect=FakeHarness): runner = EvalRunner( @@ -104,7 +104,7 @@ def __init__(self, agent_type=None, workspace=None, model=None, **kwargs): def run(self, prompt, output_dir): captured_prompts.append(prompt) - return "", type("T", (), {"model_dump": lambda self: {}})(), "", "" + return "", TimingData(), "", "" with patch("agent_skill_eval.runner.get_harness", side_effect=FakeHarness): runner = EvalRunner( @@ -155,19 +155,7 @@ def run(self, prompt, output_dir): ) return ( "I did the work", - type( - "T", - (), - { - "model_dump": lambda self: { - "total_tokens": 0, - "input_tokens": 0, - "output_tokens": 0, - "cached_tokens": 0, - "duration_ms": 0, - } - }, - )(), + TimingData(), "", "", ) @@ -650,19 +638,7 @@ def test_run_saves_evals_meta(self, tmp_path, evals_path): with patch("agent_skill_eval.runner.get_harness") as mock_harness: mock_harness.return_value.run.return_value = ( "ok", - type( - "T", - (), - { - "model_dump": lambda self: { - "total_tokens": 0, - "input_tokens": 0, - "output_tokens": 0, - "cached_tokens": 0, - "duration_ms": 0, - } - }, - )(), + TimingData(), "", "", ) @@ -697,19 +673,7 @@ def test_run_writes_run_meta_per_config(self, tmp_path, evals_path): with patch("agent_skill_eval.runner.get_harness") as mock_harness: mock_harness.return_value.run.return_value = ( "ok", - type( - "T", - (), - { - "model_dump": lambda self: { - "total_tokens": 0, - "input_tokens": 0, - "output_tokens": 0, - "cached_tokens": 0, - "duration_ms": 0, - } - }, - )(), + TimingData(), "", "", ) @@ -750,19 +714,7 @@ def test_two_agents_produce_separate_output_dirs(self, tmp_path, evals_path): with patch("agent_skill_eval.runner.get_harness") as mock_harness: mock_harness.return_value.run.return_value = ( "ok", - type( - "T", - (), - { - "model_dump": lambda self: { - "total_tokens": 0, - "input_tokens": 0, - "output_tokens": 0, - "cached_tokens": 0, - "duration_ms": 0, - } - }, - )(), + TimingData(), "", "", )