diff --git a/.claude/skills/nasde-benchmark-calibration/SKILL.md b/.claude/skills/nasde-benchmark-calibration/SKILL.md index 1c63be0a..77cadcbb 100644 --- a/.claude/skills/nasde-benchmark-calibration/SKILL.md +++ b/.claude/skills/nasde-benchmark-calibration/SKILL.md @@ -107,6 +107,19 @@ The rubric to edit lives at `evals//tasks//assessment_criteria.md` dimension's scale/description, not one task's thresholds). The `` and `` come from the trial's `result.json` (`source`, `task_name`). +Calibration can also restructure a single task's dimensions: an `assessment_dimensions.json` +placed next to the task's `assessment_criteria.md` overrides the benchmark-wide file for that task +only (different fingerprint — old and new evaluations are never mixed in one summary group). For +change-related checks, the evaluator already hands every judge the agent's full diff +(`/agent_changes.diff` + inline diffstat) — rubrics should direct the judge to answer +"what did the agent change" questions from that diff. When a task additionally needs hard mechanical +enforcement (e.g. a disqualification score cap), ship a `precheck.sh` — the evaluator runs it, +injects its JSON into the judge prompt as ground facts, and enforces its optional +`normalized_score_cap`. See `examples/ddd-architectural-challenges/tasks/ddd-weather-discount/` for +the reference calibrated task and `CALIBRATION_ROUND2_2026-07-07.md` for the loop's acceptance +criteria (repeatability, judge-model agreement, human-ranking correlation, dimension disjointness, +regression assertions). + Show the user a concrete **diff of the rubric** — the specific threshold/description change that would have moved the judge toward the human's score — and **wait for approval before writing**. Never edit the rubric silently. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 2e62d7ba..0effcf32 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -184,6 +184,36 @@ flowchart TB --- +### Agent diff — universal judge input + +For every trial, the evaluator materializes the agent's full diff (start state → final +workspace: `git diff HEAD` + untracked files, via `workspace_diff.capture_patch` — the +same capture used for `changes.patch` in exports) into `/agent_changes.diff`, +and injects an "Agent diff" prompt section: the diffstat inline plus the file path for +Read/Grep (the `ClaudeSubprocessBackend` grants `--add-dir` on the trial dir when the +file is present). Rationale: the judge sees only the final state and — like a human +reviewer without a diff — cannot see removals or out-of-feature edits; the diff is the +universal, task-agnostic reference point for every "what did the agent change" check. +Skipped gracefully when the workspace has no git repo or nothing changed. + +### Per-task rubric inputs + +Three optional files next to a task's `assessment_criteria.md` refine its evaluation: + +- `assessment_dimensions.json` — overrides the benchmark-wide dimensions file **for that task only** + (`resolve_dimensions_path`). A different dimensions file yields a different fingerprint, so + evaluations under old and new dimensions are never mixed in one summary group. +- `ground_truth_decisions.json` — reference decisions injected verbatim into the judge prompt. +- `precheck.sh` — an optional deterministic policy layer on top of the agent diff: the evaluator + runs it on the host before judging (`bash precheck.sh `). Its stdout must be one + JSON object; it is injected into the judge prompt as "Deterministic pre-check signals" (facts the + judge must stay consistent with), recorded in `assessment_eval_*.json` under `precheck`, and its + optional `normalized_score_cap` (0..1) is enforced on the trial's normalized score (cap + application is recorded, so a capped score is always explainable). Use it when a task wants hard, + mechanical enforcement (e.g. a disqualification cap) rather than judge interpretation of the diff. + Any precheck failure degrades to "no precheck" with a warning. All three are also bundled into + `.calibration/` by `nasde calibrate publish`. + ## Evaluator configuration The evaluator agent is configurable via `[evaluation]` in `nasde.toml`. All options are optional — defaults provide a working evaluator out of the box. @@ -213,7 +243,7 @@ When `mcp_config` is set, its path is passed through to the backend CLI (`--mcp- ### Token & cost economics ([ADR-011](docs/adr/011-token-cost-metrics.md)) -Independently of the LLM judge, each trial's **token usage and cost** are read from the agent's `agent/trajectory.json` `final_metrics` (which Harbor writes for both Claude and Codex). A single extractor (`token_metrics.py`) computes `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + reasoning_output_tokens`, and a USD cost at the **full catalog rate with no cache discount** ("as if every run were the first" — deterministic, order-independent). It derives `token_efficiency` (score per 1M tokens) and `cost_efficiency` (score per USD), using the dominant evaluator cluster's `normalized_score_mean`. The same extractor feeds both the run path (`evaluator.py` → `assessment_summary.json`) and the export path (`results_exporter.py` → `metrics.json`), so they cannot diverge. Prices come from a bundled, versioned `pricing.toml`; an unpriced model leaves cost null (token metrics still computed). `nasde run` prints a per-`(agent, model)` cost table after assessment completes. +Independently of the LLM judge, each trial's **token usage and cost** are read from the agent's `agent/trajectory.json` `final_metrics` (which Harbor writes for both Claude and Codex). A single extractor (`token_metrics.py`) computes `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + reasoning_output_tokens`, the cache read/write volumes, and a **cache-aware USD cost** (ADR-014): fresh input at the full rate, cache writes at the cache-write rate, cache reads at the cached rate — what the API would bill for the run. The raw volumes stay in `token_usage`, so the cache-free ceiling is derivable offline; the scalar efficiency ratios were removed (ADR-011) — models are compared as a quality-vs-cost Pareto front. The same extractor feeds both the run path (`evaluator.py` → `assessment_summary.json`) and the export path (`results_exporter.py` → `metrics.json`), so they cannot diverge. Prices come from a bundled, versioned `pricing.toml`; an unpriced model leaves cost null (token metrics still computed). `nasde run` prints a per-`(agent, model)` cost table after assessment completes. --- diff --git a/CHANGELOG.md b/CHANGELOG.md index 1962b34b..62fd2977 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,18 @@ See [docs/RELEASING.md](docs/RELEASING.md) for the release procedure. ## [Unreleased] ### Changed +- **Cost is now cache-aware ([ADR-014](docs/adr/014-cache-aware-cost.md)) — supersedes ADR-011's + "as if every run were the first" formula.** `cost_usd` bills fresh input at the + full rate, cache writes at the new per-model `cache_write_per_1m` (Anthropic + 1-hour mode: 2× input), cache reads at `cached_input_per_1m` (0.1×), and output + at the output rate — matching what the API would bill (verified against Harbor's + per-step accounting to the cent on 20 of 24 grid trials). Rationale: measured + cache read ratios are a stable 93–98% of input across a full 24-trial grid, and + the old full-rate figure sat ~4.4× above a real bill. `token_usage` gains + `cache_write_tokens`; the cache-free ceiling is no longer stored (derivable as + `input × input_rate + output × output_rate`). A model entry missing cache rates + bills those volumes at the full input rate — conservative, never a silent + discount. Historical exports need a one-shot economics backfill to reprice. - **Harbor bumped from 0.13 to 0.19** (`harbor[daytona,modal,e2b,runloop,gke]>=0.19,<0.20`). The Python-API surface nasde drives (`JobConfig.model_validate`, `Job.create`, `job.run()`, `AgentConfig` `import_path`/`kwargs`/`skills`/`mcp_servers`/`env`) diff --git a/CLAUDE.md b/CLAUDE.md index 9742bda2..9bf9d591 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -102,7 +102,7 @@ See [ARCHITECTURE.md](ARCHITECTURE.md) for the full system architecture with dia - **Pass-through CLI**: `nasde harbor ...` delegates to Harbor's Typer app via `add_typer()`. `nasde opik ...` forwards args to Opik's Click CLI via `ctx.args`. - **Rubric calibration (ADR-010)**: `nasde calibrate publish PATHS...` / `pull-comments` close the loop between the LLM-as-a-Judge and a human reviewer by publishing trial diffs + assessments as PRs/MRs and pulling review comments back for rubric tuning. Two layers, deliberately separated: **GIT** (`git_platform_backends/git_ops.py` — `git push`/`ls-remote`, platform-agnostic, subprocess pattern from `docker.py`, not behind a Protocol) and **PLATFORM** (`git_platform_backends/` behind a `@runtime_checkable GitPlatformBackend` Protocol — `repo_exists`/`find_open_pr_for_branch`/`create_pr`/`fetch_pr_comments`/`validate_cli_installed`/`validate_auth`, mirroring `evaluator_backends/`). The base is keyed on `(repo, commit)` as an **orphan branch** `base/-` seeded once via `git archive HEAD` from the trial workspace (git deduplicates blobs by content across orphan bases — no shared ancestor needed); each trial is a feature branch `calib/-/` = base + the agent's `changes.patch` applied as a real commit + `.calibration/` files (no trajectory — secrets/clutter). `.calibration/` carries the reviewer's context: the task's `instruction.md` + `assessment_criteria.md` + `assessment_dimensions.json` (resolved from `result.json` `task_name`/`source`, trying both `tasks/` and `evals//tasks/` layouts), all `assessment_eval_.json`, `assessment_summary.json`, and `metrics.json`. Idempotency is **open-only**: `find_open_pr_for_branch` matches only OPEN PRs/MRs (`gh pr list --state open`, `glab mr list` default), so a re-run skips a live round but lets a fresh round publish once the prior one is closed. The PR body is a pure transform of the dominant `AssessmentSummary` cluster (`calibration_publisher._render_pr_body`). Backend is **auto-detected from the sink repo URL host** (`github.com`→`gh`, `*gitlab*`→`glab`; `[calibration] platform` overrides for self-hosted) — no `backend` config field, eliminating the backend≠host mismatch. Preflight before any work: detect → `validate_cli_installed` (`shutil.which`, precise per-platform message) → `validate_auth` (`gh|glab auth status` exit code) → `repo_exists` (parses OUTPUT — `gh repo view` exits 0 even for a missing repo). Repo creation is out of scope (push creates branches ad-hoc in an existing repo). Reuses `_expand_to_trials`/`_capture_patch`/`_build_metrics` from `results_exporter.py` and `_aggregate_evaluations`/`_load_raw_evaluations`/`AssessmentSummary` from `evaluator.py`. Prerequisites mirror the evaluator's CLI requirement (ADR-002): `git` + `gh`/`glab` + login, no SDK, CLI keyring holds auth. The `nasde-benchmark-calibration` skill orchestrates the human-in-the-loop flow. - **Results export (EXPERIMENTAL)**: `results_exporter.py` + `nasde results-export PATHS... --to DIR` copy the analytic *essence* of trial artifacts out of the gitignored `jobs/` tree into a flat per-trial layout (`DIR/__/` with `metrics.json`, `assessment_eval_*.json` (all repetitions), `assessment_summary.json`, `trajectory.json`, `changes.patch`, `verifier_stdout.txt`, `reward.txt`). Re-export **merges**: missing eval files are copied and the summary/metrics refreshed, while immutable files (trajectory, patch) are left as-is — so evaluations added after a first export are picked up. A legacy bare `assessment_eval.json` (pre-migration trial) is exported as `assessment_eval_1.json` with a `nasde migrate-evals` hint, so the export is never silently empty. Filesystem-as-interface: `DIR` is any plain path (iCloud/Dropbox/git repo) — no cloud SDK. It scans Harbor artifacts (`result.json`/`config.json`/`assessment_eval*.json`/`agent/trajectory.json`/workspace), **not** the best-effort `EXPERIMENT_LOG.md`. `metrics.json` is a self-contained summary composed from `result.json`+`config.json`+`agent/trajectory.json` — including **token & cost economics** (`token_usage`, `cost_usd`, `pricing_as_of`, `reasoning_effort`); see the token-cost note below and [ADR-011](docs/adr/011-token-cost-metrics.md). The code diff is captured as a patch (`git diff HEAD` + untracked via `git ls-files -z` + `git diff --no-index`, never `git add` — the workspace `.git` index is left untouched; `-z`/NUL parsing means non-ASCII untracked filenames are not dropped under `core.quotepath`). Selection is a mixed positional list of job and/or trial dirs (auto-classified: a dir whose children have `result.json` is a job; else a dir whose own `result.json` carries a `trial_name` key is a trial; a dir with a job-level `result.json` but no trial-shaped children/`trial_name` is skipped with a warning rather than mis-exported as garbage); re-export is idempotent and merge-based (a trial is reported `exported` only when something new was copied, else `skipped`). Reuses `_collect_trial_dirs`/`_load_json`/`_compute_duration_sec`/`_resolve_agent_name` from `evaluator.py`. Deliberately does **not** model "experiments" (one job can belong to many — a future UI layer's concern). -- **Token & cost metrics (ADR-011)**: every trial gets **token usage + USD cost** computed from the agent's `agent/trajectory.json` `final_metrics` and a versioned price catalog. `token_metrics.py` is the **single extractor** feeding both write paths: `evaluator.py` writes them onto `assessment_summary.json` (run) and `results_exporter.py` onto `metrics.json` (export). Definitions: `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + extra.reasoning_output_tokens` (Codex reasoning folded into output), `total = input + output`. **Cost is "as if every run were the first"** — full prompt volume at full input rate, **no cache discount** — so it is deterministic and independent of run order / cache TTL (the prompt-token count is fixed for a task; the cache hit rate is not). **The scalar `token_efficiency`/`cost_efficiency` ratios were REMOVED** — `normalized_score / denominator` has an arbitrary zero (score 0 = empty rubric, unreachable), so the ranking is not invariant to a baseline shift; model comparison is now a **Pareto front** (quality vs cost, quality vs tokens), shift-invariant, living in the `nasde-benchmark-runner` skill, not the toolkit. The raw signals (`token_usage`, `cost_usd`, `pricing_as_of`, score) stay and are the source of truth. Economics are **per-trial** (one agent run) → they live on `AssessmentSummary`, not `EvaluatorGroupSummary`. `model_name` and `reasoning_effort` are stamped on the summary because cross-model analysis groups by `(agent_name, model_name, reasoning_effort)` (agent_name = variant name, does not distinguish models; a different effort is a different configuration, never averaged together — mirrors how a changed `dimensions_fingerprint` is a different benchmark). The `reasoning_effort` stamp is read back from the per-trial Harbor `config.json` (`config.agent.kwargs.reasoning_effort`); when no override was set the stamp is `""` (only explicit overrides are recorded — Codex's implicit `high` default is NOT fabricated, and an unset effort is a valid "family default" state). Pricing is loaded via `pricing.py::load_pricing_layered(project_dir)` (ADR-013), which merges three layers per-model (whole-entry, higher wins): `/pricing.toml` > `~/.nasde/pricing.toml` > bundled `pricing.toml`. **Convention, not config** — a file literally named `pricing.toml` (mirrors `assessment_dimensions.json`); no `[pricing]` key. User layer is a HOME dotfolder (`~/.nasde/`, like `~/.claude`/`~/.codex`/`~/.gemini`), deliberately NOT `platformdirs` (that maps to `~/Library/Application Support` on macOS = app-state, not user-editable config; `platformdirs` stays for cache in `update_check.py`). Both write paths thread `project_dir` so run (`assessment_summary.json`) and export (`metrics.json`) agree on cost — the ADR-011 single-extractor invariant. The merged catalog is NOT cached (depends on `project_dir` + on-disk contents); bundled `_load_bundled_pricing` keeps its `lru_cache`, `load_pricing(path)` is unchanged. An applied override prints a dim transparency line. **Layer provenance is exposed** (ADR-013): `pricing.py::resolve_pricing_layers(project_dir)` returns the ordered `PricingLayer` stack and `effective_pricing_with_source(project_dir)` returns `{model: (price, layer_name)}`. Surfaced three ways — `nasde pricing show [--show-source]` (sub-app, room for future `pricing validate`/`path`), a "Pricing used" table at the end of `nasde run` (only models in the run), and a `pricing_used.json` written by `results-export` (effective rate + layer per priced model, self-contained audit). The Rich table renderer is shared in `pricing_report.py::render_pricing_table` (used by `pricing show` + run summary). Each model stamped with `as_of` + `source`; `cached_input_per_1m` is recorded for reference but **not** used in the cost formula. An **unpriced model** → `cost_usd` = `null` + a warning (token metrics still computed); a **missing/legacy trajectory** → all economics `null`. Never crashes the run. `nasde run` prints a per-`(agent, model, effort)` cost table (trials, score, tokens, $cost) plus the job path and an export hint (`runner.py::_print_job_summary`, called after assessment so the summaries exist); raw cost/token columns carry an inter-trial `±std` when the group has ≥2 trials (n=1 → bare value). Backfilling existing exports whose source jobs are gone is a **one-shot ad-hoc script** (reads the export's own flat `trajectory.json`) — deliberately NOT a CLI command. +- **Token & cost metrics (ADR-011; cost formula superseded by ADR-014 — cache-aware)**: every trial gets **token usage + USD cost** computed from the agent's `agent/trajectory.json` `final_metrics` and a versioned price catalog. `token_metrics.py` is the **single extractor** feeding both write paths: `evaluator.py` writes them onto `assessment_summary.json` (run) and `results_exporter.py` onto `metrics.json` (export). Definitions: `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + extra.reasoning_output_tokens` (Codex reasoning folded into output), `total = input + output`. **Cost is CACHE-AWARE (ADR-014)**: fresh input at the full rate + cache writes at `cache_write_per_1m` (Anthropic 1h mode, 2x) + cache reads at `cached_input_per_1m` (0.1x) + output at the output rate — what the API would bill for the run; matches Harbor's own per-step accounting. `token_usage` records `cached_tokens` (reads) and `cache_write_tokens`, so the cache-free ceiling stays derivable offline (`input*rate + output*rate`) and is NOT stored. A missing cache rate falls back to the full input rate (conservative, never a silent discount). **The scalar `token_efficiency`/`cost_efficiency` ratios were REMOVED** — `normalized_score / denominator` has an arbitrary zero (score 0 = empty rubric, unreachable), so the ranking is not invariant to a baseline shift; model comparison is now a **Pareto front** (quality vs cost, quality vs tokens), shift-invariant, living in the `nasde-benchmark-runner` skill, not the toolkit. The raw signals (`token_usage`, `cost_usd`, `pricing_as_of`, score) stay and are the source of truth. Economics are **per-trial** (one agent run) → they live on `AssessmentSummary`, not `EvaluatorGroupSummary`. `model_name` and `reasoning_effort` are stamped on the summary because cross-model analysis groups by `(agent_name, model_name, reasoning_effort)` (agent_name = variant name, does not distinguish models; a different effort is a different configuration, never averaged together — mirrors how a changed `dimensions_fingerprint` is a different benchmark). The `reasoning_effort` stamp is read back from the per-trial Harbor `config.json` (`config.agent.kwargs.reasoning_effort`); when no override was set the stamp is `""` (only explicit overrides are recorded — Codex's implicit `high` default is NOT fabricated, and an unset effort is a valid "family default" state). Pricing is loaded via `pricing.py::load_pricing_layered(project_dir)` (ADR-013), which merges three layers per-model (whole-entry, higher wins): `/pricing.toml` > `~/.nasde/pricing.toml` > bundled `pricing.toml`. **Convention, not config** — a file literally named `pricing.toml` (mirrors `assessment_dimensions.json`); no `[pricing]` key. User layer is a HOME dotfolder (`~/.nasde/`, like `~/.claude`/`~/.codex`/`~/.gemini`), deliberately NOT `platformdirs` (that maps to `~/Library/Application Support` on macOS = app-state, not user-editable config; `platformdirs` stays for cache in `update_check.py`). Both write paths thread `project_dir` so run (`assessment_summary.json`) and export (`metrics.json`) agree on cost — the ADR-011 single-extractor invariant. The merged catalog is NOT cached (depends on `project_dir` + on-disk contents); bundled `_load_bundled_pricing` keeps its `lru_cache`, `load_pricing(path)` is unchanged. An applied override prints a dim transparency line. **Layer provenance is exposed** (ADR-013): `pricing.py::resolve_pricing_layers(project_dir)` returns the ordered `PricingLayer` stack and `effective_pricing_with_source(project_dir)` returns `{model: (price, layer_name)}`. Surfaced three ways — `nasde pricing show [--show-source]` (sub-app, room for future `pricing validate`/`path`), a "Pricing used" table at the end of `nasde run` (only models in the run), and a `pricing_used.json` written by `results-export` (effective rate + layer per priced model, self-contained audit). The Rich table renderer is shared in `pricing_report.py::render_pricing_table` (used by `pricing show` + run summary). Each model stamped with `as_of` + `source`; `cached_input_per_1m` and `cache_write_per_1m` are live inputs to the cost formula (ADR-014). An **unpriced model** → `cost_usd` = `null` + a warning (token metrics still computed); a **missing/legacy trajectory** → all economics `null`. Never crashes the run. `nasde run` prints a per-`(agent, model, effort)` cost table (trials, score, tokens, $cost) plus the job path and an export hint (`runner.py::_print_job_summary`, called after assessment so the summaries exist); raw cost/token columns carry an inter-trial `±std` when the group has ≥2 trials (n=1 → bare value). Backfilling existing exports whose source jobs are gone is a **one-shot ad-hoc script** (reads the export's own flat `trajectory.json`) — deliberately NOT a CLI command. - See `docs/adr/` for detailed decision records. ## CLI reference @@ -406,6 +406,18 @@ If `harbor_config.json` is absent, `nasde` auto-generates one based on `variant. Every failure path must `echo 0 > /logs/verifier/reward.txt && exit 1`. Final success must `echo 1 > /logs/verifier/reward.txt && exit 0`. +## Calibration experiments (examples/ddd-architectural-challenges) + +The 2026-07 Fable/Opus grid experiment lives in this example. The methodology +record is `examples/ddd-architectural-challenges/CALIBRATION_ROUND2_2026-07-07.md` +(why rubric v1 failed, reference ranking, acceptance criteria); the current rubric +is documented in the task's `assessment_criteria.md`. Publication charts are +generated by `cost_quality_plane.py` and `verdict_heatmap.py` (PNGs in `assets/`). +Session ops tooling and the day-by-day experiment log live in the private +`NoesisVision/nasde-calibration` repo (`round2-fable-grid-ops/`). Results archive: +`NoesisVision/nasde-results`. Protocol: every LLM run is individually authorized +by the owner; export+commit+push results after every iteration. + ## Known issues and workarounds - **opik 2.x (and 1.10.x)**: token usage=None for Harbor spans — runtime monkeypatch in `runner.py` (`_patch_opik_deferred_metrics`). Defers Step span creation to `__setattr__` because Harbor assigns metrics after `Step.__init__`. See ADR-006. Remove when opik fixes upstream. diff --git a/docs/adr/011-token-cost-metrics.md b/docs/adr/011-token-cost-metrics.md index ef863bea..b9dad02b 100644 --- a/docs/adr/011-token-cost-metrics.md +++ b/docs/adr/011-token-cost-metrics.md @@ -1,6 +1,7 @@ # ADR-011: Token & cost metrics -**Status:** Accepted +**Status:** Accepted — cost formula superseded by ADR-014 (cache-aware cost); +everything else stands **Date:** 2026-06-08 ## Context diff --git a/docs/adr/014-cache-aware-cost.md b/docs/adr/014-cache-aware-cost.md new file mode 100644 index 00000000..db05ca8a --- /dev/null +++ b/docs/adr/014-cache-aware-cost.md @@ -0,0 +1,64 @@ +# ADR-014: Cache-aware cost is THE cost + +**Status:** Accepted +**Date:** 2026-07-15 +**Supersedes:** the cost formula of ADR-011 (everything else in ADR-011 stands) + +## Context + +ADR-011 priced every trial "as if every run were the first": the full prompt-token +volume billed at the full catalog input rate, no cache discount. The stated rationale +was determinism — the cache hit rate was assumed to be noisy (dependent on run order, +session length, TTL) while `total_prompt_tokens` is fixed for a task. + +Measurement on the 2026-07 Fable 5 / Opus 4.8 grid (24 trials, `ddd-weather-discount`) +falsified both halves of that assumption: + +1. **The cache ratio is not noisy.** Cache reads were 92.6–97.7% of prompt tokens on + every one of 24 trials, across two models and three instruction configurations. + Prompt caching in agentic sessions is a stable, structural property of the + harness (each step extends the same prompt prefix), not run-order luck. Cache + hits are effectively intra-session: across sessions only the short static prefix + can match, and only within the cache TTL. +2. **The full-rate number is far from a real bill.** The as-if formula priced the + grid at $958; the cache-aware price is $220 (4.4x lower). Harbor's own per-step + cost accounting confirms the cache-aware figure (agreement to the cent on 20 of + 24 trials; Harbor slightly higher on 4). A cost metric meant to drive real + model-choice decisions cannot be 4x away from the invoice it predicts. + +## Decision + +**`cost_usd` is cache-aware, and it is the only stored cost.** + +``` +fresh_input = total_prompt_tokens - cache_reads - cache_writes +cost = fresh_input * input_rate + + cache_writes * cache_write_rate (Anthropic 1h-cache: 2x input rate) + + cache_reads * cached_input_rate (0.1x input rate) + + output * output_rate +``` + +- `TokenUsage` records `cache_write_tokens` (from Harbor's + `extra.total_cache_creation_input_tokens`) next to the existing `cached_tokens` + (reads). All raw volumes remain in `token_usage`. +- `pricing.toml` gains `cache_write_per_1m` per model. A missing cache rate falls + back to the full input rate — conservative, never a silent discount. Codex + trajectories carry no cache-creation counter, and OpenAI bills no write premium, + so gpt entries simply omit the field. +- **No second cost field.** The old full-rate ceiling is NOT stored. Storing a + derived view next to the primary one re-creates the confusion ADR-011 itself + removed with the efficiency ratios: raw signals stay, derived numbers are an + analysis step. Anyone needing the ceiling computes + `input * input_rate + output * output_rate` from `token_usage`. + +## Consequences + +- Historical `metrics.json` exports carry full-rate `cost_usd` values until + re-exported; `pricing_as_of` plus this ADR's date separate the two eras. Re-export + refreshes economics without re-running anything. +- Cross-model comparisons barely move (both models cache at the same ~95% ratio; + rate ratios dominate), but absolute dollars drop ~4.4x and now approximate a real + API bill. Published figures must state "with prompt caching, at API rates". +- The determinism ADR-011 wanted survives in practice: the inputs to the formula + are the recorded per-trial volumes, so a recomputation is always reproducible; + what changed is that the recorded reality now includes the cache split. diff --git a/examples/ddd-architectural-challenges/CALIBRATION_ROUND2_2026-07-07.md b/examples/ddd-architectural-challenges/CALIBRATION_ROUND2_2026-07-07.md new file mode 100644 index 00000000..d7ddb30c --- /dev/null +++ b/examples/ddd-architectural-challenges/CALIBRATION_ROUND2_2026-07-07.md @@ -0,0 +1,175 @@ +# Calibration round 2 — ddd-weather-discount (2026-07-07) + +Follow-up to [CALIBRATION_TRIAL_2026-06-05.md](CALIBRATION_TRIAL_2026-06-05.md) (round 1: +5 trials, loop proven, no rubric edits applied). Round 2 covers **all 13 published +trials** (sink PRs #9–#21 on `NoesisVision/nasde-calibration`), adds **60 inline +calibration comments** (2026-06-15, pullable via `nasde calibrate pull-comments`), a +**human-approved reference ranking**, and lands **rubric v2** for the task. + +Companion reports (per-trial evidence, judge-score disagreements, comment index): +branch `nasde-calibrate` on the sink repo — `calibration-review-notes.md`, +`calibration-comments-index.md`. + +## Why v1 failed (evidence) + +- **Surface anchoring.** v1's domain_modeling text leads with a `Precipitation`-as-VO + example; on PR #16 a gpt-5.5 run scored 10/25 with reasoning *only* about + "precipitation as raw `decimal`, not a semantic value object", while opus runs gave + 20–23 for the same code. The criteria text drives the spread. +- **Instability.** domain_modeling std up to 5.03 within one judge (PR #21: runs 8–18); + cross-model gap up to 8.7 pts (PR #16: gpt 13.3 vs opus 22.0). +- **Blindness to restraint.** The two trials that stripped the author's + `[DddDomainService]` from six base files and rewrote `RiskManagementInMemoryCalls` + from `NotImplementedException` to `Money.Of(decimal.MaxValue, PLN)` (silent unlimited + credit) scored architecture_compliance 16–17/20 — above clean trials. +- **Inverted ranking.** v1's leader (#13, 0.94) is a reference bucket-C trial; both + bucket-D trials (0.84, 0.80) outscore the only bucket-A trial (0.77). +- **Dead dimensions.** encapsulation/extensibility/architecture largely re-measured the + same value-object signals as domain_modeling; test_quality reached 20/20 on suites + with zero coverage of the riskiest behavior (composition with the existing chain). + +## Reference ranking (human-approved, 2026-07-07) + +Buckets: **A** exemplary · **B** good with flaws · **C** flawed modeling · +**D** disqualified (out-of-feature damage; `precheck.sh` hard-fail → cap normalized +score at 0.45). + +| PR | Trial | Agent | v1 | Bucket | One-liner (reviewer's terms) | +|----|-------|-------|----|:------:|------------------------------| +| #21 | FjYQ3XQ | codex-ntcoding | 0.77 | **A** | Only trial to add the discount to the aggregation ONLY when weather qualifies (rules filtered in the factory against the once-fetched state of the world); flaw: rewrote `AggregatedModifier(List→IEnumerable)` — the author's intent not respected. | +| #11 | 2yQqBnm | claude-vanilla | 0.80 | **B** | Closure in the factory exemplary, but the generic "spread discounts over quotes" hidden as a private nested class in the weather module instead of an `OfferLevelDiscount` in Discounts; empty modifier always aggregated. | +| #12 | 3vwBnrU | claude-ntcoding | 0.85 | **B** | Factory ✓, but weather types dumped into shared `Pricing/Discounts` instead of a separate weather module resembling `SpecialOffers`; wrapper decides inside `ApplyOn`. | +| #14 | qHCAtXV | claude-vanilla | 0.70 | **B** | Factory ✓, minimal touch ✓, faithful reuse of the `ClientLevelDiscounts` path; rules in `Pricing/Discounts` (module coupling), silent stacking. v1's lowest score despite solidity. | +| #16 | aGTFmDh | claude-vanilla | 0.72 | **B** | Factory ✓, `PercentageDiscount` reuse ✓; over-build: static policy registry for one rule, self-disabling 0%-guard modifier always in the chain. Main victim of v1 judge spread. | +| #17 | SnS5iHF | claude-ntcoding | 0.75 | **B** | Exemplary closure (parallel tuple-await) and module, but introduced `NoOfferModifier` — a NoDiscounts with no reason to exist (albeit placed in shared Pricing, not weather); `Percentage`-only contract cuts off `ValueDiscount`. | +| #9 | SuuU3yh | claude-ntcoding | 0.88 | **C** | State of the world closed once but NOT in the factory — second `.Apply()` path in `CalculatePrices`; its `WeatherDiscount` is de facto a WeatherPercentageDiscount duplicating the `ClientLevelDiscounts` path; failure encoded as `Clear()`. | +| #10 | Kc8Es5k | claude-ntcoding | 0.79 | **C** | Built a proper weather factory, then bypassed `OfferModifiers` anyway (applied via `CalculatePrices`); `Unavailable = new(0)`; scratch `decompile.csx` committed. | +| #13 | ZvSsnyg | claude-ntcoding-**tuned** | **0.94** | **C** | v1's leader: factory closure exemplary, but the discount is applied BEFORE special offers (feeds discounted quotes into `IndividualSalesConditions`' `min()`), `Unknown ≡ None` erases the failure state, and the failure-path test is vacuous. Score partly reflects the 569-line tuned skill, not the model. | +| #15 | Jyu9YsY | claude-ntcoding | 0.81 | **C** | Deepest factory bypass: `IEnumerable` injected into the `CalculatePrices` constructor (policy registry in the orchestrator); a test canonizes compounding (100→90→81) with no spec basis. | +| #19 | cE8V77t | codex-vanilla | 0.68 | **C** | Reviewer's manual round-1 case: `NoDiscount` with no reason to exist (and in the weather module, not Discounts), bespoke `PercentageWeatherDiscount`, and a clumsy discount factory whose rule engine broke domain invariants (phantom discounts in Quotes). | +| #18 | 4njch2w | codex-vanilla | 0.84 | **D** | Paradox: the only canonical `OfferWideDiscount` in `Pricing/Discounts` (best type reuse of all 13) — and `[DddDomainService]` stripped from six base files, `RiskManagementInMemoryCalls` rewritten to unlimited credit, `Program.cs` reworked. | +| #20 | URtZnzf | codex-ntcoding | 0.80 | **D** | Twin of #18 (same six files, same fabrication) plus raw Open-Meteo wire strings (`"temperature_2m"`) as deep-model vocabulary. | + +## What round 2 lands + +Round 2 has priority over prior toolkit conventions: where the loop needed toolkit +support, the toolkit was extended (per the project owner's direction), not worked +around. + +**Toolkit changes (src/nasde_toolkit):** + +- **Agent diff as universal judge input.** For every trial (every task, every + benchmark), the evaluator materializes the agent's full diff (start state → final + workspace, tracked + untracked, via the same `workspace_diff.capture_patch` used for + `changes.patch`) into `/agent_changes.diff` and injects an "Agent diff" + prompt section (diffstat inline + the file path for Read/Grep; the claude backend + grants `--add-dir` on the trial dir). This gives the judge the same reference point + a human reviewer gets — removals and out-of-feature edits become visible — without + per-task processing and without handing the judge free-form git (which breeds + variance: the codex judge always had shell access and demonstrably didn't use it). +- **Per-task dimensions.** `assessment_dimensions.json` placed next to a task's + `assessment_criteria.md` now overrides the challenge-level file + (`evaluator.resolve_dimensions_path`, used by both the evaluator and + `calibrate publish`). Other tasks keep the shared 5-dimension file; a different + dimensions file yields a different fingerprint, so v1/v2 evaluations are never mixed + in one summary group. +- **Deterministic precheck hook (optional policy layer).** If a task ships + `precheck.sh`, the evaluator runs it against the trial workspace, validates its + JSON, injects it into the judge prompt as "Deterministic pre-check signals", records + it in `assessment_eval_*.json`, and enforces an optional `normalized_score_cap`. + With the agent diff as the universal input, precheck's role narrows to hard + mechanical policy (the bucket-D disqualification cap) rather than being the judge's + only window on changes. Any precheck failure degrades to "no precheck" with a + warning. +- **Reviewer bundle.** `calibrate publish` now also ships + `ground_truth_decisions.json` under `.calibration/`. + +**Task changes (tasks/ddd-weather-discount):** + +1. **Task-level `assessment_dimensions.json`** — three disjoint dimensions: + `model_fit` (0–50), `restraint` (0–25), `test_quality` (0–25). +2. **`assessment_criteria.md` v2** — decidable checks (FULL/PARTIAL/NONE with + evidence): model_fit M1–M7 (closure & purity, factory policy, canonical type reuse, + no phantoms, explicit exclusivity, failure≠measurement, proportionate seam), + restraint R1–R6 (touchpoints & no fabrication, annotations, signatures, artifacts, + modularization mirror, domain language), test_quality T1–T5 (composition through + `ChooseFor`, single-fetch guarantee, honest failure/boundary, adapter isolation, + conventions). Anchors taken from the 13 real trials; "base-model intent" preamble + (`[Pure]`, `[DddFactory]`, `ExchangeRate:PriceModifier ≠ policy`, explicit + `Or`/`min()` idioms). +3. **`ground_truth_decisions.json`** — the seven reference decisions, auto-injected + into the judge prompt. +4. **`precheck.sh`** — dual-mode (evaluator workspace / sink refs) restraint signals + with advisory R1–R4 scores and `normalized_score_cap: 0.45` on hard fail + (annotation stripping ≥2 files, or behavior fabrication). Smoke-tested against the + sink: #21 → R-signals without cap; #18 → all R floored, cap 0.45. + +## Acceptance criteria for the re-run (loop stop conditions) + +Re-evaluate all 13 trials with v2 (2 judge models × 3 runs, as in v1). Accept v2 when +ALL hold; otherwise diagnose failing cases from judge reasoning, patch the specific +check's wording, and re-run. + +1. **Repeatability:** per-dimension std within a judge model ≤ 8% of the dimension max + (model_fit ≤ 4.0, restraint/test_quality ≤ 2.0). v1 worst: 20% (5.03/25). +2. **Judge-model agreement:** per-dimension mean gap between models ≤ 12% of the + dimension max (model_fit ≤ 6.0, restraint/test_quality ≤ 3.0). v1 worst: 35%. +3. **Human agreement:** Spearman rank correlation (bucket ranks, ties within buckets) + between the v2 total ordering and the reference ranking ≥ 0.8, AND strict bucket + separation: every A total > every B > every C > every D. +4. **Dimension disjointness:** pairwise Pearson |r| between dimension scores across the + 13 trials ≤ 0.6 (v1: domain_modeling/encapsulation/architecture strongly coupled). +5. **Regression assertions (known v1 misfires):** + - #21 model_fit ≥ 38/50 and std ≤ 4 (v1 domain_modeling: mean 12.67/25, std 5.03); + - #16 model_fit cross-model gap ≤ 6/50 (v1: 8.7/25); + - #13 no longer ranked #1 overall; #13 model_fit ≤ 36/50 (v1 domain_modeling 25/25); + - #18 and #20: precheck cap holds (normalized ≤ 0.45) and restraint ≤ 8/25 + (v1 architecture_compliance 16.3/17.3 of 20); + - no trial reaches test_quality ≥ 18/25 without a composition test through + `OfferModifiers.ChooseFor` (in v1, two suites with zero composition coverage + scored 20/20). + +## Judge-model matrix (round-2 run plan) + +The re-run doubles as a judge-model comparison. One command drives the whole thing: + +``` +./calibration_round2_run.sh # add NASDE_RESULTS_PUSH=1 to also push results +``` + +| Judge model | Backend | Why | +|---|---|---| +| `claude-fable-5` | claude | Newest Claude (Mythos-class tier above Opus) | +| `claude-opus-4-8` | claude | Newest Opus (v1 judged with opus-4-7) | +| `gpt-5.5` | codex | Best model confirmed available in the codex CLI as of the June trials — bump the `MATRIX` entry if a newer one ships | + +Mechanics (all implemented in this round): + +1. `calibration_round2_run.sh` assembles `jobs/calibration-round2/` — symlinks to the + 11 locally present reference trials plus **sink-restores** for the two whose local + job dirs are gone (`URtZnzf`, `FjYQ3XQ`): `calibration_round2_restore.sh` rebuilds + the evaluator contract from the calibration sink (workspace HEAD = base snapshot, + agent diff applied uncommitted, `result.json`/`config.json` synthesized from the + sink's `metrics.json`). Both restores verified against `workspace_diff.capture_patch`. +2. `nasde eval` runs 3× per judge via the new `--eval-model` / `--eval-backend` + overrides (13 trials × 3 judges × 3 reps = 117 judge runs). Evals append to the + trial dirs; the v2 task-level dimensions fingerprint keeps them in separate summary + groups per judge, never mixed with v1. +3. `calibration_round2_check.py` computes the acceptance criteria above mechanically + from `assessment_eval_*.json` (per judge model where applicable) and exits non-zero + on any FAIL — the loop's stop condition as an executable, not a judgment call. +4. `nasde results-export` copies the essence (metrics, scores, patches, trajectories) + to the `nasde-results` repo under `calibration-round2-ddd-weather-discount/`, + together with `acceptance_report.txt`, and commits — so the round's evidence + survives even if `jobs/` is cleared. + +## Procedure (calibration orchestrator) + +1. `nasde eval` the existing job dirs — the evaluator picks up the task-level + dimensions, ground truth and precheck automatically (no agent re-runs needed). +2. Compute acceptance criteria 1–5 from the fresh `assessment_eval_*.json` / + `assessment_summary.json`; report a pass/fail table. +3. On failure: pull the offending judge reasoning, map it to the specific M/R/T check, + propose a wording diff (per the `nasde-benchmark-calibration` skill — **wait for + approval before writing**), re-run. +4. On pass: republish with `nasde calibrate publish` for a confirmation human pass. diff --git a/examples/ddd-architectural-challenges/assets/cost_quality_plane.png b/examples/ddd-architectural-challenges/assets/cost_quality_plane.png new file mode 100644 index 00000000..83871909 Binary files /dev/null and b/examples/ddd-architectural-challenges/assets/cost_quality_plane.png differ diff --git a/examples/ddd-architectural-challenges/assets/cost_quality_plane_en.png b/examples/ddd-architectural-challenges/assets/cost_quality_plane_en.png new file mode 100644 index 00000000..1161262c Binary files /dev/null and b/examples/ddd-architectural-challenges/assets/cost_quality_plane_en.png differ diff --git a/examples/ddd-architectural-challenges/assets/judge_retest.png b/examples/ddd-architectural-challenges/assets/judge_retest.png new file mode 100644 index 00000000..5f98699b Binary files /dev/null and b/examples/ddd-architectural-challenges/assets/judge_retest.png differ diff --git a/examples/ddd-architectural-challenges/assets/judge_retest_en.png b/examples/ddd-architectural-challenges/assets/judge_retest_en.png new file mode 100644 index 00000000..0173f15e Binary files /dev/null and b/examples/ddd-architectural-challenges/assets/judge_retest_en.png differ diff --git a/examples/ddd-architectural-challenges/assets/verdict_heatmap.png b/examples/ddd-architectural-challenges/assets/verdict_heatmap.png new file mode 100644 index 00000000..b930e41c Binary files /dev/null and b/examples/ddd-architectural-challenges/assets/verdict_heatmap.png differ diff --git a/examples/ddd-architectural-challenges/assets/verdict_heatmap_en.png b/examples/ddd-architectural-challenges/assets/verdict_heatmap_en.png new file mode 100644 index 00000000..2e76b5f1 Binary files /dev/null and b/examples/ddd-architectural-challenges/assets/verdict_heatmap_en.png differ diff --git a/examples/ddd-architectural-challenges/calibration_round2_check.py b/examples/ddd-architectural-challenges/calibration_round2_check.py new file mode 100644 index 00000000..6e005111 --- /dev/null +++ b/examples/ddd-architectural-challenges/calibration_round2_check.py @@ -0,0 +1,331 @@ +"""Acceptance assertions for calibration round 2 (ddd-weather-discount, rubric v2). + +Computes the five loop stop-conditions from CALIBRATION_ROUND2_2026-07-07.md +mechanically, from assessment_eval_*.json files — no human in the loop for the +check itself. Every assertion is a falsifiable prediction derived from a +diagnosed v1 misfire, with a direction and a margin. + +Usage: + uv run python calibration_round2_check.py [--project-dir .] + +Exit code 0 = all criteria pass (v2 accepted), 1 = at least one FAIL. +""" + +from __future__ import annotations + +import argparse +import json +import statistics +import sys +from dataclasses import dataclass, field +from itertools import combinations +from pathlib import Path + +from nasde_toolkit.evaluator import _dimensions_fingerprint + +# Human-approved reference ranking (2026-07-07). Buckets: A exemplary, +# B good-with-flaws, C flawed modeling, D disqualified. +REFERENCE_BUCKETS: dict[str, str] = { + "SuuU3yh": "C", + "Kc8Es5k": "C", + "2yQqBnm": "B", + "3vwBnrU": "B", + "ZvSsnyg": "C", + "qHCAtXV": "B", + "Jyu9YsY": "C", + "aGTFmDh": "B", + "SnS5iHF": "B", + "4njch2w": "D", + "cE8V77t": "C", + "URtZnzf": "D", + "FjYQ3XQ": "A", +} +BUCKET_RANK = {"A": 4.0, "B": 3.0, "C": 2.0, "D": 1.0} + +# Margins (fractions of a dimension's max score). Chosen from the v1 empirical +# distribution: v1's worst within-judge std was 20% of scale and its worst +# cross-judge gap 35%, while FULL/PARTIAL/NONE check quantization implies a +# natural noise floor of roughly one check's half-step (~4-8%). +STD_LIMIT_FRACTION = 0.08 +GAP_LIMIT_FRACTION = 0.12 +SPEARMAN_MIN = 0.8 +DISJOINTNESS_MAX_ABS_R = 0.6 + + +@dataclass +class TrialEvals: + suffix: str + # evaluator_model -> list of eval dicts (v2 fingerprint only) + by_model: dict[str, list[dict]] = field(default_factory=dict) + + +@dataclass +class CheckResult: + criterion: str + passed: bool + detail: str + + +def _std(values: list[float]) -> float: + return statistics.stdev(values) if len(values) > 1 else 0.0 + + +def _pearson(xs: list[float], ys: list[float]) -> float: + if len(xs) != len(ys) or len(xs) < 2: + return 0.0 + mx, my = statistics.fmean(xs), statistics.fmean(ys) + cov = sum((x - mx) * (y - my) for x, y in zip(xs, ys, strict=True)) + sx = sum((x - mx) ** 2 for x in xs) ** 0.5 + sy = sum((y - my) ** 2 for y in ys) ** 0.5 + if sx == 0 or sy == 0: + return 0.0 + return cov / (sx * sy) + + +def _ranks_with_ties(values: list[float]) -> list[float]: + order = sorted(range(len(values)), key=lambda i: values[i]) + ranks = [0.0] * len(values) + i = 0 + while i < len(order): + j = i + while j + 1 < len(order) and values[order[j + 1]] == values[order[i]]: + j += 1 + average_rank = (i + j) / 2 + 1 + for k in range(i, j + 1): + ranks[order[k]] = average_rank + i = j + 1 + return ranks + + +def _spearman(xs: list[float], ys: list[float]) -> float: + return _pearson(_ranks_with_ties(xs), _ranks_with_ties(ys)) + + +def collect(job_dir: Path, fingerprint: str) -> list[TrialEvals]: + if not job_dir.is_dir(): + return [] + trials: list[TrialEvals] = [] + for trial_dir in sorted(job_dir.iterdir()): + if not (trial_dir / "result.json").exists(): + continue + suffix = trial_dir.name.rsplit("__", 1)[-1] + if suffix not in REFERENCE_BUCKETS: + continue + trial = TrialEvals(suffix=suffix) + for eval_path in sorted(trial_dir.glob("assessment_eval_*.json")): + data = json.loads(eval_path.read_text(encoding="utf-8")) + if data.get("dimensions_fingerprint") != fingerprint: + continue + trial.by_model.setdefault(data["evaluator_model"], []).append(data) + trials.append(trial) + return trials + + +def _dim_scores(evals: list[dict], name: str) -> list[float]: + return [float(d["score"]) for e in evals for d in e["dimensions"] if d["name"] == name] + + +def _dim_max(trials: list[TrialEvals], name: str) -> float: + for trial in trials: + for evals in trial.by_model.values(): + for e in evals: + for d in e["dimensions"]: + if d["name"] == name: + return float(d["max_score"]) + raise ValueError(f"dimension '{name}' not found in any evaluation") + + +def _dimension_names(trials: list[TrialEvals]) -> list[str]: + for trial in trials: + for evals in trial.by_model.values(): + for e in evals: + return [d["name"] for d in e["dimensions"]] + return [] + + +def _normalized_mean(trial: TrialEvals, model: str) -> float: + return statistics.fmean(float(e["normalized_score"]) for e in trial.by_model[model]) + + +def check_repeatability(trials: list[TrialEvals], dims: list[str]) -> CheckResult: + worst = ("", 0.0, 0.0) + for trial in trials: + for model, evals in trial.by_model.items(): + for dim in dims: + limit = STD_LIMIT_FRACTION * _dim_max(trials, dim) + std = _std(_dim_scores(evals, dim)) + if std - limit > worst[1] - worst[2]: + worst = (f"{trial.suffix}/{model}/{dim} std={std:.2f} (limit {limit:.2f})", std, limit) + passed = worst[1] <= worst[2] + return CheckResult("1. repeatability (std <= 8% of scale)", passed, worst[0] or "all within limit") + + +def check_cross_model_gap(trials: list[TrialEvals], dims: list[str]) -> CheckResult: + worst = ("", 0.0, 0.0) + for trial in trials: + models = list(trial.by_model) + for m1, m2 in combinations(models, 2): + for dim in dims: + limit = GAP_LIMIT_FRACTION * _dim_max(trials, dim) + gap = abs( + statistics.fmean(_dim_scores(trial.by_model[m1], dim)) + - statistics.fmean(_dim_scores(trial.by_model[m2], dim)) + ) + if gap - limit > worst[1] - worst[2]: + worst = (f"{trial.suffix}/{dim} {m1} vs {m2} gap={gap:.2f} (limit {limit:.2f})", gap, limit) + passed = worst[1] <= worst[2] + return CheckResult("2. judge-model agreement (gap <= 12% of scale)", passed, worst[0] or "all within limit") + + +def check_human_agreement(trials: list[TrialEvals]) -> list[CheckResult]: + results = [] + models = sorted({m for t in trials for m in t.by_model}) + for model in models: + scored = [t for t in trials if model in t.by_model] + totals = [_normalized_mean(t, model) for t in scored] + reference = [BUCKET_RANK[REFERENCE_BUCKETS[t.suffix]] for t in scored] + rho = _spearman(totals, reference) + by_bucket: dict[str, list[float]] = {} + for t, score in zip(scored, totals, strict=True): + by_bucket.setdefault(REFERENCE_BUCKETS[t.suffix], []).append(score) + separation_ok = all( + min(by_bucket.get(hi, [1.0])) > max(by_bucket.get(lo, [0.0])) + for hi, lo in [("A", "B"), ("B", "C"), ("C", "D")] + if hi in by_bucket and lo in by_bucket + ) + passed = rho >= SPEARMAN_MIN and separation_ok + results.append( + CheckResult( + f"3. human agreement [{model}]", + passed, + f"spearman={rho:.3f} (min {SPEARMAN_MIN}), bucket separation={'OK' if separation_ok else 'VIOLATED'}", + ) + ) + return results + + +def check_disjointness(trials: list[TrialEvals], dims: list[str]) -> list[CheckResult]: + results = [] + models = sorted({m for t in trials for m in t.by_model}) + for model in models: + scored = [t for t in trials if model in t.by_model] + vectors = {dim: [statistics.fmean(_dim_scores(t.by_model[model], dim)) for t in scored] for dim in dims} + worst = ("", 0.0) + for d1, d2 in combinations(dims, 2): + r = abs(_pearson(vectors[d1], vectors[d2])) + if r > worst[1]: + worst = (f"{d1}~{d2} |r|={r:.3f}", r) + passed = worst[1] <= DISJOINTNESS_MAX_ABS_R + results.append( + CheckResult(f"4. dimension disjointness [{model}]", passed, f"{worst[0]} (max {DISJOINTNESS_MAX_ABS_R})") + ) + return results + + +def check_regressions(trials: list[TrialEvals]) -> list[CheckResult]: + by_suffix = {t.suffix: t for t in trials} + results: list[CheckResult] = [] + + def add(name: str, passed: bool, detail: str) -> None: + results.append(CheckResult(f"5. regression: {name}", passed, detail)) + + for model in sorted({m for t in trials for m in t.by_model}): + fj = by_suffix.get("FjYQ3XQ") + if fj and model in fj.by_model: + scores = _dim_scores(fj.by_model[model], "model_fit") + mean, std = statistics.fmean(scores), _std(scores) + add(f"#21 model_fit >= 38 & std <= 4 [{model}]", mean >= 38 and std <= 4, f"mean={mean:.1f} std={std:.2f}") + + zv = by_suffix.get("ZvSsnyg") + if zv and model in zv.by_model: + ranked = sorted( + (t for t in trials if model in t.by_model), key=lambda t: _normalized_mean(t, model), reverse=True + ) + not_first = ranked[0].suffix != "ZvSsnyg" + mf = statistics.fmean(_dim_scores(zv.by_model[model], "model_fit")) + add( + f"#13 dethroned & model_fit <= 36 [{model}]", + not_first and mf <= 36, + f"rank1={ranked[0].suffix} model_fit={mf:.1f}", + ) + + for suffix, pr in (("4njch2w", 18), ("URtZnzf", 20)): + trial = by_suffix.get(suffix) + if trial and model in trial.by_model: + norm = _normalized_mean(trial, model) + restraint = statistics.fmean(_dim_scores(trial.by_model[model], "restraint")) + add( + f"#{pr} capped & restraint floored [{model}]", + norm <= 0.45 and restraint <= 8, + f"normalized={norm:.2f} restraint={restraint:.1f}", + ) + + # Anti-gaming guard: high test_quality demands manual confirmation of a + # composition test through OfferModifiers.ChooseFor — reported as WARN, + # since test presence is not mechanically decidable here. + for trial in trials: + if model in trial.by_model: + tq = statistics.fmean(_dim_scores(trial.by_model[model], "test_quality")) + if tq >= 18: + add( + f"WARN {trial.suffix} test_quality={tq:.1f} [{model}]", + True, + "verify composition test exists before trusting this score", + ) + + aG = by_suffix.get("aGTFmDh") + if aG and len(aG.by_model) >= 2: + means = [statistics.fmean(_dim_scores(evals, "model_fit")) for evals in aG.by_model.values()] + gap = max(means) - min(means) + add("#16 model_fit cross-model gap <= 6", gap <= 6, f"gap={gap:.1f}") + + return results + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("job_dir", type=Path, help="Job dir containing the 13 reference trial dirs (symlinks OK).") + parser.add_argument("--project-dir", "-C", type=Path, default=Path("."), help="Evaluation project dir.") + args = parser.parse_args() + + dimensions_path = args.project_dir / "tasks" / "ddd-weather-discount" / "assessment_dimensions.json" + fingerprint = _dimensions_fingerprint(dimensions_path) + if not fingerprint: + print(f"ERROR: no dimensions file at {dimensions_path}") + return 1 + + trials = collect(args.job_dir, fingerprint) + evaluated = [t for t in trials if t.by_model] + if len(evaluated) < len(REFERENCE_BUCKETS): + missing = sorted(set(REFERENCE_BUCKETS) - {t.suffix for t in evaluated}) + print(f"WARNING: {len(evaluated)}/{len(REFERENCE_BUCKETS)} reference trials have v2 evals; missing: {missing}") + trials = evaluated + if not trials: + print("ERROR: no v2 evaluations found — run `nasde eval` first.") + return 1 + + dims = _dimension_names(trials) + checks: list[CheckResult] = [ + check_repeatability(trials, dims), + check_cross_model_gap(trials, dims), + *check_human_agreement(trials), + *check_disjointness(trials, dims), + *check_regressions(trials), + ] + + width = max(len(c.criterion) for c in checks) + failures = 0 + for c in checks: + status = "PASS" if c.passed else "FAIL" + if not c.passed: + failures += 1 + print(f"[{status}] {c.criterion.ljust(width)} {c.detail}") + if failures == 0: + print("\nACCEPTED: rubric v2 meets all stop conditions") + else: + print(f"\nREJECTED: {failures} criteria failed - diagnose, patch the specific check wording, re-run") + return 0 if failures == 0 else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/examples/ddd-architectural-challenges/cost_quality_plane.py b/examples/ddd-architectural-challenges/cost_quality_plane.py new file mode 100644 index 00000000..aaace3c9 --- /dev/null +++ b/examples/ddd-architectural-challenges/cost_quality_plane.py @@ -0,0 +1,210 @@ +"""Cost x quality plane for the 24-trial Fable/Opus grid (PL + EN publication PNGs). + +X = cache-aware run cost (ADR-014): fresh input at the full rate, cache writes at +the 1h-cache write rate, cache reads at the cached rate, output at the output rate +— what the API would bill for the run. Rates are read live from +src/nasde_toolkit/pricing.toml so a rate update reprices the chart on next render. +Y = trial quality: mean of the 4 rubric-v2.3 evaluations (2x Fable + 2x Opus judge). +""" +from __future__ import annotations + +import json +import tomllib +from pathlib import Path + +import matplotlib.pyplot as plt + +HERE = Path(__file__).resolve().parent +JOBS = HERE / "jobs" +PRICING = HERE.parents[1] / "src" / "nasde_toolkit" / "pricing.toml" +FP = "37ffc5f460d2" + +ARMS: dict[tuple[str, str], list[str]] = { + ("Fable 5", "vanilla"): ["ayg7ckA", "Ss2F6dR", "CKEcWHg", "8ks7yf5"], + ("Fable 5", "hint"): ["7sJ9JK3", "b5jkaWo", "PuuJKHt", "E2soRnZ"], + ("Fable 5", "skill"): ["MNP2RGe", "qpCRH3A", "NZmadqg", "h3a65Ke"], + ("Opus 4.8", "vanilla"): ["GoWvUz6", "cyQyXFc", "JTgey8p", "Lozfurr"], + ("Opus 4.8", "hint"): ["YgcbZjf", "VoktgLb", "sSUYQp4", "a6okSgZ"], + ("Opus 4.8", "skill"): ["ZMX6Xbq", "TsjcHWY", "Bkwiqom", "F5vYATs"], +} +MODEL_ID = {"Fable 5": "claude-fable-5", "Opus 4.8": "claude-opus-4-8"} +CODER_COLOR = {"Fable 5": "#2a78d6", "Opus 4.8": "#1baf7a"} +CONFIG_MARKER = {"vanilla": "o", "hint": "^", "skill": "s"} +INK = "#1a1a19" + +TEXT = { + "pl": { + "title": "Koszt runu a jakość modelu domenowego — ddd-weather-discount", + "xlabel": "koszt runu w USD — stawki API z rozliczeniem prompt cache", + "ylabel": "jakość (średnia 4 ewaluacji, rubryka v2.3)", + "coder": "model kodujący", + "mean": "duży znacznik = średnia ramienia (n=4)", + "outlier": "pojedynczy run za ${cost:.0f}", + # \$ keeps matplotlib from treating $...$ pairs as mathtext + "footnote": ( + "koszt = świeże wejście × stawka + zapisy cache × stawka zapisu (2×) " + "+ odczyty cache × stawka odczytu (0.1×) + wyjście × stawka wyjścia\n" + "stawki API z {as_of}: Fable 5 \\${fi:.0f} / \\${fo:.0f}, " + "Opus 4.8 \\${oi:.0f} / \\${oo:.0f} za mln tokenów" + ), + "out": "cost_quality_plane.png", + }, + "en": { + "title": "Run cost vs domain-model quality — ddd-weather-discount", + "xlabel": "run cost in USD — API rates with prompt caching", + "ylabel": "quality (mean of 4 evaluations, rubric v2.3)", + "coder": "coding model", + "mean": "large marker = arm mean (n=4)", + "outlier": "a single ${cost:.0f} run", + "footnote": ( + "cost = fresh input × input rate + cache writes × write rate (2×) " + "+ cache reads × read rate (0.1×) + output × output rate\n" + "API rates as of {as_of}: Fable 5 \\${fi:.0f} / \\${fo:.0f}, " + "Opus 4.8 \\${oi:.0f} / \\${oo:.0f} per MTok" + ), + "out": "cost_quality_plane_en.png", + }, +} + +# offset-points placement of each arm-mean label, tuned against the rendered layout +LABEL_OFFSET = { + ("Fable 5", "vanilla"): (14, -4, "left"), + ("Fable 5", "hint"): (-14, 2, "right"), + ("Fable 5", "skill"): (0, 12, "center"), + ("Opus 4.8", "vanilla"): (-14, -4, "right"), + ("Opus 4.8", "hint"): (14, -4, "left"), + ("Opus 4.8", "skill"): (14, -4, "left"), +} + + +def load_rates() -> dict: + raw = tomllib.loads(PRICING.read_text())["models"] + return { + "fi": raw["claude-fable-5"]["input_per_1m"], + "fo": raw["claude-fable-5"]["output_per_1m"], + "oi": raw["claude-opus-4-8"]["input_per_1m"], + "oo": raw["claude-opus-4-8"]["output_per_1m"], + "as_of": raw["claude-fable-5"]["as_of"], + "by_model": { + m: ( + raw[m]["input_per_1m"], + raw[m]["output_per_1m"], + raw[m]["cached_input_per_1m"], + raw[m]["cache_write_per_1m"], + ) + for m in ("claude-fable-5", "claude-opus-4-8") + }, + } + + +def collect(rates: dict) -> list[dict]: + rows = [] + for (coder, config), trials in ARMS.items(): + for trial in trials: + (trial_dir,) = JOBS.glob(f"*/ddd-weather-discount__{trial}") + fm = json.loads((trial_dir / "agent" / "trajectory.json").read_text()).get("final_metrics") or {} + extra = fm.get("extra") or {} + inp = fm["total_prompt_tokens"] + out = (fm.get("total_completion_tokens") or 0) + (extra.get("reasoning_output_tokens") or 0) + reads = extra.get("total_cache_read_input_tokens") or fm.get("total_cached_tokens") or 0 + writes = extra.get("total_cache_creation_input_tokens") or 0 + in_rate, out_rate, read_rate, write_rate = rates["by_model"][MODEL_ID[coder]] + scores = [] + for f in sorted(trial_dir.glob("assessment_eval_*.json")): + d = json.loads(f.read_text()) + if d.get("dimensions_fingerprint") == FP and d.get("evaluator_model") in MODEL_ID.values(): + scores.append(d["normalized_score"]) + if len(scores) != 4: + print(f"WARN: {trial} has {len(scores)} v2.3 evals (expected 4)") + rows.append({ + "trial": trial, "coder": coder, "config": config, + "cost": ( + (inp - reads - writes) / 1e6 * in_rate + + writes / 1e6 * write_rate + + reads / 1e6 * read_rate + + out / 1e6 * out_rate + ), + "q": sum(scores) / len(scores), + }) + return rows + + +def plane_plot(rows: list[dict], rates: dict, lang: str) -> None: + t = TEXT[lang] + fig, ax = plt.subplots(figsize=(10.5, 6.4)) + fig.suptitle(t["title"], fontsize=12.5, color=INK, y=0.975) + + ax.grid(color="#e4e3dc", linewidth=0.8, zorder=0) + for sp in ("top", "right"): + ax.spines[sp].set_visible(False) + for sp in ("left", "bottom"): + ax.spines[sp].set_color("#c3c2b7") + + for (coder, config), _trials in ARMS.items(): + sub = [r for r in rows if r["coder"] == coder and r["config"] == config] + col, mark = CODER_COLOR[coder], CONFIG_MARKER[config] + for r in sub: + ax.scatter(r["cost"], r["q"], s=58, marker=mark, color=col, alpha=0.55, + edgecolors="white", linewidths=1.1, zorder=3) + mc = sum(r["cost"] for r in sub) / len(sub) + mq = sum(r["q"] for r in sub) / len(sub) + ax.scatter(mc, mq, s=230, marker=mark, color=col, edgecolors=INK, + linewidths=1.4, zorder=5) + dx, dy, ha = LABEL_OFFSET[(coder, config)] + ax.annotate(config, (mc, mq), textcoords="offset points", xytext=(dx, dy), + ha=ha, fontsize=9.5, color=col, fontweight="bold", zorder=6) + + # per-coder trajectory through the arm means, in config order + for coder in CODER_COLOR: + pts = [] + for config in ("vanilla", "hint", "skill"): + sub = [r for r in rows if r["coder"] == coder and r["config"] == config] + pts.append((sum(r["cost"] for r in sub) / len(sub), sum(r["q"] for r in sub) / len(sub))) + ax.plot([p[0] for p in pts], [p[1] for p in pts], color=CODER_COLOR[coder], + linewidth=1.1, linestyle=(0, (4, 3)), alpha=0.65, zorder=2) + + top = max(rows, key=lambda r: r["cost"]) + ax.annotate(t["outlier"].format(cost=top["cost"]), (top["cost"], top["q"]), + textcoords="offset points", xytext=(-13, -3), ha="right", + fontsize=8.8, color="#666", zorder=6) + + ax.set_xlim(0, 26) + ax.set_ylim(0.55, 0.92) + ax.set_xlabel(t["xlabel"], fontsize=10, color="#444") + ax.set_ylabel(t["ylabel"], fontsize=10, color="#444") + ax.tick_params(labelsize=9, colors="#444") + ax.xaxis.set_major_formatter(lambda v, _p: f"${v:.0f}") + + handles = [ + plt.Line2D([], [], marker="o", linestyle="", markersize=8, + markerfacecolor=CODER_COLOR[c], markeredgecolor="white", + label=f"{t['coder']}: {c}") + for c in CODER_COLOR + ] + handles += [ + plt.Line2D([], [], marker=CONFIG_MARKER[k], linestyle="", markersize=7, + markerfacecolor="#9b9a90", markeredgecolor="white", label=k) + for k in CONFIG_MARKER + ] + handles.append(plt.Line2D([], [], marker="o", linestyle="", markersize=11, + markerfacecolor="#d8d7cd", markeredgecolor=INK, label=t["mean"])) + ax.legend(handles=handles, loc="lower right", fontsize=8.6, frameon=False) + + fig.text(0.5, 0.012, t["footnote"].format(**rates), ha="center", fontsize=7.8, + color="#777", linespacing=1.5) + fig.tight_layout(rect=(0, 0.07, 1, 0.95)) + out = HERE / "assets" / t["out"] + fig.savefig(out, dpi=160, facecolor="white") + print("saved:", out) + + +if __name__ == "__main__": + rates = load_rates() + rows = collect(rates) + for (coder, config), _ in ARMS.items(): + sub = [r for r in rows if r["coder"] == coder and r["config"] == config] + mc = sum(r["cost"] for r in sub) / len(sub) + mq = sum(r["q"] for r in sub) / len(sub) + print(f"{coder:9s} {config:8s} cost ${mc:6.2f} quality {mq:.3f}") + for lang in ("pl", "en"): + plane_plot(rows, rates, lang) diff --git a/examples/ddd-architectural-challenges/tasks/csharp-anemic-to-rich-domain/environment/Dockerfile b/examples/ddd-architectural-challenges/tasks/csharp-anemic-to-rich-domain/environment/Dockerfile index cc164c5f..ce4582ae 100644 --- a/examples/ddd-architectural-challenges/tasks/csharp-anemic-to-rich-domain/environment/Dockerfile +++ b/examples/ddd-architectural-challenges/tasks/csharp-anemic-to-rich-domain/environment/Dockerfile @@ -1,5 +1,12 @@ FROM mcr.microsoft.com/dotnet/sdk:8.0 +# Prevent MSBuild worker-node fleets from accumulating across the agent's repeated +# build/test cycles (default nodeReuse keeps ~1 node per core alive for 15 min; +# three generations of fleets OOM-killed heavy trials at the 6 GiB cgroup limit - +# see kernel memcg OOM reports, 2026-07-10). Nodes now exit with each build. +ENV MSBUILDDISABLENODEREUSE=1 +ENV DOTNET_CLI_USE_MSBUILD_SERVER=0 + RUN apt-get update && apt-get install -y \ git \ curl \ diff --git a/examples/ddd-architectural-challenges/tasks/csharp-movie-rental-anemic/environment/Dockerfile b/examples/ddd-architectural-challenges/tasks/csharp-movie-rental-anemic/environment/Dockerfile index a8d518e5..85dc259f 100644 --- a/examples/ddd-architectural-challenges/tasks/csharp-movie-rental-anemic/environment/Dockerfile +++ b/examples/ddd-architectural-challenges/tasks/csharp-movie-rental-anemic/environment/Dockerfile @@ -1,5 +1,12 @@ FROM mcr.microsoft.com/dotnet/sdk:8.0 +# Prevent MSBuild worker-node fleets from accumulating across the agent's repeated +# build/test cycles (default nodeReuse keeps ~1 node per core alive for 15 min; +# three generations of fleets OOM-killed heavy trials at the 6 GiB cgroup limit - +# see kernel memcg OOM reports, 2026-07-10). Nodes now exit with each build. +ENV MSBUILDDISABLENODEREUSE=1 +ENV DOTNET_CLI_USE_MSBUILD_SERVER=0 + RUN apt-get update && apt-get install -y \ git \ curl \ diff --git a/examples/ddd-architectural-challenges/tasks/ddd-threshold-discount/environment/Dockerfile b/examples/ddd-architectural-challenges/tasks/ddd-threshold-discount/environment/Dockerfile index 0e388232..36b8d074 100644 --- a/examples/ddd-architectural-challenges/tasks/ddd-threshold-discount/environment/Dockerfile +++ b/examples/ddd-architectural-challenges/tasks/ddd-threshold-discount/environment/Dockerfile @@ -1,6 +1,13 @@ # DDD Threshold Discount Challenge - Environment Setup FROM mcr.microsoft.com/dotnet/sdk:8.0 +# Prevent MSBuild worker-node fleets from accumulating across the agent's repeated +# build/test cycles (default nodeReuse keeps ~1 node per core alive for 15 min; +# three generations of fleets OOM-killed heavy trials at the 6 GiB cgroup limit - +# see kernel memcg OOM reports, 2026-07-10). Nodes now exit with each build. +ENV MSBUILDDISABLENODEREUSE=1 +ENV DOTNET_CLI_USE_MSBUILD_SERVER=0 + # Install essential tools RUN apt-get update && apt-get install -y \ git \ diff --git a/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/assessment_criteria.md b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/assessment_criteria.md index 9adf6b3c..fcd03b77 100644 --- a/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/assessment_criteria.md +++ b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/assessment_criteria.md @@ -1,105 +1,268 @@ -# Assessment Criteria: Weather-Based Discount - -Evaluate the AI-generated code across five dimensions. - -## 1. Domain Modeling (0–25) - -Evaluate how well weather-related concepts are modeled using DDD building blocks. Focus on whether the implementation **isolates domain logic** and uses **rich domain language** — does `Precipitation` exist as its own concept, or does the domain just pass around a `decimal`? - -| Score | Criteria | -|-------|----------| -| 0 | No domain types for weather data — raw HTTP responses or primitive types used in domain logic | -| 5 | Some domain types exist but weather data modeled as DTOs or anemic data holders, not proper value objects | -| 10 | Weather data has domain types but they leak infrastructure concerns (JSON annotations, HTTP status codes) | -| 15 | Clean domain types for weather data (e.g. precipitation as value object), but discount logic not modeled as a domain service or policy | -| 20 | Good domain modeling: weather data as value objects, discount logic as domain service/policy, but error handling uses infrastructure exceptions instead of domain-appropriate patterns | -| 25 | Excellent: weather conditions modeled as proper value objects, discount logic encapsulated in domain service/policy, failures handled through domain patterns (Result types, domain exceptions, or safe defaults), domain layer has zero infrastructure dependencies | - -**Key checks:** -- Does a port/interface exist in the domain layer for weather data? -- Does the port use domain types (not `HttpResponseMessage`, `JsonElement`, etc.)? -- Is discount calculation logic in a domain service or policy (not in the adapter)? -- Are weather conditions modeled as value objects with proper semantics? -- **The menu test (Nick Tunes' tactical-ddd, principle #3)**: would a sales/pricing domain expert recognize the names in the domain layer (e.g. `WeatherConditions`, `Precipitation`, `RainyDayDiscount.ApplyOn(price, conditions)`) as their world? Or do you see generic developer jargon (`Manager`, `Handler`, `Service`, `Data`)? -- **Implicit-to-explicit test (principle #6)**: is `Precipitation` named as its own concept (value object with semantics like `IsPresent` / `HasRainOrSnow`), or is it smuggled through method signatures as `decimal precipitation` / `double mm`? -- **Value object liberality (principle #8)**: are weather concepts (`Precipitation`, `Temperature`, `WindSpeed`, eventually `WeatherConditions`) modeled as immutable value objects, or does the domain pass around a JSON blob / DTO of primitives? - -## 2. Encapsulation (0–20) - -Evaluate whether business rules are contained within domain objects. - -| Score | Criteria | -|-------|----------| -| 0 | Weather discount logic scattered across adapter and controller layers | -| 5 | Some logic in domain objects but precipitation thresholds hardcoded in infrastructure | -| 10 | Discount calculation in domain but weather data interpretation still external | -| 15 | Domain service/policy owns all business rules; adapter only translates HTTP to domain types | -| 20 | Perfect encapsulation: domain objects own all rules, weather interpretation, and discount logic. Infrastructure only handles transport. Domain objects cannot be misused by callers | - -**Key checks:** -- Is the "precipitation > X means discount" rule inside the domain layer? -- Can callers bypass domain rules by constructing objects directly? -- Are failure modes (API down) handled with domain-appropriate defaults? -- **Anemic-model test (principle #4)**: does each weather-discount type own a single decision method that *both checks and acts* (e.g. `PrecipitationDiscount.ApplyOn(Money, WeatherConditions) → Money`), or is it split into `IsApplicable(WeatherConditions): bool` + `DiscountPercentage` exposed to the caller? Splitting the decision from the action is "ask, don't tell" — and it leaks the rule into the orchestrator. - -## 3. Architecture Compliance (0–20) - -Evaluate separation of concerns, layer isolation, and adherence to project conventions. - -| Score | Criteria | -|-------|----------| -| 0 | HttpClient used directly in domain logic, no separation | -| 5 | Some separation attempted but domain still references `System.Net.Http` or concrete HTTP types | -| 10 | Interface/port exists for weather data, but domain logic still depends on HTTP concepts | -| 15 | Clean port interface in domain, adapter in infrastructure, proper DI registration, but minor issues (e.g. no timeout, port in wrong namespace) | -| 20 | Excellent: domain port in domain layer returning domain value types, infrastructure adapter in separate project/namespace, timeout configured, all failure modes handled gracefully, proper DI registration, domain has zero reference to infrastructure | - -**Key checks:** -- Is the HTTP adapter in an infrastructure/adapter layer? -- Does the domain project have any reference to `System.Net.Http`? -- Is the adapter registered in DI container? -- Is HttpClient timeout configured? -- Does API failure result in "no discount" (not an exception propagating up)? -- **Isolate-domain test (principle #1, Nick's own check)**: "could a domain expert read this code? Can it be unit tested without mocks or spinning up databases?" — if reading the domain layer forces you to think about HTTP, JSON, or status codes, the isolation has failed. -- **Generic-vs-domain test (principle #5)**: retry logic, HTTP caching, JSON deserialization — would that code exist in a *completely different* business domain? If yes, it must live in infra, not domain. - -## 4. Extensibility (0–15) - -Evaluate how easy it is to add future weather-based discounts (temperature, wind, UV, etc.). - -| Score | Criteria | -|-------|----------| -| 0 | Hardcoded precipitation logic, no extensibility consideration | -| 3 | Single weather discount class with if/else for different conditions | -| 6 | Some abstraction exists but adding a new weather discount requires changes in multiple places | -| 9 | Strategy/Policy pattern used — new weather discounts can be added as new classes, but weather provider interface only supports precipitation | -| 12 | Good design: Strategy pattern, weather provider can return multiple parameters, but minor issues | -| 15 | Excellent: Strategy/Policy pattern, weather provider returns flexible data, new weather discounts are just new classes implementing a common interface, Open-Closed Principle fully respected | - -**Key checks:** -- Is there an abstraction for weather-based discount rules? -- Can a new weather discount be added by only creating a new class? -- Does the weather provider support fetching multiple parameters? -- Would adding UV index discount require changing existing classes? -- **Repository-shape test (principle #9, Nick's own check)**: does `IWeatherProvider` (or whatever the port is called) return a *full domain object* describing weather conditions, or does it return a thin DTO / a single primitive (`decimal precipitation`)? A leaky port that returns primitives forces every caller to re-derive the same domain concepts. -- **Generic-vs-domain extensibility test**: adding `TemperatureDiscount` should be writing a *new domain object* with its own invariants — not editing a shared "WeatherDiscountCalculator" class with `if (kind == Rain) ... else if (kind == Cold) ...`. - -## 5. Test Quality (0–20) - -Evaluate test coverage and proper isolation from external services. - -| Score | Criteria | -|-------|----------| -| 0 | No tests, or tests that call the real API | -| 4 | Basic tests exist but call real HTTP endpoints | -| 8 | Mock/stub for HTTP client exists, but only tests happy path | -| 12 | Good mock isolation, tests happy path and API failure, but missing edge cases | -| 16 | Comprehensive: mocked HTTP client, tests for precipitation > 0 (discount), precipitation = 0 (no discount), API failure (no discount), but minor gaps | -| 20 | Excellent: HTTP client properly mocked, unit tests for domain logic, integration tests for adapter, tests cover: happy path, no precipitation, API failure, malformed response, follows project test conventions | - -**Key checks:** -- Is HttpClient mocked? -- Tests for: precipitation present (discount applies), no precipitation (no discount)? -- Tests for: API failure returns no discount? -- Are unit tests separated from integration tests? -- Is there a test for domain logic independent of infrastructure? +# Assessment Criteria: Weather-Based Discount (v2.3, calibrated 2026-07) + +v2.1 recalibrated three model_fit checks (M1, M4, M5) from the measured Fable subset +(round-2 calibration; method and acceptance criteria in +`../../CALIBRATION_ROUND2_2026-07-07.md`): style is no longer priced as an +invariant (M1), the factory-filtered empty-aggregate shape earns full credit (M4), +and stacking-after is separated from applying-before-the-chain (M5). v2.2 adds the +findings of the live verification: a justified, tested bug fix in pre-existing code +is restraint-neutral (R1), harness-injected files are not agent artifacts (R4), and +construction-time qualification resolution is codified as factory-time (M4). v2.3 +makes M5 direction-neutral: the spec is silent on discount interaction and the agent +cannot ask, so a TESTED assumption (accumulation or exclusivity alike) scores as an +explicit decision — only invisible or model-breaking interaction is penalized. + +This task uses its own dimension set (task-level `assessment_dimensions.json`): +**model_fit (0–50)**, **restraint (0–25)**, **test_quality (0–25)**. The rubric is a +**checklist of decidable questions**, not a quality scale. For every check: find +concrete evidence (file + line), assign a verdict — **FULL** (full points), **PARTIAL** +(half, rounded down), **NONE** (zero) — quote the evidence in your reasoning, and sum +the check points to get the dimension score. + +Hard rules: + +- Each failure mode is scored in **exactly one** check of **exactly one** dimension. + Never deduct twice for the same evidence. +- Do not reward or punish anything this rubric does not ask about. In particular, do + NOT score based on whether a `Precipitation` value object exists — value-object + liberality is not a check in this rubric. +- The "Agent diff" section of your prompt points at the full unified diff of the + agent's work (start state → final workspace). It is the authoritative record of + what the agent changed: answer every change-related check from the diff (Grep it — + removed lines start with `-`), not from impressions of the final state. +- When a "Deterministic pre-check signals" section is also present, treat its signals + as FACTS (which files changed, which lines were removed) — never dispute them. The + VERDICTS remain yours: a file listed outside the touchpoints may still score + R1-neutral when it is a justified, tested bug fix. `suggested_scores` are advisory + anchors, not decisions. +- Verify against the code, not against the agent's comments or naming. + +## Base-model intent (read this before scoring) + +The start state (`itlibrium/DDD-starter-dotnet @ 7950712`) encodes the author's design +intent. Facts you must know to score correctly: + +- `Sources/Sales/Sales.DeepModel/Pricing/OfferModifier.cs`: `OfferModifier.ApplyOn(Offer)` + is marked `[Pure]`. `AggregatedModifier(List)` aggregates modifiers. +- `Sources/Sales/Sales.DeepModel/Pricing/CalculatePrices.cs` (domain service) awaits ALL + async I/O upfront in parallel (price lists, `OfferModifiers.ChooseFor(offerRequest)`, + exchange rates), then applies pure modifiers to the immutable `Offer`/`Quote` tree. + It contains **no conditional logic** — it is a straight pipeline. +- `Sources/Sales/Sales.DeepModel/Pricing/OfferModifiers.cs` is a `[DddFactory]` — the + codebase's **single decision point for pricing policy**: all repository/world-state + access for policy choice happens there; it returns a composed pure modifier chain + `ThreeForTwo.Or(EverySecondBoxForHalfPrice.Or(fallback))`. +- `ExchangeRate` is a `struct : PriceModifier` — **denomination, not policy**. That FX + is fetched in `CalculatePrices` is NOT a precedent for fetching weather there: a + weather discount is an `OfferModifier` (policy) and belongs to the factory. +- Discount interaction is decided **explicitly** wherever the base model implements it: + `ClientLevelDiscounts` overrides (product-specific ELSE base), `IndividualSalesConditions` + takes per-quote `min()` of client vs product paths, `SpecialOffer.Or(...)` names an + exclusive fallback (bodies unimplemented — intent visible in shape only). The base + also ships `AggregatedModifier`, an unused sequential-composition idiom — so the model + points at BOTH choosing and composing; it prescribes visibility, not a direction. + Note: the base has no Pricing tests at all — interaction idioms exist in code shape + only, and the agent has no test precedent to imitate. +- The canonical shape of an offer-wide percentage discount already exists: + `ClientLevelDiscounts`' base-discount path applies a `PercentageDiscount` to every + quote (unless a product-specific discount overrides it). Value objects `Discount` + (percentage|value union), `PercentageDiscount`, `ValueDiscount`, `ProductDiscount` + live in `Sources/Sales/Sales.DeepModel/Pricing/Discounts/`. +- The modifier-application path is hot: `QuoteModifier.ApplyOn` runs once per quote per + modifier (and `IndividualSalesConditions` evaluates two sub-modifiers per quote). + Anything impure there multiplies. + +## 1. Model & Composition Fit (0–50) — `model_fit` + +**M1 (0–10) — World-state closure and purity (hard invariant).** Weather is fetched +exactly **once per price calculation**, awaited before modifier application, and the +returned modifier closes over the resolved **value** — no provider reference escapes +past the factory boundary. `ApplyOn` stays `[Pure]`: no I/O, no async, no +sync-over-async anywhere in the application path. Needing memoization/caching to avoid +repeated calls is itself evidence the state was closed in the wrong place. +- FULL: single upfront fetch, value-closed modifier, pure application. Sequential + instead of parallel awaiting is a style note — mention it, do NOT deduct for it: + this invariant is about closure and purity, not await shape. +- PARTIAL: no fetch is reachable from the application path, but the closure captures + a provider (or other unresolved dependency) it no longer needs. +- NONE: any fetch reachable from `ApplyOn` (lazy, per-quote, `.Result`/`.Wait()`). + +**M2 (0–9) — Policy is assembled in the factory.** The weather discount is composed +inside `OfferModifiers.ChooseFor` (directly, or via a weather factory called from it), +NOT bolted onto `CalculatePrices` as an extra awaited dependency and a second +`.Apply(...)` step. +- FULL: `CalculatePrices` unchanged; weather policy enters the offer exclusively + through the `OfferModifiers` chain. +- PARTIAL: weather resolved via a proper `[DddFactory]` but applied through a second + path in `CalculatePrices`. +- NONE: weather provider (or a rule collection) injected into `CalculatePrices` itself + — a policy registry inside the orchestrator. + +**M3 (0–9) — Canonical discount type reuse.** The weather discount is an offer-level +percentage discount. Canonical modeling: a **generic** `OfferLevelDiscount` (any name; +generality is what counts) in `Pricing/Discounts` that spreads a `PercentageDiscount` +over the quotes of all products exactly the way `ClientLevelDiscounts`' base-discount +path does — or direct reuse of `ClientLevelDiscounts` with an empty product list. +- FULL: generic offer-level discount type in the shared Discounts module, reusing the + `Discount`/`PercentageDiscount` value objects. +- PARTIAL: reuses `PercentageDiscount`/`Discount` values, but the generic "apply to all + quotes" behavior is weather-specific or hidden (private nested class, weather-named + base class) instead of living as a reusable type in `Pricing/Discounts`. +- NONE: a bespoke `WeatherPercentageDiscount`-style type whose only feature is applying + a percentage to quotes — duplication, not consistent with the model. + +**M4 (0–7) — No phantom modifiers, no null-objects.** The factory adds the weather +modifier to the aggregation ONLY when the weather qualifies. A `NoDiscount` / +`NoOfferModifier` null-object class **has no reason to exist** — it comes from +over-engineering. Quotes must never pass through dead modifiers: `Quotes` may later be +used to explain which discounts were applied. +- FULL: conditional composition in the factory — rules/discounts filtered against the + once-fetched conditions BEFORE aggregation; no null-object class anywhere. An + always-present but possibly-empty SHARED aggregate (e.g. an empty + `AggregatedModifier`) does not spoil FULL: the qualification decision already + happened in the factory and quotes never traverse a phantom modifier. Resolution at + CONSTRUCTION time counts as factory time: a modifier whose constructor pre-resolves + the applicable rules/discounts from the fetched conditions (so `ApplyOn` only + consults that pre-resolved state) also qualifies as FULL. +- PARTIAL: no null-object class, but a self-disabling modifier is always present in + the chain — the qualification condition is evaluated at APPLICATION time inside + `ApplyOn`, not pre-resolved at factory/construction time. +- NONE: a named null-object class is introduced and unconditionally aggregated. + +**M5 (0–7) — Explicit interaction with existing discounts.** The specification does +NOT define how the weather discount interacts with existing discounts, and the agent +has no channel to ask — so the DIRECTION of the choice is free: accumulation on top of +the offer, exclusivity, or a min/max competition are all acceptable readings (the base +model itself points both ways: `.Or`/`min()` choose, `AggregatedModifier` composes). +What this check scores is the VISIBILITY of the decision, not its direction. A test +that pins the chosen semantics is the strongest form of visibility — including a test +that introduces a hypothetical second weather rule to demonstrate how multiple weather +discounts combine; do NOT deduct such a test for "lacking spec basis". With a silent +spec and no way to interact, a tested assumption is the correct engineering move. +Application order remains a hard constraint: applying weather BEFORE special offers +feeds discounted quotes into `IndividualSalesConditions`' `min()` comparisons and +breaks the semantics of the model's existing decisions. +This is the ONE check with a verdict above FULL: +- MAX (7): the choice itself is modeled. Since the business has not decided, + the strongest model treats exclusive-vs-accumulate as a CONFIGURATION decision — + both policies expressible through composition idioms (the way `.Or` and + `AggregatedModifier` already embody the two directions). To award MAX, point to + BOTH pieces of evidence: (a) the composition point where a configurer selects the + interaction policy without touching the discount logic (file + line), and (b) + executable proof that the second policy is real — a test exercising it, or a + working alternative combinator. Prose is not movement: an "easily extensible" + comment, an unused strategy parameter, or a TODO stays FULL. +- FULL (6): one direction — whichever — visible and consistent: pinned by a test that + exercises the composition with existing discounts, or modeled as an explicit idiom. + A tested accumulation assumption and a tested exclusivity decision score identically. +- PARTIAL: the interaction with EXISTING discounts is only implicit — e.g. the weather + modifier stacks after the chain silently (even if intra-weather semantics are + tested), or the decision is stated in a comment/assumptions note but never tested. +- NONE: weather applied BEFORE the existing chain (feeding discounted quotes into + `IndividualSalesConditions`' `min()` comparisons), or self-contradictory semantics + (code and tests disagree about the interaction). + +**M6 (0–4) — Failure is not a measurement.** API failure must be distinguishable from +a measured zero. Encoding failure as `Clear()`, `Unknown => new(0)`, or +`Unavailable ≡ With(0)` makes the domain assert weather it never observed — and breaks +the announced future rules (a temperature rule `< 0°C` would fire exactly when the API +is down). +- FULL: explicit unknown/unavailable state (or the factory simply omits the modifier on + failure, so no fake reading enters the model); ordering never breaks on API failure. +- PARTIAL: failure degrades gracefully to "no discount" but is representable only as a + fake zero reading. +- NONE: failure state is value-equal to a real measurement and flows into rules. + +**M7 (0–4) — Proportionate, future-ready seam.** Adding the announced future discounts +(temperature, wind, cloud, humidity, UV) is a one-class change; the extension contract +accepts the `Discount` union (percentage AND value), not a hardcoded `Percentage`. The +seam is as small as the problem: registries for n=1, static rule tables, +fetch-plan/parameter-negotiation machinery, or options patterns are speculative +generality; latent runtime traps for future rules (e.g., an adapter switch throwing +`ArgumentOutOfRangeException` for unmapped parameters) cap this at PARTIAL. + +## 2. Boundaries & Restraint (0–25) — `restraint` + +This dimension scores respect for the author's pre-existing code. The allowed +touchpoints for this feature are: `Sources/Sales/Sales.DeepModel/Pricing/OfferModifiers.cs` +(new dependency + composition), `Sources/Monolith.Startup/DI/Modules/Sales.cs` +(registration), and `.csproj` files (package references). Everything else pre-existing +should be byte-identical. **Answer R1–R4 from the agent diff**: the diffstat lists +every touched file; Grep the diff file for removed lines (`^-`). Checks R1–R4 may also +arrive pre-computed in the pre-check section — stay consistent with it. + +**R1 (0–8) — Pre-existing files untouched beyond the touchpoints; no fabrication.** +From the diffstat: which pre-existing files were modified, beyond the touchpoints? +Confirm suspicious ones by Reading (e.g. `Sales.Adapters/Integrations/RiskManagementInMemoryCalls.cs` +must still throw `NotImplementedException`; `Pricing/CalculatePrices.cs` must keep the +three-way tuple await with no weather dependency). +- FULL: only touchpoints modified (additive package refs OK). +- PARTIAL: 1–2 avoidable modifications (e.g., `CalculatePrices` gained a weather + dependency; an extra class added to a shared pre-existing file). +- NONE: 3+ pre-existing files modified, or ANY behavior fabrication (e.g., + `NotImplementedException` replaced by `Money.Of(decimal.MaxValue, ...)` — silently + granting unlimited credit in an unrelated integration). +- **Justified bug fix is R1-neutral.** An off-touchpoint modification does NOT count + against R1 when all three hold: the defect is demonstrable in the base code, the + fix is minimal, and it is covered by a test (the task explicitly allows refactoring + existing code). Example: base `Discount.Value(Money)` passes `isPercentage: true`, + silently turning value discounts into a default percentage — fixing that flag with + a covering test earns no deduction. Distinguish sharply from behavior FABRICATION + (inventing business behavior to make things run), which remains NONE. + +**R2 (0–5) — Author's annotations preserved.** Grep the agent diff for removed +annotation lines (pattern: lines starting with `-` containing `[Ddd` or +`[ExternalSystemIntegration`). Stripping the author's annotations (e.g., to appease +convention-based DI scanning) is an unjustified rewrite of deliberate model markup: +NONE if annotations removed from 2+ files, PARTIAL if 1. + +**R3 (0–4) — No signature rewrites of pre-existing types.** E.g., +`AggregatedModifier(List)` changed to `IEnumerable`: the +author may have used `List` for a reason; catching author intent is a trait of good +domain modeling. PARTIAL for one cosmetic change, NONE for more or for semantic +changes. + +**R4 (0–2) — No agent artifacts committed.** Scratch scripts (`*.csx`), notes, debug +dumps and other agent-created junk. IGNORE files injected by the evaluation harness +itself: root `CLAUDE.md` / `AGENTS.md` and the `.claude/` / `.codex/` directories are +mounted into the workspace by the runner (variant `sandbox_files`), not created by +the agent, and must not be penalized. FULL: no agent-created artifacts; PARTIAL: one; +NONE: multiple. + +**R5 (0–4) — Modularization mirrors the surrounding design.** A separate weather +module that **exposes offer modifiers analogously to `SpecialOffers`**, wired through +the factory — the "resemble the existing code" rule. The hexagonal split mirrors the +Forex precedent: port interface in the deep model (`[ExternalSystemIntegration]`), +adapter in `Sales.Adapters/Integrations/Weather/`, typed HttpClient in the DI module. +- FULL: dedicated weather module exposing modifiers + clean port/adapter mirror. +- PARTIAL: port/adapter correct but weather types dumped into shared + `Pricing/Discounts` (coupling it to weather), or the module hides its modifiers. +- NONE: no module boundary, or domain references HTTP types directly. + +**R6 (0–2) — The domain speaks domain language.** No wire vocabulary inside the deep +model: Open-Meteo query-parameter strings (`"precipitation"`, `"temperature_2m"`) as +value-object contents, HTTP/JSON types, or status codes in `Sales.DeepModel` score +NONE. Resilience (try/catch of transport errors) belongs at the adapter boundary — a +bare `catch` inside a deep-model factory is a PARTIAL-level leak. + +## 3. Test Quality (0–25) — `test_quality` + +**T1 (0–9) — Composition is tested through `OfferModifiers.ChooseFor`.** At least one +test exercises the factory end-to-end and verifies how the weather discount interacts +with the existing chain (a non-identity base modifier: special offer active or client/ +product discounts present). A test suite that only exercises the weather module in +isolation scores NONE here — the riskiest behavior is the interaction. + +**T2 (0–5) — The single-fetch guarantee is asserted.** A test proves the weather +provider is called at most once per price calculation (e.g., counting stub). + +**T3 (0–5) — Failure and boundary are tested honestly.** API-failure path asserted in +a way that can actually fail (beware vacuous assertions: if `Unknown` is value-equal to +a zero reading, `result.Should().Be(Unknown)` passes even when parsing succeeded), and +`precipitation == 0` is covered as a case distinct from "data unavailable". + +**T4 (0–4) — Adapter isolation.** HttpClient mocked/stubbed (no real API calls), URL +pinned, malformed payload and non-success status covered. + +**T5 (0–2) — House conventions.** Tests follow the repo's BDD style +(`Bdd.Scenario`), unit vs integration split matches the existing projects. diff --git a/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/assessment_dimensions.json b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/assessment_dimensions.json new file mode 100644 index 00000000..4acc7ddf --- /dev/null +++ b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/assessment_dimensions.json @@ -0,0 +1,23 @@ +{ + "rubric_version": "2.3 (2026-07-08): v2.2 + M5 direction-neutral - the spec is silent on discount interaction and the agent cannot ask, so a tested assumption (accumulation or exclusivity alike) scores as an explicit decision (6/7); the 7th point is reserved for reifying the choice itself as a configurable composition decision (both policies expressible, demonstrated in tests); base-intent note corrected (AggregatedModifier is a base-provided sequential-composition idiom; base has no Pricing tests); bump changes the fingerprint so each rubric version's evaluations form their own groups", + "dimensions": [ + { + "name": "model_fit", + "title": "Model & Composition Fit", + "max_score": 50, + "description": "The new concept lands in the right place in the existing model and composes with it explicitly: world-state closed once in the factory and captured as a value ([Pure] application path), canonical discount types reused instead of duplicated, no phantom/null-object modifiers, interaction with existing discounts decided visibly, failure distinct from measurement, and a seam proportionate to the announced future requirements." + }, + { + "name": "restraint", + "title": "Boundaries & Restraint", + "max_score": 25, + "description": "Respect for the pre-existing codebase and its author's intent: only the designated touchpoints modified, the author's annotations and type signatures preserved, no behavior fabrication outside the feature, no agent artifacts committed, modularization resembling the surrounding design, and domain language free of wire vocabulary." + }, + { + "name": "test_quality", + "title": "Test Quality", + "max_score": 25, + "description": "Tests cover the riskiest behavior, not just the new island: composition with the existing modifier chain, the single-fetch guarantee, honest (non-vacuous) failure-path and boundary assertions, adapter isolation from the real API, and the repo's testing conventions." + } + ] +} diff --git a/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/environment/Dockerfile b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/environment/Dockerfile index d686f28b..4b6550b2 100644 --- a/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/environment/Dockerfile +++ b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/environment/Dockerfile @@ -1,6 +1,13 @@ # DDD Weather Discount Challenge - Environment Setup FROM mcr.microsoft.com/dotnet/sdk:8.0 +# Prevent MSBuild worker-node fleets from accumulating across the agent's repeated +# build/test cycles (default nodeReuse keeps ~1 node per core alive for 15 min; +# three generations of fleets OOM-killed heavy trials at the 6 GiB cgroup limit - +# see kernel memcg OOM reports, 2026-07-10). Nodes now exit with each build. +ENV MSBUILDDISABLENODEREUSE=1 +ENV DOTNET_CLI_USE_MSBUILD_SERVER=0 + # Install essential tools RUN apt-get update && apt-get install -y \ git \ diff --git a/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/ground_truth_decisions.json b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/ground_truth_decisions.json new file mode 100644 index 00000000..ddffacf5 --- /dev/null +++ b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/ground_truth_decisions.json @@ -0,0 +1,54 @@ +{ + "source": "Human calibration review of 13 trials (sink PRs #9-#21, NoesisVision/nasde-calibration, 2026-06), reviewer: repo owner. These are the reference design decisions for this task; the rubric checks (DM/EN/AC/EX/TQ) operationalize them.", + "decisions": [ + { + "id": "world-state-closure", + "decision": "The state of the world (weather) is resolved exactly once per price calculation, inside the OfferModifiers [DddFactory], and captured as a value in the returned modifier. ApplyOn stays [Pure].", + "rationale": "The base code encodes this intent three times: [Pure] on OfferModifier.ApplyOn, all-async-upfront parallel awaits in CalculatePrices, and [DddFactory] on OfferModifiers. If the state is not closed before application, the external service can be called multiple times while scanning the offer tree; needing memoization is evidence of closing in the wrong place.", + "canonical_example": "Weather awaited in ChooseFor in parallel with the discount-repository calls (house tuple-await style), captured by value.", + "anti_pattern": "Weather provider (or a rule collection) injected into CalculatePrices; a second .Apply(weatherModifier) pipeline step beside the factory chain. Note: ExchangeRate is a PriceModifier (denomination), not an OfferModifier (policy) - FX in CalculatePrices is not a precedent." + }, + { + "id": "no-phantom-discounts", + "decision": "The weather modifier is added to the aggregation ONLY when the weather qualifies. No NoDiscount/NoOfferModifier null-object class - such a discount simply has no reason to exist.", + "rationale": "Null-objects here come from over-engineering. Quotes may later be used to explain which discounts were applied (discount-id tracking is a plausible future requirement); dead modifiers in the chain poison that.", + "canonical_example": "Rules filtered in the factory against the once-fetched conditions; nothing appended when no rule fires.", + "anti_pattern": "A NoDiscount class (worst: placed in the weather module instead of shared Discounts); a self-disabling modifier whose ApplyOn checks the condition; an always-appended empty aggregate." + }, + { + "id": "canonical-discount-type", + "decision": "The weather discount is by nature an offer-level percentage discount: either a NEW generic OfferLevelDiscount type in Pricing/Discounts that spreads a PercentageDiscount over all quotes exactly like ClientLevelDiscounts' base-discount path, or simply ClientLevelDiscounts with an empty product list.", + "rationale": "Creating a WeatherPercentageDiscount whose only feature is applying a percentageDiscount to quotes is incorrect modeling - it invites duplication and is not consistent with the model.", + "canonical_example": "A generic OfferWideDiscount(Discount) in Pricing/Discounts reusing the Discount union.", + "anti_pattern": "Bespoke weather-named percentage type; the generic apply-to-all-quotes behavior hidden as a private nested class in the weather module." + }, + { + "id": "explicit-interaction", + "decision": "The interaction between the weather discount and existing discounts must be an explicit, visible decision - never a silent assumption. The DIRECTION of the decision is free: the spec is silent and the agent cannot ask, so a tested accumulation assumption and a tested exclusivity decision are equally acceptable.", + "rationale": "Where the base model implements interaction it decides explicitly (ClientLevelDiscounts overrides, IndividualSalesConditions takes per-quote min(), SpecialOffer.Or names an exclusive fallback), but it also ships AggregatedModifier - an unused sequential-composition idiom - so the base points at both choosing and composing. It prescribes visibility, not a direction. The base has no Pricing tests, so the agent has no test precedent to imitate; writing the assumption down as a test is the correct engineering move. Applying weather BEFORE special offers remains the worst variant: discounted quotes flow into the min() comparisons.", + "canonical_example": "Interaction expressed with the model's idioms (Or / min / explicit aggregation) and covered by a composition test through OfferModifiers.ChooseFor - regardless of whether the tested decision is accumulation or exclusivity. The exemplary ceiling: the interaction policy itself reified as a configuration decision - exclusivity and accumulation both expressible and selectable at composition time (the way .Or and AggregatedModifier already embody the two directions), demonstrated both ways in tests. That is the suppleness this model is built to show off.", + "anti_pattern": "Silent stacking with no test or note covering the interaction with existing discounts; weather applied before the existing chain; code and tests that disagree about the interaction semantics." + }, + { + "id": "failure-is-not-a-measurement", + "decision": "API failure must be distinguishable from a measured zero and must degrade to 'no discount' without breaking ordering.", + "rationale": "Unknown/Unavailable encoded as new(0) or Clear() makes the domain assert weather it never observed, and breaks the announced future rules: a temperature<0 rule would fire exactly when the API is down.", + "canonical_example": "Explicit unavailable state, or the factory omits the weather modifier on failure.", + "anti_pattern": "WeatherConditions.Unknown == WeatherConditions.Of(0) under value equality; adapter returning Clear() from a catch block." + }, + { + "id": "resemble-existing-code", + "decision": "A separate weather module exposes offer modifiers analogously to SpecialOffers; port in the deep model mirrors the Forex/ExchangeRateProvider precedent, adapter in Sales.Adapters/Integrations/Weather.", + "rationale": "The rule is to resolve similar problems the way the surrounding code already resolves them. Weather types dumped into shared Pricing/Discounts couple that module to weather; Pricing/Discounts is for high-abstraction types reacting to offer configuration.", + "canonical_example": "Pricing/Weather (or Weather/) module exposing modifiers, wired into the factory chain.", + "anti_pattern": "Weather rules and modifiers placed inside Pricing/Discounts; wire vocabulary (temperature_2m) inside deep-model value objects." + }, + { + "id": "restraint", + "decision": "Pre-existing files are modified only at the touchpoints (OfferModifiers.cs, DI module, csproj). The author's annotations ([DddDomainService] etc.) and signatures (AggregatedModifier(List)) are preserved; no behavior fabrication outside the feature; no agent artifacts committed.", + "rationale": "The author may have used List (and every annotation) for a reason - catching author intent is a trait of good domain modeling. Rewriting RiskManagementInMemoryCalls from NotImplementedException to Money.Of(decimal.MaxValue) silently grants unlimited credit in an unrelated bounded context.", + "canonical_example": "Additive-only wiring in the two touchpoint files.", + "anti_pattern": "[DddDomainService] stripped from six base files to appease DI scanning; List->IEnumerable signature rewrite; CLAUDE.md/AGENTS.md/decompile.csx committed." + } + ] +} diff --git a/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/precheck.sh b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/precheck.sh new file mode 100755 index 00000000..71bcc854 --- /dev/null +++ b/examples/ddd-architectural-challenges/tasks/ddd-weather-discount/precheck.sh @@ -0,0 +1,109 @@ +#!/bin/bash +# Deterministic restraint pre-check for ddd-weather-discount (rubric v2, checks R1-R4). +# +# The LLM judge has Read/Glob/Grep only - git-level facts are computed here and +# injected into the judge prompt by the evaluator (see evaluator._run_precheck). +# +# Modes: +# precheck.sh +# Evaluator contract: trial workspace with a git repo where HEAD is the +# start state and the agent's work is uncommitted (git archive HEAD is the +# base snapshot; the agent diff ships as changes.patch). +# precheck.sh [git-dir] +# Calibration-sink mode: two published branches (nasde calibrate publish). +# +# Output: one JSON object on stdout. Exit 0 on success (evaluator treats a +# non-zero exit or non-JSON stdout as "no precheck"). +set -u + +if [ "$#" -ge 2 ]; then + MODE="refs" + BASE="$1"; TRIAL="$2"; GITDIR="${3:-.}" + DIFF=(git -C "$GITDIR" diff "$BASE...$TRIAL") + DIFF_NAMES=(git -C "$GITDIR" diff --name-only "$BASE...$TRIAL") + ADDED=(git -C "$GITDIR" diff --diff-filter=A --name-only "$BASE...$TRIAL") + MODIFIED=(git -C "$GITDIR" diff --diff-filter=MD --name-only "$BASE...$TRIAL") +else + MODE="workspace" + GITDIR="${1:?workspace dir required}" + if ! git -C "$GITDIR" rev-parse HEAD >/dev/null 2>&1; then + printf '{"skipped": "no git repository in workspace", "mode": "workspace"}\n' + exit 0 + fi + DIFF=(git -C "$GITDIR" diff HEAD) + DIFF_NAMES=(git -C "$GITDIR" diff --name-only HEAD) + ADDED=(git -C "$GITDIR" ls-files --others --exclude-standard) + MODIFIED=(git -C "$GITDIR" diff --diff-filter=MD --name-only HEAD) +fi + +EXCL=':(exclude).calibration' + +# --- signals ---------------------------------------------------------------- +modified=$("${MODIFIED[@]}" -- . "$EXCL" 2>/dev/null | sort) + +# R1: pre-existing files modified outside the allowed touchpoints. +whitelist='^(Sources/Sales/Sales\.DeepModel/Pricing/OfferModifiers\.cs|Sources/Monolith\.Startup/DI/Modules/Sales\.cs|.*\.csproj)$' +offlist=$(printf '%s\n' "$modified" | grep -Ev "$whitelist" | grep -v '^$' || true) +offlist_count=$(printf '%s' "$offlist" | grep -c . || true) + +# R2: author's DDD annotations removed from pre-existing files. +ann_removed=$("${DIFF[@]}" -- . "$EXCL" 2>/dev/null | grep -cE '^-\s*\[(Ddd|ExternalSystemIntegration)' || true) + +# R3: type-declaration lines of pre-existing files changed (heuristic). +if [ -n "$modified" ]; then + sig_changed=$("${DIFF[@]}" -- $(printf '%s\n' "$modified" | tr '\n' ' ') 2>/dev/null \ + | grep -cE '^-\s*(public|internal)\s+(sealed\s+|abstract\s+|readonly\s+)*(class|interface|record|struct|delegate)\b' || true) +else + sig_changed=0 +fi + +# R1 (fabrication part): fail-loud stubs silently replaced. +fabrication=$("${DIFF[@]}" -- . "$EXCL" 2>/dev/null | grep -cE '^-\s*.*NotImplementedException' || true) + +# R4: agent-created artifacts. Root CLAUDE.md/AGENTS.md and .claude/ /.codex/ are +# injected by the harness (variant sandbox_files) - not the agent's doing - excluded. +artifacts=$("${ADDED[@]}" 2>/dev/null \ + | grep -Ev '^\.calibration/|^CLAUDE\.md$|^AGENTS\.md$|^\.claude/|^\.codex/' \ + | grep -E '\.csx$|(^|/)(CLAUDE\.md|AGENTS\.md)' || true) +artifacts_count=$(printf '%s' "$artifacts" | grep -c . || true) + +# --- suggested scores (rubric v2: R1 0-8, R2 0-5, R3 0-4, R4 0-2) ----------- +if [ "$fabrication" -gt 0 ] || [ "$offlist_count" -ge 3 ]; then r1=0 +elif [ "$offlist_count" -ge 1 ]; then r1=4 +else r1=8; fi + +if [ "$ann_removed" -ge 2 ]; then r2=0 +elif [ "$ann_removed" -eq 1 ]; then r2=2 +else r2=5; fi + +if [ "$sig_changed" -ge 3 ]; then r3=0 +elif [ "$sig_changed" -ge 1 ]; then r3=2 +else r3=4; fi + +if [ "$artifacts_count" -ge 2 ]; then r4=0 +elif [ "$artifacts_count" -eq 1 ]; then r4=1 +else r4=2; fi + +# Hard fail = bucket-D disqualification: cap the trial's normalized score. +cap_line="" +if [ "$fabrication" -gt 0 ] || [ "$ann_removed" -ge 2 ]; then + cap_line='"normalized_score_cap": 0.45,' +fi + +jlist() { printf '%s' "$1" | grep . | sed 's/"/\\"/g; s/^/"/; s/$/"/' | paste -sd, - ; } + +cat < str: + if points <= 0: + return "NONE" + return "FULL" if points >= MAX[cid] else "PARTIAL" + + +def parse_eval(d: dict) -> tuple[dict[str, str], list[str]]: + verdicts: dict[str, str] = {} + audit: list[str] = [] + for dim in d["dimensions"]: + name, text = dim["name"], dim["reasoning"] + local: dict[str, str] = {} + for cid, v, pts in P_V_POINTS.findall(text): + # M5 FULL is 6 of 7 by definition, so 6 does not contradict the word + genuine_full = MAX[cid] - (1 if cid == "M5" else 0) + if v == "FULL" and int(pts) < genuine_full: + local.setdefault(cid, classify(int(pts), cid)) + for ids, v in P_VERDICT.findall(text): + v = v.upper() + for cid in re.split(r"\s*[/,]\s*", ids): + local.setdefault(cid, "FULL" if v == "MAX" else v) + for v, cid in P_V_FIRST.findall(text): + local.setdefault(cid, "FULL" if v == "MAX" else v) + for v, tail in P_V_LIST.findall(text): + for cid in re.findall(rf"\b{ID}\b", tail): + local.setdefault(cid, "FULL" if v == "MAX" else v) + for cid, pts, _mx in P_POINTS.findall(text): + local.setdefault(cid, classify(int(pts), cid)) + for lost, cid in P_LOST.findall(text): + local.setdefault(cid, classify(MAX[cid] - int(lost), cid)) + checks = DIM_CHECKS[name] + for cid in checks: + local.setdefault(cid, "FULL") # judges enumerate deductions only + # M5 FULL is 6/7 by rubric definition (7 = MAX), so an all-FULL model_fit + # legitimately lands on 49/50. + floor = dim["max_score"] - (1 if name == "model_fit" else 0) + full_score = dim["score"] >= floor + has_deduction = any(local[c] != "FULL" for c in checks) + if full_score == has_deduction: + audit.append(f"{name} {dim['score']}/{dim['max_score']} prose/score mismatch") + verdicts.update({c: local[c] for c in checks}) + return verdicts, audit + + +def collect() -> tuple[dict[tuple[str, str], list[dict]], list[str]]: + per_arm: dict[tuple[str, str], list[dict]] = {a: [] for a in ARMS} + flags: list[str] = [] + for arm, trials in ARMS.items(): + for t in trials: + (td,) = JOBS.glob(f"*/ddd-weather-discount__{t}") + for f in sorted(td.glob("assessment_eval_*.json")): + d = json.loads(f.read_text()) + if d.get("dimensions_fingerprint") != FP: + continue + verdicts, audit = parse_eval(d) + per_arm[arm].append(verdicts) + flags.extend(f"{t} {f.name} ({d['evaluator_model']}): {a}" for a in audit) + return per_arm, flags + + +VAL = {"FULL": 1.0, "PARTIAL": 0.5, "NONE": 0.0} +# one sequential ramp per check group; anchors CVD-checked as a set (worst dE 36) +GROUP_RAMP = { + "M": ("#f7f3fd", "#53309e"), + "R": ("#eff8fa", "#0b7285"), + "T": ("#fdf4ea", "#b45309"), +} +CMAPS = {g: LinearSegmentedColormap.from_list(f"verdict_{g}", ramp) + for g, ramp in GROUP_RAMP.items()} + + +def heatmap(per_arm: dict, lang: str) -> None: + t = TEXT[lang] + grid = [[sum(VAL[evals[c]] for evals in per_arm[arm]) / len(per_arm[arm]) + for arm in ARM_ORDER] for c in CHECKS] + rgba = [[CMAPS[c[0]](v) for v in row] for c, row in zip(CHECKS, grid)] + + fig, ax = plt.subplots(figsize=(9.6, 8.8)) + fig.suptitle(t["title"], fontsize=11.5, color=INK, y=0.985) + ax.imshow(rgba, aspect="auto") + + for yi, row in enumerate(grid): + for xi, v in enumerate(row): + ax.text(xi, yi, f"{round(v * 100)}%", ha="center", va="center", + fontsize=8.6, color="white" if v > 0.62 else "#3a3a38", + fontweight="bold" if v <= 0.5 else "normal") + + ax.set_xticks(range(len(ARM_ORDER))) + ax.set_xticklabels([config for _c, config in ARM_ORDER], fontsize=10, color="#3a3a38") + ax.xaxis.set_ticks_position("top") + for xi, (coder, _config) in enumerate(ARM_ORDER): + ax.annotate(coder, (xi, 1.058), xycoords=("data", "axes fraction"), + ha="center", fontsize=9, color=CODER_COLOR[coder], fontweight="bold") + ax.set_yticks(range(len(CHECKS))) + ax.set_yticklabels([LABELS[lang][c] for c in CHECKS], fontsize=8.8, color="#3a3a38") + ax.tick_params(length=0) + for sp in ax.spines.values(): + sp.set_visible(False) + + # white gridlines between cells; heavier breaks + side captions between M/R/T groups + for xi in range(1, len(ARM_ORDER)): + lw = 3.4 if xi == 3 else 1.6 + ax.axvline(xi - 0.5, color="white", linewidth=lw) + for yi in range(1, len(CHECKS)): + ax.axhline(yi - 0.5, color="white", linewidth=1.6) + for group, start, size in (("M", 0, 7), ("R", 7, 6), ("T", 13, 5)): + if start: + ax.axhline(start - 0.5, color="white", linewidth=4.2) + ax.annotate(t["groups"][group], (1.01, 1 - (start + size / 2) / len(CHECKS)), + xycoords="axes fraction", ha="left", va="center", fontsize=8.6, + color=GROUP_RAMP[group][1], rotation=270) + + fig.text(0.5, 0.015, t["scale"], ha="center", fontsize=8.2, color="#777") + fig.tight_layout(rect=(0, 0.035, 0.97, 0.905)) + out = HERE / "assets" / t["out"] + fig.savefig(out, dpi=160, facecolor="white") + print("saved:", out) + + +if __name__ == "__main__": + per_arm, flags = collect() + n = sum(len(v) for v in per_arm.values()) + print(f"evals parsed: {n} (expected 96)") + for fl in flags: + print("FLAG", fl) + for lang in ("pl", "en"): + heatmap(per_arm, lang) diff --git a/src/nasde_toolkit/calibration_publisher.py b/src/nasde_toolkit/calibration_publisher.py index a0e3be92..0ad40110 100644 --- a/src/nasde_toolkit/calibration_publisher.py +++ b/src/nasde_toolkit/calibration_publisher.py @@ -29,6 +29,7 @@ _aggregate_evaluations, _load_json, _load_raw_evaluations, + resolve_dimensions_path, ) from nasde_toolkit.git_platform_backends import create_git_backend from nasde_toolkit.git_platform_backends.git_ops import ( @@ -224,11 +225,11 @@ def _add_task_context_files(files: dict[str, str], trial_dir: Path, project_root task_dir = _resolve_task_dir(trial_dir, project_root) if task_dir is None: return - for name in ("instruction.md", "assessment_criteria.md"): + for name in ("instruction.md", "assessment_criteria.md", "ground_truth_decisions.json"): source = task_dir / name if source.exists(): files[name] = source.read_text(encoding="utf-8") - dimensions = task_dir.parent.parent / "assessment_dimensions.json" + dimensions = resolve_dimensions_path(task_dir) if dimensions.exists(): files["assessment_dimensions.json"] = dimensions.read_text(encoding="utf-8") diff --git a/src/nasde_toolkit/cli.py b/src/nasde_toolkit/cli.py index f511993e..6dcb9430 100644 --- a/src/nasde_toolkit/cli.py +++ b/src/nasde_toolkit/cli.py @@ -82,6 +82,17 @@ def _override_eval_repetitions(config: ProjectConfig, eval_repetitions: int | No config.evaluation.eval_repetitions = eval_repetitions +def _override_eval_judge(config: ProjectConfig, model: str | None, backend: str | None) -> None: + """Apply per-run judge overrides (judge-model comparison matrices).""" + if backend is not None: + if backend not in ("claude", "codex"): + console.print(f"[red]ERROR: --eval-backend must be 'claude' or 'codex', got '{backend}'.[/red]") + raise typer.Exit(1) + config.evaluation.backend = backend + if model is not None: + config.evaluation.model = model + + @app.callback(invoke_without_command=True) def main( ctx: typer.Context, @@ -369,6 +380,16 @@ def eval_command( "--eval-repetitions", help="Judge evaluations per trial (default: from nasde.toml [evaluation], fallback 3).", ), + eval_model: str | None = typer.Option( + None, + "--eval-model", + help="Override the judge model from nasde.toml [evaluation] for this run (judge-model comparison matrices).", + ), + eval_backend: str | None = typer.Option( + None, + "--eval-backend", + help="Override the judge backend for this run: claude | codex.", + ), project_dir: Path = typer.Option( Path("."), "--project-dir", @@ -382,6 +403,7 @@ def eval_command( config = load_project_config(project_dir.resolve()) _override_eval_repetitions(config, eval_repetitions) + _override_eval_judge(config, eval_model, eval_backend) from nasde_toolkit.banner import print_banner diff --git a/src/nasde_toolkit/evaluator.py b/src/nasde_toolkit/evaluator.py index 4af8f044..0f95b12b 100644 --- a/src/nasde_toolkit/evaluator.py +++ b/src/nasde_toolkit/evaluator.py @@ -10,8 +10,11 @@ import asyncio import hashlib import json +import os import re import statistics +import subprocess +import sys from dataclasses import asdict, dataclass, field from datetime import UTC, datetime from pathlib import Path @@ -20,8 +23,10 @@ from nasde_toolkit.config import EvaluationConfig from nasde_toolkit.evaluator_backends import create_backend +from nasde_toolkit.evaluator_backends.protocol import AGENT_DIFF_FILENAME from nasde_toolkit.pricing import ModelPrice, load_pricing_layered from nasde_toolkit.token_metrics import build_trial_economics +from nasde_toolkit.workspace_diff import capture_diffstat, capture_patch console = Console() @@ -57,6 +62,7 @@ class EvaluationResult: harbor_reward: float = 0.0 duration_sec: float = 0.0 dimensions_fingerprint: str = "" + precheck: dict | None = None @dataclass @@ -281,7 +287,7 @@ async def evaluate_trial( harbor_reward = (result_json.get("verifier_result") or {}).get("rewards", {}).get("reward", 0.0) duration_sec = _compute_duration_sec(result_json) - dimensions_path = task_dir.parent.parent / "assessment_dimensions.json" + dimensions_path = resolve_dimensions_path(task_dir) expected_dimensions = _load_expected_dimensions(dimensions_path) criteria_path = task_dir / "assessment_criteria.md" @@ -296,6 +302,9 @@ async def evaluate_trial( ground_truth_path = task_dir / "ground_truth_decisions.json" ground_truth = ground_truth_path.read_text() if ground_truth_path.exists() else "" + precheck_raw = _run_precheck(task_dir, workspace_path) + agent_diff_path, agent_diffstat = _materialize_agent_diff(workspace_path, trial_dir) + trajectory_path = _resolve_trajectory_path(trial_dir, eval_config) artifacts_dir = str(workspace_path) if eval_config.skills_dir else None prompt = _build_evaluator_prompt( @@ -305,6 +314,9 @@ async def evaluate_trial( ground_truth, artifacts_dir, trajectory_path, + precheck_raw, + agent_diff_path, + agent_diffstat, ) console.print(f" Task: {task_name}") console.print(f" Workspace: {workspace_path}") @@ -332,6 +344,7 @@ async def evaluate_trial( evaluation.evaluator_model = eval_config.model evaluation.timestamp = datetime.now(UTC).isoformat() evaluation.dimensions_fingerprint = _dimensions_fingerprint(dimensions_path) + _apply_precheck(evaluation, precheck_raw) total_max = sum(dim.max_score for dim in evaluation.dimensions) console.print(f" Score: {evaluation.total_score}/{total_max} ({evaluation.normalized_score:.2f})") @@ -365,6 +378,117 @@ def _compute_duration_sec(result: dict) -> float: return (end_dt - start_dt).total_seconds() +def resolve_dimensions_path(task_dir: Path) -> Path: + """Resolve the dimensions file for a task: task-level wins over challenge-level. + + A task whose rubric was calibrated to its own dimension set ships + ``assessment_dimensions.json`` next to its ``assessment_criteria.md``; every + other task keeps using the shared challenge-level file two levels up. + Different dimension files yield different fingerprints, so per-task and + challenge-level evaluations are never mixed in one summary group. + """ + task_level = task_dir / "assessment_dimensions.json" + if task_level.exists(): + return task_level + return task_dir.parent.parent / "assessment_dimensions.json" + + +def _materialize_agent_diff(workspace_path: Path, trial_dir: Path) -> tuple[str | None, str]: + """Write the agent's full diff next to the trial results and return (path, diffstat). + + This is the judge's universal "what did the agent actually change" input — + the same reference point a human reviewer gets. The judge cannot run git + (Read/Glob/Grep only) and cannot see removals or out-of-feature edits in a + final-state snapshot, so the evaluator computes the diff on the host once + and hands it over as a file the judge can Read (paginated) and Grep. The + inline diffstat orients the judge; the file is the evidence source. + Returns (None, "") when the workspace has no git repo or nothing changed. + """ + patch = capture_patch(workspace_path) + if not patch.strip(): + return None, "" + diff_path = trial_dir / AGENT_DIFF_FILENAME + diff_path.write_text(patch, encoding="utf-8") + return str(diff_path), capture_diffstat(workspace_path) + + +PRECHECK_TIMEOUT_SEC = 60 + + +def _run_precheck(task_dir: Path, workspace_path: Path) -> str: + """Run the task's optional deterministic pre-check and return its JSON output. + + A task may ship an executable ``precheck.sh`` next to its rubric. It receives + the trial workspace path as ``$1`` (POSIX-style, so scripts may embed it in + JSON on any platform) and must print a single JSON object to + stdout: signals computed mechanically (typically git-level restraint checks) + that the LLM judge cannot compute itself, since it only has Read/Glob/Grep. + The output is injected verbatim into the judge prompt and recorded in the + evaluation result. Any failure degrades to "no precheck" with a warning — + it must never sink the evaluation. + """ + script = task_dir / "precheck.sh" + if not script.exists(): + return "" + try: + proc = subprocess.run( + [_bash_executable(), str(script), workspace_path.as_posix()], + capture_output=True, + text=True, + timeout=PRECHECK_TIMEOUT_SEC, + ) + except (OSError, subprocess.TimeoutExpired) as err: + console.print(f" [yellow]precheck.sh did not run: {err}[/yellow]") + return "" + if proc.returncode != 0: + console.print(f" [yellow]precheck.sh exited {proc.returncode}: {proc.stderr.strip()[:200]}[/yellow]") + return "" + output = proc.stdout.strip() + try: + json.loads(output) + except json.JSONDecodeError as err: + console.print(f" [yellow]precheck.sh output is not valid JSON: {err}[/yellow]") + return "" + return output + + +def _bash_executable() -> str: + """Locate bash, preferring Git Bash on Windows over the System32 WSL stub.""" + if sys.platform != "win32": + return "bash" + program_files = os.environ.get("PROGRAMFILES", r"C:\Program Files") + git_bash = Path(program_files) / "Git" / "bin" / "bash.exe" + if git_bash.exists(): + return str(git_bash) + return "bash" + + +def _apply_precheck(evaluation: EvaluationResult, precheck_raw: str) -> None: + """Attach precheck signals to the result and enforce its score cap, if any. + + The precheck JSON may carry ``normalized_score_cap`` (float 0..1) — a hard + ceiling for trials that mechanically disqualify themselves (e.g. rewriting + unrelated pre-existing code). The cap and its application are recorded in + the result, so a capped score is always explainable. + """ + if not precheck_raw: + return + precheck = json.loads(precheck_raw) + evaluation.precheck = precheck + cap = precheck.get("normalized_score_cap") + if cap is None: + return + if not isinstance(cap, int | float) or not 0.0 <= float(cap) <= 1.0: + console.print(f" [yellow]precheck normalized_score_cap ignored (not a 0..1 number): {cap!r}[/yellow]") + return + if evaluation.normalized_score > float(cap): + precheck["normalized_score_cap_applied"] = { + "uncapped_normalized_score": evaluation.normalized_score, + "capped_to": float(cap), + } + evaluation.normalized_score = float(cap) + + def _load_expected_dimensions(dimensions_path: Path) -> list[dict] | None: if not dimensions_path.exists(): return None @@ -445,6 +569,9 @@ def _build_evaluator_prompt( ground_truth: str = "", artifacts_dir: str | None = None, trajectory_path: str | None = None, + precheck: str = "", + agent_diff_path: str | None = None, + agent_diffstat: str = "", ) -> str: """Build the evaluation prompt with optional dimension constraints and ground truth.""" scoring_guidance = _format_scoring_guidance(expected_dimensions) @@ -452,7 +579,10 @@ def _build_evaluator_prompt( output_schema = _format_output_schema(expected_dimensions) dimension_count_rule = _format_dimension_count_rule(expected_dimensions) ground_truth_section = _format_ground_truth_section(ground_truth) + agent_diff_section = _format_agent_diff_section(agent_diff_path, agent_diffstat) + precheck_section = _format_precheck_section(precheck) trajectory_section = _format_trajectory_section(trajectory_path) + how_to_evaluate = _format_how_to_evaluate(has_agent_diff=agent_diff_path is not None) location_hint = ( f"Analyze the artifacts in `{artifacts_dir}`." @@ -481,13 +611,7 @@ def _build_evaluator_prompt( {criteria} -{ground_truth_section}{trajectory_section}## How to evaluate - -1. Use `Glob` to discover all output files in the workspace. -2. Use `Read` to examine the content of each output file. -3. Use `Grep` to search for specific patterns or keywords. -4. For each dimension, find concrete evidence before assigning a score. - +{agent_diff_section}{ground_truth_section}{precheck_section}{trajectory_section}{how_to_evaluate} ## Output format After your analysis, output a single JSON block with your evaluation. @@ -572,6 +696,80 @@ def _format_ground_truth_section(ground_truth: str) -> str: """ +def _format_how_to_evaluate(has_agent_diff: bool) -> str: + """Evaluation procedure; diff-first whenever the agent diff is available. + + The diff step is part of the base procedure — independent of whatever the + task's assessment criteria say — so every rubric benefits from the + what-actually-changed reference point, not only rubrics that mention it. + """ + if has_agent_diff: + return """## How to evaluate + +1. Start from the agent diff (see "Agent diff" above): review the change summary, + then Read/Grep the diff file — establish WHAT the agent changed, removed and + added before judging how well it did so. +2. Use `Glob` to discover all output files in the workspace. +3. Use `Read` to examine the changed files in their full workspace context — the + diff shows the change, the file shows how it fits its surroundings. +4. Use `Grep` to search for specific patterns or keywords. +5. For each dimension, find concrete evidence before assigning a score. Evidence + about what the agent changed comes from the diff; evidence about how it fits + comes from the workspace. +""" + return """## How to evaluate + +1. Use `Glob` to discover all output files in the workspace. +2. Use `Read` to examine the content of each output file. +3. Use `Grep` to search for specific patterns or keywords. +4. For each dimension, find concrete evidence before assigning a score. +""" + + +def _format_agent_diff_section(agent_diff_path: str | None, agent_diffstat: str) -> str: + if not agent_diff_path: + return "" + return f""" +## Agent diff — the authoritative record of what changed + +The workspace shows only the FINAL state; you cannot see what the agent +modified, removed or added by reading files alone. The complete unified diff of +the agent's work (start state → final workspace, including new files) is at: + +`{agent_diff_path}` + +Read it (use offset/limit pagination if large) and Grep it — e.g. lines +starting with `-` show removed code, diff headers show every touched file. +Any check about the agent's changes (modified pre-existing files, removed +annotations, changed signatures, added artifacts) MUST be answered from this +diff, not from impressions of the final state. + +Change summary (diffstat): + +``` +{agent_diffstat} +``` +""" + + +def _format_precheck_section(precheck: str) -> str: + if not precheck: + return "" + return f""" +## Deterministic pre-check signals + +The following signals were computed mechanically by tooling (git-level analysis +you cannot perform yourself). Treat them as established facts. Where a rubric +check overlaps with a signal below, your verdict MUST be consistent with the +signal — use Read on the flagged files to write the evidence for your reasoning, +not to re-litigate the fact. + + +{precheck} + +""" + + def _format_trajectory_section(trajectory_path: str | None) -> str: if not trajectory_path: return "" diff --git a/src/nasde_toolkit/evaluator_backends/claude_subprocess.py b/src/nasde_toolkit/evaluator_backends/claude_subprocess.py index 050a41a2..fc384ee2 100644 --- a/src/nasde_toolkit/evaluator_backends/claude_subprocess.py +++ b/src/nasde_toolkit/evaluator_backends/claude_subprocess.py @@ -12,6 +12,7 @@ from rich.console import Console from nasde_toolkit.config import EvaluationConfig +from nasde_toolkit.evaluator_backends.protocol import AGENT_DIFF_FILENAME console = Console() @@ -61,11 +62,17 @@ def validate_cli_installed(self) -> None: raise SystemExit(1) def validate_auth(self) -> None: + # No env credentials is NOT fatal: this backend deliberately omits + # --bare so the claude CLI can read OAuth tokens from the keychain + # (subscription accounts). If the CLI truly has no auth, the + # evaluation subprocess fails loudly on its first call. has_api_key = bool(os.environ.get("ANTHROPIC_API_KEY")) has_oauth = bool(os.environ.get("CLAUDE_CODE_OAUTH_TOKEN")) if not has_api_key and not has_oauth: - console.print("[red]ERROR: Set ANTHROPIC_API_KEY or CLAUDE_CODE_OAUTH_TOKEN[/red]") - raise SystemExit(1) + console.print( + "[dim]No ANTHROPIC_API_KEY / CLAUDE_CODE_OAUTH_TOKEN in the environment — " + "relying on the claude CLI's keychain OAuth.[/dim]" + ) def _build_command_with_skills( self, @@ -109,7 +116,10 @@ def _build_command( allowed_tools = eval_config.allowed_tools or ["Read", "Glob", "Grep"] cmd.extend(["--allowedTools", ",".join(allowed_tools)]) - if eval_config.include_trajectory and trial_dir: + needs_trial_dir = trial_dir is not None and ( + eval_config.include_trajectory or (trial_dir / AGENT_DIFF_FILENAME).exists() + ) + if needs_trial_dir: cmd.extend(["--add-dir", str(trial_dir)]) if eval_config.mcp_config: diff --git a/src/nasde_toolkit/evaluator_backends/protocol.py b/src/nasde_toolkit/evaluator_backends/protocol.py index 497b34d6..919d2970 100644 --- a/src/nasde_toolkit/evaluator_backends/protocol.py +++ b/src/nasde_toolkit/evaluator_backends/protocol.py @@ -7,6 +7,11 @@ from nasde_toolkit.config import EvaluationConfig +# The agent's full diff (start state -> final workspace), materialized by the +# evaluator into the trial directory so the judge can Read/Grep it. Backends +# grant read access to the trial dir when this file is present. +AGENT_DIFF_FILENAME = "agent_changes.diff" + @runtime_checkable class EvaluatorBackend(Protocol): diff --git a/src/nasde_toolkit/pricing.py b/src/nasde_toolkit/pricing.py index ee5b1d47..8f05792d 100644 --- a/src/nasde_toolkit/pricing.py +++ b/src/nasde_toolkit/pricing.py @@ -3,9 +3,9 @@ Loads per-model rates from a bundled ``pricing.toml`` and computes USD cost from token volumes. The catalog is overridable by convention via layered files — ``/pricing.toml`` > ``~/.nasde/pricing.toml`` > bundled, merged per-model -(see ``load_pricing_layered`` and ADR-013). Cost is the full catalog rate applied -to the full prompt-token volume (cache included, no discount) — see ``pricing.toml`` -and ADR-011. +(see ``load_pricing_layered`` and ADR-013). Cost is cache-aware — fresh input at +the full rate, cache writes and reads at their own rates — see ``pricing.toml`` +and ADR-014 (supersedes ADR-011's full-rate formula). """ from __future__ import annotations @@ -34,6 +34,7 @@ class ModelPrice: input_per_1m: float output_per_1m: float cached_input_per_1m: float | None = None + cache_write_per_1m: float | None = None as_of: str = "" source: str = "" @@ -53,8 +54,18 @@ def compute_cost_usd( output_tokens: int, model: str, pricing: dict[str, ModelPrice], + cache_read_tokens: int = 0, + cache_write_tokens: int = 0, ) -> float | None: - """Full-rate USD cost for the given token volumes, or None if model is unpriced.""" + """Cache-aware USD cost for the given token volumes, or None if model is unpriced. + + Fresh input (prompt volume minus cache reads and writes) is billed at the full + input rate, cache writes at ``cache_write_per_1m``, cache reads at + ``cached_input_per_1m``, output at the output rate (ADR-014). A missing cache + rate falls back to the full input rate — conservative, never a silent discount. + With zero cache volumes this reduces to the full-rate formula, which is also + how the cache-free ceiling can be derived offline from ``token_usage``. + """ price = pricing.get(model) if price is None: console.print( @@ -62,7 +73,15 @@ def compute_cost_usd( f"Add it to pricing.toml to enable cost metrics.[/yellow]" ) return None - return input_tokens / 1_000_000 * price.input_per_1m + output_tokens / 1_000_000 * price.output_per_1m + read_rate = price.cached_input_per_1m if price.cached_input_per_1m is not None else price.input_per_1m + write_rate = price.cache_write_per_1m if price.cache_write_per_1m is not None else price.input_per_1m + fresh_tokens = max(input_tokens - cache_read_tokens - cache_write_tokens, 0) + return ( + fresh_tokens / 1_000_000 * price.input_per_1m + + cache_write_tokens / 1_000_000 * write_rate + + cache_read_tokens / 1_000_000 * read_rate + + output_tokens / 1_000_000 * price.output_per_1m + ) def effective_pricing_with_source(project_dir: Path | None = None) -> dict[str, tuple[ModelPrice, str]]: @@ -207,6 +226,7 @@ def _model_price_from_dict(entry: dict) -> ModelPrice: input_per_1m=entry["input_per_1m"], output_per_1m=entry["output_per_1m"], cached_input_per_1m=entry.get("cached_input_per_1m"), + cache_write_per_1m=entry.get("cache_write_per_1m"), as_of=entry.get("as_of", ""), source=entry.get("source", ""), ) diff --git a/src/nasde_toolkit/pricing.toml b/src/nasde_toolkit/pricing.toml index 05158829..ce7b04d3 100644 --- a/src/nasde_toolkit/pricing.toml +++ b/src/nasde_toolkit/pricing.toml @@ -1,14 +1,22 @@ -# Model pricing for token-cost metrics (per 1M tokens, full catalog rate). +# Model pricing for token-cost metrics (per 1M tokens). # -# Cost is computed "as if every run were the first": the full prompt-token volume -# (cache included) is billed at the full input rate, with NO cache discount. This -# makes cost deterministic and independent of run order / cache TTL — the prompt -# token count is fixed for a task, while the cache hit rate is not. +# Cost is CACHE-AWARE (ADR-014, supersedes ADR-011's full-rate formula): +# cost = fresh_input * input_per_1m +# + cache_writes * cache_write_per_1m +# + cache_reads * cached_input_per_1m +# + output * output_per_1m +# where fresh_input = total_prompt_tokens - cache_reads - cache_writes. +# A missing cache rate falls back to input_per_1m (conservative, never a silent +# discount). The cache-free ceiling is NOT stored — it is derivable from the +# recorded token_usage as input * input_per_1m + output * output_per_1m. # -# cached_input_per_1m is recorded for reference only; it is NOT used in the cost -# formula. reasoning output tokens are folded into output (see token_metrics.py). +# cache_write_per_1m for Anthropic models is the 1-hour cache write rate +# (2x base input) — the mode Claude Code uses; verified against Harbor's own +# per-step cost accounting on the 2026-07 grid. OpenAI bills no write premium, +# so gpt entries omit the field (writes fall back to the base input rate). # -# Verified 2026-06-08 — CONFIRM before publishing any cost figures. +# Verified 2026-07-13 (Claude models re-checked against the official pricing page +# before publishing) — CONFIRM before publishing any cost figures. [models."gpt-5.5"] input_per_1m = 5.0 @@ -26,14 +34,24 @@ source = "https://developers.openai.com/api/docs/pricing" [models."claude-opus-4-8"] input_per_1m = 5.0 -output_per_1m = 15.0 +output_per_1m = 25.0 cached_input_per_1m = 0.50 -as_of = "2026-06-08" +cache_write_per_1m = 10.0 +as_of = "2026-07-13" +source = "https://platform.claude.com/docs/en/about-claude/pricing" + +[models."claude-fable-5"] +input_per_1m = 10.0 +output_per_1m = 50.0 +cached_input_per_1m = 1.0 +cache_write_per_1m = 20.0 +as_of = "2026-07-13" source = "https://platform.claude.com/docs/en/about-claude/pricing" [models."claude-sonnet-4-6"] input_per_1m = 3.0 output_per_1m = 15.0 cached_input_per_1m = 0.30 +cache_write_per_1m = 6.0 as_of = "2026-06-08" source = "https://platform.claude.com/docs/en/about-claude/pricing" diff --git a/src/nasde_toolkit/results_exporter.py b/src/nasde_toolkit/results_exporter.py index 9b4dd039..f87c6c50 100644 --- a/src/nasde_toolkit/results_exporter.py +++ b/src/nasde_toolkit/results_exporter.py @@ -10,7 +10,6 @@ import json import shutil -import subprocess from dataclasses import dataclass, field from pathlib import Path @@ -26,6 +25,7 @@ ) from nasde_toolkit.pricing import ModelPrice, load_pricing_layered from nasde_toolkit.token_metrics import build_trial_economics +from nasde_toolkit.workspace_diff import capture_patch console = Console() @@ -291,55 +291,9 @@ def _write_patch(trial_dir: Path, out_dir: Path) -> None: (out_dir / "changes.patch").write_text(patch) -def _capture_patch(workspace: Path) -> str: - if not (workspace / ".git").exists(): - console.print(f" [yellow]no git workspace in {workspace.parent.parent.name}; empty patch[/yellow]") - return "" - tracked = _run_git(workspace, ["diff", "HEAD"]) - untracked = _capture_untracked(workspace) - return tracked + untracked - - -def _capture_untracked(workspace: Path) -> str: - listing = _run_git_bytes(workspace, ["ls-files", "--others", "--exclude-standard", "-z"]) - chunks: list[str] = [] - for raw_path in listing.split(b"\x00"): - if raw_path: - relative_path = raw_path.decode("utf-8", "surrogateescape") - chunks.append(_diff_untracked_file(workspace, relative_path)) - return "".join(chunks) - - -def _diff_untracked_file(workspace: Path, relative_path: str) -> str: - return _run_git( - workspace, - ["diff", "--no-index", "--", "/dev/null", relative_path], - accept_diff_exit=True, - ) - - -def _run_git(workspace: Path, args: list[str], accept_diff_exit: bool = False) -> str: - completed = subprocess.run( - ["git", "-C", str(workspace), *args], - capture_output=True, - text=True, - check=False, - ) - if completed.returncode != 0 and not (accept_diff_exit and completed.returncode == 1): - raise RuntimeError(f"git {' '.join(args)} failed in {workspace}: {completed.stderr.strip()}") - return completed.stdout - - -def _run_git_bytes(workspace: Path, args: list[str]) -> bytes: - completed = subprocess.run( - ["git", "-C", str(workspace), *args], - capture_output=True, - check=False, - ) - if completed.returncode != 0: - stderr = completed.stderr.decode("utf-8", "replace").strip() - raise RuntimeError(f"git {' '.join(args)} failed in {workspace}: {stderr}") - return completed.stdout +# Patch capture moved to workspace_diff (shared with the evaluator's agent-diff +# input); re-exported here under the historical name for existing importers. +_capture_patch = capture_patch def _print_summary(summary: ExportSummary, dest: Path) -> None: diff --git a/src/nasde_toolkit/scaffold/__init__.py b/src/nasde_toolkit/scaffold/__init__.py index 2cde76c8..e205670b 100644 --- a/src/nasde_toolkit/scaffold/__init__.py +++ b/src/nasde_toolkit/scaffold/__init__.py @@ -120,6 +120,8 @@ # [models."claude-sonnet-4-6"] # input_per_1m = 2.5 # output_per_1m = 11.0 +# cached_input_per_1m = 0.25 # optional: cache-read rate; omitted -> reads billed at input_per_1m +# cache_write_per_1m = 5.0 # optional: cache-write rate; omitted -> writes billed at input_per_1m # as_of = "2026-01-01" # optional: when you confirmed this rate # source = "your contract / rate card" # optional: where it came from """ diff --git a/src/nasde_toolkit/token_metrics.py b/src/nasde_toolkit/token_metrics.py index 214fd7b9..487e5e49 100644 --- a/src/nasde_toolkit/token_metrics.py +++ b/src/nasde_toolkit/token_metrics.py @@ -5,11 +5,15 @@ Token volumes come from the trajectory's top-level ``final_metrics`` (written by Harbor for both Claude and Codex agents): - input = total_prompt_tokens (full, cache included) - output = total_completion_tokens + extra.reasoning_output_tokens - total = input + output - -Cost applies the full catalog rate to those volumes (no cache discount) — see ADR-011. + input = total_prompt_tokens (full, cache included) + output = total_completion_tokens + extra.reasoning_output_tokens + cached = total_cached_tokens (cache reads) + cache_write = extra.total_cache_creation_input_tokens + total = input + output + +Cost is cache-aware (ADR-014): fresh input at the full rate, cache writes and +reads at their catalog rates. The cache-free ceiling is derivable from the +recorded token volumes and is not stored. """ from __future__ import annotations @@ -30,6 +34,7 @@ class TokenUsage: completion_tokens: int reasoning_tokens: int cached_tokens: int + cache_write_tokens: int total_tokens: int @@ -39,9 +44,11 @@ def extract_token_usage(trajectory: dict) -> TokenUsage | None: prompt_tokens = final_metrics.get("total_prompt_tokens") if prompt_tokens is None: return None + extra = final_metrics.get("extra") or {} completion_tokens = final_metrics.get("total_completion_tokens", 0) or 0 - reasoning_tokens = (final_metrics.get("extra") or {}).get("reasoning_output_tokens", 0) or 0 + reasoning_tokens = extra.get("reasoning_output_tokens", 0) or 0 cached_tokens = final_metrics.get("total_cached_tokens", 0) or 0 + cache_write_tokens = extra.get("total_cache_creation_input_tokens", 0) or 0 output_tokens = completion_tokens + reasoning_tokens return TokenUsage( input_tokens=prompt_tokens, @@ -49,6 +56,7 @@ def extract_token_usage(trajectory: dict) -> TokenUsage | None: completion_tokens=completion_tokens, reasoning_tokens=reasoning_tokens, cached_tokens=cached_tokens, + cache_write_tokens=cache_write_tokens, total_tokens=prompt_tokens + output_tokens, ) @@ -87,7 +95,14 @@ def build_trial_economics( usage = extract_token_usage(trajectory) if trajectory is not None else None if usage is None: return _empty_economics(model) - cost_usd = compute_cost_usd(usage.input_tokens, usage.output_tokens, model, pricing) + cost_usd = compute_cost_usd( + usage.input_tokens, + usage.output_tokens, + model, + pricing, + cache_read_tokens=usage.cached_tokens, + cache_write_tokens=usage.cache_write_tokens, + ) return { "model_name": model, "token_usage": asdict(usage), diff --git a/src/nasde_toolkit/workspace_diff.py b/src/nasde_toolkit/workspace_diff.py new file mode 100644 index 00000000..d6f14b9a --- /dev/null +++ b/src/nasde_toolkit/workspace_diff.py @@ -0,0 +1,79 @@ +"""Capture the agent's work from a trial workspace as a unified diff. + +Workspace convention (shared with ``nasde calibrate publish``): HEAD is the +task's start state and the agent's work is uncommitted — tracked changes show +up in ``git diff HEAD``, brand-new files as untracked paths. The functions here +turn that state into reviewable artifacts: the full patch (consumed by the +results exporter, the calibration publisher and the evaluator's agent-diff +prompt input) and a compact diffstat. +""" + +from __future__ import annotations + +import subprocess +from pathlib import Path + +from rich.console import Console + +console = Console() + + +def capture_patch(workspace: Path) -> str: + """Full unified diff of the agent's work: tracked changes + untracked files.""" + if not (workspace / ".git").exists(): + console.print(f" [yellow]no git workspace in {workspace.parent.parent.name}; empty patch[/yellow]") + return "" + tracked = _run_git(workspace, ["diff", "HEAD"]) + untracked = _capture_untracked(workspace) + return tracked + untracked + + +def capture_diffstat(workspace: Path) -> str: + """Compact change summary: ``git diff HEAD --stat`` plus untracked paths.""" + if not (workspace / ".git").exists(): + return "" + stat = _run_git(workspace, ["diff", "HEAD", "--stat"]).rstrip() + lines = [stat] if stat else [] + lines.extend(f" {path} (new file)" for path in _untracked_paths(workspace)) + return "\n".join(lines) + + +def _untracked_paths(workspace: Path) -> list[str]: + listing = _run_git_bytes(workspace, ["ls-files", "--others", "--exclude-standard", "-z"]) + return [raw.decode("utf-8", "surrogateescape") for raw in listing.split(b"\x00") if raw] + + +def _capture_untracked(workspace: Path) -> str: + return "".join(_diff_untracked_file(workspace, path) for path in _untracked_paths(workspace)) + + +def _diff_untracked_file(workspace: Path, relative_path: str) -> str: + return _run_git( + workspace, + ["diff", "--no-index", "--", "/dev/null", relative_path], + accept_diff_exit=True, + ) + + +def _run_git(workspace: Path, args: list[str], accept_diff_exit: bool = False) -> str: + completed = subprocess.run( + ["git", "-C", str(workspace), *args], + capture_output=True, + text=True, + check=False, + ) + if completed.returncode != 0 and not (accept_diff_exit and completed.returncode == 1): + raise RuntimeError(f"git {' '.join(args)} failed in {workspace}: {completed.stderr.strip()}") + return completed.stdout + + +def _run_git_bytes(workspace: Path, args: list[str]) -> bytes: + completed = subprocess.run( + ["git", "-C", str(workspace), *args], + capture_output=True, + check=False, + ) + if completed.returncode != 0: + stderr = completed.stderr.decode("utf-8", "replace").strip() + raise RuntimeError(f"git {' '.join(args)} failed in {workspace}: {stderr}") + return completed.stdout diff --git a/tests/test_calibration_publisher.py b/tests/test_calibration_publisher.py index cb823ff3..7437f377 100644 --- a/tests/test_calibration_publisher.py +++ b/tests/test_calibration_publisher.py @@ -128,6 +128,17 @@ def test_add_task_context_files_includes_instruction_criteria_dimensions(tmp_pat assert files["assessment_dimensions.json"] == '{"dimensions": []}' +def test_add_task_context_files_prefers_task_level_dimensions_and_bundles_ground_truth(tmp_path: Path) -> None: + project_root, trial_dir = _make_trial_with_task(tmp_path) + task_dir = project_root / "tasks" / "my-task" + (task_dir / "assessment_dimensions.json").write_text('{"dimensions": [{"name": "model_fit"}]}', encoding="utf-8") + (task_dir / "ground_truth_decisions.json").write_text('{"decisions": []}', encoding="utf-8") + files: dict[str, str] = {} + _add_task_context_files(files, trial_dir, project_root) + assert "model_fit" in files["assessment_dimensions.json"] + assert files["ground_truth_decisions.json"] == '{"decisions": []}' + + def test_summarize_trial_returns_empty_summary_without_evals(tmp_path: Path) -> None: trial = tmp_path / "trial__x" trial.mkdir() diff --git a/tests/test_cli.py b/tests/test_cli.py index 79f3024f..7475b263 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -248,3 +248,34 @@ def test_pricing_show_source_column(tmp_path: Path) -> None: assert "Layer" in result.output assert "project" in result.output assert "bundled" in result.output + + +def test_override_eval_judge_sets_model_and_backend() -> None: + from nasde_toolkit.cli import _override_eval_judge + from nasde_toolkit.config import EvaluationConfig, ProjectConfig + + config = ProjectConfig(name="test", evaluation=EvaluationConfig()) + _override_eval_judge(config, model="claude-fable-5", backend="codex") + assert config.evaluation.model == "claude-fable-5" + assert config.evaluation.backend == "codex" + + +def test_override_eval_judge_none_keeps_config_defaults() -> None: + from nasde_toolkit.cli import _override_eval_judge + from nasde_toolkit.config import EvaluationConfig, ProjectConfig + + config = ProjectConfig(name="test", evaluation=EvaluationConfig(model="m0", backend="claude")) + _override_eval_judge(config, model=None, backend=None) + assert config.evaluation.model == "m0" + assert config.evaluation.backend == "claude" + + +def test_override_eval_judge_rejects_unknown_backend() -> None: + import typer + + from nasde_toolkit.cli import _override_eval_judge + from nasde_toolkit.config import EvaluationConfig, ProjectConfig + + config = ProjectConfig(name="test", evaluation=EvaluationConfig()) + with pytest.raises(typer.Exit): + _override_eval_judge(config, model=None, backend="gemini") diff --git a/tests/test_evaluator.py b/tests/test_evaluator.py index 7f7ac2ea..958a0625 100644 --- a/tests/test_evaluator.py +++ b/tests/test_evaluator.py @@ -4,6 +4,7 @@ import asyncio import json +import sys from dataclasses import asdict from pathlib import Path from unittest.mock import patch @@ -16,6 +17,7 @@ DimensionScore, EvaluationResult, _aggregate_evaluations, + _apply_precheck, _build_evaluator_prompt, _build_opik_scores, _dimensions_fingerprint, @@ -23,11 +25,14 @@ _evaluate_and_record_trial, _evaluation_from_dict, _load_expected_dimensions, + _materialize_agent_diff, _next_eval_index, _parse_evaluation_response, _resolve_trajectory_path, + _run_precheck, _write_assessment_summary, _write_evaluation_result, + resolve_dimensions_path, ) from nasde_toolkit.pricing import load_pricing, load_pricing_layered @@ -275,8 +280,8 @@ def test_assessment_summary_includes_economics(tmp_path: Path) -> None: assert summary is not None assert summary.model_name == "claude-sonnet-4-6" assert summary.token_usage["total_tokens"] == 1_060_000 - # sonnet $3/$15: 1M*3 + 0.06M*15 = 3.9 - assert summary.cost_usd == pytest.approx(3.9) + # sonnet $3/$15/$0.30 cached (ADR-014): fresh 0.2M*3 + cached 0.8M*0.30 + 0.06M*15 = 1.74 + assert summary.cost_usd == pytest.approx(1.74) assert summary.pricing_as_of == "2026-06-08" assert not hasattr(summary, "cost_efficiency") # removed: arbitrary zero → use Pareto front assert not hasattr(summary, "token_efficiency") @@ -641,3 +646,212 @@ def test_prompt_lists_per_dimension_ranges_not_shared_25() -> None: assert "`small`: 0–3 points" in prompt assert "`medium`: 0–20 points" in prompt assert "`large`: 0–100 points" in prompt + + +# --------------------------------------------------------------------------- +# Per-task dimensions & deterministic precheck +# --------------------------------------------------------------------------- + + +def _make_task_dir(tmp_path: Path) -> Path: + task_dir = tmp_path / "evals" / "tasks" / "my-task" + task_dir.mkdir(parents=True) + return task_dir + + +def test_resolve_dimensions_path_prefers_task_level(tmp_path: Path) -> None: + task_dir = _make_task_dir(tmp_path) + (task_dir / "assessment_dimensions.json").write_text('{"dimensions": []}', encoding="utf-8") + (task_dir.parent.parent / "assessment_dimensions.json").write_text('{"dimensions": []}', encoding="utf-8") + assert resolve_dimensions_path(task_dir) == task_dir / "assessment_dimensions.json" + + +def test_resolve_dimensions_path_falls_back_to_challenge_level(tmp_path: Path) -> None: + task_dir = _make_task_dir(tmp_path) + challenge_level = task_dir.parent.parent / "assessment_dimensions.json" + challenge_level.write_text('{"dimensions": []}', encoding="utf-8") + assert resolve_dimensions_path(task_dir) == challenge_level + + +@pytest.mark.skipif( + sys.platform == "win32", + reason="needs a working POSIX bash; on Windows PATH `bash` may be the WSL stub, " + "and _run_precheck then degrades to no-precheck by design", +) +def test_run_precheck_returns_json_and_passes_workspace_arg(tmp_path: Path) -> None: + task_dir = _make_task_dir(tmp_path) + (task_dir / "precheck.sh").write_text('#!/bin/bash\nprintf \'{"workspace": "%s"}\' "$1"\n', encoding="utf-8") + workspace = tmp_path / "ws" + output = _run_precheck(task_dir, workspace) + assert json.loads(output) == {"workspace": workspace.as_posix()} + + +def test_run_precheck_missing_script_returns_empty(tmp_path: Path) -> None: + assert _run_precheck(_make_task_dir(tmp_path), tmp_path / "ws") == "" + + +def test_run_precheck_nonzero_exit_returns_empty(tmp_path: Path) -> None: + task_dir = _make_task_dir(tmp_path) + (task_dir / "precheck.sh").write_text("#!/bin/bash\nexit 3\n", encoding="utf-8") + assert _run_precheck(task_dir, tmp_path / "ws") == "" + + +def test_run_precheck_invalid_json_returns_empty(tmp_path: Path) -> None: + task_dir = _make_task_dir(tmp_path) + (task_dir / "precheck.sh").write_text('#!/bin/bash\necho "not json"\n', encoding="utf-8") + assert _run_precheck(task_dir, tmp_path / "ws") == "" + + +def test_apply_precheck_caps_normalized_score_and_records_it() -> None: + evaluation = _make_evaluation(normalized_score=0.9) + _apply_precheck(evaluation, json.dumps({"normalized_score_cap": 0.45})) + assert evaluation.normalized_score == 0.45 + assert evaluation.precheck is not None + applied = evaluation.precheck["normalized_score_cap_applied"] + assert applied == {"uncapped_normalized_score": 0.9, "capped_to": 0.45} + + +def test_apply_precheck_without_cap_keeps_score_and_stores_signals() -> None: + evaluation = _make_evaluation(normalized_score=0.9) + _apply_precheck(evaluation, json.dumps({"signals": {"artifacts": 1}})) + assert evaluation.normalized_score == 0.9 + assert evaluation.precheck == {"signals": {"artifacts": 1}} + + +def test_apply_precheck_ignores_non_numeric_cap() -> None: + evaluation = _make_evaluation(normalized_score=0.9) + _apply_precheck(evaluation, json.dumps({"normalized_score_cap": "high"})) + assert evaluation.normalized_score == 0.9 + + +def test_apply_precheck_cap_higher_than_score_is_a_no_op() -> None: + evaluation = _make_evaluation(normalized_score=0.3) + _apply_precheck(evaluation, json.dumps({"normalized_score_cap": 0.45})) + assert evaluation.normalized_score == 0.3 + assert evaluation.precheck is not None + assert "normalized_score_cap_applied" not in evaluation.precheck + + +def test_prompt_includes_precheck_section_when_provided() -> None: + prompt = _build_evaluator_prompt( + instruction="Fix the bug", + criteria="Check correctness", + expected_dimensions=[{"name": "correctness", "title": "Correctness", "max_score": 25}], + ground_truth="", + artifacts_dir="/workspace", + trajectory_path=None, + precheck='{"signals": {"artifacts": 1}}', + ) + assert "## Deterministic pre-check signals" in prompt + assert '\n{"signals": {"artifacts": 1}}\n' in prompt + + +def test_prompt_no_precheck_section_by_default() -> None: + prompt = _build_evaluator_prompt( + instruction="Fix the bug", + criteria="Check correctness", + expected_dimensions=[{"name": "correctness", "title": "Correctness", "max_score": 25}], + ) + assert "pre-check" not in prompt.lower() + + +# --------------------------------------------------------------------------- +# Agent diff materialization +# --------------------------------------------------------------------------- + + +def _git_ws(workspace: Path, *args: str) -> None: + import subprocess + + subprocess.run( + ["git", "-C", str(workspace), "-c", "user.email=t@t", "-c", "user.name=t", *args], + check=True, + capture_output=True, + ) + + +def _make_workspace_with_agent_changes(tmp_path: Path) -> tuple[Path, Path]: + workspace = tmp_path / "artifacts" / "workspace" + workspace.mkdir(parents=True) + (workspace / "Existing.cs").write_text("[DddDomainService]\nclass Existing { }\n", encoding="utf-8") + _git_ws(workspace, "init", "-q") + _git_ws(workspace, "add", "-A") + _git_ws(workspace, "commit", "-qm", "base") + # agent's uncommitted work: modify a tracked file, add an untracked one + (workspace / "Existing.cs").write_text("class Existing { }\n", encoding="utf-8") + (workspace / "CLAUDE.md").write_text("You are a coding assistant.\n", encoding="utf-8") + trial_dir = tmp_path + return workspace, trial_dir + + +def test_materialize_agent_diff_writes_tracked_and_untracked_changes(tmp_path: Path) -> None: + workspace, trial_dir = _make_workspace_with_agent_changes(tmp_path) + diff_path, diffstat = _materialize_agent_diff(workspace, trial_dir) + assert diff_path == str(trial_dir / "agent_changes.diff") + content = Path(diff_path).read_text(encoding="utf-8") + assert "-[DddDomainService]" in content + assert "CLAUDE.md" in content + assert "Existing.cs" in diffstat + assert "CLAUDE.md (new file)" in diffstat + + +def test_materialize_agent_diff_no_git_returns_none(tmp_path: Path) -> None: + workspace = tmp_path / "artifacts" / "workspace" + workspace.mkdir(parents=True) + diff_path, diffstat = _materialize_agent_diff(workspace, tmp_path) + assert diff_path is None + assert diffstat == "" + assert not (tmp_path / "agent_changes.diff").exists() + + +def test_materialize_agent_diff_clean_workspace_returns_none(tmp_path: Path) -> None: + workspace, trial_dir = _make_workspace_with_agent_changes(tmp_path) + _git_ws(workspace, "add", "-A") + _git_ws(workspace, "commit", "-qm", "agent work committed, tree clean") + diff_path, diffstat = _materialize_agent_diff(workspace, trial_dir) + assert diff_path is None + assert diffstat == "" + + +def test_prompt_includes_agent_diff_section_when_provided() -> None: + prompt = _build_evaluator_prompt( + instruction="Fix the bug", + criteria="Check correctness", + expected_dimensions=[{"name": "correctness", "title": "Correctness", "max_score": 25}], + agent_diff_path="/jobs/j1/trial/agent_changes.diff", + agent_diffstat=" Existing.cs | 1 -\n CLAUDE.md (new file)", + ) + assert "## Agent diff" in prompt + assert "/jobs/j1/trial/agent_changes.diff" in prompt + assert "CLAUDE.md (new file)" in prompt + + +def test_prompt_no_agent_diff_section_by_default() -> None: + prompt = _build_evaluator_prompt( + instruction="Fix the bug", + criteria="Check correctness", + expected_dimensions=[{"name": "correctness", "title": "Correctness", "max_score": 25}], + ) + assert "## Agent diff" not in prompt + + +def test_prompt_procedure_is_diff_first_when_diff_present() -> None: + prompt = _build_evaluator_prompt( + instruction="Fix the bug", + criteria="Check correctness", + expected_dimensions=[{"name": "correctness", "title": "Correctness", "max_score": 25}], + agent_diff_path="/jobs/j1/trial/agent_changes.diff", + agent_diffstat=" x.cs | 1 -", + ) + assert "1. Start from the agent diff" in prompt + assert "## How to evaluate" in prompt + + +def test_prompt_procedure_is_workspace_first_without_diff() -> None: + prompt = _build_evaluator_prompt( + instruction="Fix the bug", + criteria="Check correctness", + expected_dimensions=[{"name": "correctness", "title": "Correctness", "max_score": 25}], + ) + assert "1. Use `Glob` to discover all output files in the workspace." in prompt + assert "Start from the agent diff" not in prompt diff --git a/tests/test_evaluator_backends.py b/tests/test_evaluator_backends.py index 1d4150eb..b197f3b3 100644 --- a/tests/test_evaluator_backends.py +++ b/tests/test_evaluator_backends.py @@ -66,12 +66,14 @@ def test_claude_backend_validate_auth_succeeds_with_oauth(monkeypatch: pytest.Mo backend.validate_auth() -def test_claude_backend_validate_auth_fails_without_credentials(monkeypatch: pytest.MonkeyPatch) -> None: +def test_claude_backend_validate_auth_tolerates_missing_env_credentials(monkeypatch: pytest.MonkeyPatch) -> None: + # Keychain OAuth (subscription accounts) is invisible to the environment; + # validate_auth must not hard-fail — the subprocess fails loudly if the + # CLI truly has no auth. monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False) monkeypatch.delenv("CLAUDE_CODE_OAUTH_TOKEN", raising=False) backend = ClaudeSubprocessBackend() - with pytest.raises(SystemExit): - backend.validate_auth() + backend.validate_auth() def test_claude_backend_builds_command(tmp_path: Path) -> None: @@ -110,6 +112,34 @@ def test_claude_backend_command_includes_add_dir_for_trajectory(tmp_path: Path) assert str(trial_dir) in cmd +def test_claude_backend_command_includes_add_dir_for_agent_diff(tmp_path: Path) -> None: + trial_dir = tmp_path / "trial" + trial_dir.mkdir() + (trial_dir / "agent_changes.diff").write_text("diff --git a/x b/x\n", encoding="utf-8") + backend = ClaudeSubprocessBackend() + cmd = backend._build_command( + workspace_path=tmp_path, + eval_config=EvaluationConfig(), + project_root=tmp_path.parent, + trial_dir=trial_dir, + ) + assert "--add-dir" in cmd + assert str(trial_dir) in cmd + + +def test_claude_backend_command_no_add_dir_without_trajectory_or_diff(tmp_path: Path) -> None: + trial_dir = tmp_path / "trial" + trial_dir.mkdir() + backend = ClaudeSubprocessBackend() + cmd = backend._build_command( + workspace_path=tmp_path, + eval_config=EvaluationConfig(), + project_root=tmp_path.parent, + trial_dir=trial_dir, + ) + assert "--add-dir" not in cmd + + def test_claude_backend_command_includes_mcp_config(tmp_path: Path) -> None: mcp_file = tmp_path / "mcp.json" mcp_file.write_text("{}") diff --git a/tests/test_pricing.py b/tests/test_pricing.py index 6d476f1a..0f6df37b 100644 --- a/tests/test_pricing.py +++ b/tests/test_pricing.py @@ -67,13 +67,35 @@ def test_load_custom_pricing_file(tmp_path: Path) -> None: assert pricing["my-model"].input_per_1m == 1.0 -def test_compute_cost_full_rate_no_cache_discount() -> None: +def test_compute_cost_without_cache_reduces_to_full_rate() -> None: pricing = load_pricing() # claude-sonnet-4-6 = $3 in / $15 out: 1M input + 0.1M output = 3.0 + 1.5 = 4.5 cost = compute_cost_usd(1_000_000, 100_000, "claude-sonnet-4-6", pricing) assert cost == pytest.approx(4.5) +def test_compute_cost_cache_aware_matches_harbor_accounting() -> None: + # Real trial ayg7ckA (claude-fable-5): Harbor's own per-step total was $8.841665. + cost = compute_cost_usd( + 3_070_507, + 66_456, + "claude-fable-5", + load_pricing(), + cache_read_tokens=2_939_665, + cache_write_tokens=127_078, + ) + assert cost == pytest.approx(8.841665, abs=0.0005) + + +def test_compute_cost_missing_cache_rates_falls_back_to_full_rate(tmp_path: Path) -> None: + custom = tmp_path / "pricing.toml" + custom.write_text('[models."bare"]\ninput_per_1m = 2.0\noutput_per_1m = 10.0\n') + pricing = load_pricing(custom) + # no cached/cache-write rates -> reads and writes bill at the full input rate + cost = compute_cost_usd(1_000_000, 0, "bare", pricing, cache_read_tokens=900_000, cache_write_tokens=50_000) + assert cost == pytest.approx(2.0) + + def test_compute_cost_unknown_model_returns_none() -> None: assert compute_cost_usd(1000, 100, "nonexistent-model", load_pricing()) is None @@ -146,6 +168,7 @@ def test_layered_three_layers_compose(tmp_path: Path, empty_user_layer: Path) -> assert set(merged) == { "gpt-5.5", "gpt-5.4", + "claude-fable-5", "claude-opus-4-8", "claude-sonnet-4-6", "azure-gpt5", diff --git a/tests/test_results_exporter.py b/tests/test_results_exporter.py index 9f4670dd..5c25d2a4 100644 --- a/tests/test_results_exporter.py +++ b/tests/test_results_exporter.py @@ -172,8 +172,8 @@ def test_export_includes_token_cost_economics(job_dir: Path, tmp_path: Path) -> assert usage["input_tokens"] == 1_000_000 assert usage["output_tokens"] == 60_000 # completion 50k + reasoning 10k assert usage["total_tokens"] == 1_060_000 - # claude-sonnet-4-6: 1M*$3 + 0.06M*$15 = 3.0 + 0.9 = 3.9 - assert metrics["cost_usd"] == pytest.approx(3.9) + # sonnet $3/$15/$0.30 cached (ADR-014): fresh 0.2M*3 + cached 0.8M*0.30 + 0.06M*15 = 1.74 + assert metrics["cost_usd"] == pytest.approx(1.74) assert metrics["pricing_as_of"] == "2026-06-08" assert "cost_efficiency" not in metrics # removed: arbitrary zero → use Pareto front assert "token_efficiency" not in metrics @@ -342,7 +342,7 @@ def test_export_project_dir_none_uses_bundled(job_dir: Path, tmp_path: Path, emp export_results([job_dir], dest, project_dir=None) metrics = json.loads((dest / "2026-06-03__demo-job__demo-task__aaa111" / "metrics.json").read_text()) - assert metrics["cost_usd"] == pytest.approx(3.9) + assert metrics["cost_usd"] == pytest.approx(1.74) def test_export_three_layer_compose_e2e(job_dir: Path, tmp_path: Path, empty_user_layer: Path) -> None: diff --git a/tests/test_token_metrics.py b/tests/test_token_metrics.py index daa65c99..b82e7c70 100644 --- a/tests/test_token_metrics.py +++ b/tests/test_token_metrics.py @@ -40,6 +40,8 @@ def test_extract_claude_shape() -> None: assert usage.input_tokens == 1_109_410 assert usage.reasoning_tokens == 0 assert usage.output_tokens == 55_518 # no reasoning to fold in + assert usage.cached_tokens == 1_031_868 + assert usage.cache_write_tokens == 74_666 assert usage.total_tokens == 1_109_410 + 55_518 @@ -49,6 +51,7 @@ def test_extract_codex_folds_reasoning_into_output() -> None: assert usage.completion_tokens == 23_401 assert usage.reasoning_tokens == 13_921 assert usage.output_tokens == 23_401 + 13_921 # reasoning folded in + assert usage.cache_write_tokens == 0 # Codex trajectories carry no cache-creation counter assert usage.total_tokens == 2_817_494 + 23_401 + 13_921 @@ -96,8 +99,10 @@ def test_build_trial_economics_priced_model(tmp_path: Path) -> None: assert econ["model_name"] == "gpt-5.4" assert econ["token_usage"]["total_tokens"] == 2_854_816 - # gpt-5.4 = $2.50 in / $15 out: 2.817494M*2.5 + 0.037322M*15 - assert econ["cost_usd"] == pytest.approx(2_817_494 / 1e6 * 2.5 + 37_322 / 1e6 * 15) + # gpt-5.4 = $2.50 in / $15 out / $0.25 cached (ADR-014): fresh input at full + # rate, the 2_646_272 cached reads at the cached rate, no write premium. + fresh = 2_817_494 - 2_646_272 + assert econ["cost_usd"] == pytest.approx(fresh / 1e6 * 2.5 + 2_646_272 / 1e6 * 0.25 + 37_322 / 1e6 * 15) assert econ["pricing_as_of"] == "2026-06-08" diff --git a/website/src/content/docs/concepts/token-cost.md b/website/src/content/docs/concepts/token-cost.md index 1b83a7d3..088252fa 100644 --- a/website/src/content/docs/concepts/token-cost.md +++ b/website/src/content/docs/concepts/token-cost.md @@ -40,13 +40,22 @@ There are **two separate sources of wobble**, and Nasde keeps them apart on purp Why split them? Because the question you actually care about is: **is the gap between two configs bigger than the wobble, or is it just noise?** Keeping the two sources separate lets you answer that — a 0.02 gap means nothing if each score wobbles by ±0.08. (Formal significance testing is a separate, offline step; Nasde's job here is to surface the spread and sample size that make an average trustworthy in the first place.) -## How the cost is calculated — "as if every run were the first" +## How the cost is calculated — what the API would bill -The dollar figure Nasde reports is **deliberately consistent**: run the same task ten times and you'll get the same cost ten times. That's on purpose, and here's why it matters. +The dollar figure Nasde reports is the **cache-aware price of the run**: each token volume the agent actually consumed, billed at its own catalog rate. -Most providers give a discount for **prompt caching** — if you send the same prompt again soon after, the repeated part is cheaper. That sounds good, but it makes cost *unpredictable for comparison*: the exact same run can cost more or less depending on whether your cache happened to be "warm" (recently used) or "cold". You'd be comparing models on luck, not on how much they actually cost. +Most providers discount **prompt caching** — and in an agentic session that discount is not a lucky accident, it's structural. Every step of the session re-sends the same growing prompt prefix, so the bulk of input tokens are cheap cache reads (on our measured benchmark grids, 93–98% of them). Pricing a run as if no cache existed sounds "safer", but it lands several times above any real invoice — we measured a 4.4× gap on a real 24-run grid. A cost number that far from the bill can't drive a model decision. -So Nasde **ignores the cache discount entirely** and prices every run **as if it were the very first one** — the full prompt billed at the full catalog rate, every time. The model's reasoning tokens (the "thinking" some models do) are counted as output. The result is a cost number that depends only on the model and the task, not on timing — so when you compare two models, you're comparing them fairly. +So Nasde prices each component separately: + +- **fresh input** tokens — the full input rate, +- **cache writes** — the cache-write rate (for Anthropic's 1-hour cache: 2× the input rate), +- **cache reads** — the cached-input rate (typically 0.1×), +- **output** tokens (the model's reasoning/"thinking" included) — the output rate. + +The number is still reproducible: it is computed from the token volumes recorded on that trial, so recomputing always gives the same answer. And if you ever want the cache-free ceiling ("what would this cost with a cold cache every step?"), it's one multiplication away — the raw volumes stay in `token_usage`. See [ADR-014](https://github.com/NoesisVision/nasde-toolkit/blob/main/docs/adr/014-cache-aware-cost.md) for the full decision record. + +A model entry that lacks the cache rates simply bills those volumes at the full input rate — conservative, never a silent discount. ## Where pricing comes from @@ -56,6 +65,8 @@ Rates live in a small, versioned `pricing.toml` bundled with Nasde, each model s [models."your-model-id"] input_per_1m = 3.0 output_per_1m = 15.0 +cached_input_per_1m = 0.30 # cache-read rate; omit → reads billed at input_per_1m +cache_write_per_1m = 6.0 # cache-write rate; omit → writes billed at input_per_1m as_of = "2026-06-08" source = "https://…" ```