Calibration round 2: rubric v2, agent-diff judge input, per-task dimensions, judge-matrix run plan

.claude/skills/nasde-benchmark-calibration/SKILL.md

@@ -107,6 +107,19 @@ The rubric to edit lives at `evals/<source>/tasks/<task>/assessment_criteria.md`
 dimension's scale/description, not one task's thresholds). The `<source>` and `<task>` come from the
 trial's `result.json` (`source`, `task_name`).
 
+Calibration can also restructure a single task's dimensions: an `assessment_dimensions.json`
+placed next to the task's `assessment_criteria.md` overrides the benchmark-wide file for that task
+only (different fingerprint — old and new evaluations are never mixed in one summary group). For
+change-related checks, the evaluator already hands every judge the agent's full diff
+(`<trial>/agent_changes.diff` + inline diffstat) — rubrics should direct the judge to answer
+"what did the agent change" questions from that diff. When a task additionally needs hard mechanical
+enforcement (e.g. a disqualification score cap), ship a `precheck.sh` — the evaluator runs it,
+injects its JSON into the judge prompt as ground facts, and enforces its optional
+`normalized_score_cap`. See `examples/ddd-architectural-challenges/tasks/ddd-weather-discount/` for
+the reference calibrated task and `CALIBRATION_ROUND2_2026-07-07.md` for the loop's acceptance
+criteria (repeatability, judge-model agreement, human-ranking correlation, dimension disjointness,
+regression assertions).
+
 Show the user a concrete **diff of the rubric** — the specific threshold/description change that would
 have moved the judge toward the human's score — and **wait for approval before writing**. Never edit
 the rubric silently.

ARCHITECTURE.md

@@ -184,6 +184,36 @@ flowchart TB
 
 ---
 
+### Agent diff — universal judge input
+
+For every trial, the evaluator materializes the agent's full diff (start state → final
+workspace: `git diff HEAD` + untracked files, via `workspace_diff.capture_patch` — the
+same capture used for `changes.patch` in exports) into `<trial>/agent_changes.diff`,
+and injects an "Agent diff" prompt section: the diffstat inline plus the file path for
+Read/Grep (the `ClaudeSubprocessBackend` grants `--add-dir` on the trial dir when the
+file is present). Rationale: the judge sees only the final state and — like a human
+reviewer without a diff — cannot see removals or out-of-feature edits; the diff is the
+universal, task-agnostic reference point for every "what did the agent change" check.
+Skipped gracefully when the workspace has no git repo or nothing changed.
+
+### Per-task rubric inputs
+
+Three optional files next to a task's `assessment_criteria.md` refine its evaluation:
+
+- `assessment_dimensions.json` — overrides the benchmark-wide dimensions file **for that task only**
+  (`resolve_dimensions_path`). A different dimensions file yields a different fingerprint, so
+  evaluations under old and new dimensions are never mixed in one summary group.
+- `ground_truth_decisions.json` — reference decisions injected verbatim into the judge prompt.
+- `precheck.sh` — an optional deterministic policy layer on top of the agent diff: the evaluator
+  runs it on the host before judging (`bash precheck.sh <workspace-path>`). Its stdout must be one
+  JSON object; it is injected into the judge prompt as "Deterministic pre-check signals" (facts the
+  judge must stay consistent with), recorded in `assessment_eval_*.json` under `precheck`, and its
+  optional `normalized_score_cap` (0..1) is enforced on the trial's normalized score (cap
+  application is recorded, so a capped score is always explainable). Use it when a task wants hard,
+  mechanical enforcement (e.g. a disqualification cap) rather than judge interpretation of the diff.
+  Any precheck failure degrades to "no precheck" with a warning. All three are also bundled into
+  `.calibration/` by `nasde calibrate publish`.
+
 ## Evaluator configuration
 
 The evaluator agent is configurable via `[evaluation]` in `nasde.toml`. All options are optional — defaults provide a working evaluator out of the box.
@@ -213,7 +243,7 @@ When `mcp_config` is set, its path is passed through to the backend CLI (`--mcp-
 
 ### Token & cost economics ([ADR-011](docs/adr/011-token-cost-metrics.md))
 
-Independently of the LLM judge, each trial's **token usage and cost** are read from the agent's `agent/trajectory.json` `final_metrics` (which Harbor writes for both Claude and Codex). A single extractor (`token_metrics.py`) computes `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + reasoning_output_tokens`, and a USD cost at the **full catalog rate with no cache discount** ("as if every run were the first" — deterministic, order-independent). It derives `token_efficiency` (score per 1M tokens) and `cost_efficiency` (score per USD), using the dominant evaluator cluster's `normalized_score_mean`. The same extractor feeds both the run path (`evaluator.py` → `assessment_summary.json`) and the export path (`results_exporter.py` → `metrics.json`), so they cannot diverge. Prices come from a bundled, versioned `pricing.toml`; an unpriced model leaves cost null (token metrics still computed). `nasde run` prints a per-`(agent, model)` cost table after assessment completes.
+Independently of the LLM judge, each trial's **token usage and cost** are read from the agent's `agent/trajectory.json` `final_metrics` (which Harbor writes for both Claude and Codex). A single extractor (`token_metrics.py`) computes `input = total_prompt_tokens` (full, cache included), `output = total_completion_tokens + reasoning_output_tokens`, the cache read/write volumes, and a **cache-aware USD cost** (ADR-014): fresh input at the full rate, cache writes at the cache-write rate, cache reads at the cached rate — what the API would bill for the run. The raw volumes stay in `token_usage`, so the cache-free ceiling is derivable offline; the scalar efficiency ratios were removed (ADR-011) — models are compared as a quality-vs-cost Pareto front. The same extractor feeds both the run path (`evaluator.py` → `assessment_summary.json`) and the export path (`results_exporter.py` → `metrics.json`), so they cannot diverge. Prices come from a bundled, versioned `pricing.toml`; an unpriced model leaves cost null (token metrics still computed). `nasde run` prints a per-`(agent, model)` cost table after assessment completes.
 
 ---
 

CHANGELOG.md

@@ -10,6 +10,18 @@ See [docs/RELEASING.md](docs/RELEASING.md) for the release procedure.
 ## [Unreleased]
 
 ### Changed
+- **Cost is now cache-aware ([ADR-014](docs/adr/014-cache-aware-cost.md)) — supersedes ADR-011's
+  "as if every run were the first" formula.** `cost_usd` bills fresh input at the
+  full rate, cache writes at the new per-model `cache_write_per_1m` (Anthropic
+  1-hour mode: 2× input), cache reads at `cached_input_per_1m` (0.1×), and output
+  at the output rate — matching what the API would bill (verified against Harbor's
+  per-step accounting to the cent on 20 of 24 grid trials). Rationale: measured
+  cache read ratios are a stable 93–98% of input across a full 24-trial grid, and
+  the old full-rate figure sat ~4.4× above a real bill. `token_usage` gains
+  `cache_write_tokens`; the cache-free ceiling is no longer stored (derivable as
+  `input × input_rate + output × output_rate`). A model entry missing cache rates
+  bills those volumes at the full input rate — conservative, never a silent
+  discount. Historical exports need a one-shot economics backfill to reprice.
 - **Harbor bumped from 0.13 to 0.19** (`harbor[daytona,modal,e2b,runloop,gke]>=0.19,<0.20`).
   The Python-API surface nasde drives (`JobConfig.model_validate`, `Job.create`,
   `job.run()`, `AgentConfig` `import_path`/`kwargs`/`skills`/`mcp_servers`/`env`)