From 6925116ea7941238e277499f2dccec4788326600 Mon Sep 17 00:00:00 2001 From: Szymon Janikowski Date: Fri, 11 Sep 2026 13:18:29 +0200 Subject: [PATCH 1/2] chore: release v0.6.0 Move [Unreleased] under 0.6.0 (2026-09-11), add the missing CHANGELOG entry for the calibration round 2 engine changes (#75: agent diff as judge input, per-task dimensions, precheck.sh hook, eval --eval-model/--eval-backend) and the [#76] link-ref on the Harbor 0.19 entry. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UwjBNRxR1vkNGBcETRzdf2 --- CHANGELOG.md | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5df6cf8..73b9cd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ See [docs/RELEASING.md](docs/RELEASING.md) for the release procedure. ## [Unreleased] +## [0.6.0] — 2026-09-11 + ### Added - Bundled price catalog: `claude-fable-5-1` ($10/$50 per 1M tokens, 1-hour cache write $20, **cache read $0.25 — 0.025x base input, not the 0.1x every other @@ -22,6 +24,24 @@ See [docs/RELEASING.md](docs/RELEASING.md) for the release procedure. pricing page 2026-07-24; the other Claude entries re-verified at the same date. ([#77]) +- **Judge inputs from calibration round 2 ([#75]).** The evaluator now + materializes the agent's full diff into `/agent_changes.diff` (shared + `workspace_diff.capture_patch`, the same code that produces `changes.patch` on + export); the judge prompt gets the diffstat inline plus the file for + Read/Grep, and the base "How to evaluate" procedure is diff-first whenever a + diff exists (the Claude backend grants `--add-dir` on the trial dir). + A task-level `assessment_dimensions.json` now overrides the benchmark-wide + file (`resolve_dimensions_path`) — a different rubric yields a different + fingerprint, so per-task evals are never averaged with old ones. An optional + task-level `precheck.sh` runs on the host before the judge: its JSON output + is injected into the prompt as ground facts and recorded in each + `assessment_eval_.json`; a `normalized_score_cap` (0..1) in that JSON is + enforced as a hard ceiling, with the uncapped score kept alongside so a + capped result stays explainable. `nasde eval` gains `--eval-model` / + `--eval-backend` per-run judge overrides (judge-model comparison matrices), + and `nasde calibrate publish` bundles the resolved per-task dimensions plus + `ground_truth_decisions.json` into `.calibration/`. + ### Changed - **Cost is now cache-aware ([ADR-014](docs/adr/014-cache-aware-cost.md)) — supersedes ADR-011's "as if every run were the first" formula.** `cost_usd` bills fresh input at the @@ -35,7 +55,7 @@ See [docs/RELEASING.md](docs/RELEASING.md) for the release procedure. `input × input_rate + output × output_rate`). A model entry missing cache rates bills those volumes at the full input rate — conservative, never a silent discount. Historical exports need a one-shot economics backfill to reprice. -- **Harbor bumped from 0.13 to 0.19** (`harbor[daytona,modal,e2b,runloop,gke]>=0.19,<0.20`). +- **Harbor bumped from 0.13 to 0.19** (`harbor[daytona,modal,e2b,runloop,gke]>=0.19,<0.20`, [#76]). The Python-API surface nasde drives (`JobConfig.model_validate`, `Job.create`, `job.run()`, `AgentConfig` `import_path`/`kwargs`/`skills`/`mcp_servers`/`env`) is unchanged; the native skill-injection contract (ADR-012) moved from @@ -610,7 +630,8 @@ Initial release under the **nasde-toolkit** name (rebrand from - `v0.1.0` represents the first public-oriented baseline; earlier commits on the `sdlc-eval-kit` history are not cataloged here. -[Unreleased]: https://github.com/NoesisVision/nasde-toolkit/compare/v0.5.0...HEAD +[Unreleased]: https://github.com/NoesisVision/nasde-toolkit/compare/v0.6.0...HEAD +[0.6.0]: https://github.com/NoesisVision/nasde-toolkit/compare/v0.5.0...v0.6.0 [0.5.0]: https://github.com/NoesisVision/nasde-toolkit/compare/v0.4.0...v0.5.0 [0.4.0]: https://github.com/NoesisVision/nasde-toolkit/compare/v0.3.3...v0.4.0 [0.3.3]: https://github.com/NoesisVision/nasde-toolkit/compare/v0.3.2...v0.3.3 @@ -655,6 +676,8 @@ Initial release under the **nasde-toolkit** name (rebrand from [#71]: https://github.com/NoesisVision/nasde-toolkit/pull/71 [#73]: https://github.com/NoesisVision/nasde-toolkit/pull/73 [#74]: https://github.com/NoesisVision/nasde-toolkit/pull/74 +[#75]: https://github.com/NoesisVision/nasde-toolkit/pull/75 +[#76]: https://github.com/NoesisVision/nasde-toolkit/pull/76 [#77]: https://github.com/NoesisVision/nasde-toolkit/pull/77 [#78]: https://github.com/NoesisVision/nasde-toolkit/pull/78 [#80]: https://github.com/NoesisVision/nasde-toolkit/pull/80 From 598a25470a0ec2e3dc78683840c3963d83fee96f Mon Sep 17 00:00:00 2001 From: Szymon Janikowski Date: Sat, 12 Sep 2026 10:48:12 +0200 Subject: [PATCH 2/2] changelog: tighten the 0.6.0 entries Trim the 0.6.0 section to concise, user-facing entries: one bullet per feature with the value it brings, no implementation history or internal verification notes. Same facts, 76 lines -> 47. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UwjBNRxR1vkNGBcETRzdf2 --- CHANGELOG.md | 103 ++++++++++++++++++--------------------------------- 1 file changed, 37 insertions(+), 66 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 73b9cd0..7be7cc4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,78 +12,49 @@ See [docs/RELEASING.md](docs/RELEASING.md) for the release procedure. ## [0.6.0] — 2026-09-11 ### Added -- Bundled price catalog: `claude-fable-5-1` ($10/$50 per 1M tokens, 1-hour cache - write $20, **cache read $0.25 — 0.025x base input, not the 0.1x every other - Claude model uses**), verified against the official Anthropic pricing page - 2026-09-04; the other Claude entries re-verified at the same date. ([#80]) -- Bundled price catalog: `claude-sonnet-5` ($2/$10 per 1M tokens, cache read - $0.20 / 1-hour cache write $4), verified against the official Anthropic - pricing page 2026-08-24. ([#78]) -- Bundled price catalog: `claude-opus-5` ($5/$25 per 1M tokens, cache read - $0.50 / 1-hour cache write $10), verified against the official Anthropic - pricing page 2026-07-24; the other Claude entries re-verified at the same - date. ([#77]) - -- **Judge inputs from calibration round 2 ([#75]).** The evaluator now - materializes the agent's full diff into `/agent_changes.diff` (shared - `workspace_diff.capture_patch`, the same code that produces `changes.patch` on - export); the judge prompt gets the diffstat inline plus the file for - Read/Grep, and the base "How to evaluate" procedure is diff-first whenever a - diff exists (the Claude backend grants `--add-dir` on the trial dir). - A task-level `assessment_dimensions.json` now overrides the benchmark-wide - file (`resolve_dimensions_path`) — a different rubric yields a different - fingerprint, so per-task evals are never averaged with old ones. An optional - task-level `precheck.sh` runs on the host before the judge: its JSON output - is injected into the prompt as ground facts and recorded in each - `assessment_eval_.json`; a `normalized_score_cap` (0..1) in that JSON is - enforced as a hard ceiling, with the uncapped score kept alongside so a - capped result stays explainable. `nasde eval` gains `--eval-model` / - `--eval-backend` per-run judge overrides (judge-model comparison matrices), - and `nasde calibrate publish` bundles the resolved per-task dimensions plus - `ground_truth_decisions.json` into `.calibration/`. +- Bundled prices for `claude-fable-5-1` ($10/$50 per 1M tokens; cache read + $0.25 = 0.025x input, unlike the 0.1x of other Claude models), + `claude-sonnet-5` ($2/$10) and `claude-opus-5` ($5/$25), verified against the + Anthropic pricing page on 2026-09-04. Trials on these models get `cost_usd` + out of the box. ([#77], [#78], [#80]) +- The judge now reviews the agent's actual diff first: the evaluator writes it + to `/agent_changes.diff` and hands it to the judge, so scores are + grounded in what changed rather than in the agent's own narrative. ([#75]) +- Per-task `assessment_dimensions.json`: a task can override the benchmark-wide + dimensions. Different rubrics get different fingerprints and are never + averaged together. ([#75]) +- Optional task-level `precheck.sh`: deterministic host-side checks whose JSON + output is given to the judge as ground facts and recorded in every + evaluation. A `normalized_score_cap` in that output enforces a hard ceiling + for mechanically disqualified trials; the uncapped score is kept alongside. + ([#75]) +- `nasde eval --eval-model` / `--eval-backend` swap the judge for one run + without editing `nasde.toml`, for judge-model comparison matrices. + `nasde calibrate publish` now bundles the per-task dimensions and + `ground_truth_decisions.json` for reviewers. ([#75]) ### Changed -- **Cost is now cache-aware ([ADR-014](docs/adr/014-cache-aware-cost.md)) — supersedes ADR-011's - "as if every run were the first" formula.** `cost_usd` bills fresh input at the - full rate, cache writes at the new per-model `cache_write_per_1m` (Anthropic - 1-hour mode: 2× input), cache reads at `cached_input_per_1m` (0.1×), and output - at the output rate — matching what the API would bill (verified against Harbor's - per-step accounting to the cent on 20 of 24 grid trials). Rationale: measured - cache read ratios are a stable 93–98% of input across a full 24-trial grid, and - the old full-rate figure sat ~4.4× above a real bill. `token_usage` gains - `cache_write_tokens`; the cache-free ceiling is no longer stored (derivable as - `input × input_rate + output × output_rate`). A model entry missing cache rates - bills those volumes at the full input rate — conservative, never a silent - discount. Historical exports need a one-shot economics backfill to reprice. -- **Harbor bumped from 0.13 to 0.19** (`harbor[daytona,modal,e2b,runloop,gke]>=0.19,<0.20`, [#76]). - The Python-API surface nasde drives (`JobConfig.model_validate`, `Job.create`, - `job.run()`, `AgentConfig` `import_path`/`kwargs`/`skills`/`mcp_servers`/`env`) - is unchanged; the native skill-injection contract (ADR-012) moved from - `BaseInstalledAgent._build_register_skills_command` to per-agent implementations - upstream but keeps the same `$HOME/.agents/skills` / `~/.gemini/skills` targets. - Unlocks ~10 new built-in Harbor agents (pi, grok-build, langgraph, deerflow, - eve, mimo, computer-1, acp, vibe, …) and agent `skills` entries as git - references (`org/name@ref`). The dependency tree also slims down (pandas, - numpy, pyarrow, mcp no longer pulled in). Verified: 463 tests green, pip-audit - clean, and a full smoke run (`ddd-threshold-discount`, claude-vanilla) with - non-null cache-aware token/cost economics on a fresh 0.19 trajectory. -- Added the NASDE branding source assets and updated the README and - documentation website to use the new brand. ([#73]) +- Cost is now cache-aware ([ADR-014](docs/adr/014-cache-aware-cost.md)): + `cost_usd` bills cache reads and writes at their per-model rates, matching the + real API bill instead of a ~4x overestimate. `token_usage` gains + `cache_write_tokens`; a model without cache rates falls back to the full input + rate, never a silent discount. Historical exports need a one-shot backfill to + reprice. +- Harbor bumped from 0.13 to 0.19 ([#76]): unlocks ~10 new built-in Harbor + agents and git-referenced agent skills (`org/name@ref`), with a smaller + dependency tree. Nothing changes in nasde's config or CLI. +- NASDE branding applied to the README and the documentation website. ([#73]) ### Fixed -- Quality Gate no longer shows a spurious red ✗ on `main` after a release: the - concurrency group is now scoped per calling workflow so the publish-embedded - gate no longer cancels the standalone push-triggered one. ([#74]) -- `uv tool install` / `pip install` from a fresh resolver no longer produces a - `nasde` that dies on `ModuleNotFoundError: No module named 'typer._click'`: - the `typer` floor is raised from `>=0.16` to `>=0.26` — `cli.py` has imported - `typer._click` (the vendored Click) since the harbor 0.13 bump, but the floor - was never lifted, so an unlocked resolve could still pick typer 0.25. ([#82]) +- Fresh installs (`uv tool install`, `pip install`) could fail with + `ModuleNotFoundError: No module named 'typer._click'`; the `typer` floor is + raised to `>=0.26`. ([#82]) +- Quality Gate no longer shows a spurious red ✗ on `main` after a release. + ([#74]) ### Security -- **Pinned `aiohttp>=3.14.3`, `cryptography>=50.0.0` and `h2>=4.4.1`** (all - transitive) to address PYSEC-2026-3545 / 3546 / 3547 (aiohttp), - PYSEC-2026-3552 (cryptography) and PYSEC-2026-3628 (h2). ([#81]) +- Pinned `aiohttp>=3.14.3`, `cryptography>=50.0.0` and `h2>=4.4.1` (transitive) + for PYSEC-2026-3545/3546/3547, PYSEC-2026-3552 and PYSEC-2026-3628. ([#81]) ## [0.5.0] — 2026-06-24