feat: show the per-finding "Why the gate failed" explanation in the diff index.html

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.23.0 — 2026-11-02
+
+- show the per-finding "Why the gate failed" explanation in the diff index.html. When a diff has at least one blocking change, `evalarc diff --output DIR` now adds a 'Why the gate failed' section to DIR/index.html, the report the CLI links to. The section sits after the Changed checks table and before the remaining sections. It contains the numeric threshold sentence and a table with these columns: Finding (kind, plus '(within noise)' when flagged, shown as a ✗ tone label so it does not rely on color), Case / check, Baseline, Current, and Failing attempt (the first failing current attempt's value and evidence, each clipped to 120 characters). Removed checks show the baseline attempt labelled 'not recorded in the current run; baseline:'. Rows with no detail show 'no attempt detail recorded'. A metadata line gives each input's file name and a 12-character sha256 prefix. Every value is HTML-escaped. summary.md and index.html now build this section from the same shared rows, so they cannot drift. summary.md text, diff.json and exit codes are unchanged. When nothing blocks, index.html is unchanged.
+
 ## 0.22.1 — 2026-10-28
 
 - fix: refuse to compare Inspect runs scored at different numeric pass thresholds. evalarc.results_diff.diff() now refuses to compare two Inspect runs whose numeric scores were judged against different pass thresholds. The Inspect loader turns every finite numeric score into a pass or fail with the threshold given to load_results(path, threshold=...). So a baseline loaded at 0.8 and a current run loaded at 0.5 could report checks as improved, or let the gate pass, when the agent had not changed. The new check runs after the existing format and task checks. It raises ValueError('inputs are not comparable: numeric pass threshold 0.8 (baseline) != 0.5 (current); numeric Inspect scores were judged against different pass thresholds. Load both runs with the same threshold.') and returns no partial result. These comparisons behave as before: equal thresholds (1 and 1.0 count as equal), Inspect runs with only letter-grade, boolean or missing score values, a baseline run dict without a threshold key, and all promptfoo and JUnit comparisons. docs/ci-gate.md has a new section, 'Numeric pass threshold must match (Inspect)'.

CITATION.cff

@@ -4,8 +4,8 @@ title: "EvalArc: Auditable Evaluations for AI Agents"
 type: software
 authors:
   - name: EvalArc contributors
-version: 0.22.1
-date-released: 2026-10-28
+version: 0.23.0
+date-released: 2026-11-02
 license: MIT
 repository-code: "https://github.com/noteflowai/evalarc"
 url: "https://noteflowai.github.io/evalarc/"

README.md

@@ -68,13 +68,13 @@ and the upgrade gate fails. These checks share two format/schema failures.
    installs the published wheel, downloads the records and rebuilds the comparison.
    Verification exits 0 for consistency; comparison exits 1 for the regression.
 
-Install the released reviewer from [PyPI](https://pypi.org/project/evalarc/0.22.1/)
+Install the released reviewer from [PyPI](https://pypi.org/project/evalarc/0.23.0/)
 in a fresh virtual environment:
 
 ```bash
 python3 -m venv .venv
 . .venv/bin/activate
-python -m pip install evalarc==0.22.1
+python -m pip install evalarc==0.23.0
 evalarc --version
 ```
 
@@ -126,7 +126,7 @@ because checks missing from it cannot be compared; rerun that evaluation to
 completion and compare again.
 
 ```yaml
-- uses: noteflowai/evalarc@v0.22.1 # or a full commit SHA
+- uses: noteflowai/evalarc@v0.23.0 # or a full commit SHA
   with:
     baseline: evals/baseline.json
     current: results/current.json

README.zh-CN.md

@@ -52,13 +52,13 @@ EvalArc 是面向 Agent 评测的 Python 命令行复核工具：逐项对比已
 3. **本地复算。** [首次复核指南](docs/first-review.zh-CN.md)使用发布的 wheel 和原始记录重建报告；
    复核退出 0 表示一致，对照退出 1 表示发现退步。
 
-在新的虚拟环境中，从 [PyPI](https://pypi.org/project/evalarc/0.22.1/)
+在新的虚拟环境中，从 [PyPI](https://pypi.org/project/evalarc/0.23.0/)
 安装已发布的复核工具：
 
 ```bash
 python3 -m venv .venv
 . .venv/bin/activate
-python -m pip install evalarc==0.22.1
+python -m pip install evalarc==0.23.0
 evalarc --version
 ```
 
@@ -105,7 +105,7 @@ Anthropic《Demystifying evals for AI agents》的建议在 EvalArc 中如何落
 准确率从 0.625 升至 0.8125，同时有三个检查项丢失通过。
 
 ```yaml
-- uses: noteflowai/evalarc@v0.22.1 # 或完整的提交 SHA
+- uses: noteflowai/evalarc@v0.23.0 # 或完整的提交 SHA
   with:
     baseline: evals/baseline.json
     current: results/current.json

docs/ci-gate.md

@@ -84,8 +84,15 @@ Inputs: baseline `baseline.json` sha256 `<first 12 hex>`, current `current.json`
 - The `Inputs:` line appears only when both runs record a source name and
   digest. The full digests stay in `diff.json`.
 
-The section only explains the result. It never changes the gate, the exit code,
-`diff.json` or `index.html`. A summary with no blocking change has no such
+`index.html`, the report the CLI prints after `Report:`, shows the same
+section below the Changed checks table. It has the same columns, threshold
+sentence and `Inputs:` digest line, built from the same rows. Each finding is
+marked with a ✗ symbol as well as a color, and the table scrolls inside a
+keyboard-focusable region. Unlike `summary.md`, the HTML lists every blocking
+finding with no row limit, as its Changed checks table already does.
+
+The section only explains the result. It never changes the gate, the exit code
+or `diff.json`. A summary or `index.html` with no blocking change has no such
 section and is the same as before. That covers a passing gate, an incomplete run
 with no changed checks, and a failure of the opt-in regression suite only.
 

docs/ci-gate.zh-CN.md

@@ -97,6 +97,14 @@ JUnit 用例）时，若至少一侧出现了通过与失败并存的实际波
 每个检查项的尝试次数（Inspect `--epochs`、promptfoo `--repeat`）。这对应“收益必须超过
 评测噪声才值得采纳”的做法。
 
+## 门禁失败原因
+
+至少有一个变化阻断门禁时，`summary.md` 与 `index.html` 都会在变化表之后加上
+“Why the gate failed”一节：逐条给出阻断类型、是否在采样噪声内、第一次失败的已记录尝试
+（`removed` 行显示基线尝试，`less_covered` 行显示已评估次数），以及数值阈值和两份输入
+SHA-256 的前 12 位。两者使用同一组行；`summary.md` 沿用变化表的行数上限，`index.html`
+列出全部阻断行。这一节只解释结果，不改变门禁、退出码或 `diff.json`；没有阻断变化时不出现。
+
 ## 留出集：改动是否泛化
 
 若改动是针对部分评测用例调出来的，仅凭这些用例无法说明它在别处也有帮助。用一个小 JSON

package.json

@@ -1,6 +1,6 @@
 {
   "name": "evalarc-evidence-site",
-  "version": "0.22.1",
+  "version": "0.23.0",
   "private": true,
   "description": "Browser checks and the TypeScript report front end for EvalArc",
   "scripts": {

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "evalarc"
-version = "0.22.1"
+version = "0.23.0"
 description = "Find agent regressions behind better scores: compare changed checks, review recorded tool actions and verify evidence offline."
 readme = "README.md"
 requires-python = ">=3.11"

src/evalarc/__init__.py

@@ -1,3 +1,3 @@
 """Auditable evaluations for AI agents."""
 
-__version__ = "0.22.1"
+__version__ = "0.23.0"