diff --git a/CHANGELOG.md b/CHANGELOG.md index f50f200..51b2aaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # Changelog +## 0.23.0 — 2026-11-02 + +- show the per-finding "Why the gate failed" explanation in the diff index.html. When a diff has at least one blocking change, `evalarc diff --output DIR` now adds a 'Why the gate failed' section to DIR/index.html, the report the CLI links to. The section sits after the Changed checks table and before the remaining sections. It contains the numeric threshold sentence and a table with these columns: Finding (kind, plus '(within noise)' when flagged, shown as a ✗ tone label so it does not rely on color), Case / check, Baseline, Current, and Failing attempt (the first failing current attempt's value and evidence, each clipped to 120 characters). Removed checks show the baseline attempt labelled 'not recorded in the current run; baseline:'. Rows with no detail show 'no attempt detail recorded'. A metadata line gives each input's file name and a 12-character sha256 prefix. Every value is HTML-escaped. summary.md and index.html now build this section from the same shared rows, so they cannot drift. summary.md text, diff.json and exit codes are unchanged. When nothing blocks, index.html is unchanged. + ## 0.22.1 — 2026-10-28 - fix: refuse to compare Inspect runs scored at different numeric pass thresholds. evalarc.results_diff.diff() now refuses to compare two Inspect runs whose numeric scores were judged against different pass thresholds. The Inspect loader turns every finite numeric score into a pass or fail with the threshold given to load_results(path, threshold=...). So a baseline loaded at 0.8 and a current run loaded at 0.5 could report checks as improved, or let the gate pass, when the agent had not changed. The new check runs after the existing format and task checks. It raises ValueError('inputs are not comparable: numeric pass threshold 0.8 (baseline) != 0.5 (current); numeric Inspect scores were judged against different pass thresholds. Load both runs with the same threshold.') and returns no partial result. These comparisons behave as before: equal thresholds (1 and 1.0 count as equal), Inspect runs with only letter-grade, boolean or missing score values, a baseline run dict without a threshold key, and all promptfoo and JUnit comparisons. docs/ci-gate.md has a new section, 'Numeric pass threshold must match (Inspect)'. diff --git a/CITATION.cff b/CITATION.cff index c744a3c..83859e1 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -4,8 +4,8 @@ title: "EvalArc: Auditable Evaluations for AI Agents" type: software authors: - name: EvalArc contributors -version: 0.22.1 -date-released: 2026-10-28 +version: 0.23.0 +date-released: 2026-11-02 license: MIT repository-code: "https://github.com/noteflowai/evalarc" url: "https://noteflowai.github.io/evalarc/" diff --git a/README.md b/README.md index 3c09397..a73a647 100644 --- a/README.md +++ b/README.md @@ -68,13 +68,13 @@ and the upgrade gate fails. These checks share two format/schema failures. installs the published wheel, downloads the records and rebuilds the comparison. Verification exits 0 for consistency; comparison exits 1 for the regression. -Install the released reviewer from [PyPI](https://pypi.org/project/evalarc/0.22.1/) +Install the released reviewer from [PyPI](https://pypi.org/project/evalarc/0.23.0/) in a fresh virtual environment: ```bash python3 -m venv .venv . .venv/bin/activate -python -m pip install evalarc==0.22.1 +python -m pip install evalarc==0.23.0 evalarc --version ``` @@ -126,7 +126,7 @@ because checks missing from it cannot be compared; rerun that evaluation to completion and compare again. ```yaml -- uses: noteflowai/evalarc@v0.22.1 # or a full commit SHA +- uses: noteflowai/evalarc@v0.23.0 # or a full commit SHA with: baseline: evals/baseline.json current: results/current.json diff --git a/README.zh-CN.md b/README.zh-CN.md index 8859a33..12894f8 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -52,13 +52,13 @@ EvalArc 是面向 Agent 评测的 Python 命令行复核工具：逐项对比已 3. **本地复算。** [首次复核指南](docs/first-review.zh-CN.md)使用发布的 wheel 和原始记录重建报告； 复核退出 0 表示一致，对照退出 1 表示发现退步。 -在新的虚拟环境中，从 [PyPI](https://pypi.org/project/evalarc/0.22.1/) +在新的虚拟环境中，从 [PyPI](https://pypi.org/project/evalarc/0.23.0/) 安装已发布的复核工具： ```bash python3 -m venv .venv . .venv/bin/activate -python -m pip install evalarc==0.22.1 +python -m pip install evalarc==0.23.0 evalarc --version ``` @@ -105,7 +105,7 @@ Anthropic《Demystifying evals for AI agents》的建议在 EvalArc 中如何落 准确率从 0.625 升至 0.8125，同时有三个检查项丢失通过。 ```yaml -- uses: noteflowai/evalarc@v0.22.1 # 或完整的提交 SHA +- uses: noteflowai/evalarc@v0.23.0 # 或完整的提交 SHA with: baseline: evals/baseline.json current: results/current.json diff --git a/docs/ci-gate.md b/docs/ci-gate.md index 7f90abb..8ba5a19 100644 --- a/docs/ci-gate.md +++ b/docs/ci-gate.md @@ -84,8 +84,15 @@ Inputs: baseline `baseline.json` sha256 ``, current `current.json` - The `Inputs:` line appears only when both runs record a source name and digest. The full digests stay in `diff.json`. -The section only explains the result. It never changes the gate, the exit code, -`diff.json` or `index.html`. A summary with no blocking change has no such +`index.html`, the report the CLI prints after `Report:`, shows the same +section below the Changed checks table. It has the same columns, threshold +sentence and `Inputs:` digest line, built from the same rows. Each finding is +marked with a ✗ symbol as well as a color, and the table scrolls inside a +keyboard-focusable region. Unlike `summary.md`, the HTML lists every blocking +finding with no row limit, as its Changed checks table already does. + +The section only explains the result. It never changes the gate, the exit code +or `diff.json`. A summary or `index.html` with no blocking change has no such section and is the same as before. That covers a passing gate, an incomplete run with no changed checks, and a failure of the opt-in regression suite only. diff --git a/docs/ci-gate.zh-CN.md b/docs/ci-gate.zh-CN.md index 307675e..b2e8c3f 100644 --- a/docs/ci-gate.zh-CN.md +++ b/docs/ci-gate.zh-CN.md @@ -97,6 +97,14 @@ JUnit 用例）时，若至少一侧出现了通过与失败并存的实际波 每个检查项的尝试次数（Inspect `--epochs`、promptfoo `--repeat`）。这对应“收益必须超过 评测噪声才值得采纳”的做法。 +## 门禁失败原因 + +至少有一个变化阻断门禁时，`summary.md` 与 `index.html` 都会在变化表之后加上 +“Why the gate failed”一节：逐条给出阻断类型、是否在采样噪声内、第一次失败的已记录尝试 +（`removed` 行显示基线尝试，`less_covered` 行显示已评估次数），以及数值阈值和两份输入 +SHA-256 的前 12 位。两者使用同一组行；`summary.md` 沿用变化表的行数上限，`index.html` +列出全部阻断行。这一节只解释结果，不改变门禁、退出码或 `diff.json`；没有阻断变化时不出现。 + ## 留出集：改动是否泛化 若改动是针对部分评测用例调出来的，仅凭这些用例无法说明它在别处也有帮助。用一个小 JSON diff --git a/package-lock.json b/package-lock.json index 8871241..f9132ca 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "evalarc-evidence-site", - "version": "0.22.1", + "version": "0.23.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "evalarc-evidence-site", - "version": "0.22.1", + "version": "0.23.0", "devDependencies": { "@axe-core/playwright": "4.13.0", "playwright": "1.63.0", diff --git a/package.json b/package.json index 6b54a72..7f5a5f5 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "evalarc-evidence-site", - "version": "0.22.1", + "version": "0.23.0", "private": true, "description": "Browser checks and the TypeScript report front end for EvalArc", "scripts": { diff --git a/pyproject.toml b/pyproject.toml index 8e3a494..f311c83 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "evalarc" -version = "0.22.1" +version = "0.23.0" description = "Find agent regressions behind better scores: compare changed checks, review recorded tool actions and verify evidence offline." readme = "README.md" requires-python = ">=3.11" diff --git a/src/evalarc/__init__.py b/src/evalarc/__init__.py index 757c79d..f7d5d56 100644 --- a/src/evalarc/__init__.py +++ b/src/evalarc/__init__.py @@ -1,3 +1,3 @@ """Auditable evaluations for AI agents.""" -__version__ = "0.22.1" +__version__ = "0.23.0" diff --git a/src/evalarc/results_diff.py b/src/evalarc/results_diff.py index dbe552b..5e3cd39 100644 --- a/src/evalarc/results_diff.py +++ b/src/evalarc/results_diff.py @@ -295,7 +295,7 @@ def render_markdown(result: dict, limit: int = 50) -> str: def render_html(result: dict, destination: Path) -> None: - from evalarc.report import _card, _esc, _page, _tone, _verdict + from evalarc.report import _card, _esc, _page, _table, _tone, _verdict before, after = result["baseline"], result["current"] passed = ( @@ -388,6 +388,8 @@ def render_html(result: dict, destination: Path) -> None: body += "" else: body += "

Every recorded check has the same pass count.

" + # Empty unless a change blocks the gate, so other reports keep their exact HTML. + body += _blocking_html(result, _esc, _table, _tone) if result.get("generalization") or result.get("leakage"): from evalarc.generalization import render_html as split_html @@ -1353,16 +1355,30 @@ def _clip(value: object) -> str: return text -def _attempt_text(entry: object) -> str | None: - """'value V · evidence E' for one recorded attempt; absent parts are left out.""" +def _attempt_parts(entry: object) -> tuple | None: + """(value, evidence) of one recorded attempt, each clipped or None; None if neither.""" if not isinstance(entry, dict): return None + value = _clip(entry["value"]) if entry.get("value") is not None else None + evidence = _clip(entry["evidence"]) if entry.get("evidence") not in (None, "") else None + if value is None and evidence is None: + return None + return value, evidence + + +def _attempt_markdown_parts(value: str | None, evidence: str | None) -> str: parts = [] - if entry.get("value") is not None: - parts.append("value " + _cell(_clip(entry["value"]))) - if entry.get("evidence") not in (None, ""): - parts.append("evidence " + _cell(_clip(entry["evidence"]))) - return " · ".join(parts) or None + if value is not None: + parts.append("value " + _cell(value)) + if evidence is not None: + parts.append("evidence " + _cell(evidence)) + return " · ".join(parts) + + +def _attempt_text(entry: object) -> str | None: + """'value V · evidence E' for one recorded attempt; absent parts are left out.""" + parts = _attempt_parts(entry) + return _attempt_markdown_parts(*parts) if parts else None def _first_attempt(detail: object, failing: bool) -> dict | None: @@ -1374,25 +1390,53 @@ def _first_attempt(detail: object, failing: bool) -> dict | None: return None -def _failing_attempt(row: dict) -> str: - """The smallest recorded example that explains why one blocking row fails.""" +_REMOVED_PREFIX = "not recorded in the current run; baseline: " + + +def _attempt_record(row: dict) -> dict: + """Unescaped parts of the smallest recorded example for one blocking row. + + Returns {prefix, value, evidence, text}: text is set when no value/evidence pair + is shown. Malformed or missing detail falls back to text and never raises. + """ kind = row.get("kind") + prefix = None + text = None + parts = None if kind == "removed": - shown = _attempt_text(_first_attempt(row.get("baseline_detail"), failing=False)) - return "not recorded in the current run; baseline: " + (shown or _NO_ATTEMPT_DETAIL) - if kind == "less_covered": + prefix = _REMOVED_PREFIX + parts = _attempt_parts(_first_attempt(row.get("baseline_detail"), failing=False)) + elif kind == "less_covered": + text = _NO_ATTEMPT_DETAIL before, after = row.get("baseline"), row.get("current") if isinstance(before, dict) and isinstance(after, dict): old, new = before.get("assessed"), after.get("assessed") if isinstance(old, int) and isinstance(new, int): - return f"assessed {new} of {old} baseline attempts" - return _NO_ATTEMPT_DETAIL - shown = _attempt_text(_first_attempt(row.get("current_detail"), failing=True)) - return shown or _NO_ATTEMPT_DETAIL + text = f"assessed {new} of {old} baseline attempts" + else: + parts = _attempt_parts(_first_attempt(row.get("current_detail"), failing=True)) + if text is None and parts is None: + text = _NO_ATTEMPT_DETAIL + value, evidence = parts if parts else (None, None) + return {"prefix": prefix, "value": value, "evidence": evidence, "text": text} + + +def _attempt_markdown(record: dict) -> str: + body = ( + record["text"] + if record["text"] is not None + else _attempt_markdown_parts(record["value"], record["evidence"]) + ) + return (record["prefix"] or "") + body -def _provenance_line(result: dict) -> str | None: - """Digest prefixes of both inputs, or None when either digest was not recorded.""" +def _failing_attempt(row: dict) -> str: + """The smallest recorded example that explains why one blocking row fails.""" + return _attempt_markdown(_attempt_record(row)) + + +def _provenance_parts(result: dict) -> list | None: + """(label, name, digest prefix) for both inputs, or None when either was not recorded.""" parts = [] for label in ("baseline", "current"): summary = result.get(label) @@ -1404,41 +1448,123 @@ def _provenance_line(result: dict) -> str | None: return None if not _HEX_DIGEST.fullmatch(digest): return None - parts.append(f"{label} {_cell(name)} sha256 `{digest[:12]}`") - return "Inputs: " + ", ".join(parts) + parts.append((label, name, digest[:12])) + return parts + + +def _provenance_line(result: dict) -> str | None: + """Digest prefixes of both inputs, or None when either digest was not recorded.""" + parts = _provenance_parts(result) + if parts is None: + return None + return "Inputs: " + ", ".join( + f"{label} {_cell(name)} sha256 `{prefix}`" for label, name, prefix in parts + ) + + +_THRESHOLD_SENTENCE = ( + "Numeric scores pass at threshold {}; categorical verdicts use the source tool's pass mark." +) + + +def _threshold_text(result: dict) -> str: + """The numeric pass threshold as the headline formats it, or '—' when unusable.""" + threshold = result.get("numeric_pass_threshold") + numeric = isinstance(threshold, (int, float)) and not isinstance(threshold, bool) + return _value({"value": threshold}) if numeric and math.isfinite(threshold) else "—" + + +def _blocking_rows(result: dict) -> list: + """Unescaped parts of each blocking change, shared by summary.md and index.html.""" + rows = [] + for row in result.get("changes") or []: + if not (isinstance(row, dict) and row.get("kind") in BLOCKING): + continue + finding = row["kind"].replace("_", " ") + if row.get("within_sampling_noise") is True: + finding += " (within noise)" + rows.append( + { + "finding": finding, + "case_id": row.get("case_id"), + "check": row.get("check"), + "baseline": row.get("baseline"), + "current": row.get("current"), + "attempt": _attempt_record(row), + } + ) + return rows + + +def _blocking_html(result: dict, esc, table, tone) -> str: + """The index.html 'Why the gate failed' section, or '' when no change blocks.""" + rows = _blocking_rows(result) + if not rows: + return "" + cells = [] + for row in rows: + attempt = row["attempt"] + if attempt["text"] is not None: + shown = esc(attempt["text"]) + else: + parts = [] + if attempt["value"] is not None: + parts.append(f"value {esc(attempt['value'])}") + if attempt["evidence"] is not None: + parts.append(f"evidence {esc(attempt['evidence'])}") + shown = " · ".join(parts) + cells.append( + [ + tone(row["finding"], "bad"), + f"{esc(row['case_id'])} / {esc(row['check'])}", + esc(_fraction(row["baseline"])), + esc(_fraction(row["current"])), + esc(attempt["prefix"] or "") + shown, + ] + ) + out = ( + "

Why the gate failed

" + + esc(_THRESHOLD_SENTENCE.format(_threshold_text(result))) + + "

" + + table( + "Why the gate failed", + ["Finding", "Case / check", "Baseline", "Current", "Failing attempt"], + cells, + ) + ) + provenance = _provenance_parts(result) + if provenance: + out += ( + '

Inputs: ' + + ", ".join( + f"{esc(label)} {esc(name)} sha256 {esc(prefix)}" + for label, name, prefix in provenance + ) + + "

" + ) + return out def _blocking_markdown(result: dict, limit: int) -> list: """Lines explaining each blocking change, or [] so other summaries stay unchanged.""" - rows = [ - row - for row in result.get("changes") or [] - if isinstance(row, dict) and row.get("kind") in BLOCKING - ] + rows = _blocking_rows(result) if not rows: return [] - threshold = result.get("numeric_pass_threshold") - numeric = isinstance(threshold, (int, float)) and not isinstance(threshold, bool) - shown = _value({"value": threshold}) if numeric and math.isfinite(threshold) else "—" lines = [ "#### Why the gate failed", "", - f"Numeric scores pass at threshold {shown}; categorical verdicts use the source " - "tool's pass mark.", + _THRESHOLD_SENTENCE.format(_threshold_text(result)), "", "| Finding | Case / check | Baseline | Current | Failing attempt |", "| --- | --- | ---: | ---: | --- |", ] for row in rows[:limit]: - finding = row["kind"].replace("_", " ") - if row.get("within_sampling_noise") is True: - finding += " (within noise)" # Case and check share one cell: together they name the finding, and the row # never repeats the changes table's standalone case cell. lines.append( - f"| {finding} | {_cell(row.get('case_id'))} / {_cell(row.get('check'))} | " - f"{_fraction(row.get('baseline'))} | {_fraction(row.get('current'))} | " - f"{_failing_attempt(row)} |" + f"| {row['finding']} | {_cell(row['case_id'])} / {_cell(row['check'])} | " + f"{_fraction(row['baseline'])} | {_fraction(row['current'])} | " + f"{_attempt_markdown(row['attempt'])} |" ) if len(rows) > limit: lines.append(f"\n{len(rows) - limit} more blocking finding(s) are in `diff.json`.") diff --git a/tests/test_feature_e5db40f1ad73.py b/tests/test_feature_e5db40f1ad73.py new file mode 100644 index 0000000..14e9c83 --- /dev/null +++ b/tests/test_feature_e5db40f1ad73.py @@ -0,0 +1,144 @@ +"""index.html shows the same 'Why the gate failed' explanation as summary.md.""" + +import copy +import json +from pathlib import Path + +from evalarc.cli import main +from evalarc.results_diff import render_html + +EXAMPLES = Path(__file__).resolve().parents[1] / "examples/results-diff" +SECTION = "

Why the gate failed

" +REGION = '
' + + +def _run(tmp_path, baseline, current, name="out"): + output = tmp_path / name + code = main(["diff", str(baseline), str(current), "--output", str(output)]) + return code, output + + +def _inspect(tmp_path): + return _run(tmp_path, EXAMPLES / "inspect/baseline.json", EXAMPLES / "inspect/current.json") + + +def _section(html): + start = html.index(SECTION) + return html[start : html.index("

Inputs

", start)] + + +def _rows(fragment): + body = fragment.split("", 1)[1].split("", 1)[0] + return [row for row in body.split("") if row] + + +def test_index_html_explains_each_blocking_finding(tmp_path): + code, output = _inspect(tmp_path) + assert code == 1 + html = (output / "index.html").read_text(encoding="utf-8") + assert html.count(SECTION) == 1 + changed = html.index("

Changed checks

") + why = html.index(SECTION) + assert changed < why < html.index("

Inputs

") + section = _section(html) + assert "Numeric scores pass at threshold 1;" in section + rows = _rows(section) + assert len(rows) == 3 + for check in ("includes", "match"): + row = next(r for r in rows if f"refund-duplicate / {check}" in r) + assert '✗ regressed' in row + flaky = next(r for r in rows if "cancel-pending / match" in r) + assert '✗ less reliable (within noise)' in flaky + assert "value I · evidence cancel:1006:queued" in flaky + recorded = json.loads((output / "diff.json").read_text()) + for label in ("baseline", "current"): + prefix = recorded[label]["source"]["sha256"][:12] + assert f"{label} {label}.json sha256 {prefix}" in section + + +def test_summary_and_diff_json_are_unchanged(tmp_path): + _, output = _inspect(tmp_path) + summary = (output / "summary.md").read_text(encoding="utf-8") + assert ( + "| less reliable (within noise) | `cancel-pending` / `match` | 2/2 | 1/2 | " + "value `I` · evidence `cancel:1006:queued` |" + ) in summary.splitlines() + assert ( + "Numeric scores pass at threshold 1; categorical verdicts use the source tool's pass mark." + in summary + ) + recorded = json.loads((output / "diff.json").read_text()) + assert recorded["gate_passed"] is False + assert recorded["blocking_changes"] == 3 + + +def test_passing_gate_has_no_explanation(tmp_path): + path = EXAMPLES / "inspect/current.json" + code, output = _run(tmp_path, path, path) + assert code == 0 + assert "Why the gate failed" not in (output / "index.html").read_text(encoding="utf-8") + + +def test_html_escapes_clips_and_falls_back(tmp_path): + _, output = _inspect(tmp_path) + result = json.loads((output / "diff.json").read_text()) + template = next(r for r in result["changes"] if r["kind"] == "regressed") + + def synthetic(kind, case_id, check, **detail): + return { + "kind": kind, + "case_id": case_id, + "check": check, + "baseline": copy.deepcopy(template["baseline"]), + "current": copy.deepcopy(template["current"]), + **detail, + } + + long_value = "v" * 130 + result["changes"] += [ + synthetic( + "regressed", + "synthetic-escape", + "", + current_detail=[{"passed": False, "value": long_value, "evidence": "refund & ok"}], + ), + synthetic( + "removed", + "synthetic-removed", + "match", + baseline_detail=[{"passed": True, "value": "C", "evidence": "kept:1"}], + ), + synthetic("unassessed", "synthetic-empty", "match"), + ] + destination = tmp_path / "synthetic.html" + render_html(result, destination) + html = destination.read_text(encoding="utf-8") + section = _section(html) + assert "<script>x" in section + assert "refund & ok" in section + assert "" not in html + assert "value " + "v" * 120 + "…" in section + assert "v" * 121 not in html + assert ( + "not recorded in the current run; baseline: value C · " + "evidence kept:1" + ) in section + empty = next(r for r in _rows(section) if "synthetic-empty" in r) + assert "no attempt detail recorded" in empty + assert len(_rows(section)) == 6 + + +def test_explanation_table_is_an_accessible_region(tmp_path): + _, output = _inspect(tmp_path) + html = (output / "index.html").read_text(encoding="utf-8") + assert html.count(REGION) == 1 + assert html.count('aria-label="Why the gate failed"') == 1 + table = html[html.index(REGION) :] + table = table[: table.index("")] + assert table.count('') == 5 and "" not in table + for header in ("Finding", "Case / check", "Baseline", "Current", "Failing attempt"): + assert f'{header}' in table + rows = _rows(table) + assert rows + for row in rows: + assert row.startswith('✗ ')