diff --git a/CHANGELOG.md b/CHANGELOG.md index 51b2aaa..9d7bac8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # Changelog +## 0.23.1 — 2026-11-07 + +- fix: reject JUnit testcases without a name instead of merging them into one check. evalarc diff now rejects JUnit XML that contains a `` element whose name attribute is missing, empty or whitespace-only. Before this fix, the JUnit loader built IDs such as `corpus::None` or `corpus::` for these elements and merged every unnamed testcase in a suite into one check. A swapped failure ([fail, pass] in the baseline, [pass, fail] in the current run) then tallied 1/2 on both sides, was classified as unchanged, and the gate exited 0. Loading now raises a ValueError that names the 0-based position of the testcase among its suite's direct testcase children and the suite name, or (unnamed suite) when the suite has no name. evalarc diff reports this through its existing unusable-input path: it exits 2 and prints the message on stderr, whichever input file contains the unnamed testcase. Named testcases keep their exact IDs, verdicts and evidence. Identically named classname::name entries still load as repeated attempts of one check. docs/ci-gate.md and docs/ci-gate.zh-CN.md list this case among the exit-2 causes and give the recovery step. + ## 0.23.0 — 2026-11-02 - show the per-finding "Why the gate failed" explanation in the diff index.html. When a diff has at least one blocking change, `evalarc diff --output DIR` now adds a 'Why the gate failed' section to DIR/index.html, the report the CLI links to. The section sits after the Changed checks table and before the remaining sections. It contains the numeric threshold sentence and a table with these columns: Finding (kind, plus '(within noise)' when flagged, shown as a ✗ tone label so it does not rely on color), Case / check, Baseline, Current, and Failing attempt (the first failing current attempt's value and evidence, each clipped to 120 characters). Removed checks show the baseline attempt labelled 'not recorded in the current run; baseline:'. Rows with no detail show 'no attempt detail recorded'. A metadata line gives each input's file name and a 12-character sha256 prefix. Every value is HTML-escaped. summary.md and index.html now build this section from the same shared rows, so they cannot drift. summary.md text, diff.json and exit codes are unchanged. When nothing blocks, index.html is unchanged. diff --git a/CITATION.cff b/CITATION.cff index 83859e1..21d5c54 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -4,8 +4,8 @@ title: "EvalArc: Auditable Evaluations for AI Agents" type: software authors: - name: EvalArc contributors -version: 0.23.0 -date-released: 2026-11-02 +version: 0.23.1 +date-released: 2026-11-07 license: MIT repository-code: "https://github.com/noteflowai/evalarc" url: "https://noteflowai.github.io/evalarc/" diff --git a/README.md b/README.md index a73a647..d93b097 100644 --- a/README.md +++ b/README.md @@ -68,13 +68,13 @@ and the upgrade gate fails. These checks share two format/schema failures. installs the published wheel, downloads the records and rebuilds the comparison. Verification exits 0 for consistency; comparison exits 1 for the regression. -Install the released reviewer from [PyPI](https://pypi.org/project/evalarc/0.23.0/) +Install the released reviewer from [PyPI](https://pypi.org/project/evalarc/0.23.1/) in a fresh virtual environment: ```bash python3 -m venv .venv . .venv/bin/activate -python -m pip install evalarc==0.23.0 +python -m pip install evalarc==0.23.1 evalarc --version ``` @@ -126,7 +126,7 @@ because checks missing from it cannot be compared; rerun that evaluation to completion and compare again. ```yaml -- uses: noteflowai/evalarc@v0.23.0 # or a full commit SHA +- uses: noteflowai/evalarc@v0.23.1 # or a full commit SHA with: baseline: evals/baseline.json current: results/current.json diff --git a/README.zh-CN.md b/README.zh-CN.md index 12894f8..55630ba 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -52,13 +52,13 @@ EvalArc 是面向 Agent 评测的 Python 命令行复核工具：逐项对比已 3. **本地复算。** [首次复核指南](docs/first-review.zh-CN.md)使用发布的 wheel 和原始记录重建报告； 复核退出 0 表示一致，对照退出 1 表示发现退步。 -在新的虚拟环境中，从 [PyPI](https://pypi.org/project/evalarc/0.23.0/) +在新的虚拟环境中，从 [PyPI](https://pypi.org/project/evalarc/0.23.1/) 安装已发布的复核工具： ```bash python3 -m venv .venv . .venv/bin/activate -python -m pip install evalarc==0.23.0 +python -m pip install evalarc==0.23.1 evalarc --version ``` @@ -105,7 +105,7 @@ Anthropic《Demystifying evals for AI agents》的建议在 EvalArc 中如何落 准确率从 0.625 升至 0.8125，同时有三个检查项丢失通过。 ```yaml -- uses: noteflowai/evalarc@v0.23.0 # 或完整的提交 SHA +- uses: noteflowai/evalarc@v0.23.1 # 或完整的提交 SHA with: baseline: evals/baseline.json current: results/current.json diff --git a/docs/ci-gate.md b/docs/ci-gate.md index 8ba5a19..793c41d 100644 --- a/docs/ci-gate.md +++ b/docs/ci-gate.md @@ -138,7 +138,7 @@ Exit codes: **0** no check lost passes, **1** at least one did, **2** the files could not be read or compared (different formats, different Inspect task names, no samples, malformed input, an Inspect sample repeated in the same epoch, an Inspect sample that is not an object or has no integer or string id, a -promptfoo result row repeated). +promptfoo result row repeated, a JUnit testcase without a name). Treat 2 as a broken pipeline, not a regression. An Inspect log that contains the same sample ID twice in one epoch (an integer @@ -194,6 +194,23 @@ pass rates. Re-export with `promptfoo eval --output` (or remove the copied row in a reviewed commit) and compare again. Rows without an integer `testIdx` are not checked, and a copy whose `testIdx` was edited is not detected. +A JUnit `` whose `name` attribute is missing, empty or only +whitespace exits 2. The message names the suite and gives the testcase's +zero-based position among that suite's testcases. A suite without a name is +shown as `(unnamed suite)`: + +```text +evalarc: JUnit testcase at position 0 in suite 'corpus' has no name attribute; each testcase needs a name to be compared. Re-export the results with test names or fix the XML. +``` + +A testcase without a name cannot be paired across runs. Earlier versions gave +every unnamed testcase in a suite the same invented ID, such as +`corpus::None`. Those testcases merged into one check, so a failure that moved +from one unnamed testcase to another read as unchanged and the gate passed. +Re-export from the runner with test names, or fix the reporter so each +`` has a `name`, then compare again. Testcases with the same +non-empty classname and name still load as repeated attempts of one check. + ### Numeric pass threshold must match (Inspect) The Inspect loader turns each finite numeric score into a pass or a fail by diff --git a/docs/ci-gate.zh-CN.md b/docs/ci-gate.zh-CN.md index b2e8c3f..4e3bc24 100644 --- a/docs/ci-gate.zh-CN.md +++ b/docs/ci-gate.zh-CN.md @@ -47,7 +47,7 @@ epoch 中都对重复订单退款（`refund-duplicate` 退化），`cancel-pendi `regressed` 或 `less_reliable`。删除一个失败的检查项也会失败， 否则删掉它就能让门禁变绿。退出码：**0** 无检查项丢失通过，**1** 至少一项丢失， **2** 文件无法读取或无法比较（格式不同、Inspect 任务名不同、没有样本、格式错误、 -Inspect 样本在同一 epoch 中重复、Inspect 样本不是对象或没有整数/字符串 id、promptfoo 结果行重复）。同一 epoch 中重复出现的样本 ID（整数 `7` 与字符串 +Inspect 样本在同一 epoch 中重复、Inspect 样本不是对象或没有整数/字符串 id、promptfoo 结果行重复、JUnit testcase 没有 name）。同一 epoch 中重复出现的样本 ID（整数 `7` 与字符串 `"7"` 视为同一 ID）会以退出码 2 报告并给出样本与 epoch：若把副本算作额外尝试，会夸大覆盖并 掩盖 `less_covered`。这类日志通常来自手工合并、拼接或编辑；请用 `inspect log dump LOG.eval > LOG.json` 重新导出，或重新运行评测。同一 ID 出现在不同 epoch @@ -58,6 +58,7 @@ promptfoo `--output` 文件中同一结果行（相同的 `testIdx`、`promptIdx `testIdx`，因此重复运行仍按同一测试的多次尝试读取；`testIdx` 未变的行是合并、拼接或编辑留下的 副本，若计入会夸大尝试次数与通过率。请用 `promptfoo eval --output` 重新导出（或在经过审阅的 提交中删除副本行）后再比较。没有整数 `testIdx` 的行不做此检查，`testIdx` 被改过的副本也无法发现。 +JUnit `` 的 `name` 属性缺失、为空或只有空白时，同样以退出码 2 报告，并给出所属套件和该 testcase 在套件内从 0 开始的位置；没有名称的套件显示为 `(unnamed suite)`，例如 `JUnit testcase at position 0 in suite 'corpus' has no name attribute`。没有名称的 testcase 无法在两次运行间配对。此前同一套件中所有无名 testcase 会得到同一个虚构 ID（如 `corpus::None`）并合并为一个检查项，失败从一个无名 testcase 移到另一个时会被读作不变，门禁通过。请让测试运行器带测试名重新导出，或修复报告程序使每个 `` 都有 `name`，然后再比较。classname 与 name 都相同且非空的 testcase 仍按同一检查项的多次尝试读取。 2 表示流水线本身有问题，不是退化。 ## 回归语料 diff --git a/package-lock.json b/package-lock.json index f9132ca..bc0a1de 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "evalarc-evidence-site", - "version": "0.23.0", + "version": "0.23.1", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "evalarc-evidence-site", - "version": "0.23.0", + "version": "0.23.1", "devDependencies": { "@axe-core/playwright": "4.13.0", "playwright": "1.63.0", diff --git a/package.json b/package.json index 7f5a5f5..2a4f8bb 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "evalarc-evidence-site", - "version": "0.23.0", + "version": "0.23.1", "private": true, "description": "Browser checks and the TypeScript report front end for EvalArc", "scripts": { diff --git a/pyproject.toml b/pyproject.toml index f311c83..d6c472c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "evalarc" -version = "0.23.0" +version = "0.23.1" description = "Find agent regressions behind better scores: compare changed checks, review recorded tool actions and verify evidence offline." readme = "README.md" requires-python = ">=3.11" diff --git a/src/evalarc/__init__.py b/src/evalarc/__init__.py index f7d5d56..9d91754 100644 --- a/src/evalarc/__init__.py +++ b/src/evalarc/__init__.py @@ -1,3 +1,3 @@ """Auditable evaluations for AI agents.""" -__version__ = "0.23.0" +__version__ = "0.23.1" diff --git a/src/evalarc/results_diff.py b/src/evalarc/results_diff.py index 5e3cd39..1c37147 100644 --- a/src/evalarc/results_diff.py +++ b/src/evalarc/results_diff.py @@ -844,9 +844,20 @@ def _junit(raw: bytes) -> dict: names = set() for suite in suites: names.add(suite.get("name")) - for case in suite.findall("testcase"): + for position, case in enumerate(suite.findall("testcase")): + name = case.get("name") + if name is None or not name.strip(): + # An unnamed testcase cannot be paired across runs, and every unnamed + # testcase in a suite would merge into one check and hide a swap. + suite_name = suite.get("name") + shown = repr(suite_name) if suite_name else "(unnamed suite)" + raise ValueError( + f"JUnit testcase at position {position} in suite {shown} has no name " + "attribute; each testcase needs a name to be compared. Re-export the " + "results with test names or fix the XML." + ) owner = case.get("classname") or suite.get("name") or "" - case_id = f"{owner}::{case.get('name')}" if owner else str(case.get("name")) + case_id = f"{owner}::{name}" if owner else name outcome, evidence = True, None for tag, value in (("failure", False), ("error", None), ("skipped", None)): element = case.find(tag) diff --git a/tests/test_feature_5e40b6f9743a.py b/tests/test_feature_5e40b6f9743a.py new file mode 100644 index 0000000..127c0af --- /dev/null +++ b/tests/test_feature_5e40b6f9743a.py @@ -0,0 +1,100 @@ +"""SYNTHETIC JUnit identity cases: a testcase without a name cannot be compared. + +Every input is hand-written XML in tmp_path. The cases pin how evalarc diff treats +JUnit testcases whose name attribute is missing, empty or whitespace-only. They say +nothing about which real exporters emit such testcases. +""" + +import pytest + +from evalarc.cli import main +from evalarc.results_diff import diff, load_results + +MISSING = object() +HINT = ( + "each testcase needs a name to be compared. " + "Re-export the results with test names or fix the XML." +) + + +def _not_a_test(function): + # pytest collects module-level functions whose names start with "test"; this + # XML builder is a helper, not a test. + function.__test__ = False + return function + + +@_not_a_test +def testcase(name, failed): + attribute = "" if name is MISSING else f' name="{name}"' + body = '' if failed else "" + return f'{body}' + + +def suite_xml(cases, suite="corpus", wrap=False): + label = "" if suite is None else f' name="{suite}"' + body = "".join(testcase(name, failed) for name, failed in cases) + xml = f"{body}" + if wrap: + xml = f"{xml}" + return f'\n{xml}\n' + + +def write(path, text): + path.write_text(text) + return path + + +def message(position, suite="'corpus'"): + return f"JUnit testcase at position {position} in suite {suite} has no name attribute" + + +def test_swapped_failure_between_unnamed_testcases_exits_2(tmp_path, capsys): + # Before the fix both files tallied 1/2 for one merged check and the gate passed. + baseline = write(tmp_path / "baseline.xml", suite_xml([(MISSING, True), (MISSING, False)])) + current = write(tmp_path / "current.xml", suite_xml([(MISSING, False), (MISSING, True)])) + assert main(["diff", str(baseline), str(current)]) == 2 + err = capsys.readouterr().err + assert f"{message(0)}; {HINT}" in err + + +@pytest.mark.parametrize( + "blank", ["", " "], ids=["synthetic-empty-name", "synthetic-whitespace-name"] +) +def test_blank_name_is_rejected_on_either_side(tmp_path, capsys, blank): + named = write(tmp_path / "named.xml", suite_xml([("test_a", False), ("test_b", False)])) + broken = write(tmp_path / "broken.xml", suite_xml([("test_a", False), (blank, False)])) + with pytest.raises(ValueError) as error: + load_results(broken, "junit") + assert message(1) in str(error.value) + for pair in ((named, broken), (broken, named)): + assert main(["diff", *map(str, pair)]) == 2 + assert message(1) in capsys.readouterr().err + + +def test_unnamed_suite_under_testsuites_is_reported(tmp_path): + path = write(tmp_path / "suites.xml", suite_xml([(MISSING, False)], suite=None, wrap=True)) + with pytest.raises(ValueError, match=r"in suite \(unnamed suite\) has no name attribute"): + load_results(path, "junit") + + +def test_named_testcases_keep_their_ids_and_show_the_swap(tmp_path, capsys): + baseline = write(tmp_path / "baseline.xml", suite_xml([("test_a", True), ("test_b", False)])) + current = write(tmp_path / "current.xml", suite_xml([("test_a", False), ("test_b", True)])) + before, after = load_results(baseline, "junit"), load_results(current, "junit") + assert sorted(before["cases"]) == ["corpus::test_a", "corpus::test_b"] + result = diff(before, after) + assert sorted((row["kind"], row["case_id"]) for row in result["changes"]) == [ + ("improved", "corpus::test_a"), + ("regressed", "corpus::test_b"), + ] + assert main(["diff", str(baseline), str(current)]) == 1 + assert "no name attribute" not in capsys.readouterr().err + + +def test_identically_named_testcases_are_still_repeated_attempts(tmp_path): + path = write(tmp_path / "repeats.xml", suite_xml([("test_a", False), ("test_a", True)])) + loaded = load_results(path, "junit") + assert sorted(loaded["cases"]) == ["corpus::test_a"] + attempts = loaded["cases"]["corpus::test_a"]["passed"] + assert [attempt["passed"] for attempt in attempts] == [True, False]