fix: name the archive and member when an Inspect .eval log has a malformed header.json or sample

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.23.2 — 2026-11-09
+
+- fix: name the archive and member when an Inspect .eval log has a malformed header.json or sample. evalarc diff now explains a malformed member in an Inspect .eval archive. The error names the archive, the member and the problem, plus a recovery step. Before, it printed a bare JSON decoder position or an internal Python TypeError. A header.json or samples/*.json member that is not valid UTF-8 JSON reports '<archive> member '<name>' is not valid JSON: <decoder message>'. A header.json that is not a JSON object reports 'must be a JSON object (got <type>)'. Member names are quoted and escaped like sample IDs, so control characters show as escapes such as \x1b. A displayed name longer than 80 characters is cut and ends in '...'. The command still exits 2, prints nothing to stdout, creates no --output folder and leaves the --markdown file untouched: an existing file keeps its bytes and a missing one is not created. Valid archives load exactly as before. docs/ci-gate.md and docs/ci-gate.zh-CN.md describe the messages.
+
 ## 0.23.1 — 2026-11-07
 
 - fix: reject JUnit testcases without a name instead of merging them into one check. evalarc diff now rejects JUnit XML that contains a `<testcase>` element whose name attribute is missing, empty or whitespace-only. Before this fix, the JUnit loader built IDs such as `corpus::None` or `corpus::` for these elements and merged every unnamed testcase in a suite into one check. A swapped failure ([fail, pass] in the baseline, [pass, fail] in the current run) then tallied 1/2 on both sides, was classified as unchanged, and the gate exited 0. Loading now raises a ValueError that names the 0-based position of the testcase among its suite's direct testcase children and the suite name, or (unnamed suite) when the suite has no name. evalarc diff reports this through its existing unusable-input path: it exits 2 and prints the message on stderr, whichever input file contains the unnamed testcase. Named testcases keep their exact IDs, verdicts and evidence. Identically named classname::name entries still load as repeated attempts of one check. docs/ci-gate.md and docs/ci-gate.zh-CN.md list this case among the exit-2 causes and give the recovery step.

CITATION.cff

@@ -4,8 +4,8 @@ title: "EvalArc: Auditable Evaluations for AI Agents"
 type: software
 authors:
   - name: EvalArc contributors
-version: 0.23.1
-date-released: 2026-11-07
+version: 0.23.2
+date-released: 2026-11-09
 license: MIT
 repository-code: "https://github.com/noteflowai/evalarc"
 url: "https://noteflowai.github.io/evalarc/"

README.md

@@ -68,13 +68,13 @@ and the upgrade gate fails. These checks share two format/schema failures.
    installs the published wheel, downloads the records and rebuilds the comparison.
    Verification exits 0 for consistency; comparison exits 1 for the regression.
 
-Install the released reviewer from [PyPI](https://pypi.org/project/evalarc/0.23.1/)
+Install the released reviewer from [PyPI](https://pypi.org/project/evalarc/0.23.2/)
 in a fresh virtual environment:
 
 ```bash
 python3 -m venv .venv
 . .venv/bin/activate
-python -m pip install evalarc==0.23.1
+python -m pip install evalarc==0.23.2
 evalarc --version
 ```
 
@@ -126,7 +126,7 @@ because checks missing from it cannot be compared; rerun that evaluation to
 completion and compare again.
 
 ```yaml
-- uses: noteflowai/evalarc@v0.23.1 # or a full commit SHA
+- uses: noteflowai/evalarc@v0.23.2 # or a full commit SHA
   with:
     baseline: evals/baseline.json
     current: results/current.json

README.zh-CN.md

@@ -52,13 +52,13 @@ EvalArc 是面向 Agent 评测的 Python 命令行复核工具：逐项对比已
 3. **本地复算。** [首次复核指南](docs/first-review.zh-CN.md)使用发布的 wheel 和原始记录重建报告；
    复核退出 0 表示一致，对照退出 1 表示发现退步。
 
-在新的虚拟环境中，从 [PyPI](https://pypi.org/project/evalarc/0.23.1/)
+在新的虚拟环境中，从 [PyPI](https://pypi.org/project/evalarc/0.23.2/)
 安装已发布的复核工具：
 
 ```bash
 python3 -m venv .venv
 . .venv/bin/activate
-python -m pip install evalarc==0.23.1
+python -m pip install evalarc==0.23.2
 evalarc --version
 ```
 
@@ -105,7 +105,7 @@ Anthropic《Demystifying evals for AI agents》的建议在 EvalArc 中如何落
 准确率从 0.625 升至 0.8125，同时有三个检查项丢失通过。
 
 ```yaml
-- uses: noteflowai/evalarc@v0.23.1 # 或完整的提交 SHA
+- uses: noteflowai/evalarc@v0.23.2 # 或完整的提交 SHA
   with:
     baseline: evals/baseline.json
     current: results/current.json

docs/ci-gate.md

@@ -141,6 +141,28 @@ Inspect sample that is not an object or has no integer or string id, a
 promptfoo result row repeated, a JUnit testcase without a name).
 Treat 2 as a broken pipeline, not a regression.
 
+An Inspect `.eval` archive with an unreadable member also exits 2, writes no
+`--output` folder and appends no Markdown. The message names the archive and
+the member. The member name is shown quoted and escaped the same way as sample
+IDs, so a control character never reaches the terminal (an ESC character
+appears as `\x1b`), and a displayed name longer than 80 characters is cut
+and ends in `...`. Inside an archive, member header.json must be a JSON object,
+and it and every `samples/*.json` member must be valid UTF-8 JSON. A member
+that is not valid UTF-8 JSON gives the first message below. A `header.json` that is not a JSON object gives the
+second:
+
+```text
+evalarc: current.eval member 'header.json' is not valid JSON: Expecting value: line 1 column 1 (char 0). Re-export the log from Inspect (inspect log dump LOG.eval > LOG.json) or rerun the evaluation instead of editing the archive.
+evalarc: current.eval member 'header.json' must be a JSON object (got array): the archive is not a usable Inspect log. Re-export it from Inspect or rerun the evaluation.
+```
+
+Earlier versions printed only the decoder position or an internal Python
+error, such as `list indices must be integers or slices, not str`. To recover,
+convert the archive with `inspect log dump LOG.eval > LOG.json`. If Inspect
+cannot read the archive either, rerun the evaluation. One bad member rejects the
+whole input. A sample member that parses but is not an object keeps the
+`sample at position N is not an object` message described below.
+
 An Inspect log that contains the same sample ID twice in one epoch (an integer
 `7` and a string `"7"` count as the same ID) exits 2 and names the sample
 and epoch. Counting the copy as an extra attempt would overstate coverage and

docs/ci-gate.zh-CN.md

@@ -47,7 +47,7 @@ epoch 中都对重复订单退款（`refund-duplicate` 退化），`cancel-pendi
 `regressed` 或 `less_reliable`。删除一个失败的检查项也会失败，
 否则删掉它就能让门禁变绿。退出码：**0** 无检查项丢失通过，**1** 至少一项丢失，
 **2** 文件无法读取或无法比较（格式不同、Inspect 任务名不同、没有样本、格式错误、
-Inspect 样本在同一 epoch 中重复、Inspect 样本不是对象或没有整数/字符串 id、promptfoo 结果行重复、JUnit testcase 没有 name）。同一 epoch 中重复出现的样本 ID（整数 `7` 与字符串
+Inspect 样本在同一 epoch 中重复、Inspect 样本不是对象或没有整数/字符串 id、promptfoo 结果行重复、JUnit testcase 没有 name、Inspect `.eval` 归档成员不是有效 JSON 或其 header.json 不是对象）。同一 epoch 中重复出现的样本 ID（整数 `7` 与字符串
 `"7"` 视为同一 ID）会以退出码 2 报告并给出样本与 epoch：若把副本算作额外尝试，会夸大覆盖并
 掩盖 `less_covered`。这类日志通常来自手工合并、拼接或编辑；请用
 `inspect log dump LOG.eval > LOG.json` 重新导出，或重新运行评测。同一 ID 出现在不同 epoch
@@ -59,6 +59,14 @@ promptfoo `--output` 文件中同一结果行（相同的 `testIdx`、`promptIdx
 副本，若计入会夸大尝试次数与通过率。请用 `promptfoo eval --output` 重新导出（或在经过审阅的
 提交中删除副本行）后再比较。没有整数 `testIdx` 的行不做此检查，`testIdx` 被改过的副本也无法发现。
 JUnit `<testcase>` 的 `name` 属性缺失、为空或只有空白时，同样以退出码 2 报告，并给出所属套件和该 testcase 在套件内从 0 开始的位置；没有名称的套件显示为 `(unnamed suite)`，例如 `JUnit testcase at position 0 in suite 'corpus' has no name attribute`。没有名称的 testcase 无法在两次运行间配对。此前同一套件中所有无名 testcase 会得到同一个虚构 ID（如 `corpus::None`）并合并为一个检查项，失败从一个无名 testcase 移到另一个时会被读作不变，门禁通过。请让测试运行器带测试名重新导出，或修复报告程序使每个 `<testcase>` 都有 `name`，然后再比较。classname 与 name 都相同且非空的 testcase 仍按同一检查项的多次尝试读取。
+Inspect `.eval` 归档中有成员无法读取时，同样以退出码 2 报告，不创建 `--output` 目录，也不追加 Markdown。错误信息给出归档文件名和成员名；成员名与样本 ID 一样加引号并转义显示，控制字符不会原样输出到终端（ESC 显示为 `\x1b`），显示长度超过 80 个字符的名称会被截断并以 `...` 结尾。`header.json` 或 `samples/*.json` 成员不是有效的 UTF-8 JSON 时给出下面第一条信息，`header.json` 不是 JSON 对象时给出第二条：
+
+```text
+evalarc: current.eval member 'header.json' is not valid JSON: Expecting value: line 1 column 1 (char 0). Re-export the log from Inspect (inspect log dump LOG.eval > LOG.json) or rerun the evaluation instead of editing the archive.
+evalarc: current.eval member 'header.json' must be a JSON object (got array): the archive is not a usable Inspect log. Re-export it from Inspect or rerun the evaluation.
+```
+
+此前只会输出解码位置或 Python 内部错误（如 `list indices must be integers or slices, not str`）。恢复方法：用 `inspect log dump LOG.eval > LOG.json` 转换归档；若 Inspect 也无法读取，则重新运行评测。一个损坏的成员会使整个输入被拒绝。能解析但不是对象的样本成员仍报告 `sample at position N is not an object`。
 2 表示流水线本身有问题，不是退化。
 
 ## 回归语料

package.json

@@ -1,6 +1,6 @@
 {
   "name": "evalarc-evidence-site",
-  "version": "0.23.1",
+  "version": "0.23.2",
   "private": true,
   "description": "Browser checks and the TypeScript report front end for EvalArc",
   "scripts": {

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "evalarc"
-version = "0.23.1"
+version = "0.23.2"
 description = "Find agent regressions behind better scores: compare changed checks, review recorded tool actions and verify evidence offline."
 readme = "README.md"
 requires-python = ">=3.11"

src/evalarc/__init__.py

@@ -1,3 +1,3 @@
 """Auditable evaluations for AI agents."""
 
-__version__ = "0.23.1"
+__version__ = "0.23.2"

src/evalarc/results_diff.py

@@ -448,6 +448,19 @@ def _json(raw: bytes, path: Path) -> object:
         raise ValueError(f"{path.name} is not JSON: {error}") from None
 
 
+def _archive_json(path: Path, archive: zipfile.ZipFile, item: zipfile.ZipInfo) -> object:
+    """Parse one .eval member; on failure name the archive and the escaped member."""
+    raw = archive.read(item)
+    try:
+        return json.loads(raw)
+    except (UnicodeDecodeError, json.JSONDecodeError) as error:
+        raise ValueError(
+            f"{path.name} member {_shown_id(item.filename)} is not valid JSON: {error}. "
+            "Re-export the log from Inspect (inspect log dump LOG.eval > LOG.json) or rerun "
+            "the evaluation instead of editing the archive."
+        ) from None
+
+
 def _inspect_document(path: Path, raw: bytes) -> dict:
     if raw[:4] != b"PK\x03\x04":
         document = _json(raw, path)
@@ -462,9 +475,16 @@ def _inspect_document(path: Path, raw: bytes) -> dict:
             header = PurePosixPath("header.json")
             if header not in members:
                 raise ValueError(f"{path.name} has no header.json; the run may be incomplete")
-            document = json.loads(archive.read(members[header]))
+            document = _archive_json(path, archive, members[header])
+            if not isinstance(document, dict):
+                shown = _shown_id(members[header].filename)
+                raise ValueError(
+                    f"{path.name} member {shown} must be a JSON object "
+                    f"(got {_json_type(document)}): the archive is not a usable Inspect log. "
+                    "Re-export it from Inspect or rerun the evaluation."
+                )
             document["samples"] = [
-                json.loads(archive.read(item))
+                _archive_json(path, archive, item)
                 for name, item in sorted(members.items())
                 if name.parent == PurePosixPath("samples") and name.suffix == ".json"
             ]

tests/test_feature_1e31601a8900.py

@@ -0,0 +1,162 @@
+"""SYNTHETIC .eval archives: a malformed member exits 2 and names the archive and member.
+
+Every archive is built here from the bundled Inspect example or from hand-written
+bytes. The cases pin how evalarc diff reports these defects. They say nothing about
+which corruptions real Inspect exports produce.
+"""
+
+import json
+import zipfile
+from pathlib import Path
+
+import pytest
+
+from evalarc.cli import main
+from evalarc.results_diff import load_results
+
+ROOT = Path(__file__).resolve().parents[1]
+EXAMPLES = ROOT / "examples/results-diff"
+JSON_LOG = EXAMPLES / "inspect/current.json"
+POSITIONS = pytest.mark.parametrize("archive_first", [True, False], ids=["baseline", "current"])
+
+
+def header_and_samples():
+    document = json.loads(JSON_LOG.read_text())
+    samples = document.pop("samples")
+    return document, samples
+
+
+def build(path, header, members=()):
+    with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as output:
+        output.writestr("header.json", header)
+        for name, body in members:
+            output.writestr(name, body)
+    return path
+
+
+def run_diff(tmp_path, capsys, archive, archive_first=False, summary=None):
+    """Run evalarc diff with --output and --markdown and check nothing was written.
+
+    summary: bytes to pre-create the --markdown file with, or None to leave it absent.
+    """
+    pair = (archive, JSON_LOG) if archive_first else (JSON_LOG, archive)
+    output = tmp_path / "review"
+    markdown = tmp_path / "job-summary.md"
+    if summary is not None:
+        markdown.write_bytes(summary)
+    args = ["diff", str(pair[0]), str(pair[1]), "--output", str(output)]
+    code = main(args + ["--markdown", str(markdown)])
+    captured = capsys.readouterr()
+    assert code == 2, captured.err
+    assert captured.out == ""
+    assert not output.exists()
+    if summary is None:
+        assert not markdown.exists()
+    else:
+        assert markdown.read_bytes() == summary
+    assert "Traceback" not in captured.err
+    assert f"evalarc: {archive.name} member " in captured.err
+    return captured.err
+
+
+@POSITIONS
+def test_invalid_json_header_names_the_archive_and_member(tmp_path, capsys, archive_first):
+    archive = build(tmp_path / "broken-header.eval", "not json")
+    existing = b"## Earlier step\n\nkept as is\n"
+    err = run_diff(tmp_path, capsys, archive, archive_first, summary=existing)
+    assert f"{archive.name} member 'header.json' is not valid JSON" in err
+    assert "inspect log dump LOG.eval > LOG.json" in err
+    decoder_lines = [line for line in err.splitlines() if "Expecting value: line 1" in line]
+    assert decoder_lines
+    assert all(archive.name in line for line in decoder_lines)
+
+
+@POSITIONS
+@pytest.mark.parametrize(
+    "header,kind,internal",
+    [
+        ("[1, 2]", "array", "list indices must be integers"),
+        ('"text"', "string", "does not support item assignment"),
+        ("7", "number", "does not support item assignment"),
+        ("null", "null", "does not support item assignment"),
+    ],
+    ids=["array", "string", "number", "null"],
+)
+def test_non_object_header_is_explained(tmp_path, capsys, archive_first, header, kind, internal):
+    archive = build(tmp_path / "odd-header.eval", header)
+    err = run_diff(tmp_path, capsys, archive, archive_first)
+    assert f"{archive.name} member 'header.json' must be a JSON object (got {kind})" in err
+    assert "Re-export it from Inspect or rerun the evaluation" in err
+    assert internal not in err
+
+
+def test_invalid_utf8_sample_names_the_member(tmp_path, capsys):
+    header = json.dumps(header_and_samples()[0])
+    members = [("samples/refund_epoch_1.json", b'{"id": "\xc3\x28"}')]
+    archive = build(tmp_path / "bad-sample.eval", header, members)
+    err = run_diff(tmp_path, capsys, archive)
+    assert "member 'samples/refund_epoch_1.json' is not valid JSON" in err
+
+
+def test_member_name_control_characters_are_escaped(tmp_path, capsys):
+    header = json.dumps(header_and_samples()[0])
+    members = [("samples/bad\x1b[31m.json", "not json")]
+    archive = build(tmp_path / "escape.eval", header, members)
+    err = run_diff(tmp_path, capsys, archive)
+    assert "not valid JSON" in err
+    assert "bad" in err
+    assert "member 'samples/bad\\x1b[31m.json' is not valid JSON" in err
+    raw = [ch for ch in err if (ord(ch) < 32 or ord(ch) == 127) and ch not in "\r\n"]
+    assert raw == []
+
+
+def test_long_member_name_is_cut_at_80_characters(tmp_path, capsys):
+    header = json.dumps(header_and_samples()[0])
+    name = "samples/" + "x" * 100 + ".json"
+    archive = build(tmp_path / "long.eval", header, [(name, "not json")])
+    err = run_diff(tmp_path, capsys, archive, summary=b"")
+    shown = "'samples/" + "x" * 68 + "..."
+    assert len(shown) == 80
+    assert f"member {shown} is not valid JSON" in err
+    assert name not in err
+
+
+def test_parsed_non_object_sample_keeps_the_position_error(tmp_path, capsys):
+    header = json.dumps(header_and_samples()[0])
+    archive = build(tmp_path / "array-sample.eval", header, [("samples/a.json", "[1]")])
+    assert main(["diff", str(JSON_LOG), str(archive)]) == 2
+    assert "sample at position 0 is not an object (got array)" in capsys.readouterr().err
+
+
+def test_valid_archive_still_loads_like_the_json_log(tmp_path, capsys):
+    header, samples = header_and_samples()
+    members = [
+        (f"samples/{item['id']}_epoch_{item['epoch']}.json", json.dumps(item)) for item in samples
+    ]
+    archive = build(tmp_path / "current.eval", json.dumps(header), members)
+    assert load_results(archive)["cases"] == load_results(JSON_LOG)["cases"]
+    assert main(["diff", str(EXAMPLES / "inspect/baseline.json"), str(archive)]) == 1
+    assert "regressed: refund-duplicate / match" in capsys.readouterr().out
+
+
+def test_ci_gate_docs_explain_the_archive_messages():
+    text = (ROOT / "docs/ci-gate.md").read_text(encoding="utf-8")
+    start = text.index("Exit codes: **0**")
+    for phrase in (
+        "member header.json must be a JSON object",
+        "member 'header.json' is not valid JSON",
+        "member 'header.json' must be a JSON object (got array)",
+        "is not valid JSON",
+        "inspect log dump LOG.eval",
+    ):
+        assert text.find(phrase, start) > start, phrase
+
+
+def test_zh_ci_gate_docs_explain_the_archive_messages():
+    text = (ROOT / "docs/ci-gate.zh-CN.md").read_text(encoding="utf-8")
+    for phrase in (
+        "member 'header.json' is not valid JSON",
+        "member 'header.json' must be a JSON object (got array)",
+        "inspect log dump LOG.eval > LOG.json",
+    ):
+        assert phrase in text, phrase