[noteflowai]

An engineer gating with an Inspect .eval archive gets an unexplained error when a member of the archive is unreadable. The gate already refuses the input with exit 2, so missing evidence cannot pass. The message, however, is either a bare JSON decoder position or a Python TypeError such as 'list indices must be integers or slices, not str'. Neither tells the reviewer which file or member is broken, or what to do. EA-02 asks that malformed records be explained, not just rejected. The JSON-log path already does this; the .eval path does not. The roadmap is stale (updated 2026-09-29), so this work is limited to an existing input path and adds no capability.

Scope: only the zip (.eval) branch of _inspect_document in src/evalarc/results_diff.py, plus a short note in docs/ci-gate.md. JSON-format Inspect logs, promptfoo, JUnit, diff classification, the schema and all output files are unchanged.

Member display: every member name in these messages is formatted with the existing _shown_id helper, which the existing sample-id errors use for untrusted values. Its truncation length was not inspected, so the plan relies only on the property the tests check: no raw control character from a member name reaches stderr. The archive name stays path.name, unescaped, like the existing archive errors. It comes from the user's own command-line argument. Member content never appears in a message.

Parsing: a small helper reads each decoded member (header.json and each samples/*.json) and calls json.loads. On json.JSONDecodeError or UnicodeDecodeError it raises ValueError: "{archive} member {member} is not valid JSON: {decoder message}. Re-export the log from Inspect (inspect log dump LOG.eval > LOG.json) or rerun the evaluation instead of editing the archive." The helper does not catch NotImplementedError or BadZipFile, so the existing zstd guidance and BadZipFile message still apply.

Header type: if the parsed header.json is not a JSON object, the code raises ValueError before the samples are assigned: "{archive} member header.json must be a JSON object (got {type}): the archive is not a usable Inspect log. Re-export it from Inspect or rerun the evaluation." {type} comes from the existing _json_type helper.

Sample members that parse as JSON but are not objects keep the existing 'sample at position N is not an object' error from _inspect.

Unchanged: the declared-size bomb check, the missing header.json message, member ordering and how samples are collected.

CLI contract: when either input to evalarc diff BASELINE CURRENT [--output DIR] [--markdown FILE] [--json] has one of these defects, the command prints 'evalarc: ' to stderr, writes nothing to stdout, exits 2, creates no --output folder and appends no Markdown. Valid archives produce the same cases and exit codes as before.

Documentation: docs/ci-gate.md gets a short paragraph directly after the existing 'Exit codes' paragraph. It shows one example message for an invalid member and one for a non-object header.json, and gives the recovery step (inspect log dump LOG.eval > LOG.json, or rerun the evaluation).

Acceptance:

• A synthetic .eval archive whose header.json is not valid JSON makes main(["diff", baseline, archive, "--output", dir]) return 2, with the archive in the baseline position and in the current position. stderr contains the archive file name, 'header.json' and 'not valid JSON'. stderr contains no 'Traceback' and no 'Expecting value: line 1' without the archive name on the same line. The --output directory does not exist afterwards.
• A synthetic .eval archive whose header.json is a JSON array returns 2. stderr contains the archive name, 'header.json' and 'must be a JSON object (got array)'. stderr contains neither 'list indices must be integers' nor 'Traceback'. A string header gives 'got string' and no 'does not support item assignment'.
• A synthetic .eval archive with a valid header.json and a samples/refund_epoch_1.json member that is invalid UTF-8 returns 2. stderr names the archive and contains 'refund_epoch_1.json' and 'not valid JSON'. A second archive whose only sample member is named 'samples/bad\u001b[31m.json' and holds invalid JSON also returns 2. Its stderr contains no character in U+0000..U+001F or U+007F other than line endings, and still contains 'bad'.
• A valid synthetic .eval archive built from examples/results-diff/inspect/current.json loads with cases equal to the JSON log. Diffing examples/results-diff/inspect/baseline.json against it still returns 1. tests/test_results_diff.py and tests/test_feature_cb4fe5a9ab91.py pass unchanged.
• docs/ci-gate.md has a paragraph after the 'Exit codes' paragraph that contains the phrases 'member header.json must be a JSON object' and 'is not valid JSON', and the recovery command 'inspect log dump LOG.eval'. The new test checks these phrases.

One specialist feature. Behavioral acceptance fails on the base and passes on the implementation; project checks and independent model review passed. GPU results are included only when actually executed. Automatically developed and reviewed; no human review is claimed.