sieve: an implementation that cannot be run is not a failing implementation

[leonacostaok]

Not for auto-merge — pqc-tools merges by hand.

What went wrong

A conformance run on dandelionlabs-io/leonacostaok-blog reported this:

Sieve ml-kem-768: FAIL - 35 failing check(s)
  high · correctness/roundtrip[0]        harness error: SUT exited with code 1
  high · implicit-rejection/trial[0]     harness error (possible crash on bad ct): SUT exited with code 1
  high · sizes/decaps-ct-too-short       SUT crashed/hung on a wrong-length input ...
  ... 32 more

Every one of those is false. --impl pointed at node ./my-impl.js, which does not exist in that repo, so nothing was ever tested. The report named AF-02 and AF-05 bug classes against code that never executed, and the 35 details between them did not contain the words "cannot find module".

Reproduce on main:

cd "$(mktemp -d)" && npx @quantakrypto/sieve --impl "node ./nope.js" --param ml-kem-768

What this changes

A new ERROR verdict, decided from the response count. The runner already tracks every protocol response it decodes. Zero of them at the end of a run means the SUT never did anything the battery could judge, so every failure recorded against it is an artefact. Those results are replaced by a single harness category and overall: "ERROR".

ERROR outranks FAIL because if the implementation never ran, nothing else in the report is a statement about it.

Two cases are deliberately not errors, and both are tested:

• A SUT that replies {"ok": false} to everything. It is alive and speaking the protocol, so its refusals are a genuine conformance signal and the full battery runs.
• A SUT that dies part-way. By then it had answered, so its recorded failures are real.

This costs nothing on a healthy run: it reads a counter rather than spending a probe request. (An earlier revision of this PR did issue a preflight keygen; that was wasted work and is gone.)

The cause is now reported. SutCrashError already captured the child's stderr, but all 14 category call sites discarded it and reported (err as Error).message, which is only ever exited with code N. They now go through a new exported describeSutError.

Picking the right stderr line needs care: Node buries Error: Cannot find module between an internal loader frame and its version banner, while Python opens with Traceback and puts the exception last. Neither end alone works, so it anchors on the first and last lines that name a failure:

node     SUT exited with code 1: Error: Cannot find module '/repo/my-impl.js'
python   SUT exited with code 1: Traceback (most recent call last): | RuntimeError: boom in my kem
missing  failed to spawn SUT: spawn no-such-binary ENOENT (no stderr)
hangs    SUT started but never returned a protocol response (every request timed out)

The last of those is the one case with no crash and no stderr. It says so plainly rather than dressing up silence as a diagnosis.

Same run as the bug report, after:

[FAIL] harness - the implementation under test could not be run, so no conformance checks were performed
      - FAIL sut-startup: SUT exited with code 1: Error: Cannot find module '/repo/my-impl.js'

checks: 0 pass, 1 fail, 0 skip
OVERALL: ERROR

35 findings to 1, and it names the file.

Known trade-off

A SUT that spawns but never replies still times out on every probe before the run concludes, which with the default 10s timeout and 32 iterations is slow. That is unchanged from main — it is not a regression, just not something this PR fixes. A crashed process is unaffected: it rejects immediately, so the whole run takes ~170ms.

Compatibility

Additive, SemVer minor. SieveReport.overall gains a third value; three symbols are exported (describeSutError, HARNESS_CATEGORY, Verdict); no existing field changes shape.

CLI exit codes are unchanged. Only PASS exits 0, so ERROR exits 1 exactly as FAIL did and existing CI gates behave identically. Consumers that want to tell them apart read .overall. I chose not to add a distinct exit code for that reason — happy to if you would rather have one.

Verified

npm run build, npm run lint, npm test across all 8 packages: 1257 tests, 0 failures. docs/API.md and docs/api-surface.json regenerated for the three new exports.

New packages/sieve/test/unrunnable-sut.test.ts covers: ERROR plus a single harness check and no invented bug classes; the detail naming the missing module; a live-but-broken SUT still getting a normal FAIL verdict; a hung SUT reporting the silence honestly; ERROR outranking FAIL; the Node-vs-Python stderr selection; and a guard that a run needing no requests sends none, so nothing probes the SUT speculatively.

Related

The website ingests Sieve reports and had to be defended separately, since it must keep working against published Sieve versions that predate this: quantakrypto/website#115 treats counts.pass == 0 as "never ran", records it as failed rather than a verdict, and keeps it out of badges and the posture series.