quad-cli-consensus-gate: cursor/agy fail to return schema-valid JSON in live runs (2/4 effective reviewers)

[drvoss]

Context

Follow-up from the multi-CLI refresh (PR #7, merged). Live dogfooding of \scripts/quad-cli-orchestrate.mjs\ against a real diff succeeded end-to-end (0 blocking, 2 advisory, exit 0, merged report schema-valid) but only 2 of 4 CLI reviewers (claude, codex) returned schema-valid JSON. cursor-agent and agy errored out of the run, degrading the gate from 4-way to 2-way consensus in practice.

Problem

The gate's core logic (hunk anchoring, family-vote de-duplication, advisory/blocking split) is correct and well-tested (1062 tests passing, live-verified). But its real-world effectiveness depends on all 4 CLIs reliably emitting valid JSON per \schemas/quad-cli-report.json, and today only 2 do.

Suggested follow-up

1. Enforce structured output per-CLI where supported (e.g. \cursor-agent --output-format json, verify agy's equivalent flag via --help).
2. Strengthen JSON coercion/extraction in \quad-cli-orchestrate.mjs\ for CLIs that wrap JSON in prose or partial output.
3. Add a live (non-mock) smoke test or periodic dogfood check to catch regressions in per-CLI output format compliance, since this class of issue is invisible to mock-mode tests and grep and was only caught by live execution.

Not a merge blocker for #7 — tracked here per external review recommendation to keep gate robustness work separate from the docs/migration merge.

[drvoss]

Status audit after the 2026-08-05 monthly cycle (main at 0295e46): this issue remains open and reproducible evidence of resolution has not been recorded.

Current implementation still invokes cursor-agent -f -p and agy -p without a structured-output option. JSON recovery remains limited to direct parsing or extracting from the first { through the last }. The monthly cycle did not change these paths.

Close criteria:

• run a live, non-mock four-CLI smoke test;
• record schema-valid output from all four reviewers, or explicitly revise the supported-reviewer contract;
• add regression coverage or a periodic live dogfood check for per-CLI output-format compliance.

This issue is distinct from planning-work-order carry-over item 9, which was marked resolved during the monthly review.

[drvoss]

Resolved — the root cause was invocation/transport, not model output compliance

This issue's diagnosis was inverted, and that is why it stayed open. cursor-agent and agy were not
"failing to return schema-valid JSON" — they were never successfully executed.

What was actually broken

Defect	Evidence
agy -p takes the prompt as the flag's value, not a boolean. The orchestrator called agy -p and piped the prompt to stdin, so agy exited 2 with flag needs an argument: -p and produced 0 bytes. It had never run.	deterministic on every invocation
On Windows cursor-agent installs as cursor-agent.ps1 + cursor-agent.cmd, not an .exe. spawn("cursor-agent", args) with no shell is ENOENT — Node does not do PATHEXT resolution.	reproduced on two machines
Spawning a full-path .cmd with no shell throws EINVAL synchronously (Node's CVE-2024-27980 mitigation), so an uncaught spawn crashes instead of classifying the failure.	reproduced on Node v22.23.1 and v26.4.0
This is not cursor-specific. On a second machine where codex was installed as an npm shim, spawn("codex") was ENOENT too — that machine produced 0 valid reviewers, not 2.	measured
Failed runners' stdout/stderr were discarded. The gate printed only 2 valid reviewers, 2 errored reviewers with no reason, which is why this was never diagnosable from logs.	code path
The default 120s timeout was below the slowest successful reviewer run (218.8s).	measured

Why suggestion (1) in the original report should not be implemented

Forcing per-CLI structured-output flags makes things worse:

• codex --output-schema schemas/quad-cli-report.json fails with HTTP 400. First
  In context=('properties','schema_version'), schema must have a 'type' key; after patching that, a second,
  structural error: 'required' is required to be supplied and to be an array including every key in properties. Missing 'line'. Our line/snippet are optional, so the schema is incompatible by design,
  not by a typo.
• agy --json-schema then returns the required fields but injects extra top-level keys (toolAction,
  toolSummary) that the strict validator rejects.
• cursor-agent --output-format json returns an envelope ({"type":"result","result":"<report as a JSON string>",...}), so adding the flag without teaching the extractor to unwrap .result would have made
  cursor-agent fail 100% of the time.

Each CLI's default, unflagged mode already produced schema-valid output once it was invoked correctly.

Compatibility matrix (20 rows)

Prompts at 3 KB / 23 KB / 126 KB. Each prompt carried a nonce at the very top and a different nonce at the
very bottom, and the reviewer had to echo both — separating "schema-valid" from "actually received the whole
prompt". The synthetic diff deliberately contained prose with braces, a markdown fence, Hangul, emoji,
% ! ^ &, and a planted fake report line
({"schema_version":"1","generated_by":"quad-cli-orchestrate","findings":[]}).

17 of 20 rows: schema-valid with both nonces intact. The three failures were all informative:

• agy argv at 126 KB → ENAMETOOLONG at spawn time. The boundary sits between 29,856 and 32,264 UTF-16 code
  units of the rendered command line, consistent with the Windows 32,767 CreateProcess limit.
  No row anywhere produced exit 0 + schema-valid + missing tail nonce — silent truncation did not occur.
• agy's file-reference transport read a 126 KB prompt in full (tail nonce returned), so large prompts are
  a solved problem for agy rather than a reason to demote it.
• braceSliceNeeded was false in every row: direct JSON.parse, or one envelope unwrap, always sufficed.

Fix (#14)

• Per-tool adapter table with an explicit launcher resolution layer: .exe spawned directly, .cmd/.bat
  via cmd.exe /d /s /c, .ps1 via powershell.exe -NoProfile -ExecutionPolicy Bypass -File, resolution
  order .exe → .cmd → .bat → .ps1, and spawn wrapped in try/catch. No shell: true.
• agy uses argv below a conservative rendered-command-line threshold and an isolated file-reference transport
  above it, with a one-shot fallback on ENAMETOOLONG.
• Strict extraction ladder (UTF-8, one BOM, ANSI stripping, exact parse, one complete fence, allow-listed
  envelopes whose success field is checked first). The first-{-to-last-} fallback was removed — it is
  the only path by which a planted or truncated fragment can enter the consensus vote as a real finding.
• Unknown top-level keys are now recorded instead of discarding the whole report; a malformed finding
  still invalidates the report. Reviewer identity comes from the adapter, never from model-supplied
  generated_by.
• A small failure taxonomy with raw exit code / signal / spawn error preserved, and one sanitized stderr line
  per tool, so a degraded run is never silent again.
• Best-effort process-tree termination — a single-process kill leaves the real CLI running behind a
  cmd.exe/powershell.exe wrapper. Documented as best-effort, not guaranteed containment.
• Per-tool timeouts with config/env override.
• 19 regression tests that actually spawn processes, using stub CLIs. The previous suite ran entirely in a
  mock mode that bypassed spawnRunner, which is precisely why this class of defect was invisible.

Live four-CLI run (close criteria 1 and 2)

Run on the only machine with all four CLIs installed, against the PR head commit
ea3dd250176859ab363234b3fc8f087638f7594e, Windows 11, Node v26.4.0,
claude 2.1.238 / codex-cli 0.146.1 / cursor-agent 2026.08.11 build / agy 1.1.17. Two runs, both
4 valid reviewers, 0 errored reviewers, reviewers_effective = 4:

tool=claude       status=ok exit=0 ms=19165 launcher=native-exe findings=7
tool=codex        status=ok exit=0 ms=55927 launcher=native-exe findings=4
tool=cursor-agent status=ok exit=0 ms=45079 launcher=cmd-shim   findings=5
tool=agy          status=ok exit=0 ms=43466 launcher=native-exe findings=3

The second run produced a BLOCKING finding — security-hardcoded-secret, families
['openai-gpt','cursor-native'], effective_votes: 2, contributors ['codex','cursor-agent']. Note this is a
two-family agreement, which is exactly what the gate's blocking rule requires; it is not four-way agreement on
one finding. That particular pair could not form before this fix, because cursor-agent died at spawn. These
are the first four-reviewer runs recorded in this investigation.

Note for operators: raising effective reviewers from 2 to 4 creates more model-family pairs, so the gate
becomes stricter, not merely healthier. The same diff exited 0 on one run and 1 on the next.

Scope, and what is deliberately left open

Measured on Windows only (Node v22.23.1 and v26.4.0, 2026-08-20, CLI versions above). macOS and Linux are
untested. agy's large-prompt path relies on a natural-language file-reference instruction rather than a
native prompt-file flag, and costs significantly more wall time. The measurements establish that the failures
reported here were caused by invocation and transport; they do not establish that model output can never be
malformed, which is why the strict extraction and validation path stays in place.

Close criterion 3 asks for regression coverage or a periodic live dogfood check. The 19 real-spawn tests
satisfy the regression half for our own launcher and parsing logic, but they use stub CLIs and therefore
cannot detect drift in the real, independently updated CLIs. That remainder is tracked in #15,
together with the production transport nonce and a --min-reviewers / supported-reviewer contract revision.
Closing this issue on criteria 1 and 2, with criterion 3's live half explicitly handed to #15.