Skip to content

axi respond reports i/o timeout for a ruling the daemon already executed; adjudicators re-send and double-apply #8

Description

@Blakeolson21

Found while adjudicating the parked gate estate on 2026-08-27 (Austin). Every one of three no-slop axi respond --action fix calls returned a non-zero exit with read unix ->~/.no-mistakes/socket: i/o timeout, at two different points (classify gate execution context: read response: ... and respond to review: read response: ...). All three rulings had in fact been ACCEPTED AND EXECUTED by the daemon. board24-shorts-policy advanced to review,fixing round "fix 4" with the head moving c57edc34 -> 42855aa2 and the reviewer log quoting the exact instruction text from the call that had reported failure; hooks-officer-sweep-1 did the same. The client reported failure for work that succeeded.

Mechanism, from ~/.no-mistakes/logs/daemon.log: the daemon logs ipc request method=respond and proceeds, then logs ERROR write response: write unix ...socket->: write: broken pipe because the client already hit its read deadline and disconnected. Under estate load these broken-pipe errors are continuous. Cheap calls are unaffected - axi status returns in about 1.9s throughout - so this is not a wedged daemon. The expensive path is classify gate execution context, which per the binary's strings calls list active runs and list steps for active run; with roughly 30 concurrent runs that enumeration exceeds the client read deadline while the mutation behind it still commits.

Why this is worse than a cosmetic error. The reported failure invites a retry, and the retry double-applies a ruling that already landed - burning a fix round on a gate whose convergence budget is the thing being rationed, and corrupting the round history that the non-convergence detector reads. There is also no reliable way for a caller to tell landed from not-landed: immediately after a timed-out respond, axi status still showed the old round counts and findings: N awaiting for a ruling that had landed, because the fix agent had not started yet. A guard keying on status: fix_review also produced a FALSE NEGATIVE in the other direction, because axi status stops emitting the gate: block once the step flips to fixing. The only signal that actually worked was polling for the review,fixing step status plus a moved head, several minutes later.

Asks, in priority order. (1) Make respond idempotent under retry: accept a caller-supplied idempotency key, or reject a respond that targets a step already past the round it was issued against, so a duplicate send is a no-op rather than a funded round. (2) Decouple the mutation ack from the blocking follow-on - acknowledge acceptance of the respond immediately with the run id and round number, then stream or let the caller poll, instead of holding one socket read across both the enumeration and the fix. (3) Make the read deadline proportional to the work, or make classify gate execution context cheap enough not to enumerate every active run in the estate for a single-run operation. (4) Until then, make the timeout message state that the ruling may have been accepted and name the check that distinguishes the two states, rather than presenting as a clean failure.

Cross-reference: MO#300-#303 and #305-#308 (the eight redesign tickets filed from this same adjudication pass) were unaffected, since they go through gh rather than the daemon.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions