A refusal the operator could act on arrives as a transport failure, and the sender is told nothing

[fujibee]

Title: A refusal the operator could act on arrives as a transport failure, and the sender is told nothing

---

A remote server may refuse a write with a status that means "the caller must do something" rather than "try again later" — 402 payment_required is the case that exists today, and any server implementation may return it. The engine does not distinguish that from a transport failure, records no reason, and send has already returned success to the agent. From every side it looks like the message went out.

What was measured

scripts/internal/remote-sync.mjs, on integration/remote @ d2e5f43.

402 appears nowhere.

grep -nE '402|payment_required' scripts/internal/remote-sync.mjs
  -> 0 matches

Not retrying it is right, and is the only thing about it that is right.

:1598  export function isRetryable(error) {
:1599    if (error?.retryable === true) return true;
:1600    return [408, 429, 500, 502, 503, 504].includes(error?.status);

A 402 raises HTTP 402 payment_required out of request() (:1051-1056) and lands where HTTP 500 internal and HTTP 403 forbidden land. The one refusal a person can act on is shaped exactly like the ones they cannot.

Nothing records why a cycle failed. #756/#760 added a cycle stamp, so status can say whether anything has synced. That answers "is it working?". Nothing answers "why not" — there is no last_error equivalent in the engine.

And the sender already saw success. send commits to the local store and returns; the push happens later. A refusal there reaches nobody.

agent      send            -> local commit -> SUCCESS
engine     push            -> refused      -> raised, not recorded
person     status          -> "nothing has synced"
                           -> no reason

That is worse than looking stopped. It looks sent.

What "fixed" looks like

1. Classify a refusal the caller can act on, separately from a transport failure.

Not retryable — the answer will not change by asking again. But not "an error" in the transport sense either: the server has made a decision and stated it. The engine needs a third outcome beside succeeded and failed, will retry: refused, and here is what the server said.

The distinction is about the class of status, not about one number. A server may refuse for reasons this protocol never enumerates.

2. Keep what the server said, verbatim, and make it readable.

The refusal carries a status and a code. Store both, and the time. Do not translate them into a sentence about why — the engine cannot know why, and a sentence it invents will be wrong for some server.

This is the same shape as the cycle stamp #760 added: one more fact beside it, not a new mechanism.

3. The agent must be able to answer when asked.

Someone asks their agent "why isn't this syncing?" and the agent should be able to say: the last cycle was refused by the server, with this status and this code, at this time. Today the agent has nothing to read.

So the recorded fact has to sit somewhere the agent's own surfaces reach — the same place it learns anything else about the engine — not only in a command a human types.

4. Assume nothing about which server is on the other end.

agmsg talks to a remote. It may be self-hosted, someone else's, or a service. The engine must not carry knowledge of any particular one: no product name, no URL, no advice about subscribing, no mapping from a status to a business meaning.

What it may do is repeat what the server said and say where the operator of that server would be reached — which the config already knows, because the endpoint is in it. Interpretation belongs to whoever runs the server, not to the client.

402 is the case that exists today. The fix should be right for a self-hosted server that returns it for its own reasons.

5. Decide what send does — this is a design question, not an implementation detail.

• Keep accepting sends and let them queue — the current behaviour, and defensible: local-first means the local store is the record, and nothing is lost. But then the person must be able to find out, or they accumulate silently.
• Refuse the send — honest at the moment of the action, and wrong for a local-first tool: the message belongs locally whether or not any server will take it.

The first is probably right, which makes 2 and 3 the whole fix. A queue nobody can ask about is the defect. A queue with a readable reason is a feature.

Not in scope

Why a particular server refused, and what a person should do about it. That belongs to whoever operates that server.

[fujibee]

Two measurements that change this issue. Both make it worse and narrower at the same time.

The engine does not fail the cycle. It exits.

scripts/internal/remote-sync.mjs:2464  (runLoop)
    } catch (error) {
      try { await eventCall("cycle.error", { message, code }) } catch {}
      if (!isRetryableCall(error)) throw error;

A non-retryable error leaves the loop. main() rejects, and the top-level handler writes a fatal event, prints to stderr, and sets exitCode = 1. The process ends.

So a refusal does not degrade syncing — it stops the engine. status afterwards reports:

<team>  connected (engine stopped — run: remote.sh sync start <team>) since <date>

That line invites the one action that cannot work: starting it again produces the same refusal and the same exit. The status output actively points the wrong way, and it will keep doing so for as long as the server's answer stands.

The reason IS recorded. Nothing reads it.

event() (:1369-1376) writes JSON lines to stdout, and sync start captures them:

scripts/remote.sh:1770   logfile="$CONNECTION_ROOT/run/remote-sync.$team.log"

So {"event":"fatal","message":"HTTP 402 payment_required","code":"payment_required"} is on disk, with a timestamp, at a known path.

It is not missing. It is unread. status does not open that file — it reads a pidfile and the cycle stamp. Nothing surfaces the last fatal, and nothing an agent can consult mentions it.

What this does to the fix

Two of the four points in the issue body get smaller and one gets larger.

• "Keep what the server said" — largely done. The status and code reach a durable file already. What is missing is a defined place to read it from, not a new place to write it.
• "The agent must be able to answer" — this is now the centre. The fact exists and no surface exposes it.
• New, and not in the original body: the engine should not exit on a refusal the caller can act on. Exiting is right for a malformed config; it is wrong for a server that has stated a decision, because it converts a recoverable condition into a dead process whose status line recommends a no-op.

Whether the engine should stay up and keep refusing, or exit with a distinguishable reason that status reports honestly, is the design question. Both are defensible; "exit, and tell the person to start it again" is not.

[fujibee]

Every line number above is wrong. The findings are not. Corrected against integration/remote @ d2e5f43, which is what both comments claim to cite.

                        stated   actual
isRetryable               1598     1848
  non-retryable throws    2464     2766
request() error raise  1051-56   1298-1303
event() writes         1369-76   1620-1627
fatal, top level             —     3362
engine log path            1770     1509   (also 1309)

The content each one supports was re-checked at d2e5f43 and holds:

402 / payment_required in remote-sync.mjs        0 occurrences
isRetryable                [408, 429, 500, 502, 503, 504]   — 402 absent
runLoop                    if (!isRetryableCall(error)) throw error
event()                    stdout, plus AGMSG_SYNC_LOG_FILE when set
sync start                 $CONNECTION_ROOT/run/remote-sync.<team>.log
remote.sh sync             only `start` — no stop, no status, no once

How this happened, because it decides how much to trust the rest

I read the files from a shared checkout at /Users/Shared/agmsg-remote-src without anchoring to a ref. That working tree sits at d427434 — 230 commits behind integration/remote. Every grep -n I ran answered correctly about a tree from before most of this year's sync work, and printed no warning, because there was nothing wrong with the command.

I then wrote @ d2e5f43 above the results, which is the head I had fetched, not the one I had read.

The conclusions survived by luck. The code those line numbers point at could have been deleted, moved, or rewritten in 230 commits, and nothing in my method would have caught it. The reason this is a correction rather than a retraction is that the seat taking the work re-derived the numbers instead of trusting them.

Cite by anchored content, not by line. git show <ref>:<path> for anything quoted, and quote enough of the line that a reader can find it after it moves.

[fujibee]

Landed on integration/remote as #792, merge commit 5d3cc9eba4bbaf22ef94e2d652722ff552c2fe13. Closed by hand: Closes does not fire for a pull request that does not target the default branch.