Fix queued chat message parity#935
Conversation
Mirror web UI queue behavior by assigning stable per-message send run IDs and forwarding them as chat.send idempotency keys. Reconcile queued prompts by identity instead of prompt text so duplicate user messages remain distinct. Handle deferred in_flight admissions by requeueing with bounded backoff, failing visibly when the gateway never admits the run, and preventing ambiguous assistant retransmits from being replayed across queued-user boundaries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Codex review: needs maintainer review before merge. Reviewed July 8, 2026, 4:34 PM ET / 20:34 UTC. Summary Reproducibility: yes. source-reproducible: the PR includes focused regressions and live WinUI/UIA/MCP proof for duplicate queued messages, in_flight retry, timeout handling, and queue cancel behavior. I did not run a live current-main failing reproduction in this read-only review. Review metrics: 3 noteworthy metrics.
Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Risk before merge
Maintainer options:
Next step before merge
Maintainer decision needed
Security Review detailsBest possible solution: Have the active reviewer re-check current head, then merge once the changes-requested review is resolved and required checks are green. Do we have a high-confidence way to reproduce the issue? Yes, source-reproducible: the PR includes focused regressions and live WinUI/UIA/MCP proof for duplicate queued messages, in_flight retry, timeout handling, and queue cancel behavior. I did not run a live current-main failing reproduction in this read-only review. Is this the best way to solve the issue? Yes, the current approach looks like the narrow maintainable fix: stable queued run IDs feed chat.send idempotency keys, local queue identity drives reconciliation/cancel, and MCP docs/tests cover the new command surface. The remaining blocker is human re-review, not a new implementation path. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 0d11a8ae6a49. Label changesLabel justifications:
Evidence reviewedWhat I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (6 earlier review cycles)
|
Catch and warn on unexpected exceptions from fire-and-forget queued send drain dispatch so failures are diagnosable instead of becoming unobserved task exceptions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Allow native tray chat users to remove queued or failed queued messages by local queue identity before they are dispatched. Wire the cancel action through the chat provider and composer, and add accessible state-specific controls for pending and failed cards. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Expose native chat queue inspection and cancellation through local MCP app.chat tools, include queued entries in chat snapshots, and document the new winnode surface. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Disable the model and reasoning controls while an assistant turn is active or queued messages remain, leaving the session selector available for UX testing. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep queued run/idempotency mappings and local echo suppression alive while an in_flight send waits for retry, and track accepted deferred run IDs for lifecycle promotion/reset gating. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
shanselman
left a comment
There was a problem hiding this comment.
Caleb, first: this is a strong cleanup PR. The hard parts here are exactly the right things to be thinking about: stable queue identity instead of prompt-text matching, idempotency keys that survive retries, bounded retry terminality, and cancel/remove actions keyed by local queue identity. That is good engineering progress, especially given this is cleaning up a prematurely-merged draft path.
I did a dual-model adversarial pass and one issue looks real enough that I want to hold merge until it is fixed. The good news: it is a focused state-machine cleanup, and the fix should be straightforward.
Blocking before merge
Deferred in_flight retries can leave stale local-echo suppressors
When a queued send gets chat.send status in_flight, the provider requeues and retries with the same stable send/idempotency key. That part is good. The gap is that each retry path can enqueue another LocalSentText suppressor for the same queued message, while the consume/remove paths only clear one matching entry.
Why it matters: after the first gateway echo is consumed, a leftover suppressor for the same queued message/text can still be present. A later legitimate same-text user message from another client could be treated as “our echo” and dropped from the timeline.
Places to look:
OpenClawChatDataProvider.csEnqueueLocalEchoLockedTryConsumeLocalEchoLockedRemovePendingLocalEchoLocked- deferred admission path around
RequeueDeferredAdmissionLocked
Suggested fix:
- Make local echo suppressor tracking idempotent by
QueuedMessageId. - Before enqueueing a new
LocalSentText, remove/replace any existing entry for that same queued message id. - When removing/consuming by queued message id, clear all entries for that id, not just the first one.
- Add a regression test: queued send gets
in_flight, retries, first same-text gateway echo is consumed/reconciled, then a separate later same-text remote user message should render as a distinct user entry.
A good done signal would be a test with a name like:
QueuedSend_InFlightRetry_DoesNotSuppressLaterSameTextRemoteUserMessageNon-blocking follow-ups
These should not block this PR unless you want to handle them while you are in the area:
- Failed queue cards currently keep model/reasoning dropdowns disabled because the composer gates on
queuedMessages.Count, not just pendingQueued/Sendingwork. A failed card is terminal, so it probably should not block changing model/reasoning before retry. app.chat.queue.cancelwithoutthreadIdcan fail afterapp.chat.queue.listreturns all threads, because cancel defaults to the current compose/default thread. Either requiringthreadIdin docs or scanning all queued threads when omitted would make automation friendlier.
Also confirmed: app.chat.queue.list / app.chat.queue.cancel being local MCP-only Windows Hub app commands is okay. They are not gateway RPCs, and the docs correctly describe them as local tray automation hooks. The cross-platform gateway contract surface touched here is the existing chat.send.idempotencyKey, which is the right place to align with OpenClaw queue behavior.
Once the blocker fix + regression test is in and CI is green, I expect this to be mergeable.
Treat timeout chat.send ACKs as terminal failures and keep in-flight retry local echo suppressors idempotent per queued message id. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Disable message option dropdowns based on active or pending sends instead of raw queue count, allow failed queue cards without blocking new direct sends, and require threadId/sessionKey for app.chat.queue.cancel. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Clear stale queued-drain guards when queue state is emptied, clean up idle local-run tracking after cancel, and document ephemeral queue identifiers. Add targeted queue regressions for stale drain guards, remote backfill after cancel, and deferred in-flight cancel behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
shanselman
left a comment
There was a problem hiding this comment.
Reviewed follow-up changes with >90% confidence. The prior stale local-echo suppressor blocker is fixed by queued-message-id dedupe/removal and covered by QueuedSend_InFlightRetry_DoesNotSuppressLaterSameTextRemoteUserMessage. Current CI is green; MCP docs/proof now align with the required threadId contract.
Summary
chat.send.idempotencyKey.in_flightas deferred admission with bounded backoff, visible failure on exhaustion, safe handling for active-run/direct-send races, and preserved run/idempotency correlation through lifecycle promotion or terminal cleanup.Validation
./build.ps1— passeddotnet test ./tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj --no-restore— 2716 passed / 31 skippeddotnet test ./tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj --no-restore— 1604 passeddotnet test ./tests/OpenClaw.WinNode.Cli.Tests/OpenClaw.WinNode.Cli.Tests.csproj --no-restore— 120 passedGateway status contract
ChatSendAckStatusdefinition:"started" | "in_flight" | "ok" | "timeout" | "error".in_flightis the deferred non-terminal admission status.timeoutanderrorare terminal failure statuses,startedis accepted/admitted, andokis completed/idempotent success.Real behavior proof
hellomessages through the tray chat UI; all three drained successfully, leaving 0 queued cards and 3 visiblehellouser turns.GatewayProtocolLiveRoundTripTests.ChatSend_UsesCallerProvidedIdempotencyKeyOnWire, provingOpenClawGatewayClientsends caller-providedidempotencyKey.QueuedSend_InFlightAckThenLifecycleBeforeRetry_PromotesWithoutResend, coveringin_flightfollowed by matchinglifecycle.startbefore retry, with no resend and no duplicate user echo.in_flightcancel behavior without unsafechat.abort.MCP proof
winnode --list-toolsadvertised both queue tools from this PR:Controlled proof run
proof-155257queued two duplicate outgoing messages, listed both over MCP, canceled onlyq2, and confirmedq3remained queued:UIA transcript
After the MCP cancel, UIA saw the remaining duplicate queued card and exactly one cancel control for that sibling:
Screenshot of queued messages:

Review