Skip to content

🚥 fix: Pause Rejected Steer Recoveries - #16306

Merged
danny-avila merged 3 commits into
devfrom
lia/investigate-steer-recovery
Sep 25, 2026
Merged

danny-avila merged 3 commits into
devfrom
lia/investigate-steer-recovery

Conversation

@lia-by-librechat

@lia-by-librechat lia-by-librechat Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A rejected parked-steer recovery was restored to the local queue with the same recovery and request IDs. The next completed run submitted it again. Cancelling the chip could race queue drain, letting a late rejection restore a source that had already been cancelled.

This PR stops those automatic retries without turning uncertain delivery into a fresh ordinary send. It covers steer recovery only, not durable-worker or subagent-discovery polling.

Behavior

  • Recovery-specific start rejections retain the user's original text, attachments, quotes, and binding in a Recovery paused row. Neither queue drain nor Send now submits a held recovery.
  • Cancellation reserves the source before its HTTP request. Confirmed cancellation suppresses late restoration and leaves the current row held until its guarded Edit/Remove action succeeds. A newer composer draft cannot accidentally turn cancelled words into an ordinary auto-send.
  • The row offers Copy to composer for review and an explicitly tab-local dismissal. Copying does not send anything. Dismissal does not claim to cancel server work. A refused copy shows an error rather than a success toast; the held row remains untouched. Once dismissed, a retained completed run-end boundary releases exactly one ordinary successor without submitting the rejected recovery.
  • Feature-owned Jotai safety state survives remounts and tab reloads without persisting message content. An interrupted cancellation reloads as held and can be retried. Storage failures retain the in-memory guard.
  • Redis and in-memory admission failures expose a specific reason: source_missing, protocol_mismatch, owner_mismatch, invalid_payload, or payload_mismatch. The existing RECOVERY_PAYLOAD_MISMATCH wire code remains compatible with older clients. Logs include conversation/source identity and reason, not payload content.
  • Rate limiting keeps the existing retry behavior. Legacy conversion/redelivery cannot make a held source sendable or resurrect a dismissed source.
recovery rejection -> hold source -> restore original row -> stop automatic drain
cancel request     -> reserve source before HTTP
  confirmed        -> suppress stale restores; keep held until Edit/Remove succeeds
  uncertain        -> keep held for review or another cancellation attempt

A missing source is not proof of non-delivery. A genuine payload mismatch leaves its source parked. Neither outcome is automatically downgraded to a fresh message.

Verification

Verified at merge-resolved pushed head ea41f6d3a8ec49ae8de3d01a3d2574bba1958bde, merged with current dev at 34938ebcbad520be862fb3862fbfd757b9b026ad without a force-push. Resolved the overlapping controller test mock by retaining both the steer-recovery failure helper and the new agent-error metadata helper.

Check Result
Client hooks, recovery controls, and tab safety state 357 passed
Recovery, ownership, protocol rollout, and steering lifecycle, including a real isolated Redis server 134 passed
Resumable HTTP-controller tests (merged controller and new dev cases) 160 passed
Targeted ESLint, Prettier, import sorting, package validation, and circular-dependency checks Passed

Validation limitations:

  • Ran npx tsc --noEmit in both changed TypeScript workspaces. Both report local baseline errors. Repeated the checks in a clean worktree at the current base with the same dependency overlay; this patch adds no diagnostics.
  • Lighthouse CI on the merged head is in progress. The previous head ebfc7b7bd2e6b8eac771a37aca4704323a786193 failed its 4,500 ms median LCP budget at 4,548.585 ms. The earlier PR head passed at 4,349.916 ms. The latest merge push reruns this gate; do not treat it as passed until its result is available.
  • On the previous head, a local npm run lighthouse did not reach browser measurement because an isolated-declaration TS9010 in unchanged packages/api/src/stream/jobStoreCapabilities.ts:11 also reproduced in its clean base worktree. GitHub CI builds using its own dependencies; the local tooling blocker is not a diagnosis of the earlier measured CI LCP failure.
  • Redis Cluster, browser E2E, and the optional full config-migration/unused-key/unused-package scans were not run.

The original historical rejecting branch remains unproven. These regressions establish the current failure modes and corrected behavior, not a reconstruction of expired production state.

@lia-by-librechat

Copy link
Copy Markdown
Contributor Author

Ready for review of exact head f3f9309079ef54bb41dfa373102748de50288fe1.

This head pauses rejected steer recoveries without resending them as ordinary messages, reserves cancellation before queue drain, preserves cancelled rows when a guarded edit cannot complete, and suppresses late restoration/redelivery. The server keeps the legacy error code while adding specific rejection reasons and payload-free diagnostics.

Verification at this head: 641 focused tests passed, including real Redis and the full resumable HTTP-controller test file. Targeted static checks passed. Both TypeScript workspaces reproduce baseline diagnostics with no additions from this patch. Lighthouse was attempted but its prerequisite API declaration build fails at the same unchanged jobStoreCapabilities.ts:11 as the clean base.

The invariant self-review covered queue writers and readers, direct and automatic sends, cancellation interleavings, late responses, remount/reload, disabled storage, owner checks, source preservation, and legacy protocol conversion. No external review has been received for this head yet. Please trigger the review against this SHA.

@danny-avila

Copy link
Copy Markdown
Collaborator

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-25T03:32:19.985054Z ea41f6d Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f3f9309079

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread client/src/hooks/Chat/useQueueDrain.ts Outdated
head != null &&
recoveryDisposition(jotaiStore.get(recoveryDispositionsFamily(conversationId)), head) !=
null;
const next = shouldDrain && !held ? (head ?? null) : null;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the run-end signal while recovery is held

When a completed run encounters a held recovery at the head of a queue, this branch leaves the row in place but has already consumed the one-shot run-end signal. If the user then dismisses that recovery, dismissRecovery only removes the row and does not rearm the drain, so any ordinary follow-ups behind it remain queued indefinitely until another generation happens or the user sends one manually. Keep the terminal signal parked while the head is held, or explicitly rearm it when the hold is resolved.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ebfc7b7. A completed run-end now stays parked while a recovery is held; queue-head changes wake the drain when dismissal removes that row. Tests cover blocked and confirmed-cancelled holds, new-conversation migration, aborted runs, and exactly one ordinary successor after dismissal.

Comment on lines +283 to +285
onRestoreToComposer(
message.text,
message.files,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Handle a refused copy-to-composer operation

When the composer already contains text, files, quotes, or skills—or is reserved by answer mode—onRestoreToComposer returns false, but this new Copy action ignores that result and still shows the review toast. In that common scenario, clicking “Copy to composer for review” copies nothing and provides no indication that the action failed; handle the false result with appropriate feedback instead of reporting the normal outcome.

AGENTS.md reference: AGENTS.md:L42-L44

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ebfc7b7. The Copy action now checks the composer callback's boolean return. A refused copy shows an error toast instead of a success toast and leaves the paused recovery intact; acceptance and refusal are both covered by tests.

@lia-by-librechat

Copy link
Copy Markdown
Contributor Author

Ready for review of exact head ebfc7b7bd2e6b8eac771a37aca4704323a786193.

Addressed both Codex findings on the previous head: a held recovery no longer consumes the one-shot completed run-end, and dismissing it wakes the next ordinary queued item. A refused Copy to composer now reports an error instead of claiming success, leaving the paused row untouched. Tests also cover cancelled holds, first-run migration, aborted runs, and one successor per terminal boundary. Replied to each inline finding with this commit.

Verification for this review round: 357 focused client tests passed across six suites; targeted static checks (ESLint, Prettier, imports, package validation, circular dependencies) passed. Ran npx tsc --noEmit in the changed client workspace and compared with an unchanged dev worktree using the same dependency overlay: 0 new diagnostic sites (13 current vs 16 baseline). Server files were not changed this round; the previous head had 134 backend tests (including real Redis) and 155 HTTP tests passing. Lighthouse remains blocked by the unchanged baseline API isolated-declaration failure described in the PR. CI for the new head is in progress.

@github-actions

Copy link
Copy Markdown
Contributor

Lighthouse CI failed. The last 80 log lines contain the measured budgets and assertion failures.

│ 21      │ 'http://localhost:3080/api/convos?pinned=true&limit=100'                                                        │ 2679.2889999999898 │ 3439.362999999983  │ 200    │
│ 22      │ 'http://localhost:3080/api/mcp/servers'                                                                         │ 2998.875           │ 4260.712999999989  │ 200    │
│ 23      │ 'http://localhost:3080/api/permissions/mcpServer/effective/all'                                                 │ 2999.777000000002  │ 3757.9869999999937 │ 200    │
│ 24      │ 'http://localhost:3080/api/prompts/groups?limit=10'                                                             │ 2999.9770000000135 │ 4324.834999999992  │ 200    │
│ 25      │ 'http://localhost:3080/api/keys?name=openAI'                                                                    │ 3242.837999999989  │ 3933.2369999999937 │ 200    │
│ 26      │ 'http://localhost:3080/api/presets'                                                                             │ 3244.1910000000207 │ 3944.0100000000093 │ 200    │
│ 27      │ 'http://localhost:3080/api/tags'                                                                                │ 3245.154999999999  │ 3944.2360000000044 │ 200    │
│ 28      │ 'http://localhost:3080/api/share/link/16390000-0000-4000-8000-000000000001'                                     │ 3246.4219999999914 │ 4264.311000000016  │ 200    │
│ 29      │ 'http://localhost:3080/api/messages/16390000-0000-4000-8000-000000000001'                                       │ 3247.1760000000068 │ 4439.6750000000175 │ 200    │
│ 30      │ 'http://localhost:3080/api/files/config'                                                                        │ 3247.4130000000005 │ 4198.750999999989  │ 200    │
│ 31      │ 'http://localhost:3080/api/agents/tools/web_search/auth'                                                        │ 3247.6199999999953 │ 6966.005999999994  │ 200    │
│ 32      │ 'http://localhost:3080/api/endpoints/token-config'                                                              │ 3247.8060000000114 │ 4455.6020000000135 │ 200    │
│ 33      │ 'http://localhost:3080/api/agents/tools/calls?conversationId=16390000-0000-4000-8000-000000000001'              │ 3247.9899999999907 │ 4766.822000000015  │ 200    │
│ 34      │ 'http://localhost:3080/api/agents/chat/status/16390000-0000-4000-8000-000000000001?generationProtocolVersion=2' │ 4526.885999999999  │ 4783.779999999999  │ 200    │
└─────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────┴────────────────────┴────────┘

Inspect .lighthouse HTML/JSON and e2e/lighthouse/README.md. Reuse loaded user/config data; overlap independent reads without bypassing authorization.

┌─────────┬────────────────────────────┬──────────────────────┬───────┐
│ (index) │ audit                      │ median               │ limit │
├─────────┼────────────────────────────┼──────────────────────┼───────┤
│ 0       │ 'largest-contentful-paint' │ 4548.585             │ 4500  │
│ 1       │ 'cumulative-layout-shift'  │ 0.016889534218763675 │ 0.1   │
│ 2       │ 'total-blocking-time'      │ 231.89499999999998   │ 500   │
└─────────┴────────────────────────────┴──────────────────────┴───────┘

  1) [chrome] › e2e/lighthouse/load.spec.ts:10:5 › serial database latency stays within web-vitals budgets 

    Error: Median largest-contentful-paint must stay within 4500

    expect(received).toBeLessThanOrEqual(expected)

    Expected: <= 4500
    Received:    4548.585

       at audit.ts:159

      157 |   console.table(measured);
      158 |   for (const { audit, median, limit } of measured) {
    > 159 |     expect(median, `Median ${audit} must stay within ${limit}`).toBeLessThanOrEqual(limit);
          |                                                                 ^
      160 |   }
      161 |   return results;
      162 | }
        at auditPage (/home/runner/work/LibreChat/LibreChat/e2e/lighthouse/audit.ts:159:65)
        at /home/runner/work/LibreChat/LibreChat/e2e/lighthouse/load.spec.ts:33:19

    attachment #1: screenshot (image/png) ──────────────────────────────────────────────────────────
    e2e/lighthouse/.test-results/load-serial-database-latency-stays-within-web-vitals-budgets-chrome/test-failed-1.png
    ────────────────────────────────────────────────────────────────────────────────────────────────

    Error Context: e2e/lighthouse/.test-results/load-serial-database-latency-stays-within-web-vitals-budgets-chrome/error-context.md

    attachment #3: trace (application/zip) ─────────────────────────────────────────────────────────
    e2e/lighthouse/.test-results/load-serial-database-latency-stays-within-web-vitals-budgets-chrome/trace.zip
    Usage:

        npx playwright show-trace e2e/lighthouse/.test-results/load-serial-database-latency-stays-within-web-vitals-budgets-chrome/trace.zip

    ────────────────────────────────────────────────────────────────────────────────────────────────


🤖: global teardown has been started
2026-09-24 18:07:17 �[32minfo�[39m: �[32mMongo Connection options�[39m
2026-09-24 18:07:17 �[32minfo�[39m: �[32m{�[39m
�[32m  "bufferCommands": false�[39m
�[32m}�[39m
🤖:  ✅  Connected to Database
🤖:  ✅  Found user in Database
🤖:  ✅  Deleted 1 convos & 2 messages
🤖:  ✅  Deleted user from Database
🤖: global teardown has been started
2026-09-24 18:07:18 �[32minfo�[39m: �[32mMongo Connection options�[39m
2026-09-24 18:07:18 �[32minfo�[39m: �[32m{�[39m
�[32m  "bufferCommands": false�[39m
�[32m}�[39m
🤖:  ✅  Connected to Database
🤖:  ⚠️  User not found in Database
  1 failed
    [chrome] › e2e/lighthouse/load.spec.ts:10:5 › serial database latency stays within web-vitals budgets 

Open the full run

@lia-by-librechat

Copy link
Copy Markdown
Contributor Author

CI update for exact head ebfc7b7bd2e6b8eac771a37aca4704323a786193: the Lighthouse job failed its 4,500 ms median LCP budget at 4,548.585 ms (48.585 ms over). The previous PR head passed at 4,349.916 ms. These results alone cannot attribute the difference to this patch rather than changes to dev or run-to-run variation. The earlier local Lighthouse setup failure is a separate baseline declaration-build problem; it did not cause CI's measured failure. The PR description now distinguishes them.

All other current-head CI checks passed. An exact-head rerun of the failed job was attempted but this integration was denied by GitHub (403 Resource not accessible by integration); no commit was created just to retry the gate. Lighthouse is not green or waived. The reviewed code fixes and 357 focused client tests remain unchanged. There has not yet been a new Codex review of this head.

@danny-avila

Copy link
Copy Markdown
Collaborator

@codex review the latest head

@lia-by-librechat

Copy link
Copy Markdown
Contributor Author

Ready for review of exact pushed head ea41f6d3a8ec49ae8de3d01a3d2574bba1958bde.

Merged current origin/dev (34938ebcbad520be862fb3862fbfd757b9b026ad) into PR #16306 without rebasing or force-pushing. The sole content conflict was in the resumable-controller test mock: both the PR's getSteerRecoveryFailure and dev's getAgentErrorMetadata exports are retained. GitHub now reports the PR as mergeable.

Checks on the merged content: 357 client recovery/queue/UI tests, 134 backend tests including real Redis, and 160 full resumable-controller tests passed. PR-targeted static checks passed. Ran tsc --noEmit in client and packages/api and compared each against a clean worktree at the same dev head, using the same local dependency overlay: neither workspace gained any new diagnostic sites; both local runs still report environment/baseline diagnostics. GitHub CI, including Lighthouse, has restarted for the new head. The previous head had a measured Lighthouse budget failure; this merge head's gate must be judged by its own CI result.

No newer Codex review has arrived for this SHA yet.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: ea41f6d3a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@danny-avila
danny-avila merged commit fe8a993 into dev Sep 25, 2026
36 checks passed
@danny-avila
danny-avila deleted the lia/investigate-steer-recovery branch September 25, 2026 03:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants