🧪 fix: Count System Instructions in Mock Model Usage - #16350
Merged
Merged
Conversation
The e2e fake model reported input tokens for the chat messages alone, because the SDK hands a test override model the pruned messages without the systemRunnable pipe. Once #16301 gave every memory-enabled chat a system prompt, the calibrated context snapshot put used tokens below the instructions and the gauge dropped its Messages row. Count input over the complete prompt, as a real provider bills it.
Collaborator
Author
|
Local regression check with this fix: 91/91 passed on first try, 0 flaky (chromium, memory store, retries=1, none used). Specs covered: activity-phases, agent-file-ownership, content-filters.persisted, message-tree, model-replay, quotes, compaction-rerun-controls, failed-summary-history, persisted-context-usage, subagent-context-total, tool-context-split, thread-fold, steering, tool-approvals, completion. These are the mock specs that assert on usage, cost, compaction or context, plus the long-running suites. |
danny-avila
added a commit
that referenced
this pull request
Sep 25, 2026
…16339) * 🐢 fix: Back Off Waiting Completion Wake-ups and Deliver Them When Ready * fix: Signal every readiness path and make the completion wait cap configurable * fix: Mark held completion deliveries instead of pipeline expedite, and announce store-won approval expiry * fix: Scope settle expedites to the resumed conversation and close the remaining signal gaps * fix: Announce a won approval expiry once and release subagent wake-up registrations * 🧪 ci: Count System Instructions in Mock Model Usage (#16350) The e2e fake model reported input tokens for the chat messages alone, because the SDK hands a test override model the pruned messages without the systemRunnable pipe. Once #16301 gave every memory-enabled chat a system prompt, the calibrated context snapshot put used tokens below the instructions and the gauge dropped its Messages row. Count input over the complete prompt, as a real provider bills it. --------- Co-authored-by: Lia <lia@librechat.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every PR that runs e2e against current
devfails the same three specs:usage.spec.ts:111and:167: the context-usage popover never shows its Messages rowscenarios/series-eight-theme.spec.ts:84: the Messages swatch (span.bg-series-1) is missing from the same popoverWhat broke it: #16301 (Surface Persistent Memory Before First Save). Memory is enabled in the e2e config, and an empty memory store now still adds the memory-instructions paragraph, so every mock chat now has a system prompt of about 57 tokens. #16301 skipped e2e (backend-only path filter), so the first PRs to run e2e afterwards inherited the failure.
Why a system prompt breaks the gauge in the mock only: the SDK passes a test override model the pruned messages without the
systemRunnablepipe (prepareProviderRequest({ model: this.overrideModel ?? model })). The fake model therefore saw only[human "hello"]and reportedinput_tokens: 2. The context snapshot calibrates against that usage, so the popover showed "used" as 7 (2 input + 5 output) against a 57-token system prompt. That mademessageBudget = max(0, 7 − 57) = 0, and the zero-value Messages row disappeared. A real provider bills the system prompt, so production is unaffected.Fix:
UsageEmittingFakeChatModelnow counts input over the complete prompt, using the harness's existinggetStreamAgentView. That helper already applies the agent'ssystemRunnablefor prompt assertions, for the same reason. The only other change is replacing a nested ternary lower in the file, because the pre-commit hook lints the whole file and blocked the commit on it.Testing
Run locally against
edddf92(currentdev), in-memory stream store:dev:usage.spec.ts:111fails, reproducing CI.usage.spec.tsandseries-eight-theme.spec.tspass, 8/8.