Skip to content

🧪 fix: Count System Instructions in Mock Model Usage - #16350

Merged
danny-avila merged 1 commit into
devfrom
danny-avila/fake-model-system-usage
Sep 25, 2026
Merged

danny-avila merged 1 commit into
devfrom
danny-avila/fake-model-system-usage

Conversation

@danny-avila

Copy link
Copy Markdown
Collaborator

Summary

Every PR that runs e2e against current dev fails the same three specs:

  • usage.spec.ts:111 and :167: the context-usage popover never shows its Messages row
  • scenarios/series-eight-theme.spec.ts:84: the Messages swatch (span.bg-series-1) is missing from the same popover

What broke it: #16301 (Surface Persistent Memory Before First Save). Memory is enabled in the e2e config, and an empty memory store now still adds the memory-instructions paragraph, so every mock chat now has a system prompt of about 57 tokens. #16301 skipped e2e (backend-only path filter), so the first PRs to run e2e afterwards inherited the failure.

Why a system prompt breaks the gauge in the mock only: the SDK passes a test override model the pruned messages without the systemRunnable pipe (prepareProviderRequest({ model: this.overrideModel ?? model })). The fake model therefore saw only [human "hello"] and reported input_tokens: 2. The context snapshot calibrates against that usage, so the popover showed "used" as 7 (2 input + 5 output) against a 57-token system prompt. That made messageBudget = max(0, 7 − 57) = 0, and the zero-value Messages row disappeared. A real provider bills the system prompt, so production is unaffected.

Fix: UsageEmittingFakeChatModel now counts input over the complete prompt, using the harness's existing getStreamAgentView. That helper already applies the agent's systemRunnable for prompt assertions, for the same reason. The only other change is replacing a nested ternary lower in the file, because the pre-commit hook lints the whole file and blocked the commit on it.

before:  fake input = [human "hello"]                 → input_tokens 2  → used 7 < system 57 → Messages hidden
after:   fake input = [system <instructions>, human]  → input_tokens ≈ 60 → Messages row renders

Testing

Run locally against edddf92 (current dev), in-memory stream store:

  • On dev: usage.spec.ts:111 fails, reproducing CI.
  • With 🪴 fix: Surface Persistent Memory Before First Save #16301 reverted: the same test passes, which confirms the cause.
  • With this fix: usage.spec.ts and series-eight-theme.spec.ts pass, 8/8.
  • Regression sweep over the other mock specs that assert usage, cost, compaction or context: results to follow in a comment.

The e2e fake model reported input tokens for the chat messages alone, because the SDK hands a test override model the pruned messages without the systemRunnable pipe. Once #16301 gave every memory-enabled chat a system prompt, the calibrated context snapshot put used tokens below the instructions and the gauge dropped its Messages row. Count input over the complete prompt, as a real provider bills it.
@danny-avila
danny-avila marked this pull request as ready for review September 25, 2026 01:50
@danny-avila
danny-avila merged commit 402c19c into dev Sep 25, 2026
10 checks passed
@danny-avila
danny-avila deleted the danny-avila/fake-model-system-usage branch September 25, 2026 01:50
@danny-avila

Copy link
Copy Markdown
Collaborator Author

Local regression check with this fix: 91/91 passed on first try, 0 flaky (chromium, memory store, retries=1, none used). Specs covered: activity-phases, agent-file-ownership, content-filters.persisted, message-tree, model-replay, quotes, compaction-rerun-controls, failed-summary-history, persisted-context-usage, subagent-context-total, tool-context-split, thread-fold, steering, tool-approvals, completion. These are the mock specs that assert on usage, cost, compaction or context, plus the long-running suites.

danny-avila added a commit that referenced this pull request Sep 25, 2026
…16339)

* 🐢 fix: Back Off Waiting Completion Wake-ups and Deliver Them When Ready

* fix: Signal every readiness path and make the completion wait cap configurable

* fix: Mark held completion deliveries instead of pipeline expedite, and announce store-won approval expiry

* fix: Scope settle expedites to the resumed conversation and close the remaining signal gaps

* fix: Announce a won approval expiry once and release subagent wake-up registrations

* 🧪 ci: Count System Instructions in Mock Model Usage (#16350)

The e2e fake model reported input tokens for the chat messages alone, because the SDK hands a test override model the pruned messages without the systemRunnable pipe. Once #16301 gave every memory-enabled chat a system prompt, the calibrated context snapshot put used tokens below the instructions and the gauge dropped its Messages row. Count input over the complete prompt, as a real provider bills it.

---------

Co-authored-by: Lia <lia@librechat.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant