Release 2.5.3: agent-lane stalls, API integrity, honored request controls - #243
Merged
Conversation
…okens details Template render + full-transcript BPE tokenize ran per request ahead of any GPU work — pure TTFT tax that agent clients (which resend byte-identical prefixes every turn) pay on every round-trip, and that warm session-bank hits paid before the bank lookup could even run. - mtplx/chat_encode_cache.py: content-keyed LRU (128 entries, env off-switch MTPLX_CHAT_ENCODE_CACHE=off). Key covers messages, tools, tool_choice, thinking/effort, prompt-mode flags, per-INSTANCE tokenizer uuid (weakref registry) and the CURRENT template hash — recomputed every call so template swaps can never serve a stale entry. Hits return copies; template observability is replayed on hit. - usage.completion_tokens_details.reasoning_tokens (OpenAI-standard): lets external clients separate thinking from visible output instead of under-reading the decoder from the stream (the 47-52 vs 40-44 discrepancy class an external benchmarker hit this week). - tests/test_chat_encode_cache.py: hit identity, key sensitivity, template swap invalidation, off-switch, LRU bound. Full test_server_openai.py green (an earlier per-object key memo DID cause one cross-instance pollution failure — that design was discarded for the weakref+per-call-hash form). A/B gate still owed before any ship: TTFT rungs + warm pair vs stock 2.5.2.
The footer streamed as a content delta to EVERY client (default on, only response_format-suppressed). On the OpenAI-compat API that is server-injected prose inside model output. Tonight's showdown receipts (outputs/mlxserve-showdown-20260805) measured the damage on 2.5.2: - every capped request delivered ~34 tokens MORE than usage.completion_tokens (the footer), a wire-vs-usage mismatch external tools inherit as error; - the footer chunk flushed ~430ms after the last real token — that dead gap sits inside external tools' decode window (llmprobe: first->last text frame), deflating measured tok/s ~9% on short runs; - temp-0 byte-equality across runs is impossible when the appended banner contains per-run tok/s digits (external 'fidelity/determinism' checks fail on this alone); - an external benchmarker's 'MTPLX self-reports 47-52 while the stream reads 40-44' comparison was literally reading this banner vs his own deflated client math. New contract: footer renders only for requests carrying a managed MTPLX client hint (browser/chat/app/terminal surfaces) — founder-visible UX is unchanged there. Anonymous API clients get clean model output. Operator escape hatch: MTPLX_STATS_FOOTER_SCOPE=all restores legacy behavior; --no-stats-footer still kills it everywhere. tests/test_stats_footer_scope.py: anonymous-clean, managed-keeps, scope-all restores, flag-off wins (4 passed).
… unchanged) Anonymous OpenAI-compat clients currently have body sampling params demoted to hints unless they send X-MTPLX-Allow-Client-Controls — correct for managed agent lanes, but it reads as 'MTPLX ignores temperature' to every external tool (and did, publicly, this week). This adds the capability to honor explicit anonymous body params (OpenAI semantics) behind MTPLX_CLIENT_CONTROLS_DEFAULT=honor. Managed surfaces (app/browser/opencode hints) remain server-owned in both modes. Default stays 'hints' — the flip is a founder generation-policy decision; receipts for it live in outputs/mlxserve-showdown-20260805/ANALYSIS.md.
… on the wire
External conformance/discovery tools POST {} and malformed bodies to every
OpenAI-shaped path and quote our error bodies verbatim ('a lot of python
errors while running LLM probe' — this week's DM). Reproduced with the exact
battery: /v1/completions {} fell through into generation machinery and 500'd
with 'AttributeError: ...' on the wire.
- /v1/completions: empty prompt now 400s ('prompt must not be empty') before
touching runtime state, matching the chat/messages contract.
- catch-all 500s: exception class+repr moved to the server log (with
traceback + request_id); wire body carries a generic message + request_id.
MTPLX_DEBUG_ERRORS=1 restores inline detail for local debugging.
- tests/test_endpoint_probe_hygiene.py pins the whole battery (5 passed):
clean 400s, 422 on malformed, 404s on unimplemented, no internals at 500,
debug env escape.
At request temperature 0 the target distribution is one-hot but the launch draft sampler stayed at 0.6 — acceptance collapsed to 'did the sampled draft hit argmax': measured [79/65/42]% by depth vs [91/83/67]% at 0.6 on the 27B (2026-08-05 showdown, cond-M receipts), i.e. the exact lane every external benchmark (temp:0) measures ran with a self-inflicted acceptance handicap. Greedy target output is draft-independent — coupling drafts to greedy is speed-only and cannot change generated text. Explicit draft samplers are respected; MTPLX_GREEDY_DRAFT_COUPLING=off restores old behavior. Observability: draft_sampler_greedy_coupled=true. A/B gate owed: cond-M rungs vs stock 2.5.2 (queue3).
…er's idle commit The idle postcommit's bounded foreground grace (2.0s) is a same-session bargain: the queued request profits from the commit it waits on (without it, agent tool turns pay a 2-4k block-salvage re-prefill, 3-7s TTFT — 2026-08-01 gauntlet). But the grace also let CROSS-session requests pay for commits that cannot help them: at <2s cadence every multi-session request (bench tools, subagent fan-outs, parallel chats) waited out a stranger's 0.5-3.5GB retokenized_history job and lost 30-50% decode to its residue (2026-08-05 showdown receipts: PC elapsed 0.68s == rep1 TTFT +0.75s; 0/11 cross-session commits aborted vs 2.0.1's engine-global preemption). EngineSessionManager.abort_cross_session_postcommits() sweeps every other session's pending commit at request admission (best-effort, off the scheduler-owner thread, before foreground submit). Same-session grace semantics unchanged. Env: MTPLX_POSTCOMMIT_CROSS_SESSION_YIELD=0 restores the old behavior. Observability: postcommit_cross_session_yield in request stats + console line. Tests: 5 new (sweep, same-session spare, stateless, env gate) + existing postcommit integration suite + full test_server_openai.py green.
A session-bank restore hands the generation exact-size KV buffers (clone or reference lease), so the first compiled-route call must ensure_capacity -> mx.concatenate the ENTIRE restored context per full-attention layer before it can run: an O(context-bytes) copy paid synchronously inside warm TTFT. Measured 2026-08-05 (warmforensics x3, temp-gated): turbo warm TTFT 365-961ms vs sustained/eager 117-210ms on the same restore; the restore itself is free (0.000-0.001s ram clone). Live sessions never pay this - demote() hands the padded buffers back, so re-promotion is a capacity no-op. The bank now routes the first verify round(s) after a large restore (default >= 2048 restored tokens) through the eager path and promotes from the next round, so the promotion copy lands mid-stream after the first tokens are on the wire instead of inside TTFT. - restored_tokens flows explicitly from the generation loop (PromptState cached_tokens: 0 cold, restored prefix length warm). - Non-consuming prewarm-eligibility probes skip without ticking the counter (and without promoting - the probe itself would otherwise pay the copy). - Parity modes keep full compiled coverage from round 1. - Env: MTPLX_COMPILED_VERIFY_POST_RESTORE_EAGER_ROUNDS (default 1, 0 kills), MTPLX_COMPILED_VERIFY_POST_RESTORE_MIN_TOKENS (default 2048). - Observability: fallback_reasons[post_restore_warmup]. - 5 new tests; full test_graphbank_compiled_verify.py green (59).
Temp-gated receipts 2026-08-05 (M5 Max, die <72C, adjacent arms): dense chunk 4096 gives +41-58% prefill at an 8k prompt (675-772 vs 478-490 tok/s, TTFT 16.8-17.2s -> 10.6-12.2s) and +33-38% at 32k (428-433 -> 573-590, TTFT 76s -> 56s). 8192 loses to 4096 at both rungs. History: PR #33 bumped to 4096 in May; the 128k validation split the knob (dense 4096 / repage 2048) because repage 4096 costs +14.3 GB peak (+38%) at 128k with no TTFT win - a memory-pillar regression on 96 GB and smaller Macs (tools/bench/prefill-chunk-split-128k-2026-05-09.md). Issue #63 then froze both at 2048 from OpenCode/Pi cold-prefill QA that predates die-temp gated benchmarking. This restores the validated split with fresh receipts: dense (all contexts through 128k) at 4096, repage (>128k) at 2048. Chunking is layout-only on the prefill path; both env knobs remain user-overridable. Diagnostics script mirror updated in the same change.
MTPLX_CLIENT_CONTROLS_DEFAULT flips 'hints' -> 'honor'. Anonymous (non MTPLX-managed) clients now get OpenAI-API semantics: explicit body params (temperature, top_p, top_k, enable_thinking, penalties, generation_mode, depth) are applied per-request instead of being recorded as observability hints. Managed MTPLX surfaces (app, browser chat, OpenCode and other x-mtplx-client lanes) remain server-owned in both modes, so the curated agent sampler policy is untouched. Why: external tools assume the OpenAI contract. 2026-08-05/06 receipts: llmprobe sent temperature:0, was silently served the 0.6 coding sampler, and its author published 'MTPLX does not respect temp=0' plus issue #241 (temperature and thinking toggle not passed through). The same policy made external AR baselines silently run MTP (mistakes ledger). Honoring explicit params kills that class at the source; requests that send nothing keep the server's launch/live settings exactly as before. Rollback: MTPLX_CLIENT_CONTROLS_DEFAULT=hints restores the old policy; the per-request X-MTPLX-Allow-Client-Controls opt-in still works there. Tests: hints-mode coverage pinned via env in the four policy tests, new integration test for the honor default, unit matrix updated.
Version bump plus the changelog entry for the 2.5.3 fix set: cross-session postcommit yield, post-restore eager first round, chat-encode memoization, stats-footer scoping, reasoning_tokens usage details, endpoint hygiene, anonymous client-controls honor default, dense prefill chunk 4096, greedy draft coupling.
…k+ receipt Clean-room ABBA (2026-08-06, 4k restore, fresh servers, die-temp banded): stock's first-round promotion on the restored cache measured 2.9ms server TTFT — the ensure_capacity copy is megabytes at 4k, not the 08-05 warm anomaly. That anomaly decomposes to first-shape-in-process compile traces (+postcommit stacking +hot die), which the deferral cannot fix — and the deferral's eager->compiled transition introduced one novel verify-shape trace (~100-200ms once per process; toy repro is trace-neutral, live transition shape is new). The copy scales linearly with restored context (~2 GB at 32k), so the lever may still pay for big-session restores; that experiment needs its own receipt before any default flips. Mechanism, env knobs, stats reason, and tests all stay; default rounds 1 -> 0.
… 2048" This reverts commit aba3b42.
… inversion Tonight's matched-state ABBA (the temp-gated stock re-verify cell the 08-05 ledger owed) inverted the chunk sweep: dense-4096 measured -3..-6% prefill vs stock 2048 at 8k and 32k in BOTH adjacent pairs (643-656 vs 675 @8k; 558-579 vs 594-609 @32k). The 08-05 +41-58% compared against non-temp-gated baseline arms. Default stays 2048 (revert 5d3ad54); notes and changelog now describe only what shipped.
…ate-safe encode key, footer off agent clients Independent review of the 2.5.3 diff surfaced three ship-blockers, all fixed here with regression tests: F1: the dense prefill chunk fallback in generation._prefill_chunk_size still read 4096 — the hunk had been staged into the graphbank commit instead of the chunk commit, so the chunk revert (5d3ad54) never touched it. Reachability was narrow (both profiles pin the env) but HEAD shipped the exact change the A/B inverted. Now 2048, matching the revert. F2: the chat-encode cache key had no date component while the rendered prompt embeds one (tool contract date line, strftime_now templates): an exact repeat across local midnight served yesterday's token ids until eviction. The key now carries the render day; worst case is one full cache turnover per day, which is the correct behavior. F3: footer scoping admitted every managed client hint, so OpenCode (and pi/hermes/openwebui — agents that parse assistant content) still received the TPS footer, contradicting the release notes. The footer surface is now the product-UI set only (chat, mtplx app identities); managed agent clients and UA-sniffed OpenCode are footer-free, with regression tests. Also from the same review: the cross-session postcommit sweep now runs at /v1/completions admission too (sessionless requests abort all pending commits, matching the changelog claim); NaN/Infinity sampler params get a clean 400 instead of a deep 500 now that anonymous controls are honored; release-notes wording fixed (exact-match, not prefix-match); internal receipts path dropped from a shipped docstring; dead test scaffolding removed.
…hat path Pre-merge correction (founder review of 52703d4): the /v1/completions cross-session sweep wrapped abort_cross_session_postcommits in a broad 'except Exception: pass', silently swallowing failures where the chat path lets them surface — the no-bug-masking rule. The swallow is gone; a sweep failure now propagates to the sanitized 500 handler identically to chat, and the sweep outcome lands in request observability (postcommit_cross_session_yield) for parity. Regression tests: completions invokes the sweep with except_session_id=None (sessionless: every pending commit is foreign), the env kill switch suppresses it, and a raising sweep surfaces as a 500 rather than letting the request proceed as if it had succeeded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 2.5.3. Small release focused on the agent lane and the API surface.
What's in it
the moment any request is admitted, whichever session they belong to.
Kills the occasional dead or sluggish agent turn (worst measured case:
a follow-up turn 44% slower with ~0.75s extra first-token latency).
Same-session grace preserved so streaming tool turns keep their prefix.
tokenize (77-92ms per request at agent transcript sizes; sub-millisecond
on hit). Warm follow-up latency in gate runs: 194-961ms band down to a
steady 65-74ms.
end with the TPS footer external tools counted as model output (~430ms
timing pollution, usage mismatch, temp-0 inequality). App/browser keep it.
from visible output for API clients.
explicit body params (temperature, top_p, top_k, thinking, penalties,
generation mode). Managed MTPLX clients stay server-owned. Closes A few bugs #241.
Rollback: MTPLX_CLIENT_CONTROLS_DEFAULT=hints.
depth-2 acceptance .526 -> .590.
Gates
512/8k both pairs; honor probe verified (anonymous temp=0 applied,
owner=client); zero regressions.
Two candidate optimizations were benchmarked and deliberately NOT shipped
after inversion in matched-state A/B (dense prefill chunk 4096, post-restore
eager default) — receipts in the release QA record.