Skip to content

fix: compose classifier output budgets and escalate length-stopped retries - #61

Open
ArietidsZ wants to merge 4 commits into
czottmann:mainfrom
ArietidsZ:fix/42-composed-classifier-budgets
Open

ArietidsZ wants to merge 4 commits into
czottmann:mainfrom
ArietidsZ:fix/42-composed-classifier-budgets

Conversation

@ArietidsZ

@ArietidsZ ArietidsZ commented Sep 22, 2026 •

Copy link
Copy Markdown

Refs #42
Refs #39

What changed

Both classifier stages now pass min(answer allowance + #51 reasoning reserve, model.maxTokens) as maxTokens. At explicit low that is 5296 for the detailed stage (1200 + 4096) and 4608 for the fast stage (512 + 4096). With no reasoning level set, the requests stay at 1200 and 512.

The #51 reserve is the OMP 18 table already used by the context-fit check (low 4096, xhigh/max 32768). It is not pi-ai 0.86.0's thinking-pad table (low 2048; xhigh and max clamp to high 16384). Sending the reserve keeps the completion-function argument equal to the amount the fit check reserved. It does not make that amount the provider's reasoning budget.

On a shared-ceiling provider the composed value is one output cap. Hidden reasoning can still consume it, including the room needed for classifier_decision. On a budget-thinking provider the adapter may add its own thinking budget on top of this value and then clamp to the context window, so the wire limit can exceed the reserve. At xhigh and max, a composed ceiling is the model output limit whenever that limit is at most the allowance plus the reserve (33,968 for the detailed stage, 33,280 for the fast stage).

A detailed-stage length stop used to retry the same request. It now retries once at the smaller of model.maxTokens and contextWindow - estimatedInput - 4096, and never below the first ceiling. When the escalated ceiling cannot exceed the first ceiling, no retry is sent; the action fails closed after the first length stop. On the staged path this happens when the model output limit already bounds the request; direct classifyWithRetry callers can also reach it through the estimated context room. A second length stop still fails closed. Malformed output still retries at the same ceiling. Timeout and provider errors still fail closed without a retry.

This can be a large retry. A model whose output limit is 384000 can be asked for 384000 tokens on the second attempt. classifierTimeoutMs bounds time, not that token reservation. Codex responses ignore maxTokens, so neither the composed ceiling nor the retry cap applies there.

ClassifierIoAttempt records requestMaxTokens. CHANGELOG and the classifier docs describe the ceiling as a raised cap, not as a partition between reasoning and the answer.

Why this shape

The fit check already reserved allowance + #51 reserve and the request sent only the allowance. Those two numbers now match at the completion-function boundary. That is an internal consistency fix. It is not evidence that a shared-ceiling model will finish inside the new cap. Whether the #39 truncations fit in 5296 was not measured.

There is no new configuration. #39 asks for a classifierDetailedMaxTokens knob. This PR does not add one. fastClassifierMaxTokens still changes the fast-stage allowance; it raises the sent ceiling only while the model output limit exceeds the allowance plus the reserve.

This replaces #43. That PR was built on 4d902a2, before the token estimator and the #51 reserves. It also sent model.maxTokens on the first attempt. This branch sends that only when the composed ceiling clamps to the model limit, or on a length retry.

Commit-message note: the first commit on this branch (daf6c36) says that on shared-ceiling providers reasoning "no longer consumes the decision tool call room". That claim is withdrawn; it predates the analysis above and is superseded by it. History is left intact.

Tests

The new tests pin the #51 reserve table, the composed ceilings, and the retry arguments, including the case where a base ceiling of 32000 is returned unchanged for a 10000-token window, and the skipped retry when the ceiling cannot rise (the model output limit equal to the base, and a base above the estimated context room). A staged-path test covers the production wiring: explicit xhigh with the composed ceiling clamped to model.maxTokens fails closed after a single detailed call on a length stop, verified red at the assertion level against the pre-skip source. They do not show that 5296 is enough for a shared-ceiling model. The first red check against unmodified source was an import failure from the new exports, not an assertion failure; the skip tests were verified red at the assertion level before the implementation. Full suite: 329 tests, 328 pass, one pre-existing skip. npm run check is clean.

Send each classifier stage the output room its context-fit check already
reserves: the answer allowance plus the reasoning budget. On providers
where hidden reasoning shares the output ceiling, reasoning no longer
consumes the decision tool call room.

After a length stop, the detailed retry now re-requests with the largest
context-safe ceiling (model output limit capped to context room) instead
of repeating the identical request. Attempts record requestMaxTokens.

Refs czottmann#42, czottmann#39
The composed maxTokens matches the czottmann#51 reserve. It does not partition
hidden reasoning from the answer, and the length retry is not context-safe
when the base ceiling already exceeds the window.
When the model output limit already bounds the request (at xhigh and
max, or on models whose output limit is at most the allowance plus the
reserve), the escalated retry ceiling equals the first ceiling. The
retry would repeat the identical request at up to the model output
limit. Fail closed after the first length stop instead.

Also document the fast-stage composed ceiling at xhigh and max, mark
the ceiling raise as explicit-level only, and derive the 33,968 and
33,280 figures from the allowance plus the reserve.
- flow.md: drop the "the retry keeps it" sentence; the skip falsifies
  it (the new staged-path test asserts a single call in that case).
- classifier.ts: the skip comment named only the model output limit as
  the bounding cause; the estimated context room also bounds direct
  classifyWithRetry callers.
- configuration.md: qualify the fastClassifierMaxTokens advice; raising
  the allowance does not change the sent ceiling once the model output
  limit is at most the allowance plus the reserve.
- CHANGELOG: drop the (czottmann#42) marker; this change does not fix czottmann#42.
- tests: classifyInStages skip coverage at explicit xhigh with the
  composed ceiling clamped to model.maxTokens (verified red at the
  assertion level against c372f62).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants