Proposal: request-fit routing gate for two-tier routes

[panpan0000]

Proposal: length-based difficulty proxy for chat routing

Status: draft (v2, replaces the "request-fit routing gate" framing).

Motivation

Switchyard's scorecard/difficulty classifier today only covers coding tasks. For chat routes we have no notion of "easy" vs "hard" at all — everything either goes to one target, or routing depends on a judge we haven't built for chat yet.

We want a cheap, no-training-required proxy for difficulty in chat routes, so we can send obviously-simple requests to the efficient model and obviously- demanding ones to the capable model, without waiting for a real chat scorecard.

Length is the cheapest signal we have:

• Long inputs are more likely to need a large context window.
• Small models tend to degrade faster than large ones on long-context retrieval/reasoning, even when the input technically fits.
  It is not a good proxy for difficulty in general — a one-line prompt can be a hard theorem, a 50-page doc can be a trivial extraction — but it is cheap, available today, and directionally correct for the "long input" half of the problem.

What this is not

• Not a difficulty classifier. It never says "this prompt is short, so send it to the cheap model." It only escalates on length. Absence of length escalation means "no opinion", not "confirmed easy".
• Not a replacement for a future chat scorecard. It's a stopgap for the one signal we can act on without training anything.

Options considered

1. Length threshold only (chosen for v1) — estimate input size (chars/4), escalate to the capable model above a threshold, otherwise leave the decision to whatever comes next (default target, or a judge if one is configured). Zero training, zero extra latency, immediately deployable.
2. Keyword/regex heuristics — detect phrases suggesting hard reasoning ("prove", "derive", "why does X fail") or simple tasks ("summarize", "list", "extract"). Cheap, but brittle and high maintenance; rejected for v1, worth revisiting as a cheap additional signal later.
3. Self-triage by the efficient model — have the small model judge its own confidence/difficulty before answering. No new infra, but costs an extra call (or requires prompting tricks), and ties routing quality to the small model's self-assessment, which is exactly what we don't trust for hard cases. Rejected for v1, interesting as a second-tier signal.
4. Trained chat difficulty classifier (the real fix) — a chat-side scorecard analogous to the coding one. Most accurate, highest cost (data, training, maintenance). This is the actual long-term answer; this proposal is the stopgap until it exists.
5. Do nothing — fine only until we deploy an efficient tier with a small context window; then every long chat session pays for a wasted call to the small model before falling back.

v1 ships option 1. Options 2–3 are candidate follow-ups layered on top of the same classifier slot. Option 4 is the real destination; this issue is not meant to replace or preempt that work.

Design

One classifier, matching the existing LlmTaskClassifier pattern: classifier core, FallThrough wrapper, one RouteConfig variant.

est = estimate_input_tokens(request)   # chars/4 over instructions + messages + tools
est >= escalate_over_input_tokens  → route to capable tier
otherwise                          → abstain

"Abstain" matters for composability: if a route has no judge configured, abstain is equivalent to your plain if/else — it just falls through to the default (efficient) target. If a route does have a judge configured later (e.g. a future chat scorecard), the length check runs first as a cheap pre-filter, and the judge only runs for requests that don't already need escalation. Same classifier, no special-casing needed either way.

Config:

[routes.auto]
type = "length_gate"
efficient_target = "small-model"
capable_target = "big-model"
escalate_over_input_tokens = 24000

Threshold is in tokens even though the estimator counts characters — no tokenizer dependency for v1. If chars/4 proves too inaccurate for some model family, add a per-route chars_per_token knob later.

Tests

Estimator: text, tool calls/results, multimodal blocks, instructions; behavior exactly at the threshold.
Abstain path: no judge configured → falls through to default target (equivalent to plain if/else).
Abstain path: judge configured → judge decides for non-escalated requests, is never called for escalated ones.
Server: TOML round-trip, unknown fields rejected, missing threshold rejected.

Open questions

Naming: length_gate vs something else?
v1 as a standalone route type only, or also expose the classifier for stage_router-style composition from day one?
Is chars/4 good enough across the tokenizers we actually deploy, or does v1 need the chars_per_token knob now rather than later?
Should we track this issue as a placeholder for option 4 (trained chat scorecard), or open that as a separate proposal once this ships?

Related work

See #445 / #453 for a complementary chat-routing approach: instead of a length proxy, it uses an LLM-judged capability card and in-session "regret" signals (user corrections/re-asks) to escalate. The two are meant to be composable —
length_gate is a domain-agnostic pre-filter for context-window risk, while #445's approach targets the "no verifier" difficulty-judgment problem itself.
A route could plausibly use both: length_gate as an early abstain-capable pre-check, falling through to the conversation classifier for everything else.

[nachiketb-nvidia]

Hi @panpan0000 few notes and questions on the proposal

• Why not use a standard tokenizer like tiktoken instead of chars? tiktoken has an efficent rust crate we can use directly via oncelocks and gives a better estimate
• can we change naming from "weak" and "strong" to "efficient" and "capable"
• What is the hypothesis/use case of such routing? We currently have context overflow mitigations in client. Colloquially, in our experiments, we've seen the results are overall better if the frontier model starts off rather than the smaller model
• Wouldn't this be better served as a classifier trait implementation to be used by fallthrough?

[panpan0000]

Thanks for the questions @nachiketb-nvidia —
you were right , the original write-up buried the actual motivation and over-engineered the mechanism.

I rewrote the proposal (see updated issue body).

in short :
Motivation our scorecard/difficulty classifier only covers coding. Chat routes have no difficulty signal at all.

Length is the cheapest proxy we have today (long input → more likely to need a bigger window, and small models degrade faster on long context even when it fits). #445 will another (more complex) way to resolve chat crouting.
It's not a real difficulty classifier — that's a separate, bigger effort (chat-side scorecard) — this is a stopgap.

sorry , the previous draft was over-designed and I just fixed

[bhuvan002]

Hi @panpan0000 , thank you for the proposal. Could you add a benchmarking plan please along with the baselines you wish to compare this with? All the claims should be experimentally validated.