Proposal: conversation routing without a verifier

[panpan0000]

Proposal: Conversation Routing Without a Verifier

Status: draft.

TL;DR

Switchyard's difficulty classifier (capability classifier + escalation judge) was built for coding tasks. Every rule in it assumes a machine-verifiable outcome — a test passes, a build succeeds, a checker approves.

Conversation tasks — chat, Q&A, writing, summarization, tutoring, roleplay — don't have that. There's no verifier. The only judge of quality is the user.

Right now, those requests all fall into a dead zone: primary_rule=none, capability_boundary=unmatched. The classifier spits out a p_solve with nothing grounding it, and the escalation judge has no tests or builds to read. Routing a conversation request to the efficient or capable model is basically a coin flip.

Why not just bolt on a "conversation" rule card?

Because the existing capability card is really a verifiable-outcome card. Its definition of success ("whole task correct, as judged by the final verifier") and its p_solve calibration both assume a verifier exists.

Conversation needs a different definition of success — "rated acceptable by a reference evaluator" — and its own calibration set. Mixing that into the same p_solve scale would corrupt the calibration that's already tuned for coding.

Proposal — three layers, cheapest first

1. Decide the value prop. What does efficient vs. capable even mean for conversation — draft vs. polished? Correct vs. incorrect? Everything downstream depends on this.

2. Cheap default + regret signal. Don't try to forecast difficulty from the first message. Route by a deployment-level default, and escalate based on in-session user feedback — corrections, "that's wrong," "rephrase that," "not what I asked." The user is a free, reliable verifier, so use it.

3. Only if a real classifier is justified: build a separate mode = "conversation" (alongside capability/escalation/custom), with its own prompt, schema, and rules grounded in failure modes you can actually see from the instructions — factual faithfulness, multi-step reasoning, strict format adherence, high-stakes domains, long-document fidelity, multilingual handling. Calibrate against LLM-as-judge labels, since there's no verifier to calibrate against.

Prototype through the existing custom classifier mode before promoting anything to a first-class mode.

Non-goals

• No changes to the coding capability card or escalation judge — those stay as-is for coding.
• No difficulty guessing here. The request-fit length gate (Proposal: request-fit routing gate for two-tier routes #442) is a separate, domain-agnostic mechanism and isn't part of this proposal.

Relationship to #442 / #453

• Proposal: request-fit routing gate for two-tier routes #442 (length gate proposal, implemented in feat(libsy): add request-fit routing gate for two-tier routes #443) solves a different problem: is the input too long? It has nothing to do with conversation quality or user satisfaction — it's a domain-agnostic pre-filter.
• This proposal's implementation is feat(server): add config-only conversation routing example #453: a zero-code, config-only example using the existing llm_classifier custom mode, with two routes — one that scores against a conversation capability card, and one that implements layer 2 above (default efficient, escalate on regret).
• The two are meant to be complementary, not competing. A request could pass through the length gate first (cheap, domain-agnostic context-window check), and if the gate abstains, fall through to the conversation classifier described here.

Open questions

1. Should conversation routing even use an efficient/capable binary, or a different target set (e.g. draft/careful)?
2. Which conversation classes to support first — chat, Q&A, writing, tutoring?
3. LLM-as-judge or human labels for calibration?

[panpan0000]

One more note on implementation paths — two of the three can be validated with zero new code, since the existing llm_classifier machinery already covers them:

Approach	New code?	Cost
Pre-hoc scoring card (conversation rubric)	✅ None	Write a prompt + JSON schema into llm_classifier custom mode; needs calibration
Escalate on user regret	✅ None either	Swap the escalation judge's prompt (its input already includes the user's follow-up messages); costs one judge call per turn
Escalate on regret, cheaper variant	❌ One new classifier	The judge is only called when a regret pattern is detected in the user's turn; zero cost otherwise

Suggested order: validate the two zero-code paths first; only build the third if the per-turn judge cost turns out to matter.