llm-cascade

Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey

Cascaded Framework for Cost-Aware LLM Serving

Sample code for two-tier LLM inference (cascading) on Amazon Bedrock, using a single Bedrock Converse API surface. NVIDIA Nemotron Nano handles routine support-ticket classification on every request; Anthropic Claude Sonnet handles cases the routing logic flags as harder. Includes a bake-off harness, sample data, and tests. Next.js + TypeScript.

How much of a frontier model's tool-calling ability can you hand to a model that costs 1/60th as much? Context engineering -> prompt compilation -> inference scaffolds -> distillation, each measured as % of the frontier gap closed, on a hidden split, for under $20.

A self-distilling neuro-symbolic cascade that amortises LLM cost across knowledge-graph QA and regulatory-compliance checking, with auditable Datalog proof trees.

Cost aware LLM routing over MCP, and the 417 task benchmark that decides its policy.

In-browser LLM router over 12 models from 6 providers, with a cheapest-first cascade that grades and escalates

A leakage-controlled, paired-bootstrap-rigorous study of whether an LLM's internal TopK Sparse Autoencoder (SAE) feature spaces encode a difficulty-predictive routing signal. Benchmarked on HellaSwag (rigorous predictive null) and SQuAD continuous perplexity (positive Pareto cascade), with Platt recalibration and causal ablatings.

Authoring checklist for deliberate model downshifting: a strong model writes a next-session prompt a cheaper model can execute, escalating only at pre-named forks.