Distill a repeated Jev classification task into a local model, on the fly — same answers, your hardware.
-
Updated
Nov 11, 2026 - Python
Distill a repeated Jev classification task into a local model, on the fly — same answers, your hardware.
Cost-efficient coding-agent runtime: prefix-aware prompt-cache routing, poisoning-safe semantic cache, budgeted context compression, request batching/dedup, cascading model routing with DP-optimal policies, hard budget scheduling, span-level cost attribution, sandboxed tools, synthetic workload factory. NumPy core, CPU-only torch extra.
⚖️ Kadı — a calibrated judge in front of your frontier LLM. Routine decisions answered locally with a conformal-prediction guarantee (≤ α disagreement with the oracle); everything else escalated and learned from. Zero-dependency Python.
Train a small local model for one recurring task and measure when to fall back to the original API.
Confidence-gated LLM cascades that measure how often the cheap tier is wrong on the cases it answers alone.
Support inbox that routes each ticket to the cheapest model that can handle it: fine-tuned MiniLM first, LLM only when unsure.
Execution-based routing between a small and a large LLM for text-to-SQL. A 7B answers first, a classifier on its executed SQL decides when to escalate to a 32B. Matches the 32B on BIRD Mini-Dev at 30% escalation and 29% of the cost. Pre-registered experiments.
Cost-aware routing across agentic pipelines. Empirical benchmarks of accuracy, cost, and dangerous rate.
OpsPilot: the on-call engineer for Kubernetes that never sleeps. It finds what broke, names the root cause and applies a small fix, and every command is checked before it runs. 94% of incidents solved on free-tier LLMs, with 69% fewer big-model tokens and zero unsafe commands.
Studying Minimum Sufficient Inference: when objective execution evidence can stop LLM inference without sacrificing reliability.
Adaptive Cascade Tuning via Importance Sampling
AI Evals: evaluating and improving an AI customer support agent. Error analysis, a validated LLM judge, a CI quality gate that blocks regressions, drift monitoring with human review of every alert, targeted fixes compared on a score vs cost frontier, and cost reduction with a judge cascade..
LLM serving router that prices every request in dollars and milliseconds. Semantic cache plus a confidence-gated cheap-to-GPT-4 cascade: 96.9% lower cost than always calling the strong model, within 1.6 accuracy points, median latency 450ms to sub-ms on repeated traffic. FastAPI, SQLite cost ledger, reproducible benchmark.
Distill a frontier-model task (gpt-oss-120b, banking77 intents) into a 1.5B QLoRA student served on vLLM, with a confidence router that escalates hard queries to the teacher. Measured: 90% of teacher accuracy alone, 99.7% routed at 28% escalation, break-even ~313K req/day. Spend-capped, Cloud Run + Terraform ready.
Two-tier AI log triage router written entirely in BigQuery SQL. AI.IF filters noise, AI.CLASSIFY + AI.SCORE rank blast radius, and only severity ≥ 8 incidents escalate to a stronger model via AI.GENERATE. Includes 20 fictional sample logs with built-in traps.
NSGA-II search framework for CIFAR-10 big/little dynamic inference cascades under embedded memory constraints.
Route each LLM call to the cheapest model that will get it right. Verifier-gated cascades with an offline threshold optimizer and cost accounting that counts every verifier call at current list prices.
Model cascade with a verification gate: try the cheapest model first, escalate only when tests or a cross-vendor judge reject the result.
A cost-aware decision layer for routing text requests: a cheap model first, an LLM only when confidence is low, a person when both are unsure. FastAPI service with a React dashboard, measured for accuracy, cost and latency.
To associate your repository with the model-cascade topic, visit your repo's landing page and select "manage topics."