Skip to content

bench: add route classifier benchmark and preclassifier - #17

Merged
phaedrus-on-git merged 2 commits into
mainfrom
spike/route-classifier-benchmark
Sep 26, 2026
Merged

phaedrus-on-git merged 2 commits into
mainfrom
spike/route-classifier-benchmark

Conversation

@phaedrus-on-git

@phaedrus-on-git phaedrus-on-git commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a labeled fast-vs-strong routing corpus, benchmark harness, and configurable keyword preclassifier for cheap high-confidence route decisions before the LLM classifier.

What changed:

  • Adds test/bench/router-routing.cases.json with 20 labeled routing cases.
  • Adds tools/route_classifier_bench.py for guardrail, keyword, and live OpenAI-compatible classifier comparisons.
  • Adds routing.preclassifier config with target keywords, confidence threshold, and negation phrase handling.
  • Keeps routing order conservative: explicit guardrail rules -> preclassifier -> LLM classifier -> default target.

Initial benchmark results:

  • guardrails: 10/20, ~0ms
  • keywords: 20/20, avg ~0.003ms
  • live local-coder-fast classifier: 20/20, avg ~13.6s, with first cold case ~133s and warm cases mostly 6-8s

Validation:

  • python3 tools/route_classifier_bench.py --cases test/bench/router-routing.cases.json --candidates guardrails,keywords
  • go test ./...
  • make check

@phaedrus-on-git phaedrus-on-git changed the title bench: add routing classifier benchmark harness bench: add route classifier benchmark and preclassifier Sep 26, 2026
@phaedrus-on-git
phaedrus-on-git marked this pull request as ready for review September 26, 2026 02:15
@phaedrus-on-git
phaedrus-on-git merged commit 8e98190 into main Sep 26, 2026
8 checks passed
@phaedrus-on-git
phaedrus-on-git deleted the spike/route-classifier-benchmark branch September 26, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant