Skip to content

benchmarking.md

Latest commit

 

History

History
140 lines (111 loc) · 4.15 KB

benchmarking.md

File metadata and controls

140 lines (111 loc) · 4.15 KB
Raw

Benchmarking

DevRail Router includes a small streamed benchmark harness for comparing OpenAI-compatible model aliases such as local-coder.

The harness sends fixed chat-completion cases with stream=true and stream_options.include_usage=true, then writes one JSON object per case to stdout. Each result captures:

  • case ID
  • model alias
  • router request ID, when returned
  • HTTP status
  • time to first SSE event
  • total request duration
  • response bytes
  • prompt, completion, and total tokens when the backend emits streamed usage
  • a short first-content sample for sanity checking
  • a bounded content sample for lightweight quality comparison
  • a bounded reasoning sample when a backend streams OpenAI-style reasoning_content

Local-Coder Baseline

Run the default coding-oriented cases against the llm-srv router:

go run ./cmd/devrail-router bench \
  -base-url http://llm-srv-01.mfsoho.linkridge.net:18080/v1 \
  -model local-coder \
  -cases test/bench/local-coder.cases.json \
  -max-tokens 512

Save a baseline:

go run ./cmd/devrail-router bench \
  -base-url http://llm-srv-01.mfsoho.linkridge.net:18080/v1 \
  -model local-coder \
  -cases test/bench/local-coder.cases.json \
  -max-tokens 512 \
  > local-coder-baseline.jsonl

Run the same cases against another alias, such as an experimental local-coder-parallel, by changing -model only. Keeping the case file and token cap stable makes queue wait, first-token latency, duration, and token throughput easier to compare in Grafana.

Custom Cases

Case files are JSON arrays. A case can use a simple prompt:

[
  {
    "id": "small-refactor",
    "prompt": "Refactor this Go function and explain the tradeoff."
  }
]

Or an explicit OpenAI-style message list:

[
  {
    "id": "reviewer",
    "messages": [
      {"role": "system", "content": "You are a concise Go reviewer."},
      {"role": "user", "content": "Find the highest-risk bug in this proxy."}
    ]
  }
]

Use stable, short IDs. They appear in JSONL output and make it easier to line up command results with router request IDs, logs, and Prometheus samples.

Hard-Thinking Comparison

Use test/bench/hard-thinking.cases.json when comparing a normal coding alias with a slower planner/reviewer alias:

go run ./cmd/devrail-router bench \
  -base-url http://llm-srv-01.mfsoho.linkridge.net:18080/v1 \
  -model local-coder \
  -cases test/bench/hard-thinking.cases.json \
  -max-tokens 768 \
  > local-coder-hard-thinking.jsonl

go run ./cmd/devrail-router bench \
  -base-url http://llm-srv-01.mfsoho.linkridge.net:18080/v1 \
  -model local-coder-deep \
  -cases test/bench/hard-thinking.cases.json \
  -max-tokens 768 \
  -timeout 20m \
  > local-coder-deep-hard-thinking.jsonl

The deep alias may spend early tokens on reasoning before emitting normal content, so use a larger token cap than a smoke test. Compare both timings and the content_sample field before making a slower backend automatic.

Routing Classifier Benchmarks

Use the route-classifier benchmark to compare fast-vs-strong selection policies without sending full generation requests through the router:

python3 tools/route_classifier_bench.py \
  --cases test/bench/router-routing.cases.json \
  --candidates guardrails,keywords

Add openai to compare a live OpenAI-compatible classifier model:

python3 tools/route_classifier_bench.py \
  --candidates guardrails,keywords,openai \
  --openai-base-url http://llm-srv-01.mfsoho.linkridge.net:18080/v1 \
  --openai-model local-coder-fast

The tool writes one JSON object per candidate/case to stdout and prints per candidate accuracy/timing summaries to stderr.

Router selection order is:

  1. explicit routing rules, such as prompt-size and output-token guardrails
  2. optional keyword preclassifier for high-confidence cheap decisions
  3. optional OpenAI-compatible LLM classifier for ambiguous requests
  4. the model alias default target

Use the benchmark corpus to tune the preclassifier keyword list before enabling it in deployed config. Negated phrases such as no production and without security impact should remain fall-through cases so they can reach the LLM classifier or default route.