Merge pull request #7 from CommonstackAI/feat/open-data-pipeline-site

Open data pipeline and project site

Author: Amorth

CHANGELOG.md

@@ -2,6 +2,26 @@
 
 All notable changes to **TwinRouterBench** are documented in this file.
 
+## [Unreleased]
+
+### Added
+
+- Benchmark-agnostic pipeline configuration and the stable
+  `run_pipeline(config)` / `twinrouterbench data run --config` interfaces.
+- Configurable normalized and single-turn loaders; execution, backend-judge,
+  exact-match, and contains evaluators; loader/evaluator/backend plugin contracts.
+- Generic OpenAI-compatible executor and a config-only custom QA example.
+- Pipeline/suite validation commands and rejection of inline credentials.
+
+- Versioned static data-construction package and `twinrouterbench data` CLI.
+- Paper-aligned sequential-locking downgrade search, tier-pool cascade,
+  mixed-prefix reconstruction, hardened open-ended judge, and manual audit flow.
+- SWE-bench, BFCL, mtRAG, QMSum, and PinchBench adapters with deterministic
+  offline fixtures.
+- Mock, replay, and optional live-plugin backends; guarded release-candidate
+  publication and source/license provenance validation.
+- Offline unit, end-to-end, CLI, replay-equivalence, and golden-hash tests.
+
 ## [0.1.0] - 2026-04-30
 
 ### Changed

CITATION.cff

@@ -0,0 +1,88 @@
+cff-version: 1.2.0
+message: "If you use TwinRouterBench, please cite the paper and this software."
+title: "TwinRouterBench"
+type: software
+version: 0.1.0
+repository-code: "https://github.com/CommonstackAI/TwinRouterBench"
+url: "https://commonstackai.github.io/TwinRouterBench/"
+license: Apache-2.0
+authors:
+  - family-names: Yang
+    given-names: Pei
+  - family-names: Chen
+    given-names: Wanyi
+  - family-names: Yang
+    given-names: Tongyun
+  - family-names: Feng
+    given-names: Pengbin
+  - family-names: Xing
+    given-names: Jiarong
+  - family-names: Guo
+    given-names: Wentao
+  - family-names: Yao
+    given-names: Yuhang
+  - family-names: Han
+    given-names: Yuhang
+  - family-names: Li
+    given-names: Hanchen
+  - family-names: Wang
+    given-names: Xu
+  - family-names: Wang
+    given-names: Zeyu
+  - family-names: Xiao
+    given-names: Jie
+  - family-names: Yang
+    given-names: Anjie
+  - family-names: Tian
+    given-names: Liang
+  - family-names: Ai
+    given-names: Lynn
+  - family-names: Yang
+    given-names: Eric
+  - family-names: Shi
+    given-names: Tianyu
+identifiers:
+  - type: doi
+    value: "10.48550/arXiv.2605.18859"
+    description: "arXiv-issued DOI"
+preferred-citation:
+  type: article
+  title: "TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing"
+  year: 2026
+  doi: "10.48550/arXiv.2605.18859"
+  url: "https://arxiv.org/abs/2605.18859"
+  authors:
+    - family-names: Yang
+      given-names: Pei
+    - family-names: Chen
+      given-names: Wanyi
+    - family-names: Yang
+      given-names: Tongyun
+    - family-names: Feng
+      given-names: Pengbin
+    - family-names: Xing
+      given-names: Jiarong
+    - family-names: Guo
+      given-names: Wentao
+    - family-names: Yao
+      given-names: Yuhang
+    - family-names: Han
+      given-names: Yuhang
+    - family-names: Li
+      given-names: Hanchen
+    - family-names: Wang
+      given-names: Xu
+    - family-names: Wang
+      given-names: Zeyu
+    - family-names: Xiao
+      given-names: Jie
+    - family-names: Yang
+      given-names: Anjie
+    - family-names: Tian
+      given-names: Liang
+    - family-names: Ai
+      given-names: Lynn
+    - family-names: Yang
+      given-names: Eric
+    - family-names: Shi
+      given-names: Tianyu

README.md

@@ -1,18 +1,78 @@
-# Twin Router Bench
+<div align="center">
 
-[![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/html/2605.18859)
-[![Code](https://img.shields.io/badge/Code-GitHub-24292f)](https://github.com/CommonstackAI/TwinRouterBench)
+# TwinRouterBench
+
+### Fast static supervision. Live agentic routing evaluation.
+
+**A two-track benchmark for choosing the cheapest sufficient LLM at every agent step.**
+
+[![Paper](https://img.shields.io/badge/Paper-arXiv%3A2605.18859-b31b1b)](https://arxiv.org/abs/2605.18859)
 [![Dataset](https://img.shields.io/badge/Dataset-Hugging%20Face-ffcc4d)](https://huggingface.co/datasets/Amorph/TwinRouterBench)
-[![Website](https://img.shields.io/badge/Website-Project-2ea44f)](https://commonstackai.github.io/TwinRouterBench/)
+[![Leaderboard](https://img.shields.io/badge/Leaderboard-Live-0f766e)](https://commonstackai.github.io/TwinRouterBench/)
+[![License](https://img.shields.io/badge/License-Apache--2.0-2563eb)](LICENSE)
+[![GitHub Stars](https://img.shields.io/github/stars/CommonstackAI/TwinRouterBench?style=flat&color=f97316)](https://github.com/CommonstackAI/TwinRouterBench/stargazers)
+
+[**Project Page**](https://commonstackai.github.io/TwinRouterBench/) ·
+[**Paper**](https://arxiv.org/abs/2605.18859) ·
+[**Dataset**](https://huggingface.co/datasets/Amorph/TwinRouterBench) ·
+[**Data Pipeline**](docs/DATA_GENERATION.md) ·
+[**Submit a Router**](https://github.com/CommonstackAI/TwinRouterBench/issues/new?template=leaderboard_submission.yml)
+
+</div>
 
-**Twin Router Bench** is a single benchmark suite for **per-step LLM routing**: a *router* chooses which pooled `model_id` to use on every agent step, under locked pricing and cache rules. The suite ships in one Python distribution (**`twinrouterbench`**) and one source tree (**`TwinRouterBench/`**).
+<p align="center">
+  <img src="leaderboard/assets/overview.svg" width="100%" alt="TwinRouterBench combines execution-verified static supervision with live SWE-bench routing evaluation." />
+</p>
 
-It contains **two tracks** inside the same product—same protocol and locked tables—not two separate benchmarks:
+## Why TwinRouterBench?
 
-| Track | Role | CLI entry |
-|-------|------|-----------|
-| **Static** | Fast validation on a fixed supervision bank (tier labels + nominal cost metrics). | `twinrouterbench static …` |
-| **Dynamic** | End-to-end evaluation on **SWE-bench Verified** with real tool use—**mini-swe-agent** scaffold or **editor** scaffold. | `twinrouterbench dynamic …` / `twinrouterbench swe …` |
+Most router benchmarks make one model choice for a complete prompt. Real coding,
+research, and computer-use agents make many calls under changing tool state and
+conversation history. TwinRouterBench evaluates the decision the router actually
+faces: **which model should handle the next call given the full current prefix?**
+
+| Track | What it measures | Public artifact |
+|---|---|---|
+| **Static** | Cheapest-sufficient tier at each routed step, verified through downgrade search and mixed-model execution. | **970 labels**, **520 trajectories**, **5 workloads** |
+| **Dynamic** | End-to-end task success and realized API spend while routing every live agent call. | **100 held-out SWE-bench Verified cases** |
+
+> **Headline result.** The trained router resolves **75/100** held-out cases for
+> **$25.66**, compared with **74/100** for the **$54.73** all-Opus reference—a
+> **53.1% cost reduction** with comparable task success.
+
+## Quick start
+
+```bash
+pip install -e .
+
+# Reproduce the complete offline data-construction flow without API keys.
+twinrouterbench data generate \
+  --benchmark all \
+  --backend mock \
+  --output-dir runs/data-generation/mock-all
+
+# Add and run a new benchmark using only data + JSON configuration.
+twinrouterbench data run \
+  --config configs/data_generation/custom_qa_pipeline.json \
+  --output-dir runs/data-generation/custom-qa
+```
+
+The package exposes four connected surfaces:
+
+```bash
+twinrouterbench data ...       # config-driven construction/review/publish pipeline
+twinrouterbench static ...     # deterministic offline router scoring
+twinrouterbench dynamic ...    # live mini-swe-agent evaluation
+twinrouterbench swe ...        # editor-scaffold SWE-bench evaluation
+```
+
+| Resource | Link |
+|---|---|
+| Interactive leaderboard and protocol | [Project page](https://commonstackai.github.io/TwinRouterBench/) |
+| Static question bank | [Hugging Face](https://huggingface.co/datasets/Amorph/TwinRouterBench) |
+| Data-generation methodology and CLI | [docs/DATA_GENERATION.md](docs/DATA_GENERATION.md) |
+| Leaderboard submission rules | [docs/SUBMISSION.md](docs/SUBMISSION.md) |
+| Citation metadata | [CITATION.cff](CITATION.cff) |
 
 ---
 
@@ -100,6 +160,7 @@ Ensure a checkout exists at `semantic-router/` relative to the checkout parent d
 Primary entrypoint:
 
 ```bash
+twinrouterbench data <subcommand> [args]     # construct/review/publish data
 twinrouterbench static <subcommand> [args]   # static track
 twinrouterbench dynamic <subcommand> [args]  # mini-swe-agent harness
 twinrouterbench swe <subcommand> [args]     # editor-scaffold harness
@@ -372,7 +433,9 @@ Under `TwinRouterBench/scripts/examples/`:
 | `swerouter/` | Router protocol, pricing, cache simulation, harness, and leaderboard. |
 | `data/static/` | Static track JSONL: `question_bank.jsonl`, `manifest.json`. |
 | `data/dynamic/` | Dynamic track locked JSON: `model_pool.json`, `model_pricing.json`, `ttl_policy.json`, `tier_to_model.json`, `sr_knn_to_pool.json`, plus `dynamic_heldout100_ids.txt` (paper’s 100-case SWE-bench Verified held-out split). |
-| `twinrouterbench/` | Meta-CLI dispatcher. |
+| `twinrouterbench/data_generation/` | Static data construction, review, replay, provenance, and publish pipeline. |
+| `leaderboard/` | Project page and interactive static/dynamic leaderboard published through GitHub Pages. |
+| `twinrouterbench/` | Unified CLI dispatcher. |
 | `.env.example` | Template for gateway credentials. |
 
 ---
@@ -394,15 +457,28 @@ Do **not** upload run logs to the Hugging Face dataset page.
 
 ## Citation
 
-If you use Twin Router Bench in research, please cite the associated paper. **Bibliographic details are withheld for anonymous review** and will be added after publication (no preprint URL in this release).
+If you use TwinRouterBench, please cite the paper and this repository. Machine-readable metadata is available in [`CITATION.cff`](CITATION.cff).
+
+```bibtex
+@misc{yang2026twinrouterbench,
+  title         = {TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing},
+  author        = {Pei Yang and Wanyi Chen and Tongyun Yang and Pengbin Feng and Jiarong Xing and Wentao Guo and Yuhang Yao and Yuhang Han and Hanchen Li and Xu Wang and Zeyu Wang and Jie Xiao and Anjie Yang and Liang Tian and Lynn Ai and Eric Yang and Tianyu Shi},
+  year          = {2026},
+  eprint        = {2605.18859},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.LG},
+  doi           = {10.48550/arXiv.2605.18859},
+  url           = {https://arxiv.org/abs/2605.18859}
+}
+```
 
 
 ## Implementation note (CLI forwarding)
 
-`twinrouterbench static|dynamic|swe` dispatches in-process to the existing CLIs. For debugging, you may still invoke `python -m miniswerouter.cli` or `python -m swerouter.cli` with `PYTHONPATH` set to `TwinRouterBench/`.
+`twinrouterbench data|static|dynamic|swe` dispatches in-process to the corresponding CLIs. For debugging, you may still invoke `python -m twinrouterbench.data_generation.cli`, `python -m miniswerouter.cli`, or `python -m swerouter.cli` with `PYTHONPATH` set to `TwinRouterBench/`.
 
 ---
 
 ## Appendix: migration
 
-This tree unifies the static and dynamic router benchmark tracks in one install. Use **Twin Router Bench** naming in new scripts; keep legacy console script names and pathnames only where required for backward compatibility.
+This tree unifies the static and dynamic router benchmark tracks in one install. Use **TwinRouterBench** in new documentation and scripts; keep legacy console-script names and pathnames only where backward compatibility requires them.

configs/data_generation/commonstack_smoke.json

@@ -0,0 +1,15 @@
+{
+  "run_id": "commonstack-live-smoke-2026-08-18",
+  "backend_name": "live",
+  "seed": 20260818,
+  "review_rate": 0,
+  "cascade_size": 1,
+  "max_cases_per_benchmark": 1,
+  "collector": "twinrouterbench-commonstack-live-smoke",
+  "collected_at": "2026-08-18",
+  "model_pool_path": "configs/data_generation/commonstack_smoke_models.json",
+  "generation_parameters": {
+    "temperature": 0,
+    "top_p": 1
+  }
+}

configs/data_generation/commonstack_smoke_models.json

@@ -0,0 +1,9 @@
+{
+  "version": "commonstack-live-smoke-2026-08-18",
+  "tiers": {
+    "low": ["openai/gpt-4o-mini"],
+    "mid": ["deepseek/deepseek-v3.2"],
+    "mid_high": ["google/gemini-3-flash-preview"],
+    "high": ["openai/gpt-5.4-nano-2026-03-17"]
+  }
+}

configs/data_generation/custom_qa_pipeline.json

@@ -0,0 +1,48 @@
+{
+  "schema": "twinrouterbench.data_pipeline.v1",
+  "suite_version": "custom-qa-example-v1",
+  "generation": {
+    "run_id": "custom-qa-config-only",
+    "backend_name": "mock",
+    "review_rate": 0,
+    "cascade_size": 1,
+    "collector": "config-only-example",
+    "collected_at": "2026-08-18",
+    "model_pool_path": "commonstack_smoke_models.json"
+  },
+  "backend": "mock",
+  "benchmarks": {
+    "custom_qa": {
+      "display_name": "Custom QA",
+      "scenario": "single_turn_qa",
+      "multi_step": false,
+      "manual_review": false,
+      "source": {
+        "uri": "local://custom_qa_tasks.jsonl",
+        "license": "CC0-1.0",
+        "version": "example-v1"
+      },
+      "loader": {
+        "type": "single_turn",
+        "options": {
+          "system_prompt": "Answer the question concisely.",
+          "field_map": {
+            "instance_id": "id",
+            "prompt": "question",
+            "reference": "answer",
+            "target_tier": "target_tier",
+            "hint": "hint"
+          }
+        }
+      },
+      "evaluation": {
+        "trial": "execution",
+        "final": "execution"
+      }
+    }
+  },
+  "run": {
+    "benchmarks": "all",
+    "output_dir": "../../runs/data-generation/custom-qa"
+  }
+}

configs/data_generation/custom_qa_tasks.jsonl

@@ -0,0 +1,2 @@
+{"id":"capital-france","question":"What is the capital of France?","answer":"Paris","target_tier":"low","hint":"low"}
+{"id":"explain-cache","question":"Explain why a cache TTL is useful in one sentence.","answer":"A cache TTL limits how long stale data can be reused.","target_tier":"mid","hint":"low"}

configs/data_generation/mock_all.json

@@ -0,0 +1,13 @@
+{
+  "run_id": "mock-all-v1",
+  "backend_name": "mock",
+  "seed": 20260508,
+  "review_rate": 0.1,
+  "cascade_size": 3,
+  "collector": "twinrouterbench-reference-pipeline",
+  "collected_at": "2026-05-08",
+  "generation_parameters": {
+    "temperature": 0,
+    "top_p": 1
+  }
+}

data/README.md

@@ -6,3 +6,7 @@
 | **`dynamic/`** | Dynamic | Locked JSON for SWE runs: model pool, pricing, TTL policy, tier map, SR-KNN label mapping, etc. Also `dynamic_heldout100_ids.txt` — the paper’s fixed 100 SWE-bench Verified held-out instance IDs (see top-level README). |
 
 Code defaults: `main.dataset.STATIC_DATA_DIR` → `TwinRouterBench/data/static/`; harness defaults for pool/pricing → `TwinRouterBench/data/dynamic/`.
+
+The construction code that produces static-track candidates lives under
+`twinrouterbench/data_generation/`. It writes isolated run directories and does
+not overwrite this directory. See [`../docs/DATA_GENERATION.md`](../docs/DATA_GENERATION.md).