From a2fa0a30ed61a82911d352aadd6b21e7af80ebc6 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Wed, 7 Oct 2026 15:13:38 +0200 Subject: [PATCH 01/17] Document native NePS tuning configuration and worker usage --- README.md | 59 +++++++++++++++++++++++++++++++++++++++++ configs/tune_judge.yaml | 57 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) create mode 100644 configs/tune_judge.yaml diff --git a/README.md b/README.md index 2367ca42..d4ca6ae0 100644 --- a/README.md +++ b/README.md @@ -339,6 +339,65 @@ Runs save the selected sample, judge evidence, metric battles, configuration, an `--meta_eval.split validation|test` restricts sampling to one half of the arena prompts (`--meta_eval.validation_fraction`, default 0.5). The split hashes each battle's prompt, so it does not depend on `--run.seed`. +### Tuning a judge + +The `tune-judge-*` tasks search judge configurations against the same human battles as their `meta-eval-*` counterparts, following [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Install `pip install 'judgearena[tune]'` for [NePS](https://github.com/automl/neps) 0.17 (`neural-pipeline-search`). + +[configs/tune_judge.yaml](configs/tune_judge.yaml) uses native NePS settings under `tune_judge.neps`: `pipeline_space` has dotted config names, a `type` (`Categorical`, `Float`, `Integer`, or `IntegerFidelity`), and native constructor kwargs such as `choices`, `lower`, `upper`, `prior`, `prior_confidence`, and `log` where supported. Categorical `prior` is a choice **index**. Every parameter with a `prior` must also set `prior_confidence` (the example uses `medium`). Prompt presets are scalar choices (`meta-eval-pair-score`, `arena-hard`, `default_with_explanation`); grouped overrides are unsupported. The example searches temperature continuously and uses `meta_eval.battles_per_model` as the validation fidelity. + +For example, a continuous temperature parameter under `tune_judge.neps.pipeline_space`: + +```yaml +judge.temperature: + type: Float + lower: 0.0 + upper: 1.0 + prior: 0.0 + prior_confidence: medium +``` + +- `optimizer: {name: neps_priorband, eta: 3}` uses parameter priors; `neps_hyperband` samples uniformly. Both require `objectives: [agreement]` outside `neps`. +- `optimizer: {name: mo_hyperband, eta: 3, mo_selector: epsnet}` and `primo` support `objectives: [agreement, cost_per_1k_battles]`. Agreement is maximized (NePS minimizes `1 - agreement`); cost is minimized. +- For `primo`, put native `prior_centers` settings in the optimizer mapping, keyed by `objective_0` (disagreement) and `objective_1` (cost). Each contains dotted parameter names mapped to native setting **values**, for example: + +```yaml +optimizer: + name: primo + eta: 3 + mo_selector: epsnet + prior_centers: + objective_0: + judge.model: VLLM/Qwen/Qwen2.5-7B-Instruct + judge.prompt_preset: meta-eval-pair-score + judge.temperature: 0.0 + judge.max_out_tokens: 2048 + judge.swap_mode: both + objective_1: + judge.model: VLLM/Qwen/Qwen2.5-7B-Instruct + judge.prompt_preset: meta-eval-pair-score + judge.temperature: 0.0 + judge.max_out_tokens: 1024 + judge.swap_mode: fixed +``` + +Each trial runs the matching `meta-eval-*` task on the validation split and shares its judgement cache. Cost uses input plus output tokens (counted with tiktoken), including cached outputs, and `price_per_million_tokens[judge.model]` for every searched model. The Qwen/Gemma prices are illustrative estimates; this objective estimates USD per 1,000 battles, not actual API spend. NePS cost spend limits are unsupported and rejected; use `neps.total_evaluations_to_spend` (40 in the example). + +The main process waits for active trials after `neps.run` returns under normal budget accounting, then selects the best agreement configuration per judge model at the highest fidelity for held-out testing. The native NePS completion flag is not a worker barrier. Optional `test_battles_per_model` defaults to the highest fidelity. + +With NePS 0.17, leave `sample_prior_first` at its default for `neps_priorband` and `neps_hyperband`: enabling it fails in upstream trial-ID parsing. Priors still guide sampling without that option. For `primo`, its initial design must reach the highest fidelity before it switches to Bayesian optimization. + +```bash +judgearena --config_path configs/tune_judge.yaml +# Resume with the same explicit tuning directory and settings. +judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir +# Helper workers join that directory and run only the search. +judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir --tune_judge.search_only true +``` + +`run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials and `test_results.parquet` holds the selected configurations' test results. For multiple workers on the same node, share a node-local `store_root` for SQLite judgement caching; multi-node NFS cache sharing is not supported. + +The primary process can extend the global evaluation or fidelity budget when resuming. Helpers inherit the persisted global limits. Search-space, optimizer, objective, and pricing changes require a new run directory. + ## 📈 Estimating ELO Ratings JudgeArena can estimate the ELO rating of a model by running it against opponents sampled from a human preference arena (`LMArena-100k`, `LMArena-140k`, or `ComparIA`). diff --git a/configs/tune_judge.yaml b/configs/tune_judge.yaml new file mode 100644 index 00000000..31c55945 --- /dev/null +++ b/configs/tune_judge.yaml @@ -0,0 +1,57 @@ +# NePS 0.17 judge-tuning example; install with pip install 'judgearena[tune]'. +# Run: judgearena --config_path configs/tune_judge.yaml +# Validation trials use the matching meta-eval task and its judgement cache. +task: tune-judge-lmarena-140k-en +judge: + model: VLLM/Qwen/Qwen2.5-7B-Instruct + prompt_preset: meta-eval-pair-score + temperature: 0.0 + max_out_tokens: 2048 + max_model_len: 16384 +meta_eval: + top_models: 20 +tune_judge: + neps: + pipeline_space: + judge.model: + type: Categorical + choices: [VLLM/Qwen/Qwen2.5-7B-Instruct, VLLM/google/gemma-2-9b-it] + prior: 0 # Categorical priors are choice indices. + prior_confidence: medium + judge.prompt_preset: + type: Categorical + choices: [meta-eval-pair-score, arena-hard, default_with_explanation] + prior: 0 + prior_confidence: medium + judge.temperature: + type: Float + lower: 0.0 + upper: 1.0 + prior: 0.0 + prior_confidence: medium + judge.max_out_tokens: + type: Integer + lower: 1024 + upper: 2048 + prior: 2048 + prior_confidence: medium + judge.swap_mode: + type: Categorical + choices: [fixed, both] + prior: 0 + prior_confidence: medium + meta_eval.battles_per_model: + type: IntegerFidelity + lower: 10 + upper: 90 + optimizer: {name: mo_hyperband, eta: 3, mo_selector: epsnet} + total_evaluations_to_spend: 40 + objectives: [agreement, cost_per_1k_battles] + # Estimated USD per million tokens, not measured spend or provider quotes. + price_per_million_tokens: + VLLM/Qwen/Qwen2.5-7B-Instruct: 0.12 + VLLM/google/gemma-2-9b-it: 0.14 + # Optional; null uses the highest validation fidelity for held-out testing. + test_battles_per_model: null + run_dir: null + search_only: false From ada6319583a9ffd47d459b5699b9f612fcea722e Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Wed, 7 Oct 2026 15:49:35 +0200 Subject: [PATCH 02/17] Document Pareto reports and continuing failed tuning trials --- README.md | 2 +- configs/tune_judge.yaml | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index d4ca6ae0..3a015e0e 100644 --- a/README.md +++ b/README.md @@ -394,7 +394,7 @@ judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/t judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir --tune_judge.search_only true ``` -`run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials and `test_results.parquet` holds the selected configurations' test results. For multiple workers on the same node, share a node-local `store_root` for SQLite judgement caching; multi-node NFS cache sharing is not supported. +`run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials, `pareto.parquet` reports the agreement/cost front of completed maximum-fidelity validation configurations, and `test_results.parquet` holds the selected configurations' test results. The example sets `neps.ignore_errors: true` so failed trials are recorded while the search continues. For multiple workers on the same node, share a node-local `store_root` for SQLite judgement caching; multi-node NFS cache sharing is not supported. The primary process can extend the global evaluation or fidelity budget when resuming. Helpers inherit the persisted global limits. Search-space, optimizer, objective, and pricing changes require a new run directory. diff --git a/configs/tune_judge.yaml b/configs/tune_judge.yaml index 31c55945..eee9201f 100644 --- a/configs/tune_judge.yaml +++ b/configs/tune_judge.yaml @@ -46,6 +46,7 @@ tune_judge: upper: 90 optimizer: {name: mo_hyperband, eta: 3, mo_selector: epsnet} total_evaluations_to_spend: 40 + ignore_errors: true # Record failed trials and continue the search. objectives: [agreement, cost_per_1k_battles] # Estimated USD per million tokens, not measured spend or provider quotes. price_per_million_tokens: From 41e7026f63401b52982951b21cbbef22142f011f Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Thu, 8 Oct 2026 00:11:46 +0200 Subject: [PATCH 03/17] Use fixed paper judge limits and document shared cache deployment --- README.md | 6 +++++- configs/tune_judge.yaml | 12 ++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 3a015e0e..39344abf 100644 --- a/README.md +++ b/README.md @@ -345,6 +345,8 @@ The `tune-judge-*` tasks search judge configurations against the same human batt [configs/tune_judge.yaml](configs/tune_judge.yaml) uses native NePS settings under `tune_judge.neps`: `pipeline_space` has dotted config names, a `type` (`Categorical`, `Float`, `Integer`, or `IntegerFidelity`), and native constructor kwargs such as `choices`, `lower`, `upper`, `prior`, `prior_confidence`, and `log` where supported. Categorical `prior` is a choice **index**. Every parameter with a `prior` must also set `prior_confidence` (the example uses `medium`). Prompt presets are scalar choices (`meta-eval-pair-score`, `arena-hard`, `default_with_explanation`); grouped overrides are unsupported. The example searches temperature continuously and uses `meta_eval.battles_per_model` as the validation fidelity. +Use categorical temperature choices (for example, `[0.0, 0.1, 1.0]`) when cache reuse matters: continuous sampling usually produces a new model descriptor for each value. + For example, a continuous temperature parameter under `tune_judge.neps.pipeline_space`: ```yaml @@ -394,7 +396,9 @@ judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/t judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir --tune_judge.search_only true ``` -`run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials, `pareto.parquet` reports the agreement/cost front of completed maximum-fidelity validation configurations, and `test_results.parquet` holds the selected configurations' test results. The example sets `neps.ignore_errors: true` so failed trials are recorded while the search continues. For multiple workers on the same node, share a node-local `store_root` for SQLite judgement caching; multi-node NFS cache sharing is not supported. +The example keeps the paper limits fixed across judges: 8,192 characters per candidate completion, 8,192 judge output tokens, and each model’s native context. Instructions are not truncated. A battle whose rendered prompt exceeds the judge’s context produces a failed annotation, lowering agreement and coverage; it stays in the shared battle set and its skipped output is not cached. Only parameters explicitly included in the search space vary between trials. + +`run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials, `pareto.parquet` reports the agreement/cost front of completed maximum-fidelity validation configurations, and `test_results.parquet` holds the selected configurations' test results. The example sets `neps.ignore_errors: true` so failed trials are recorded while the search continues. Shared SQLite judgement caching uses rollback journaling. Concurrent writes passed a two-node stress test on kislurm’s NFSv3; this does not guarantee correct locking on other filesystems. Never enable WAL for an NFS cache. A node-local `store_root` remains an option for workers on the same node. The primary process can extend the global evaluation or fidelity budget when resuming. Helpers inherit the persisted global limits. Search-space, optimizer, objective, and pricing changes require a new run directory. diff --git a/configs/tune_judge.yaml b/configs/tune_judge.yaml index eee9201f..982d584a 100644 --- a/configs/tune_judge.yaml +++ b/configs/tune_judge.yaml @@ -6,8 +6,10 @@ judge: model: VLLM/Qwen/Qwen2.5-7B-Instruct prompt_preset: meta-eval-pair-score temperature: 0.0 - max_out_tokens: 2048 - max_model_len: 16384 + max_out_tokens: 8192 + max_model_len: null +generation: + truncate_judge_input_chars: 8192 meta_eval: top_models: 20 tune_judge: @@ -29,12 +31,6 @@ tune_judge: upper: 1.0 prior: 0.0 prior_confidence: medium - judge.max_out_tokens: - type: Integer - lower: 1024 - upper: 2048 - prior: 2048 - prior_confidence: medium judge.swap_mode: type: Categorical choices: [fixed, both] From fc564f1a00ffc27d0a6b7f0aaf59198ad339db1a Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Thu, 8 Oct 2026 00:19:38 +0200 Subject: [PATCH 04/17] Scope context-failure documentation to vLLM chat judges --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 39344abf..9310dc1e 100644 --- a/README.md +++ b/README.md @@ -396,7 +396,7 @@ judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/t judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir --tune_judge.search_only true ``` -The example keeps the paper limits fixed across judges: 8,192 characters per candidate completion, 8,192 judge output tokens, and each model’s native context. Instructions are not truncated. A battle whose rendered prompt exceeds the judge’s context produces a failed annotation, lowering agreement and coverage; it stays in the shared battle set and its skipped output is not cached. Only parameters explicitly included in the search space vary between trials. +The example keeps the paper limits fixed across judges: 8,192 characters per candidate completion, 8,192 judge output tokens, and each model’s native context. Instructions are not truncated. For vLLM chat judges, a battle whose rendered prompt exceeds the judge’s context produces a failed annotation, lowering agreement and coverage; it stays in the shared battle set and its skipped output is not cached. Only parameters explicitly included in the search space vary between trials. `run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials, `pareto.parquet` reports the agreement/cost front of completed maximum-fidelity validation configurations, and `test_results.parquet` holds the selected configurations' test results. The example sets `neps.ignore_errors: true` so failed trials are recorded while the search continues. Shared SQLite judgement caching uses rollback journaling. Concurrent writes passed a two-node stress test on kislurm’s NFSv3; this does not guarantee correct locking on other filesystems. Never enable WAL for an NFS cache. A node-local `store_root` remains an option for workers on the same node. From 13c293ed7c959b6cd065342d1b172f8992611ef1 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Thu, 8 Oct 2026 00:28:25 +0200 Subject: [PATCH 05/17] Link SQLite guidance for network cache journaling --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9310dc1e..7b18f880 100644 --- a/README.md +++ b/README.md @@ -398,7 +398,7 @@ judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/t The example keeps the paper limits fixed across judges: 8,192 characters per candidate completion, 8,192 judge output tokens, and each model’s native context. Instructions are not truncated. For vLLM chat judges, a battle whose rendered prompt exceeds the judge’s context produces a failed annotation, lowering agreement and coverage; it stays in the shared battle set and its skipped output is not cached. Only parameters explicitly included in the search space vary between trials. -`run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials, `pareto.parquet` reports the agreement/cost front of completed maximum-fidelity validation configurations, and `test_results.parquet` holds the selected configurations' test results. The example sets `neps.ignore_errors: true` so failed trials are recorded while the search continues. Shared SQLite judgement caching uses rollback journaling. Concurrent writes passed a two-node stress test on kislurm’s NFSv3; this does not guarantee correct locking on other filesystems. Never enable WAL for an NFS cache. A node-local `store_root` remains an option for workers on the same node. +`run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials, `pareto.parquet` reports the agreement/cost front of completed maximum-fidelity validation configurations, and `test_results.parquet` holds the selected configurations' test results. The example sets `neps.ignore_errors: true` so failed trials are recorded while the search continues. Shared SQLite judgement caching uses rollback journaling. Concurrent writes passed a two-node stress test on kislurm’s NFSv3; this does not guarantee correct locking on other filesystems. Never enable [WAL](https://sqlite.org/wal.html) for an NFS cache. A node-local `store_root` remains an option for workers on the same node. The primary process can extend the global evaluation or fidelity budget when resuming. Helpers inherit the persisted global limits. Search-space, optimizer, objective, and pricing changes require a new run directory. From d946415952bc0762365c30ec77b23450b375106a Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Thu, 8 Oct 2026 16:32:56 +0200 Subject: [PATCH 06/17] Document the parameterized judge tuning workflow --- README.md | 15 +++++++++++++-- configs/tune_judge.yaml | 3 ++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 7b18f880..57bc5746 100644 --- a/README.md +++ b/README.md @@ -341,7 +341,18 @@ Runs save the selected sample, judge evidence, metric battles, configuration, an ### Tuning a judge -The `tune-judge-*` tasks search judge configurations against the same human battles as their `meta-eval-*` counterparts, following [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Install `pip install 'judgearena[tune]'` for [NePS](https://github.com/automl/neps) 0.17 (`neural-pipeline-search`). +The `tune-judge` workflow searches judge configurations against the human battles of the packaged task selected by `tune_judge.meta_eval_task`, following [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Install `pip install 'judgearena[tune]'` for [NePS](https://github.com/automl/neps) 0.17 (`neural-pipeline-search`). + +Select the benchmark and language in the configuration: + +```yaml +task: tune-judge +tune_judge: + meta_eval_task: meta-eval-lmarena-140k-en + # NePS search settings, objectives, and prices follow below. +``` + +`tune-judge` has no dataset definition of its own. `judgearena tasks list` and dataset downloads continue to use the packaged meta-eval tasks. Judge defaults and meta-eval restrictions come from the selected target; `meta_eval` holds its sampling settings. [configs/tune_judge.yaml](configs/tune_judge.yaml) uses native NePS settings under `tune_judge.neps`: `pipeline_space` has dotted config names, a `type` (`Categorical`, `Float`, `Integer`, or `IntegerFidelity`), and native constructor kwargs such as `choices`, `lower`, `upper`, `prior`, `prior_confidence`, and `log` where supported. Categorical `prior` is a choice **index**. Every parameter with a `prior` must also set `prior_confidence` (the example uses `medium`). Prompt presets are scalar choices (`meta-eval-pair-score`, `arena-hard`, `default_with_explanation`); grouped overrides are unsupported. The example searches temperature continuously and uses `meta_eval.battles_per_model` as the validation fidelity. @@ -382,7 +393,7 @@ optimizer: judge.swap_mode: fixed ``` -Each trial runs the matching `meta-eval-*` task on the validation split and shares its judgement cache. Cost uses input plus output tokens (counted with tiktoken), including cached outputs, and `price_per_million_tokens[judge.model]` for every searched model. The Qwen/Gemma prices are illustrative estimates; this objective estimates USD per 1,000 battles, not actual API spend. NePS cost spend limits are unsupported and rejected; use `neps.total_evaluations_to_spend` (40 in the example). +Each trial runs the configured `meta-eval-*` task on the validation split and shares its judgement cache. Cost uses input plus output tokens (counted with tiktoken), including cached outputs, and `price_per_million_tokens[judge.model]` for every searched model. The Qwen/Gemma prices are illustrative estimates; this objective estimates USD per 1,000 battles, not actual API spend. NePS cost spend limits are unsupported and rejected; use `neps.total_evaluations_to_spend` (40 in the example). The main process waits for active trials after `neps.run` returns under normal budget accounting, then selects the best agreement configuration per judge model at the highest fidelity for held-out testing. The native NePS completion flag is not a worker barrier. Optional `test_battles_per_model` defaults to the highest fidelity. diff --git a/configs/tune_judge.yaml b/configs/tune_judge.yaml index 982d584a..32feb3cd 100644 --- a/configs/tune_judge.yaml +++ b/configs/tune_judge.yaml @@ -1,7 +1,7 @@ # NePS 0.17 judge-tuning example; install with pip install 'judgearena[tune]'. # Run: judgearena --config_path configs/tune_judge.yaml # Validation trials use the matching meta-eval task and its judgement cache. -task: tune-judge-lmarena-140k-en +task: tune-judge judge: model: VLLM/Qwen/Qwen2.5-7B-Instruct prompt_preset: meta-eval-pair-score @@ -13,6 +13,7 @@ generation: meta_eval: top_models: 20 tune_judge: + meta_eval_task: meta-eval-lmarena-140k-en neps: pipeline_space: judge.model: From 5354ce487102f1ad8b55d97085e7d776c7c27349 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Thu, 8 Oct 2026 18:38:21 +0200 Subject: [PATCH 07/17] Document packaged tuning defaults and nested search spaces --- README.md | 97 ++++++++++++++++++++++------------------- configs/tune_judge.yaml | 53 +++++----------------- 2 files changed, 62 insertions(+), 88 deletions(-) diff --git a/README.md b/README.md index 57bc5746..785102ed 100644 --- a/README.md +++ b/README.md @@ -341,77 +341,84 @@ Runs save the selected sample, judge evidence, metric battles, configuration, an ### Tuning a judge -The `tune-judge` workflow searches judge configurations against the human battles of the packaged task selected by `tune_judge.meta_eval_task`, following [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Install `pip install 'judgearena[tune]'` for [NePS](https://github.com/automl/neps) 0.17 (`neural-pipeline-search`). +The packaged `tune-judge` task searches judge settings against human preferences, inspired by [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Install `pip install 'judgearena[tune]'` for NePS 0.17.0, then run: -Select the benchmark and language in the configuration: +```bash +judgearena --config_path configs/tune_judge.yaml +judgearena tasks show tune-judge +``` + +The task supplies method defaults: `meta-eval-lmarena-140k-en`, objectives `[agreement, cost]`, `mo_hyperband` with `eta: 3`, and 10-90 battles per model. Run settings override these defaults field by field. Selecting another optimizer name starts with that optimizer's own kwargs; provide `eta` or other settings explicitly when needed. Trials use the selected meta-eval task's judge defaults and restrictions, data, language variant, and judgement cache. `meta_eval.top_models` and other sampling settings remain optional run settings. + +[configs/tune_judge.yaml](configs/tune_judge.yaml) uses this short form: ```yaml task: tune-judge +judge: + model: VLLM/Qwen/Qwen2.5-7B-Instruct + prompt_preset: meta-eval-pair-score + temperature: 0.0 + max_out_tokens: 8192 +generation: {truncate_judge_input_chars: 8192} tune_judge: - meta_eval_task: meta-eval-lmarena-140k-en - # NePS search settings, objectives, and prices follow below. + search_space: + judge: + model: [VLLM/Qwen/Qwen2.5-7B-Instruct, VLLM/google/gemma-2-9b-it] + prompt_preset: [meta-eval-pair-score, arena-hard] + swap_mode: [fixed, both] + temperature: {lower: 0.0, upper: 1.0, prior_confidence: low} + max_out_tokens: {lower: 1024, upper: 8192, log: true} + neps: {total_evaluations_to_spend: 40} ``` -`tune-judge` has no dataset definition of its own. `judgearena tasks list` and dataset downloads continue to use the packaged meta-eval tasks. Judge defaults and meta-eval restrictions come from the selected target; `meta_eval` holds its sampling settings. +Lists, or `{choices: [...]}`, define categorical parameters. `{lower, upper}` defines a range, integer when both bounds are integers and float otherwise; use `0.0` and `1.0` for continuous temperature. `log: true` enables logarithmic sampling. Parameters are nested like the run configuration and may vary judge fields or `generation.truncate_judge_input_chars`. -[configs/tune_judge.yaml](configs/tune_judge.yaml) uses native NePS settings under `tune_judge.neps`: `pipeline_space` has dotted config names, a `type` (`Categorical`, `Float`, `Integer`, or `IntegerFidelity`), and native constructor kwargs such as `choices`, `lower`, `upper`, `prior`, `prior_confidence`, and `log` where supported. Categorical `prior` is a choice **index**. Every parameter with a `prior` must also set `prior_confidence` (the example uses `medium`). Prompt presets are scalar choices (`meta-eval-pair-score`, `arena-hard`, `default_with_explanation`); grouped overrides are unsupported. The example searches temperature continuously and uses `meta_eval.battles_per_model` as the validation fidelity. +The resolved reference judge supplies each parameter's prior when its value lies in the domain. An explicit `prior` is a **value**, including for categorical parameters. `prior_confidence` can be `low`, `medium`, or `high` and defaults to `medium`. For example: -Use categorical temperature choices (for example, `[0.0, 0.1, 1.0]`) when cache reuse matters: continuous sampling usually produces a new model descriptor for each value. +```yaml +prompt_preset: + choices: [meta-eval-pair-score, arena-hard] + prior: arena-hard + prior_confidence: high +``` -For example, a continuous temperature parameter under `tune_judge.neps.pipeline_space`: +Use categorical temperature choices such as `[0.0, 0.1, 1.0]` when cache reuse matters; continuous sampling usually produces a new model descriptor for each value. Fidelity is separate from the search space: ```yaml -judge.temperature: - type: Float - lower: 0.0 - upper: 1.0 - prior: 0.0 - prior_confidence: medium +tune_judge: + meta_eval_task: meta-eval-comparia-fr + fidelity: {battles_per_model: {lower: 10, upper: 90}} + objectives: [agreement, cost] + optimizer: {name: mo_hyperband, eta: 3, mo_selector: epsnet} ``` -- `optimizer: {name: neps_priorband, eta: 3}` uses parameter priors; `neps_hyperband` samples uniformly. Both require `objectives: [agreement]` outside `neps`. -- `optimizer: {name: mo_hyperband, eta: 3, mo_selector: epsnet}` and `primo` support `objectives: [agreement, cost_per_1k_battles]`. Agreement is maximized (NePS minimizes `1 - agreement`); cost is minimized. -- For `primo`, put native `prior_centers` settings in the optimizer mapping, keyed by `objective_0` (disagreement) and `objective_1` (cost). Each contains dotted parameter names mapped to native setting **values**, for example: +`mo_hyperband` and `primo` support agreement and cost objectives. `neps_priorband` and `neps_hyperband` require `objectives: [agreement]`. Optimizer kwargs remain native NePS settings; `primo` accepts objective-specific `prior_centers` and `prior_confidences` with dotted parameter names. Other run settings belong under `neps`, including evaluation budgets and `ignore_errors` (defaults to `true`). Managed directory/completion settings and NePS cost-spend budgets are unsupported. With NePS 0.17, leave `sample_prior_first` at its default for `neps_priorband` and `neps_hyperband` because enabling it fails in upstream trial-ID parsing. + +**Expected judge cost** counts input and output tokens with tiktoken, including cached judgements, and reports USD per 1,000 battles. Local vLLM models use the pinned [judgetuning price table](https://github.com/geoalgo/judgetuning/blob/main/judgetuning/llm_client/llm_specs.py): some entries are runtime-derived measurements, others historical provider references. Unlisted models use historical size tiers based on tensor shapes in locally cached safetensors headers. These are estimates; the tiers above 41B are extrapolated from the paper's larger-model examples. Unknown quantized models require an override because packed tensor shapes do not reveal their original parameter count. + +Hosted models use OpenRouter's published input/output prices, cached in `openrouter_pricing.json`. This supplies a reference estimate even when another provider serves the model. Optional per-model overrides take precedence: ```yaml -optimizer: - name: primo - eta: 3 - mo_selector: epsnet - prior_centers: - objective_0: - judge.model: VLLM/Qwen/Qwen2.5-7B-Instruct - judge.prompt_preset: meta-eval-pair-score - judge.temperature: 0.0 - judge.max_out_tokens: 2048 - judge.swap_mode: both - objective_1: - judge.model: VLLM/Qwen/Qwen2.5-7B-Instruct - judge.prompt_preset: meta-eval-pair-score - judge.temperature: 0.0 - judge.max_out_tokens: 1024 - judge.swap_mode: fixed +tune_judge: + price_per_million_tokens: + VLLM/org/model: 0.2 + OpenRouter/org/model: {input: 0.2, output: 0.8} ``` -Each trial runs the configured `meta-eval-*` task on the validation split and shares its judgement cache. Cost uses input plus output tokens (counted with tiktoken), including cached outputs, and `price_per_million_tokens[judge.model]` for every searched model. The Qwen/Gemma prices are illustrative estimates; this objective estimates USD per 1,000 battles, not actual API spend. NePS cost spend limits are unsupported and rejected; use `neps.total_evaluations_to_spend` (40 in the example). +A cost-objective search stops before evaluation if any searched model has no price. An agreement-only run may leave cost unknown. `prices.json` records the resolved rates and sources in the tuning directory; resuming with different resolved prices is rejected. Expected cost is separate from actual provider spend and NePS's search budget. -The main process waits for active trials after `neps.run` returns under normal budget accounting, then selects the best agreement configuration per judge model at the highest fidelity for held-out testing. The native NePS completion flag is not a worker barrier. Optional `test_battles_per_model` defaults to the highest fidelity. - -With NePS 0.17, leave `sample_prior_first` at its default for `neps_priorband` and `neps_hyperband`: enabling it fails in upstream trial-ID parsing. Priors still guide sampling without that option. For `primo`, its initial design must reach the highest fidelity before it switches to Bayesian optimization. +The primary process waits for active trials, reports the full-budget agreement/cost Pareto front, and selects the best agreement configuration per model for held-out testing. `test_battles_per_model` defaults to the highest validation fidelity. Results include `trials.parquet`, `pareto.parquet`, and `test_results.parquet`; NePS state stays under `neps/`, separate from inference caches. ```bash -judgearena --config_path configs/tune_judge.yaml -# Resume with the same explicit tuning directory and settings. +# Explicitly resume the same session. judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir -# Helper workers join that directory and run only the search. +# Helper workers join an initialized session and run validation only. judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir --tune_judge.search_only true ``` -The example keeps the paper limits fixed across judges: 8,192 characters per candidate completion, 8,192 judge output tokens, and each model’s native context. Instructions are not truncated. For vLLM chat judges, a battle whose rendered prompt exceeds the judge’s context produces a failed annotation, lowering agreement and coverage; it stays in the shared battle set and its skipped output is not cached. Only parameters explicitly included in the search space vary between trials. - -`run_dir: null` creates a tuning directory under `--run.result_folder`; set an explicit `tune_judge.run_dir` to resume or share workers. NePS state lives in `/neps/`, with meta-eval trial artifacts separately in `/trials/`. `trials.parquet` summarizes trials, `pareto.parquet` reports the agreement/cost front of completed maximum-fidelity validation configurations, and `test_results.parquet` holds the selected configurations' test results. The example sets `neps.ignore_errors: true` so failed trials are recorded while the search continues. Shared SQLite judgement caching uses rollback journaling. Concurrent writes passed a two-node stress test on kislurm’s NFSv3; this does not guarantee correct locking on other filesystems. Never enable [WAL](https://sqlite.org/wal.html) for an NFS cache. A node-local `store_root` remains an option for workers on the same node. +The example keeps paper limits fixed across judges: 8,192 characters per candidate completion, 8,192 judge output tokens, and native context. Instructions are not truncated. For vLLM chat judges, overlong prompts remain failed annotations in the shared battle set, lowering agreement and coverage; skipped outputs are not cached. Only searched settings vary. -The primary process can extend the global evaluation or fidelity budget when resuming. Helpers inherit the persisted global limits. Search-space, optimizer, objective, and pricing changes require a new run directory. +Shared rollback-mode SQLite caching was verified on kislurm NFSv3 in a two-node test; this does not guarantee locking on other filesystems. Never use WAL on NFS. Workers share an explicit tuning directory and can reuse judgements from direct runs of the target meta-eval task. ## 📈 Estimating ELO Ratings diff --git a/configs/tune_judge.yaml b/configs/tune_judge.yaml index 32feb3cd..b1b425e6 100644 --- a/configs/tune_judge.yaml +++ b/configs/tune_judge.yaml @@ -1,6 +1,4 @@ -# NePS 0.17 judge-tuning example; install with pip install 'judgearena[tune]'. -# Run: judgearena --config_path configs/tune_judge.yaml -# Validation trials use the matching meta-eval task and its judgement cache. +# Install judgearena[tune], then run: judgearena --config_path configs/tune_judge.yaml task: tune-judge judge: model: VLLM/Qwen/Qwen2.5-7B-Instruct @@ -10,46 +8,15 @@ judge: max_model_len: null generation: truncate_judge_input_chars: 8192 -meta_eval: - top_models: 20 tune_judge: - meta_eval_task: meta-eval-lmarena-140k-en + search_space: + judge: + model: [VLLM/Qwen/Qwen2.5-7B-Instruct, VLLM/google/gemma-2-9b-it] + prompt_preset: [meta-eval-pair-score, arena-hard, default_with_explanation] + swap_mode: [fixed, both] + temperature: {lower: 0.0, upper: 1.0, prior_confidence: low} + max_out_tokens: {lower: 1024, upper: 8192, log: true} neps: - pipeline_space: - judge.model: - type: Categorical - choices: [VLLM/Qwen/Qwen2.5-7B-Instruct, VLLM/google/gemma-2-9b-it] - prior: 0 # Categorical priors are choice indices. - prior_confidence: medium - judge.prompt_preset: - type: Categorical - choices: [meta-eval-pair-score, arena-hard, default_with_explanation] - prior: 0 - prior_confidence: medium - judge.temperature: - type: Float - lower: 0.0 - upper: 1.0 - prior: 0.0 - prior_confidence: medium - judge.swap_mode: - type: Categorical - choices: [fixed, both] - prior: 0 - prior_confidence: medium - meta_eval.battles_per_model: - type: IntegerFidelity - lower: 10 - upper: 90 - optimizer: {name: mo_hyperband, eta: 3, mo_selector: epsnet} total_evaluations_to_spend: 40 - ignore_errors: true # Record failed trials and continue the search. - objectives: [agreement, cost_per_1k_battles] - # Estimated USD per million tokens, not measured spend or provider quotes. - price_per_million_tokens: - VLLM/Qwen/Qwen2.5-7B-Instruct: 0.12 - VLLM/google/gemma-2-9b-it: 0.14 - # Optional; null uses the highest validation fidelity for held-out testing. - test_battles_per_model: null - run_dir: null - search_only: false +# Task defaults select meta-eval-lmarena-140k-en, agreement/cost objectives, +# mo_hyperband, and 10-90 battles per model. Each can be overridden here. From 932df22317e15885d17198562b18d84bf3e8c412 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Fri, 9 Oct 2026 09:45:01 +0200 Subject: [PATCH 08/17] Resolve OpenRouter reference prices and freeze rates per tuning session --- judgearena/tuning/pricing.py | 96 ++++++++++++++++++++++++++++--- judgearena/tuning/runner.py | 30 ++++++---- tests/test_tuning.py | 20 +++++-- tests/test_tuning_pricing.py | 108 +++++++++++++++++++++++++++++++++++ 4 files changed, 227 insertions(+), 27 deletions(-) create mode 100644 tests/test_tuning_pricing.py diff --git a/judgearena/tuning/pricing.py b/judgearena/tuning/pricing.py index be671c95..7acff2e4 100644 --- a/judgearena/tuning/pricing.py +++ b/judgearena/tuning/pricing.py @@ -1,6 +1,12 @@ """Explicit judge-token prices used by the tuning cost objective.""" +import json from dataclasses import dataclass +from pathlib import Path +from urllib.error import URLError +from urllib.request import urlopen + +_OPENROUTER_URL = "https://openrouter.ai/api/v1/models" @dataclass(frozen=True) @@ -10,19 +16,91 @@ class TokenPrice: source: str -def resolve_prices(models, overrides, *, require_cost=True) -> dict[str, TokenPrice]: - """Resolve explicit USD-per-million-token rates before searching.""" +def _fetch_openrouter(catalog_cache: Path | None) -> list[dict]: + try: + with urlopen(_OPENROUTER_URL, timeout=10) as response: + records = json.load(response)["data"] + if catalog_cache is not None: + catalog_cache.parent.mkdir(parents=True, exist_ok=True) + catalog_cache.write_text(json.dumps(records), encoding="utf-8") + return records + except (URLError, TimeoutError, OSError) as error: + if catalog_cache is not None and catalog_cache.exists(): + cached = json.loads(catalog_cache.read_text(encoding="utf-8")) + if isinstance(cached, dict): + return [ + { + "id": model_id, + "pricing": {"prompt": price[0], "completion": price[1]}, + } + for model_id, price in cached.items() + ] + if isinstance(cached, list): + return cached + raise ValueError( + "Could not fetch OpenRouter pricing and no cached catalog is available; " + "fetch the catalog on a login node or set " + "tune_judge.price_per_million_tokens" + ) from error + + +def _find_price(model: str, records: list[dict]) -> TokenPrice | None: + requested_id = model.split("/", 1)[-1].lower() + suffix = "/".join(requested_id.split("/")[-2:]) + record = next( + ( + item + for item in records + if str(item.get("hugging_face_id") or "").lower() == requested_id + ), + None, + ) + if record is None: + record = next( + (item for item in records if str(item.get("id") or "").lower() == suffix), + None, + ) + if record is None: + return None + rates = record.get("pricing", {}) + if "prompt" not in rates or "completion" not in rates: + return None + return TokenPrice( + float(rates["prompt"]) * 1_000_000, + float(rates["completion"]) * 1_000_000, + "openrouter_reference", + ) + + +def resolve_prices( + models, + overrides, + *, + catalog_cache: Path | None = None, + require_cost=True, +) -> dict[str, TokenPrice]: + """Resolve explicit or OpenRouter reference USD-per-million-token rates.""" prices = {} + unresolved = [] for model in sorted(models): - if model not in overrides: - continue - rate = overrides[model] - if isinstance(rate, (int, float)): - rate = {"input": rate, "output": rate} - prices[model] = TokenPrice(rate["input"], rate["output"], "user_override") + rate = overrides.get(model) + if rate is not None: + if isinstance(rate, (int, float)): + rate = {"input": rate, "output": rate} + prices[model] = TokenPrice( + float(rate["input"]), float(rate["output"]), "user_override" + ) + else: + unresolved.append(model) + records = _fetch_openrouter(catalog_cache) if unresolved else [] + for model in unresolved: + price = _find_price(model, records) + if price is not None: + prices[model] = price missing = sorted(set(models) - prices.keys()) if missing and require_cost: raise ValueError( - f"No token price for {missing}; set tune_judge.price_per_million_tokens" + f"No token price for {missing}; set tune_judge.price_per_million_tokens " + f"overrides for: {missing}" ) return prices diff --git a/judgearena/tuning/runner.py b/judgearena/tuning/runner.py index d6e59565..d9c477ac 100644 --- a/judgearena/tuning/runner.py +++ b/judgearena/tuning/runner.py @@ -253,21 +253,27 @@ def run_tune_judge( models.add(trial_cfg.judge.model) if "judge.model" not in specs: models.add(cfg.judge.model) - prices = resolve_prices( - models, - tuning.price_per_million_tokens, - require_cost="cost" in tuning.objectives, - ) - runner = _TrialRunner(runner.base, prices, execute_trial) tune_dir = prepare_session(cfg) price_path = tune_dir / "prices.json" - snapshot = {model: asdict(price) for model, price in prices.items()} if price_path.exists(): - stored = json.loads(price_path.read_text()) - if stored != snapshot: - raise ValueError("Resolved prices changed for this tuning session") - elif not tuning.search_only: - price_path.write_text(json.dumps(snapshot, indent=2)) + prices = { + model: TokenPrice(**price) + for model, price in json.loads(price_path.read_text()).items() + } + else: + prices = resolve_prices( + models, + tuning.price_per_million_tokens, + catalog_cache=Path(cfg.run.store_root or cfg.run.result_folder) + / "openrouter_pricing.json", + require_cost="cost" in tuning.objectives, + ) + price_path.write_text( + json.dumps( + {model: asdict(price) for model, price in prices.items()}, indent=2 + ) + ) + runner = _TrialRunner(runner.base, prices, execute_trial) logger.info("Tuning %s in %s", cfg.task, tune_dir) _search(runner, cfg, tune_dir, space) if tuning.search_only: diff --git a/tests/test_tuning.py b/tests/test_tuning.py index dff53083..05747201 100644 --- a/tests/test_tuning.py +++ b/tests/test_tuning.py @@ -98,7 +98,7 @@ def _fake_trial(config_path): @pytest.mark.parametrize( "algorithm", ["neps_priorband", "neps_hyperband", "mo_hyperband", "primo"] ) -def test_native_optimizer_smoke(tmp_path, token_counter, algorithm): +def test_native_optimizer_smoke(tmp_path, token_counter, algorithm, monkeypatch): pytest.importorskip("neps") cfg = _tuning_config(tmp_path, algorithm) calls = [] @@ -127,11 +127,16 @@ def execute(path): assert load_config(calls[-1]).meta_eval.battles_per_model == 3 before = len(calls) (folder / "test_results.parquet").unlink() - resumed = runner.run_tune_judge(cfg, task, execute_trial=execute) - pd.testing.assert_frame_equal(results, resumed) - assert len(calls) == before # Completed search and held-out artifacts are reused. helper = _tuning_config(tmp_path, search_only=True) - assert runner.run_tune_judge(helper, task, execute_trial=execute).empty + with monkeypatch.context() as patch: + patch.setattr( + runner, + "resolve_prices", + lambda *a, **kw: pytest.fail("Refetched session prices"), + ) + resumed = runner.run_tune_judge(cfg, task, execute_trial=execute) + assert runner.run_tune_judge(helper, task, execute_trial=execute).empty + pd.testing.assert_frame_equal(results, resumed) assert len(calls) == before incompatible = _tuning_config( tmp_path, price_per_million_tokens={"Dummy/judge": 2.0, "Dummy/other": 2.0} @@ -199,7 +204,10 @@ def test_cost_counts_both_cached_orientations(tmp_path, token_counter): assert "cost" not in objective -def test_prices_required_before_execution(tmp_path): +def test_prices_required_before_execution(tmp_path, monkeypatch): + from judgearena.tuning import pricing + + monkeypatch.setattr(pricing, "_fetch_openrouter", lambda _: []) pytest.importorskip("neps") cfg = _tuning_config(tmp_path, "mo_hyperband", price_per_million_tokens={}) with pytest.raises(ValueError, match="No token price"): diff --git a/tests/test_tuning_pricing.py b/tests/test_tuning_pricing.py new file mode 100644 index 00000000..18351a44 --- /dev/null +++ b/tests/test_tuning_pricing.py @@ -0,0 +1,108 @@ +import json +from urllib.error import URLError + +import pytest + +from judgearena.tuning import pricing +from judgearena.tuning.pricing import TokenPrice, resolve_prices + + +def test_overrides_do_not_fetch_catalog(monkeypatch): + def fail(*args, **kwargs): + raise AssertionError("all prices were overridden") + + monkeypatch.setattr(pricing, "urlopen", fail) + result = resolve_prices( + ["VLLM/local", "OpenRouter/hosted"], + {"VLLM/local": 1.2, "OpenRouter/hosted": {"input": 0.2, "output": 0.7}}, + ) + assert result["VLLM/local"] == TokenPrice(1.2, 1.2, "user_override") + assert result["OpenRouter/hosted"] == TokenPrice(0.2, 0.7, "user_override") + + +def test_huggingface_id_match_precedes_suffix_for_every_provider(tmp_path, monkeypatch): + records = [ + { + "id": "org/model", + "pricing": {"prompt": "0.000001", "completion": "0.000002"}, + }, + { + "id": "elsewhere/different", + "hugging_face_id": "ORG/Model", + "pricing": {"prompt": "0.000003", "completion": "0.000004"}, + }, + ] + monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _Response(records)) + cache = tmp_path / "openrouter_pricing.json" + for model in ("VLLM/org/model", "Provider/org/model"): + result = resolve_prices([model], {}, catalog_cache=cache) + assert result[model] == TokenPrice(3, 4, "openrouter_reference") + assert json.loads(cache.read_text()) == records + + +def test_suffix_match(monkeypatch): + monkeypatch.setattr( + pricing, + "urlopen", + lambda *a, **k: _Response( + [ + { + "id": "org/model", + "pricing": {"prompt": "0.000005", "completion": "0.000009"}, + } + ] + ), + ) + assert resolve_prices(["Hosted/org/model"], {})["Hosted/org/model"] == TokenPrice( + 5, 9, "openrouter_reference" + ) + + +def test_fetch_failure_uses_cached_catalog(tmp_path, monkeypatch): + cache = tmp_path / "openrouter_pricing.json" + cache.write_text( + json.dumps( + [ + { + "id": "org/model", + "pricing": {"prompt": "0.000002", "completion": "0.000006"}, + } + ] + ) + ) + monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _raise_url_error()) + price = resolve_prices(["Backend/org/model"], {}, catalog_cache=cache) + assert price["Backend/org/model"] == TokenPrice(2, 6, "openrouter_reference") + + +def test_fetch_failure_without_cache_has_actionable_error(tmp_path, monkeypatch): + monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _raise_url_error()) + with pytest.raises(ValueError, match="login node.*price_per_million_tokens"): + resolve_prices( + ["Backend/org/model"], {}, catalog_cache=tmp_path / "catalog.json" + ) + + +def test_unpriced_model_fails_for_cost_objective(monkeypatch): + monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _Response([])) + with pytest.raises(ValueError, match="unknown.*overrides"): + resolve_prices(["VLLM/unknown"], {}) + assert resolve_prices(["VLLM/unknown"], {}, require_cost=False) == {} + + +class _Response: + def __init__(self, records): + self.records = records + + def __enter__(self): + return self + + def __exit__(self, *args): + return None + + def read(self): + return json.dumps({"data": self.records}).encode() + + +def _raise_url_error(): + raise URLError("offline") From fade56d39b78f93b7a34a7f807532d603f85e9b6 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Fri, 9 Oct 2026 09:45:06 +0200 Subject: [PATCH 09/17] Document OpenRouter references, offline pricing, and field-based ranges --- README.md | 21 ++++++++++++++++----- configs/tune_judge.yaml | 5 +++++ 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 785102ed..8e172e7b 100644 --- a/README.md +++ b/README.md @@ -366,12 +366,16 @@ tune_judge: model: [VLLM/Qwen/Qwen2.5-7B-Instruct, VLLM/google/gemma-2-9b-it] prompt_preset: [meta-eval-pair-score, arena-hard] swap_mode: [fixed, both] - temperature: {lower: 0.0, upper: 1.0, prior_confidence: low} + temperature: {lower: 0, upper: 1, prior_confidence: low} max_out_tokens: {lower: 1024, upper: 8192, log: true} + # User estimates in USD per 1M tokens, not live catalog prices. + price_per_million_tokens: + VLLM/Qwen/Qwen2.5-7B-Instruct: {input: 0.12, output: 0.12} + VLLM/google/gemma-2-9b-it: {input: 0.14, output: 0.14} neps: {total_evaluations_to_spend: 40} ``` -Lists, or `{choices: [...]}`, define categorical parameters. `{lower, upper}` defines a range, integer when both bounds are integers and float otherwise; use `0.0` and `1.0` for continuous temperature. `log: true` enables logarithmic sampling. Parameters are nested like the run configuration and may vary judge fields or `generation.truncate_judge_input_chars`. +Lists, or `{choices: [...]}`, define categorical parameters. `{lower, upper}` defines a range whose type follows the corresponding run-config field: temperature is continuous even when written `{lower: 0, upper: 1}`, while `max_out_tokens` is integer. `log: true` enables logarithmic sampling. Parameters are nested like the run configuration and may vary judge fields or `generation.truncate_judge_input_chars`. The resolved reference judge supplies each parameter's prior when its value lies in the domain. An explicit `prior` is a **value**, including for categorical parameters. `prior_confidence` can be `low`, `medium`, or `high` and defaults to `medium`. For example: @@ -394,9 +398,16 @@ tune_judge: `mo_hyperband` and `primo` support agreement and cost objectives. `neps_priorband` and `neps_hyperband` require `objectives: [agreement]`. Optimizer kwargs remain native NePS settings; `primo` accepts objective-specific `prior_centers` and `prior_confidences` with dotted parameter names. Other run settings belong under `neps`, including evaluation budgets and `ignore_errors` (defaults to `true`). Managed directory/completion settings and NePS cost-spend budgets are unsupported. With NePS 0.17, leave `sample_prior_first` at its default for `neps_priorband` and `neps_hyperband` because enabling it fails in upstream trial-ID parsing. -**Expected judge cost** counts input and output tokens with tiktoken, including cached judgements, and reports USD per 1,000 battles. Local vLLM models use the pinned [judgetuning price table](https://github.com/geoalgo/judgetuning/blob/main/judgetuning/llm_client/llm_specs.py): some entries are runtime-derived measurements, others historical provider references. Unlisted models use historical size tiers based on tensor shapes in locally cached safetensors headers. These are estimates; the tiers above 41B are extrapolated from the paper's larger-model examples. Unknown quantized models require an override because packed tensor shapes do not reveal their original parameter count. +**Expected judge cost** counts input and output tokens with tiktoken, including cached judgements, and reports USD per 1,000 battles. A per-model override takes precedence and may be a scalar (same input/output rate) or separate input/output rates, in USD per 1M tokens. Without an override, the OpenRouter model catalog supplies reference rates for every provider, including vLLM: matching first uses the Hugging Face ID, then the model ID's last two path segments. Catalog prices are estimates for comparison, not the actual price charged by another provider. + +A new primary run fetches the catalog's free metadata and saves it as `{store_root}/openrouter_pricing.json`. If fetching fails, it uses that cached catalog; if neither is available, add an override or fetch the catalog on a login node and make the cache available. Resume and helper workers use the run's `prices.json` without fetching. On an internet-connected login node, prefetch the catalog into the same `run.store_root` used by the job: + +```bash +mkdir -p /path/to/store-root +curl -fsS https://openrouter.ai/api/v1/models | jq '.data' > /path/to/store-root/openrouter_pricing.json +``` -Hosted models use OpenRouter's published input/output prices, cached in `openrouter_pricing.json`. This supplies a reference estimate even when another provider serves the model. Optional per-model overrides take precedence: +Optional per-model overrides use: ```yaml tune_judge: @@ -405,7 +416,7 @@ tune_judge: OpenRouter/org/model: {input: 0.2, output: 0.8} ``` -A cost-objective search stops before evaluation if any searched model has no price. An agreement-only run may leave cost unknown. `prices.json` records the resolved rates and sources in the tuning directory; resuming with different resolved prices is rejected. Expected cost is separate from actual provider spend and NePS's search budget. +The example's Qwen2.5-7B and gemma-2-9b-it rates (0.12 and 0.14) are user-provided estimates, not claims about current catalog prices. A cost-objective search stops before evaluation if any searched model has no price, naming the unresolved models and `tune_judge.price_per_million_tokens`. An agreement-only run may leave cost unknown. `prices.json` records the resolved rates and whether each came from `user_override` or `openrouter_reference`; resuming uses this run-specific file without a catalog fetch. Expected cost is separate from actual provider spend and NePS's search budget. The primary process waits for active trials, reports the full-budget agreement/cost Pareto front, and selects the best agreement configuration per model for held-out testing. `test_battles_per_model` defaults to the highest validation fidelity. Results include `trials.parquet`, `pareto.parquet`, and `test_results.parquet`; NePS state stays under `neps/`, separate from inference caches. diff --git a/configs/tune_judge.yaml b/configs/tune_judge.yaml index b1b425e6..c505dae3 100644 --- a/configs/tune_judge.yaml +++ b/configs/tune_judge.yaml @@ -16,6 +16,11 @@ tune_judge: swap_mode: [fixed, both] temperature: {lower: 0.0, upper: 1.0, prior_confidence: low} max_out_tokens: {lower: 1024, upper: 8192, log: true} + # User-provided estimates in USD per 1M tokens (input and output); these + # are explicit overrides, not live OpenRouter catalog prices. + price_per_million_tokens: + VLLM/Qwen/Qwen2.5-7B-Instruct: {input: 0.12, output: 0.12} + VLLM/google/gemma-2-9b-it: {input: 0.14, output: 0.14} neps: total_evaluations_to_spend: 40 # Task defaults select meta-eval-lmarena-140k-en, agreement/cost objectives, From 70d139a5d978a391d22484a4e4e07f76e779fd85 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Fri, 9 Oct 2026 09:47:17 +0200 Subject: [PATCH 10/17] Clarify catalog lookup only runs for models without overrides --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8e172e7b..df9ee684 100644 --- a/README.md +++ b/README.md @@ -400,7 +400,7 @@ tune_judge: **Expected judge cost** counts input and output tokens with tiktoken, including cached judgements, and reports USD per 1,000 battles. A per-model override takes precedence and may be a scalar (same input/output rate) or separate input/output rates, in USD per 1M tokens. Without an override, the OpenRouter model catalog supplies reference rates for every provider, including vLLM: matching first uses the Hugging Face ID, then the model ID's last two path segments. Catalog prices are estimates for comparison, not the actual price charged by another provider. -A new primary run fetches the catalog's free metadata and saves it as `{store_root}/openrouter_pricing.json`. If fetching fails, it uses that cached catalog; if neither is available, add an override or fetch the catalog on a login node and make the cache available. Resume and helper workers use the run's `prices.json` without fetching. On an internet-connected login node, prefetch the catalog into the same `run.store_root` used by the job: +When a searched model has no override, a new primary run fetches the catalog's free metadata and saves it as `{store_root}/openrouter_pricing.json`. If fetching fails, it uses that cached catalog; if neither is available, add an override or fetch the catalog on a login node and make the cache available. Resume and helper workers use the run's `prices.json` without fetching. On an internet-connected login node, prefetch the catalog into the same `run.store_root` used by the job: ```bash mkdir -p /path/to/store-root From b8480ed102ac1b43983b563cda7417868c800797 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:57:16 +0200 Subject: [PATCH 11/17] Centralize OpenRouter reference pricing and native token cost calculation --- judgearena/{tuning => }/pricing.py | 42 +++++++++++++++++++++--------- judgearena/tuning/runner.py | 2 +- tests/test_tuning_pricing.py | 40 +++++++++++++++++++++++++--- 3 files changed, 68 insertions(+), 16 deletions(-) rename judgearena/{tuning => }/pricing.py (72%) diff --git a/judgearena/tuning/pricing.py b/judgearena/pricing.py similarity index 72% rename from judgearena/tuning/pricing.py rename to judgearena/pricing.py index 7acff2e4..3b465289 100644 --- a/judgearena/tuning/pricing.py +++ b/judgearena/pricing.py @@ -1,4 +1,4 @@ -"""Explicit judge-token prices used by the tuning cost objective.""" +"""Shared reference token pricing and request cost calculation.""" import json from dataclasses import dataclass @@ -6,6 +6,8 @@ from urllib.error import URLError from urllib.request import urlopen +from judgearena.usage import RequestUsage + _OPENROUTER_URL = "https://openrouter.ai/api/v1/models" @@ -16,6 +18,24 @@ class TokenPrice: source: str +def reference_cost(usage: RequestUsage, price: TokenPrice) -> float: + """Calculate USD from native input/output token counts and per-million rates.""" + missing = [ + name + for name in ("input_tokens", "output_tokens") + if getattr(usage, name) is None + ] + if missing: + fields = " and ".join(missing) + raise ValueError( + f"Cannot calculate reference cost: missing native {fields}; " + "provide provider-reported input_tokens and output_tokens usage" + ) + return ( + usage.input_tokens * price.input + usage.output_tokens * price.output + ) / 1_000_000 + + def _fetch_openrouter(catalog_cache: Path | None) -> list[dict]: try: with urlopen(_OPENROUTER_URL, timeout=10) as response: @@ -27,20 +47,16 @@ def _fetch_openrouter(catalog_cache: Path | None) -> list[dict]: except (URLError, TimeoutError, OSError) as error: if catalog_cache is not None and catalog_cache.exists(): cached = json.loads(catalog_cache.read_text(encoding="utf-8")) - if isinstance(cached, dict): - return [ - { - "id": model_id, - "pricing": {"prompt": price[0], "completion": price[1]}, - } - for model_id, price in cached.items() - ] if isinstance(cached, list): return cached + raise ValueError( + "Cached OpenRouter pricing catalog uses an unsupported format; " + "refresh the catalog or set price_per_million_tokens overrides" + ) from error raise ValueError( "Could not fetch OpenRouter pricing and no cached catalog is available; " "fetch the catalog on a login node or set " - "tune_judge.price_per_million_tokens" + "price_per_million_tokens overrides" ) from error @@ -80,6 +96,8 @@ def resolve_prices( require_cost=True, ) -> dict[str, TokenPrice]: """Resolve explicit or OpenRouter reference USD-per-million-token rates.""" + if not require_cost: + return {} prices = {} unresolved = [] for model in sorted(models): @@ -98,9 +116,9 @@ def resolve_prices( if price is not None: prices[model] = price missing = sorted(set(models) - prices.keys()) - if missing and require_cost: + if missing: raise ValueError( - f"No token price for {missing}; set tune_judge.price_per_million_tokens " + f"No token price for {missing}; set price_per_million_tokens " f"overrides for: {missing}" ) return prices diff --git a/judgearena/tuning/runner.py b/judgearena/tuning/runner.py index d9c477ac..49b80804 100644 --- a/judgearena/tuning/runner.py +++ b/judgearena/tuning/runner.py @@ -30,7 +30,7 @@ if TYPE_CHECKING: from judgearena.tasks.schema import ResolvedTaskSpec -from judgearena.tuning.pricing import TokenPrice, resolve_prices +from judgearena.pricing import TokenPrice, resolve_prices from judgearena.tuning.session import collect_trials, prepare_session logger = get_logger(__name__) diff --git a/tests/test_tuning_pricing.py b/tests/test_tuning_pricing.py index 18351a44..c13071ad 100644 --- a/tests/test_tuning_pricing.py +++ b/tests/test_tuning_pricing.py @@ -3,8 +3,9 @@ import pytest -from judgearena.tuning import pricing -from judgearena.tuning.pricing import TokenPrice, resolve_prices +from judgearena import pricing +from judgearena.pricing import TokenPrice, reference_cost, resolve_prices +from judgearena.usage import RequestUsage def test_overrides_do_not_fetch_catalog(monkeypatch): @@ -84,10 +85,43 @@ def test_fetch_failure_without_cache_has_actionable_error(tmp_path, monkeypatch) def test_unpriced_model_fails_for_cost_objective(monkeypatch): + def unexpected_fetch(*args, **kwargs): + raise AssertionError("agreement-only pricing must not fetch the catalog") + + monkeypatch.setattr(pricing, "urlopen", unexpected_fetch) + assert resolve_prices(["VLLM/unknown"], {}, require_cost=False) == {} + monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _Response([])) with pytest.raises(ValueError, match="unknown.*overrides"): resolve_prices(["VLLM/unknown"], {}) - assert resolve_prices(["VLLM/unknown"], {}, require_cost=False) == {} + + +def test_reference_cost_requires_native_counts_and_uses_reported_zero(): + with pytest.raises( + ValueError, match="missing native input_tokens and output_tokens" + ): + reference_cost( + RequestUsage(stage="judging", reasoning_tokens=12), + TokenPrice(1, 2, "ref"), + ) + + usage = RequestUsage( + stage="judging", input_tokens=0, output_tokens=3, reasoning_tokens=100 + ) + assert reference_cost(usage, TokenPrice(1_000_000, 2_000_000, "ref")) == 6 + + with pytest.raises(ValueError, match="missing native output_tokens"): + reference_cost( + RequestUsage(stage="judging", input_tokens=3), TokenPrice(1, 2, "ref") + ) + + +def test_fetch_failure_rejects_legacy_dictionary_catalog(tmp_path, monkeypatch): + cache = tmp_path / "catalog.json" + cache.write_text(json.dumps({"org/model": [1, 2]})) + monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _raise_url_error()) + with pytest.raises(ValueError, match="unsupported format.*refresh.*overrides"): + resolve_prices(["Backend/org/model"], {}, catalog_cache=cache) class _Response: From e57a55214c2b5301c697d131950758f323d9a186 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Fri, 9 Oct 2026 10:58:12 +0200 Subject: [PATCH 12/17] Document native usage requirements and separate expected cost from run spend --- README.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index df9ee684..fbd4e51f 100644 --- a/README.md +++ b/README.md @@ -398,9 +398,9 @@ tune_judge: `mo_hyperband` and `primo` support agreement and cost objectives. `neps_priorband` and `neps_hyperband` require `objectives: [agreement]`. Optimizer kwargs remain native NePS settings; `primo` accepts objective-specific `prior_centers` and `prior_confidences` with dotted parameter names. Other run settings belong under `neps`, including evaluation budgets and `ignore_errors` (defaults to `true`). Managed directory/completion settings and NePS cost-spend budgets are unsupported. With NePS 0.17, leave `sample_prior_first` at its default for `neps_priorband` and `neps_hyperband` because enabling it fails in upstream trial-ID parsing. -**Expected judge cost** counts input and output tokens with tiktoken, including cached judgements, and reports USD per 1,000 battles. A per-model override takes precedence and may be a scalar (same input/output rate) or separate input/output rates, in USD per 1M tokens. Without an override, the OpenRouter model catalog supplies reference rates for every provider, including vLLM: matching first uses the Hugging Face ID, then the model ID's last two path segments. Catalog prices are estimates for comparison, not the actual price charged by another provider. +**Expected judge cost** uses native input and output token counts from model responses, including counts persisted with cached judgements, and reports USD per 1,000 battles. It does not estimate tokens with tiktoken. Cost tuning therefore requires fresh caches populated with native usage; caches created before usage was persisted cannot supply those counts. A per-model override takes precedence and may be a scalar (same input/output rate) or separate input/output rates, in USD per 1M tokens. Without an override, the shared OpenRouter reference price resolver supplies rates for any backend. Catalog prices are estimates for comparison, not the actual price charged by another provider. -When a searched model has no override, a new primary run fetches the catalog's free metadata and saves it as `{store_root}/openrouter_pricing.json`. If fetching fails, it uses that cached catalog; if neither is available, add an override or fetch the catalog on a login node and make the cache available. Resume and helper workers use the run's `prices.json` without fetching. On an internet-connected login node, prefetch the catalog into the same `run.store_root` used by the job: +When a searched model has no override, a cost-objective run may fetch the catalog's free metadata and save it as `{store_root}/openrouter_pricing.json`. Agreement-only tuning does not fetch the catalog and leaves cost unreported. Resume and helper workers use the run's `prices.json` without fetching. On an internet-connected login node, the catalog can be prefetched into the same `run.store_root` used by the job: ```bash mkdir -p /path/to/store-root @@ -416,7 +416,9 @@ tune_judge: OpenRouter/org/model: {input: 0.2, output: 0.8} ``` -The example's Qwen2.5-7B and gemma-2-9b-it rates (0.12 and 0.14) are user-provided estimates, not claims about current catalog prices. A cost-objective search stops before evaluation if any searched model has no price, naming the unresolved models and `tune_judge.price_per_million_tokens`. An agreement-only run may leave cost unknown. `prices.json` records the resolved rates and whether each came from `user_override` or `openrouter_reference`; resuming uses this run-specific file without a catalog fetch. Expected cost is separate from actual provider spend and NePS's search budget. +The example's Qwen2.5-7B and gemma-2-9b-it rates (0.12 and 0.14) are user-provided estimates, not claims about current catalog prices. A cost-objective search stops before evaluation if any searched model has no price, naming the unresolved models and `tune_judge.price_per_million_tokens`; resolved rates are frozen for the session. `prices.json` records the resolved rates and whether each came from `user_override` or `openrouter_reference`; resuming uses this run-specific file without a catalog fetch. Expected cost includes cached judgements and describes the judge configuration. Actual provider spend counts only calls made during the current run; cache hits incur no new provider call. Both are separate from NePS's search budget. + +Cost tuning requires native input and output token counts on every reused judgement. If the cache contains rows without counts, the run errors and directs you to use a fresh `store_root` or agreement-only tuning. It does not estimate or backfill counts, or automatically regenerate cached judgements. A judge call skipped because its prompt exceeds the model context limit contributes zero generation tokens; the failed annotation remains in the battle set and affects agreement and coverage. With `ignore_errors: true`, failed judge subprocess trials are recorded and penalized with infinite objective values; pricing, usage-accounting, and configuration errors still abort the tuning run. The primary process waits for active trials, reports the full-budget agreement/cost Pareto front, and selects the best agreement configuration per model for held-out testing. `test_battles_per_model` defaults to the highest validation fidelity. Results include `trials.parquet`, `pareto.parquet`, and `test_results.parquet`; NePS state stays under `neps/`, separate from inference caches. From 7c51bfac8789600e56a55e3126ee26c95c65bce1 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Fri, 9 Oct 2026 16:28:43 +0200 Subject: [PATCH 13/17] Document token-only caches and session pricing --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fbd4e51f..8c9e8cf3 100644 --- a/README.md +++ b/README.md @@ -416,7 +416,7 @@ tune_judge: OpenRouter/org/model: {input: 0.2, output: 0.8} ``` -The example's Qwen2.5-7B and gemma-2-9b-it rates (0.12 and 0.14) are user-provided estimates, not claims about current catalog prices. A cost-objective search stops before evaluation if any searched model has no price, naming the unresolved models and `tune_judge.price_per_million_tokens`; resolved rates are frozen for the session. `prices.json` records the resolved rates and whether each came from `user_override` or `openrouter_reference`; resuming uses this run-specific file without a catalog fetch. Expected cost includes cached judgements and describes the judge configuration. Actual provider spend counts only calls made during the current run; cache hits incur no new provider call. Both are separate from NePS's search budget. +The example's Qwen2.5-7B and gemma-2-9b-it rates (0.12 and 0.14) are user-provided estimates, not claims about current catalog prices. A cost-objective search stops before evaluation if any searched model has no price, naming the unresolved models and `tune_judge.price_per_million_tokens`; resolved rates are frozen for the session. `prices.json` records the resolved rates and whether each came from `user_override` or `openrouter_reference`; resuming uses this run-specific file without a catalog fetch. Expected cost includes cached judgements and describes the judge configuration. Inference caches retain token usage without dollar charges; charges in older cache rows are discarded when restoring usage. Expected cost is recalculated from these counts using the session's frozen prices. Actual provider spend stays in the original run reports and counts only calls made during that run; cache hits incur no new provider call. Both are separate from NePS's search budget, and the tuner leaves NePS's built-in `cost` field unset. Cost tuning requires native input and output token counts on every reused judgement. If the cache contains rows without counts, the run errors and directs you to use a fresh `store_root` or agreement-only tuning. It does not estimate or backfill counts, or automatically regenerate cached judgements. A judge call skipped because its prompt exceeds the model context limit contributes zero generation tokens; the failed annotation remains in the battle set and affects agreement and coverage. With `ignore_errors: true`, failed judge subprocess trials are recorded and penalized with infinite objective values; pricing, usage-accounting, and configuration errors still abort the tuning run. From d9f0a39c4b7614368e9ce6ad790436e3335fc7f0 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Fri, 9 Oct 2026 16:50:29 +0200 Subject: [PATCH 14/17] Share pricing fixtures and parameterize catalog resolution cases --- tests/test_tuning_pricing.py | 167 ++++++++++++++++------------------- 1 file changed, 77 insertions(+), 90 deletions(-) diff --git a/tests/test_tuning_pricing.py b/tests/test_tuning_pricing.py index c13071ad..836449d1 100644 --- a/tests/test_tuning_pricing.py +++ b/tests/test_tuning_pricing.py @@ -1,4 +1,6 @@ import json +from contextlib import nullcontext +from io import BytesIO from urllib.error import URLError import pytest @@ -8,6 +10,21 @@ from judgearena.usage import RequestUsage +@pytest.fixture +def catalog(): + return [ + { + "id": "org/model", + "pricing": {"prompt": "0.000001", "completion": "0.000002"}, + }, + { + "id": "elsewhere/different", + "hugging_face_id": "ORG/Model", + "pricing": {"prompt": "0.000003", "completion": "0.000004"}, + }, + ] + + def test_overrides_do_not_fetch_catalog(monkeypatch): def fail(*args, **kwargs): raise AssertionError("all prices were overridden") @@ -21,67 +38,53 @@ def fail(*args, **kwargs): assert result["OpenRouter/hosted"] == TokenPrice(0.2, 0.7, "user_override") -def test_huggingface_id_match_precedes_suffix_for_every_provider(tmp_path, monkeypatch): - records = [ - { - "id": "org/model", - "pricing": {"prompt": "0.000001", "completion": "0.000002"}, - }, - { - "id": "elsewhere/different", - "hugging_face_id": "ORG/Model", - "pricing": {"prompt": "0.000003", "completion": "0.000004"}, - }, - ] - monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _Response(records)) - cache = tmp_path / "openrouter_pricing.json" - for model in ("VLLM/org/model", "Provider/org/model"): - result = resolve_prices([model], {}, catalog_cache=cache) - assert result[model] == TokenPrice(3, 4, "openrouter_reference") - assert json.loads(cache.read_text()) == records - - -def test_suffix_match(monkeypatch): +@pytest.mark.parametrize( + ("model", "hf_match", "expected"), + [ + ("VLLM/org/model", True, TokenPrice(3, 4, "openrouter_reference")), + ("Hosted/org/model", True, TokenPrice(3, 4, "openrouter_reference")), + ("Gateway/org/model", True, TokenPrice(3, 4, "openrouter_reference")), + ("VLLM/org/model", False, TokenPrice(1, 2, "openrouter_reference")), + ], +) +def test_reference_model_matching( + model, hf_match, expected, catalog, monkeypatch, tmp_path +): + records = catalog if hf_match else catalog[:1] monkeypatch.setattr( pricing, "urlopen", - lambda *a, **k: _Response( - [ - { - "id": "org/model", - "pricing": {"prompt": "0.000005", "completion": "0.000009"}, - } - ] - ), - ) - assert resolve_prices(["Hosted/org/model"], {})["Hosted/org/model"] == TokenPrice( - 5, 9, "openrouter_reference" + lambda *a, **k: nullcontext(BytesIO(json.dumps({"data": records}).encode())), ) - - -def test_fetch_failure_uses_cached_catalog(tmp_path, monkeypatch): - cache = tmp_path / "openrouter_pricing.json" - cache.write_text( - json.dumps( - [ - { - "id": "org/model", - "pricing": {"prompt": "0.000002", "completion": "0.000006"}, - } - ] + cache = tmp_path / "catalog.json" + assert resolve_prices([model], {}, catalog_cache=cache) == {model: expected} + assert json.loads(cache.read_text()) == records + + +@pytest.mark.parametrize( + "cache_content,error", + [ + ("catalog", None), + (None, "login node.*price_per_million_tokens"), + ({"org/model": [1, 2]}, "unsupported format.*refresh.*overrides"), + ], +) +def test_offline_catalog_resolution( + tmp_path, monkeypatch, catalog, cache_content, error +): + cache = tmp_path / "catalog.json" + if cache_content is not None: + cache.write_text( + json.dumps(catalog if cache_content == "catalog" else cache_content) ) - ) monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _raise_url_error()) - price = resolve_prices(["Backend/org/model"], {}, catalog_cache=cache) - assert price["Backend/org/model"] == TokenPrice(2, 6, "openrouter_reference") - - -def test_fetch_failure_without_cache_has_actionable_error(tmp_path, monkeypatch): - monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _raise_url_error()) - with pytest.raises(ValueError, match="login node.*price_per_million_tokens"): - resolve_prices( - ["Backend/org/model"], {}, catalog_cache=tmp_path / "catalog.json" - ) + if error: + with pytest.raises(ValueError, match=error): + resolve_prices(["VLLM/org/model"], {}, catalog_cache=cache) + else: + assert resolve_prices(["VLLM/org/model"], {}, catalog_cache=cache) == { + "VLLM/org/model": TokenPrice(3, 4, "openrouter_reference") + } def test_unpriced_model_fails_for_cost_objective(monkeypatch): @@ -91,52 +94,36 @@ def unexpected_fetch(*args, **kwargs): monkeypatch.setattr(pricing, "urlopen", unexpected_fetch) assert resolve_prices(["VLLM/unknown"], {}, require_cost=False) == {} - monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _Response([])) + monkeypatch.setattr( + pricing, + "urlopen", + lambda *a, **k: nullcontext(BytesIO(b'{"data": []}')), + ) with pytest.raises(ValueError, match="unknown.*overrides"): resolve_prices(["VLLM/unknown"], {}) -def test_reference_cost_requires_native_counts_and_uses_reported_zero(): - with pytest.raises( - ValueError, match="missing native input_tokens and output_tokens" - ): - reference_cost( +@pytest.mark.parametrize( + ("usage", "missing"), + [ + ( RequestUsage(stage="judging", reasoning_tokens=12), - TokenPrice(1, 2, "ref"), - ) + "input_tokens and output_tokens", + ), + (RequestUsage(stage="judging", input_tokens=3), "output_tokens"), + ], +) +def test_reference_cost_requires_native_counts(usage, missing): + with pytest.raises(ValueError, match=f"missing native {missing}"): + reference_cost(usage, TokenPrice(1, 2, "ref")) + +def test_reference_cost_uses_native_counts_including_zero(): usage = RequestUsage( stage="judging", input_tokens=0, output_tokens=3, reasoning_tokens=100 ) assert reference_cost(usage, TokenPrice(1_000_000, 2_000_000, "ref")) == 6 - with pytest.raises(ValueError, match="missing native output_tokens"): - reference_cost( - RequestUsage(stage="judging", input_tokens=3), TokenPrice(1, 2, "ref") - ) - - -def test_fetch_failure_rejects_legacy_dictionary_catalog(tmp_path, monkeypatch): - cache = tmp_path / "catalog.json" - cache.write_text(json.dumps({"org/model": [1, 2]})) - monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _raise_url_error()) - with pytest.raises(ValueError, match="unsupported format.*refresh.*overrides"): - resolve_prices(["Backend/org/model"], {}, catalog_cache=cache) - - -class _Response: - def __init__(self, records): - self.records = records - - def __enter__(self): - return self - - def __exit__(self, *args): - return None - - def read(self): - return json.dumps({"data": self.records}).encode() - def _raise_url_error(): raise URLError("offline") From 3a498b867fa54ae46b2f3ee1735192aba73c64dd Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Fri, 9 Oct 2026 17:10:11 +0200 Subject: [PATCH 15/17] Keep concise reference pricing contract tests --- tests/test_tuning_pricing.py | 145 ++++++++++------------------------- 1 file changed, 41 insertions(+), 104 deletions(-) diff --git a/tests/test_tuning_pricing.py b/tests/test_tuning_pricing.py index 836449d1..8600f02c 100644 --- a/tests/test_tuning_pricing.py +++ b/tests/test_tuning_pricing.py @@ -1,7 +1,6 @@ import json from contextlib import nullcontext from io import BytesIO -from urllib.error import URLError import pytest @@ -10,120 +9,58 @@ from judgearena.usage import RequestUsage -@pytest.fixture -def catalog(): - return [ - { - "id": "org/model", - "pricing": {"prompt": "0.000001", "completion": "0.000002"}, - }, - { - "id": "elsewhere/different", - "hugging_face_id": "ORG/Model", - "pricing": {"prompt": "0.000003", "completion": "0.000004"}, - }, - ] - - -def test_overrides_do_not_fetch_catalog(monkeypatch): - def fail(*args, **kwargs): - raise AssertionError("all prices were overridden") - - monkeypatch.setattr(pricing, "urlopen", fail) - result = resolve_prices( - ["VLLM/local", "OpenRouter/hosted"], - {"VLLM/local": 1.2, "OpenRouter/hosted": {"input": 0.2, "output": 0.7}}, - ) - assert result["VLLM/local"] == TokenPrice(1.2, 1.2, "user_override") - assert result["OpenRouter/hosted"] == TokenPrice(0.2, 0.7, "user_override") - - -@pytest.mark.parametrize( - ("model", "hf_match", "expected"), - [ - ("VLLM/org/model", True, TokenPrice(3, 4, "openrouter_reference")), - ("Hosted/org/model", True, TokenPrice(3, 4, "openrouter_reference")), - ("Gateway/org/model", True, TokenPrice(3, 4, "openrouter_reference")), - ("VLLM/org/model", False, TokenPrice(1, 2, "openrouter_reference")), - ], -) -def test_reference_model_matching( - model, hf_match, expected, catalog, monkeypatch, tmp_path -): - records = catalog if hf_match else catalog[:1] - monkeypatch.setattr( - pricing, - "urlopen", - lambda *a, **k: nullcontext(BytesIO(json.dumps({"data": records}).encode())), - ) - cache = tmp_path / "catalog.json" - assert resolve_prices([model], {}, catalog_cache=cache) == {model: expected} - assert json.loads(cache.read_text()) == records +def test_override_and_agreement_only_skip_catalog(monkeypatch): + monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: pytest.fail("fetched")) + assert resolve_prices(["VLLM/local"], {"VLLM/local": 1.2}) == { + "VLLM/local": TokenPrice(1.2, 1.2, "user_override") + } + assert resolve_prices(["VLLM/local"], {}, require_cost=False) == {} @pytest.mark.parametrize( - "cache_content,error", - [ - ("catalog", None), - (None, "login node.*price_per_million_tokens"), - ({"org/model": [1, 2]}, "unsupported format.*refresh.*overrides"), - ], + "provider,hf_match", [("VLLM", True), ("OpenRouter", True), ("VLLM", False)] ) -def test_offline_catalog_resolution( - tmp_path, monkeypatch, catalog, cache_content, error +def test_reference_matching_and_catalog_snapshot( + provider, hf_match, monkeypatch, tmp_path ): - cache = tmp_path / "catalog.json" - if cache_content is not None: - cache.write_text( - json.dumps(catalog if cache_content == "catalog" else cache_content) + records = [dict(id="org/model", pricing=dict(prompt="1e-6", completion="2e-6"))] + if hf_match: + records.append( + dict( + id="other/model", + hugging_face_id="ORG/Model", + pricing=dict(prompt="3e-6", completion="4e-6"), + ) ) - monkeypatch.setattr(pricing, "urlopen", lambda *a, **k: _raise_url_error()) - if error: - with pytest.raises(ValueError, match=error): - resolve_prices(["VLLM/org/model"], {}, catalog_cache=cache) - else: - assert resolve_prices(["VLLM/org/model"], {}, catalog_cache=cache) == { - "VLLM/org/model": TokenPrice(3, 4, "openrouter_reference") - } - - -def test_unpriced_model_fails_for_cost_objective(monkeypatch): - def unexpected_fetch(*args, **kwargs): - raise AssertionError("agreement-only pricing must not fetch the catalog") - - monkeypatch.setattr(pricing, "urlopen", unexpected_fetch) - assert resolve_prices(["VLLM/unknown"], {}, require_cost=False) == {} - + response = json.dumps({"data": records}).encode() monkeypatch.setattr( - pricing, - "urlopen", - lambda *a, **k: nullcontext(BytesIO(b'{"data": []}')), + pricing, "urlopen", lambda *a, **k: nullcontext(BytesIO(response)) + ) + model, cache = f"{provider}/org/model", tmp_path / "prices.json" + expected = ( + TokenPrice(3, 4, "openrouter_reference") + if hf_match + else TokenPrice(1, 2, "openrouter_reference") ) - with pytest.raises(ValueError, match="unknown.*overrides"): - resolve_prices(["VLLM/unknown"], {}) + assert resolve_prices([model], {}, catalog_cache=cache)[model] == expected + assert json.loads(cache.read_text()) == records + with pytest.raises(ValueError, match="overrides"): + resolve_prices(["VLLM/unknown"], {}, catalog_cache=cache) @pytest.mark.parametrize( - ("usage", "missing"), - [ - ( - RequestUsage(stage="judging", reasoning_tokens=12), - "input_tokens and output_tokens", - ), - (RequestUsage(stage="judging", input_tokens=3), "output_tokens"), - ], + "tokens,expected", [((0, 3), 6), ((2, 1), 4), ((None, 3), None)] ) -def test_reference_cost_requires_native_counts(usage, missing): - with pytest.raises(ValueError, match=f"missing native {missing}"): - reference_cost(usage, TokenPrice(1, 2, "ref")) - - -def test_reference_cost_uses_native_counts_including_zero(): +def test_native_counts_and_no_double_counted_reasoning(tokens, expected): usage = RequestUsage( - stage="judging", input_tokens=0, output_tokens=3, reasoning_tokens=100 + stage="judging", + input_tokens=tokens[0], + output_tokens=tokens[1], + reasoning_tokens=100, ) - assert reference_cost(usage, TokenPrice(1_000_000, 2_000_000, "ref")) == 6 - - -def _raise_url_error(): - raise URLError("offline") + price = TokenPrice(1_000_000, 2_000_000, "ref") + if expected is None: + with pytest.raises(ValueError, match="missing native"): + reference_cost(usage, price) + else: + assert reference_cost(usage, price) == expected From 075459978abedd71252ea01b323878dc09d54fb2 Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Wed, 14 Oct 2026 14:09:06 +0200 Subject: [PATCH 16/17] Shorten the judge tuning README section --- README.md | 67 +++++++++++++++++++++---------------------------------- 1 file changed, 25 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 8c9e8cf3..c7e5b0e5 100644 --- a/README.md +++ b/README.md @@ -341,20 +341,18 @@ Runs save the selected sample, judge evidence, metric battles, configuration, an ### Tuning a judge -The packaged `tune-judge` task searches judge settings against human preferences, inspired by [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Install `pip install 'judgearena[tune]'` for NePS 0.17.0, then run: +The `tune-judge` task searches judge settings against human preferences, following [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Each trial runs the matching meta-eval task, so trials share its data and judgement cache. Install NePS with `pip install 'judgearena[tune]'`, then run: ```bash judgearena --config_path configs/tune_judge.yaml -judgearena tasks show tune-judge +judgearena tasks show tune-judge # task defaults ``` -The task supplies method defaults: `meta-eval-lmarena-140k-en`, objectives `[agreement, cost]`, `mo_hyperband` with `eta: 3`, and 10-90 battles per model. Run settings override these defaults field by field. Selecting another optimizer name starts with that optimizer's own kwargs; provide `eta` or other settings explicitly when needed. Trials use the selected meta-eval task's judge defaults and restrictions, data, language variant, and judgement cache. `meta_eval.top_models` and other sampling settings remain optional run settings. - -[configs/tune_judge.yaml](configs/tune_judge.yaml) uses this short form: +By default the task tunes on `meta-eval-lmarena-140k-en`, optimizes agreement and cost with `mo_hyperband` (`eta: 3`), and uses 10 to 90 battles per model. Your config overrides these field by field. If you pick another optimizer, set its kwargs (such as `eta`) yourself. [configs/tune_judge.yaml](configs/tune_judge.yaml) is a minimal example: ```yaml task: tune-judge -judge: +judge: # reference judge, also the search prior model: VLLM/Qwen/Qwen2.5-7B-Instruct prompt_preset: meta-eval-pair-score temperature: 0.0 @@ -368,25 +366,19 @@ tune_judge: swap_mode: [fixed, both] temperature: {lower: 0, upper: 1, prior_confidence: low} max_out_tokens: {lower: 1024, upper: 8192, log: true} - # User estimates in USD per 1M tokens, not live catalog prices. - price_per_million_tokens: + price_per_million_tokens: # your estimates in USD per 1M tokens VLLM/Qwen/Qwen2.5-7B-Instruct: {input: 0.12, output: 0.12} VLLM/google/gemma-2-9b-it: {input: 0.14, output: 0.14} neps: {total_evaluations_to_spend: 40} ``` -Lists, or `{choices: [...]}`, define categorical parameters. `{lower, upper}` defines a range whose type follows the corresponding run-config field: temperature is continuous even when written `{lower: 0, upper: 1}`, while `max_out_tokens` is integer. `log: true` enables logarithmic sampling. Parameters are nested like the run configuration and may vary judge fields or `generation.truncate_judge_input_chars`. +**Search space.** A list (or `{choices: [...]}`) is categorical. `{lower, upper}` is a range whose type follows the run-config field, so `temperature` is continuous and `max_out_tokens` is an integer; `log: true` samples on a log scale. Parameters are nested like the run config and can include `generation.truncate_judge_input_chars`. -The resolved reference judge supplies each parameter's prior when its value lies in the domain. An explicit `prior` is a **value**, including for categorical parameters. `prior_confidence` can be `low`, `medium`, or `high` and defaults to `medium`. For example: +**Priors.** Your reference judge is the prior wherever its value lies in the domain. To override, set `prior` to a value (also for categoricals) and `prior_confidence` to `low`, `medium` (default) or `high`. -```yaml -prompt_preset: - choices: [meta-eval-pair-score, arena-hard] - prior: arena-hard - prior_confidence: high -``` +**Cache reuse.** A continuous value such as temperature usually creates a new model descriptor per sample, so nothing is reused. Use a categorical list like `[0.0, 0.1, 1.0]` when reuse matters. -Use categorical temperature choices such as `[0.0, 0.1, 1.0]` when cache reuse matters; continuous sampling usually produces a new model descriptor for each value. Fidelity is separate from the search space: +**Optimizer and objectives.** These are the defaults; set them only to change something: ```yaml tune_judge: @@ -396,42 +388,33 @@ tune_judge: optimizer: {name: mo_hyperband, eta: 3, mo_selector: epsnet} ``` -`mo_hyperband` and `primo` support agreement and cost objectives. `neps_priorband` and `neps_hyperband` require `objectives: [agreement]`. Optimizer kwargs remain native NePS settings; `primo` accepts objective-specific `prior_centers` and `prior_confidences` with dotted parameter names. Other run settings belong under `neps`, including evaluation budgets and `ignore_errors` (defaults to `true`). Managed directory/completion settings and NePS cost-spend budgets are unsupported. With NePS 0.17, leave `sample_prior_first` at its default for `neps_priorband` and `neps_hyperband` because enabling it fails in upstream trial-ID parsing. +- `mo_hyperband` and `primo` take `[agreement, cost]`; `neps_priorband` and `neps_hyperband` take `[agreement]`. +- Optimizer kwargs are native NePS settings. `primo` also takes per-objective `prior_centers` and `prior_confidences`. +- Other `neps.run` settings go under `neps`, such as budgets and `ignore_errors` (default `true`). Directory, completion and NePS cost-budget settings are not supported. +- Leave `sample_prior_first` unset for `neps_priorband` and `neps_hyperband`; NePS 0.17 fails on it. -**Expected judge cost** uses native input and output token counts from model responses, including counts persisted with cached judgements, and reports USD per 1,000 battles. It does not estimate tokens with tiktoken. Cost tuning therefore requires fresh caches populated with native usage; caches created before usage was persisted cannot supply those counts. A per-model override takes precedence and may be a scalar (same input/output rate) or separate input/output rates, in USD per 1M tokens. Without an override, the shared OpenRouter reference price resolver supplies rates for any backend. Catalog prices are estimates for comparison, not the actual price charged by another provider. - -When a searched model has no override, a cost-objective run may fetch the catalog's free metadata and save it as `{store_root}/openrouter_pricing.json`. Agreement-only tuning does not fetch the catalog and leaves cost unreported. Resume and helper workers use the run's `prices.json` without fetching. On an internet-connected login node, the catalog can be prefetched into the same `run.store_root` used by the job: - -```bash -mkdir -p /path/to/store-root -curl -fsS https://openrouter.ai/api/v1/models | jq '.data' > /path/to/store-root/openrouter_pricing.json -``` - -Optional per-model overrides use: - -```yaml -tune_judge: - price_per_million_tokens: - VLLM/org/model: 0.2 - OpenRouter/org/model: {input: 0.2, output: 0.8} -``` +**Cost.** Expected cost is USD per 1,000 battles, computed from the native input and output token counts of the judge responses, including counts stored with cached judgements. It is separate from NePS's search budget. -The example's Qwen2.5-7B and gemma-2-9b-it rates (0.12 and 0.14) are user-provided estimates, not claims about current catalog prices. A cost-objective search stops before evaluation if any searched model has no price, naming the unresolved models and `tune_judge.price_per_million_tokens`; resolved rates are frozen for the session. `prices.json` records the resolved rates and whether each came from `user_override` or `openrouter_reference`; resuming uses this run-specific file without a catalog fetch. Expected cost includes cached judgements and describes the judge configuration. Inference caches retain token usage without dollar charges; charges in older cache rows are discarded when restoring usage. Expected cost is recalculated from these counts using the session's frozen prices. Actual provider spend stays in the original run reports and counts only calls made during that run; cache hits incur no new provider call. Both are separate from NePS's search budget, and the tuner leaves NePS's built-in `cost` field unset. +- Prices come from `tune_judge.price_per_million_tokens` (one number, or `{input, output}`). Without an override, the OpenRouter catalog supplies an estimate, saved to `{store_root}/openrouter_pricing.json`. Agreement-only runs never fetch it. For compute nodes without internet, prefetch it on the login node: `curl -fsS https://openrouter.ai/api/v1/models | jq '.data' > /openrouter_pricing.json`. +- A searched model with no price stops the run before evaluation. Resolved rates are frozen in the run's `prices.json`, which resume and helper workers read. +- Every reused judgement needs native token counts. Caches written before usage was stored do not have them, so the run errors; use a fresh `store_root` or tune on agreement only. Counts are never estimated. +- A judge call skipped for exceeding the context limit adds zero generation tokens, and the failed annotation lowers agreement and coverage. +- With `ignore_errors: true`, failed trials get infinite objective values. Pricing, usage and configuration errors still abort the run. -Cost tuning requires native input and output token counts on every reused judgement. If the cache contains rows without counts, the run errors and directs you to use a fresh `store_root` or agreement-only tuning. It does not estimate or backfill counts, or automatically regenerate cached judgements. A judge call skipped because its prompt exceeds the model context limit contributes zero generation tokens; the failed annotation remains in the battle set and affects agreement and coverage. With `ignore_errors: true`, failed judge subprocess trials are recorded and penalized with infinite objective values; pricing, usage-accounting, and configuration errors still abort the tuning run. +**Results.** The primary process waits for active trials, reports the agreement/cost Pareto front at full budget, and tests the best-agreement configuration per model on the held-out split (`test_battles_per_model`, default the highest fidelity). Outputs are `trials.parquet`, `pareto.parquet` and `test_results.parquet`; NePS state stays in `neps/`, apart from the inference caches. -The primary process waits for active trials, reports the full-budget agreement/cost Pareto front, and selects the best agreement configuration per model for held-out testing. `test_battles_per_model` defaults to the highest validation fidelity. Results include `trials.parquet`, `pareto.parquet`, and `test_results.parquet`; NePS state stays under `neps/`, separate from inference caches. +**Resume and parallel workers.** Workers share an explicit tuning directory: ```bash -# Explicitly resume the same session. +# Resume the same session. judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir -# Helper workers join an initialized session and run validation only. +# Join as a helper that runs validation trials only. judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir --tune_judge.search_only true ``` -The example keeps paper limits fixed across judges: 8,192 characters per candidate completion, 8,192 judge output tokens, and native context. Instructions are not truncated. For vLLM chat judges, overlong prompts remain failed annotations in the shared battle set, lowering agreement and coverage; skipped outputs are not cached. Only searched settings vary. +The example keeps the paper's limits fixed across judges (8,192 characters per candidate completion, 8,192 judge output tokens, native context, no instruction truncation), so only searched settings vary. Overlong prompts for vLLM chat judges stay failed annotations and are not cached. -Shared rollback-mode SQLite caching was verified on kislurm NFSv3 in a two-node test; this does not guarantee locking on other filesystems. Never use WAL on NFS. Workers share an explicit tuning directory and can reuse judgements from direct runs of the target meta-eval task. +Rollback-mode SQLite caching on a shared mount worked in a two-node test on kislurm (NFSv3). That is no guarantee for other filesystems, and WAL must never be used on NFS. ## 📈 Estimating ELO Ratings From b486992111f2e5078b7600b2d0cceccbbeb6c3cd Mon Sep 17 00:00:00 2001 From: Erlis Lushtaku <59629249+ErlisLushtaku@users.noreply.github.com> Date: Wed, 14 Oct 2026 14:12:21 +0200 Subject: [PATCH 17/17] Use uv in the judge tuning docs --- README.md | 11 ++++++----- configs/tune_judge.yaml | 2 +- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index c7e5b0e5..911c377a 100644 --- a/README.md +++ b/README.md @@ -341,11 +341,12 @@ Runs save the selected sample, judge evidence, metric battles, configuration, an ### Tuning a judge -The `tune-judge` task searches judge settings against human preferences, following [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Each trial runs the matching meta-eval task, so trials share its data and judgement cache. Install NePS with `pip install 'judgearena[tune]'`, then run: +The `tune-judge` task searches judge settings against human preferences, following [Salinas et al., 2025](https://arxiv.org/abs/2501.17178). Each trial runs the matching meta-eval task, so trials share its data and judgement cache. Install NePS with the `tune` extra, then run: ```bash -judgearena --config_path configs/tune_judge.yaml -judgearena tasks show tune-judge # task defaults +uv sync --extra tune +uv run judgearena --config_path configs/tune_judge.yaml +uv run judgearena tasks show tune-judge # task defaults ``` By default the task tunes on `meta-eval-lmarena-140k-en`, optimizes agreement and cost with `mo_hyperband` (`eta: 3`), and uses 10 to 90 battles per model. Your config overrides these field by field. If you pick another optimizer, set its kwargs (such as `eta`) yourself. [configs/tune_judge.yaml](configs/tune_judge.yaml) is a minimal example: @@ -407,9 +408,9 @@ tune_judge: ```bash # Resume the same session. -judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir +uv run judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir # Join as a helper that runs validation trials only. -judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir --tune_judge.search_only true +uv run judgearena --config_path configs/tune_judge.yaml --tune_judge.run_dir /path/to/tune-dir --tune_judge.search_only true ``` The example keeps the paper's limits fixed across judges (8,192 characters per candidate completion, 8,192 judge output tokens, native context, no instruction truncation), so only searched settings vary. Overlong prompts for vLLM chat judges stay failed annotations and are not cached. diff --git a/configs/tune_judge.yaml b/configs/tune_judge.yaml index c505dae3..9f7b2dd4 100644 --- a/configs/tune_judge.yaml +++ b/configs/tune_judge.yaml @@ -1,4 +1,4 @@ -# Install judgearena[tune], then run: judgearena --config_path configs/tune_judge.yaml +# Install the tune extra, then run: uv run judgearena --config_path configs/tune_judge.yaml task: tune-judge judge: model: VLLM/Qwen/Qwen2.5-7B-Instruct