Refresh LUMI results and use corrected CoT evals by default

README.md

@@ -4,12 +4,12 @@ A standalone, offline dashboard for comparing model evaluation scores. Explore c
 
 **[Open the dashboard](https://openeurollm.github.io/quickdash/)** or **[try the fictional example](https://openeurollm.github.io/quickdash/demo.html)**. No installation is needed to use either page.
 
-The main page opens with **v1annealC_120k_l0fix** as A and **v2anneal_120k** as B, using the exports in [results/](results/README.md). The earlier v1 run and synthetic comparisons remain selectable. The startup pair is configured in [results/default.yaml](results/default.yaml). The default strict matching excludes results whose few-shot settings differ from the catalogue; review **Warnings**, or explicitly select relaxed matching to include them.
+The main page opens with **v1annealC_120k_l0fix** as A and **v2anneal_120k** as B, using the exports in [results/](results/README.md). Other shared checkpoints, including the merge3 models, and synthetic comparisons remain selectable. The startup pair is configured in [results/default.yaml](results/default.yaml). The default strict matching excludes results whose few-shot settings differ from the catalogue; review **Warnings**, or explicitly select relaxed matching to include them.
 
 ## Compare models
 
 1. Select shared models as A and B, or use **Add model CSV** to open your exports. Three labelled synthetic comparisons (perturbed, higher, and lower scores) are supplied for exploring the interface.
-2. Choose a **Weighting profile**. **Eval set** starts on **flagship-1** to check the expected evals and apply its language exclusions. Select **Any available** to compare shared measurements without requiring that set. Weights and eval sets are independent. The supplied sets exclude prompted Global PIQA pending scoring validation.
+2. Choose a **Weighting profile**. **Eval set** starts on **flagship-1**, using corrected CoT reasoning and code-continuation results with its expected coverage and language exclusions. Missing corrected results show incomplete coverage; original runs remain inspectable. Select **Any available** to compare all recognized shared measurements, which can include both protocols. Weights and eval sets are independent. The supplied sets exclude prompted Global PIQA pending scoring validation.
 3. Review **Warnings**, then explore the scores and breakdowns. The global catalogue determines how to interpret each eval: category, scoring field, normalization, and language assignments.
 
 **Original** is the startup weighting profile. **Code & math emphasis** gives Code and Math 20% each, with the other category weights adjusted as shown in the [configuration reference](docs/configuration.md#choose-weights-and-expected-coverage).

configs/README.md

@@ -11,6 +11,8 @@ Choose the file to edit based on what you want to change:
 
 Catalogue `metric`, `metric_filter`, and `shots` are defaults. A named set can override these for a whole eval; omitted values inherit. The dashboard can explicitly relax few-shot matching with warnings. Language exclusions use the catalogue’s assignments, including both translation endpoints. Unknown names are configuration errors. See [set configuration and matching](../docs/configuration.md#strict-and-relaxed-matching).
 
+Distinct evaluation protocols use separate catalogue entries, such as [polymath.yaml](evals/polymath.yaml) and [polymath_cot.yaml](evals/polymath_cot.yaml). The default `flagship-1` set selects the corrected CoT and code-continuation entries, without fallback to original runs. `Any available` can include both; use a named set to choose one protocol per benchmark.
+
 Put repeated language `evidence` and `note` under `language_defaults` in the eval file; individual language entries can override either field. Ordinary entries need only a canonical `language` and `tasks`: scope is inferred from the declared language fields. Keep an explicit `scope: pooled` when a pooled result is assigned to a specific language label. See [shared language metadata](../docs/configuration.md#shared-language-metadata).
 
 Each profile or set needs a distinct `name` within its directory. To change a selector's startup choice, edit that directory's `default.txt` to name one YAML file. The catalogue is selected at build time with `--catalogue`, or temporarily loaded in the browser.

configs/evals/aime24_cot.yaml

@@ -0,0 +1,22 @@
+name: aime24_cot
+category: Reasoning
+match:
+  name: aime24_cot
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0
+  max: 1
+  basis: not_applicable
+  note: CoT pass@1; retain the AIME zero floor without a random-integer guessing correction.
+  sources:
+  - https://maa.org/maa-invitational-competitions/
+languages:
+- language: eng_Latn
+  tasks:
+  - aime24_cot
+  evidence: https://maa.org/maa-invitational-competitions/
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/aime25_cot.yaml

@@ -0,0 +1,22 @@
+name: aime25_cot
+category: Reasoning
+match:
+  name: aime25_cot
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0
+  max: 1
+  basis: not_applicable
+  note: CoT pass@1; retain the AIME zero floor without a random-integer guessing correction.
+  sources:
+  - https://maa.org/maa-invitational-competitions/
+languages:
+- language: eng_Latn
+  tasks:
+  - aime25_cot
+  evidence: https://maa.org/maa-invitational-competitions/
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/amc23_cot.yaml

@@ -0,0 +1,22 @@
+name: amc23_cot
+category: Reasoning
+match:
+  name: amc23_cot
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0
+  max: 1
+  basis: not_applicable
+  note: CoT pass@1; retain the AMC23 zero floor used for open-ended mathematical answers.
+  sources:
+  - https://maa.org/student-programs/amc/
+languages:
+- language: eng_Latn
+  tasks:
+  - amc23_cot
+  evidence: https://maa.org/student-programs/amc/
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/gpqa_diamond_cot.yaml

@@ -0,0 +1,22 @@
+name: gpqa_diamond_cot
+category: Knowledge
+match:
+  name: gpqa_diamond_cot
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0.25
+  max: 1
+  basis: uniform_choice
+  note: CoT pass@1; retain the GPQA Diamond four-choice uniform-guess baseline of 0.25.
+  sources:
+  - https://huggingface.co/datasets/Idavidrein/gpqa
+languages:
+- language: eng_Latn
+  tasks:
+  - gpqa_diamond_cot
+  evidence: https://huggingface.co/datasets/Idavidrein/gpqa
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/humaneval_cont.yaml

@@ -0,0 +1,22 @@
+name: humaneval_cont
+category: Code
+match:
+  name: humaneval_cont
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0
+  max: 1
+  basis: not_applicable
+  note: Code-continuation pass@1 is graded by executable tests; no random-program baseline is defined.
+  sources:
+  - https://github.com/openai/human-eval
+languages:
+- language: eng_Latn
+  tasks:
+  - humaneval_cont
+  evidence: https://github.com/openai/human-eval
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/jeebench_cot.yaml

@@ -0,0 +1,23 @@
+name: jeebench_cot
+category: Reasoning
+match:
+  name: jeebench_cot
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0.1055
+  max: 1
+  note: CoT pass@1; retain the shared JEEBench normalization floor of 0.1055. This is the existing benchmark scoring
+    convention, based on the paper's mixed-format guessing policy for 515 questions, rather than a measured baseline
+    for this prompting protocol.
+  sources:
+  - https://aclanthology.org/2023.emnlp-main.468.pdf#page=5
+languages:
+- language: eng_Latn
+  tasks:
+  - jeebench_cot
+  evidence: https://aclanthology.org/2023.emnlp-main.468.pdf#page=5
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/livecodebench_cont.yaml

@@ -0,0 +1,22 @@
+name: livecodebench_cont
+category: Code
+match:
+  name: livecodebench_cont
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0
+  max: 1
+  basis: not_applicable
+  note: Code-continuation pass@1 is graded by executable tests; no random-program baseline is defined.
+  sources:
+  - https://github.com/LiveCodeBench/LiveCodeBench
+languages:
+- language: eng_Latn
+  tasks:
+  - livecodebench_cont
+  evidence: https://github.com/LiveCodeBench/LiveCodeBench
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/math500_cot.yaml

@@ -0,0 +1,23 @@
+name: math500_cot
+category: Math
+match:
+  name: math500_cot
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0
+  max: 1
+  basis: not_applicable
+  note: CoT pass@1 for generated mathematical answers; retain the MATH500 zero floor because no common finite answer
+    space is defined.
+  sources:
+  - https://huggingface.co/datasets/HuggingFaceH4/MATH-500
+languages:
+- language: eng_Latn
+  tasks:
+  - math500_cot
+  evidence: https://huggingface.co/datasets/HuggingFaceH4/MATH-500
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/mbpp_cont.yaml

@@ -0,0 +1,22 @@
+name: mbpp_cont
+category: Code
+match:
+  name: mbpp_cont
+metric: pass@1
+metric_filter: all
+shots: 3
+score:
+  scale: 1
+normalize:
+  min: 0
+  max: 1
+  basis: not_applicable
+  note: Code-continuation pass@1 is graded by executable tests; no random-program baseline is defined.
+  sources:
+  - https://github.com/google-research/google-research/tree/master/mbpp
+languages:
+- language: eng_Latn
+  tasks:
+  - mbpp_cont
+  evidence: https://github.com/google-research/google-research/tree/master/mbpp
+  note: English benchmark prompt language; protocol task identifier is recorded in the flag-evals-471 export.

configs/evals/polymath.yaml

@@ -1,6 +1,6 @@
 name: polymath
 category: Reasoning
-match: {regex: 'polymath_.+'}
+match: {regex: 'polymath_.+_(?:low|medium|high|top)'}
 metric: exact_match
 metric_filter: none
 shots: 0

configs/evals/polymath_cot.yaml

@@ -0,0 +1,80 @@
+name: polymath_cot
+category: Reasoning
+match:
+  regex: polymath_.+_(?:low|medium|high|top)_cot
+metric: pass@1
+metric_filter: all
+shots: 0
+score:
+  scale: 1
+normalize:
+  min: 0
+  max: 1
+  basis: not_applicable
+  note: CoT pass@1 for generated mathematical answers; no common finite answer space is defined.
+  sources:
+  - https://github.com/QwenLM/PolyMath
+aggregation:
+  components:
+  - name: low
+    match:
+      regex: polymath_.+_low_cot
+    relative_weight: 1
+  - name: medium
+    match:
+      regex: polymath_.+_medium_cot
+    relative_weight: 2
+  - name: high
+    match:
+      regex: polymath_.+_high_cot
+    relative_weight: 4
+  - name: top
+    match:
+      regex: polymath_.+_top_cot
+    relative_weight: 8
+  note: 'PolyMath difficulty-weighted pass@1 combines all four levels within each language: (low + 2 × medium +
+    4 × high + 8 × top) / 15. Exported pass@1 values are unweighted per-level scores. Each CoT level is required;
+    original-protocol rows cannot fill a missing CoT component.'
+  sources:
+  - https://qwen-polymath.github.io/#benchmark-score
+  - https://github.com/QwenLM/PolyMath/blob/main/eval/run_eval.py
+language_defaults:
+  evidence: https://huggingface.co/datasets/Qwen/PolyMath
+  note: Dataset language follows the underlying PolyMath split; the task suffix identifies the CoT protocol.
+languages:
+- language: deu_Latn
+  tasks:
+  - polymath_de_high_cot
+  - polymath_de_low_cot
+  - polymath_de_medium_cot
+  - polymath_de_top_cot
+- language: eng_Latn
+  tasks:
+  - polymath_en_high_cot
+  - polymath_en_low_cot
+  - polymath_en_medium_cot
+  - polymath_en_top_cot
+- language: spa_Latn
+  tasks:
+  - polymath_es_high_cot
+  - polymath_es_low_cot
+  - polymath_es_medium_cot
+  - polymath_es_top_cot
+- language: fra_Latn
+  tasks:
+  - polymath_fr_high_cot
+  - polymath_fr_low_cot
+  - polymath_fr_medium_cot
+  - polymath_fr_top_cot
+- language: ita_Latn
+  tasks:
+  - polymath_it_high_cot
+  - polymath_it_low_cot
+  - polymath_it_medium_cot
+  - polymath_it_top_cot
+- language: por_Latn
+  tasks:
+  - polymath_pt_high_cot
+  - polymath_pt_low_cot
+  - polymath_pt_medium_cot
+  - polymath_pt_top_cot

configs/sets/flagship-1.yaml

@@ -3,28 +3,29 @@ name: flagship-1
 mode: fixed
 exclude_languages: [kat_Geor]
 notes:
+  - Use corrected CoT reasoning and code-continuation protocols; original-protocol results are excluded without fallback.
   - Require the catalogue tasks for each listed eval, inheriting its scoring defaults.
   - Exclude Georgian results because of a vocabulary error. Exclude English X-CSQA to avoid overlap with CommonsenseQA.
 evals:
   - name: bigbench_cs_algorithms
-  - name: HumanEval
-  - name: LiveCodeBench
-  - name: mbpp
+  - name: humaneval_cont
+  - name: livecodebench_cont
+  - name: mbpp_cont
   - name: bigbench_dyck_languages
   - name: bigbench_operators
   - name: bigbench_repeat_copy_logic
   - name: gsm8k
   - name: mgsm
-  - name: MATH500
-  - name: AIME24
-  - name: AIME25
-  - name: AMC23
-  - name: JEEBench
+  - name: math500_cot
+  - name: aime24_cot
+  - name: aime25_cot
+  - name: amc23_cot
+  - name: jeebench_cot
   - name: agieval_lsat_ar
-  - name: polymath
+  - name: polymath_cot
   - name: arc_challenge
   - name: arc_easy
-  - name: GPQADiamond
+  - name: gpqa_diamond_cot
   - name: include_base_44
   - name: jeopardy
   - name: mmlu