From 08875c27b444ad918d5428c444b31156c98793e1 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Mon, 12 Oct 2026 08:20:57 -0700 Subject: [PATCH 01/19] Upload lm_eval results for qwen 3 8B (mmlu + wikitext Signed-off-by: Daniel Korzekwa --- examples/llm_qat/README.md | 2 +- examples/llm_qat/evaluation/README.md | 18 + .../llm_qat/evaluation/nel_configs/mmlu.yaml | 36 + .../evaluation/nel_configs/wikitext.yaml | 36 + ...al_results_2026-10-08T10-29-11.704256.json | 4030 +++++++++++++++++ .../mmlu_full/nel_config.yaml | 58 + ...al_results_2026-10-08T07-23-16.887993.json | 136 + .../wikitext_full/nel_config.yaml | 41 + ...al_results_2026-10-08T11-35-03.360012.json | 4030 +++++++++++++++++ .../mmlu_full/nel_config.yaml | 58 + ...al_results_2026-10-08T07-23-24.097523.json | 136 + .../wikitext_full/nel_config.yaml | 41 + ...al_results_2026-10-08T09-24-34.263592.json | 4030 +++++++++++++++++ .../mmlu_full/nel_config.yaml | 57 + ...al_results_2026-10-08T07-22-39.797863.json | 136 + .../wikitext_full/nel_config.yaml | 40 + 16 files changed, 12884 insertions(+), 1 deletion(-) create mode 100644 examples/llm_qat/evaluation/README.md create mode 100644 examples/llm_qat/evaluation/nel_configs/mmlu.yaml create mode 100644 examples/llm_qat/evaluation/nel_configs/wikitext.yaml create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json create mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml diff --git a/examples/llm_qat/README.md b/examples/llm_qat/README.md index cd88ae30498..a73447f9f03 100644 --- a/examples/llm_qat/README.md +++ b/examples/llm_qat/README.md @@ -309,7 +309,7 @@ The cached dataset is stored under `.dataset_cache/tokenized/` by default (confi ## Results -\[Coming Soon\] +See [Qwen3-8B evaluation results](evaluation/README.md) for the teacher, PTQ, and PTQ → QAD comparison. ## Native Fake-Quantized Evaluation diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md new file mode 100644 index 00000000000..29a948e49b8 --- /dev/null +++ b/examples/llm_qat/evaluation/README.md @@ -0,0 +1,18 @@ +# Evaluation + +NeMo Evaluator Launcher (NEL) configurations and evaluation artifacts for the [Hugging Face QAT/QAD tutorial](../README.md). + +- `nel_configs/`: shared benchmark recipes. +- `results///`: each evaluation's `nel_config.yaml` and original `lm_eval_results_*.json` generated by lm-evaluation-harness. Per-example samples are not stored here. + +## Results + +Qwen3-8B was evaluated using NEL with vLLM on one H100 per evaluation. QAD starts from the PTQ checkpoint and uses the BF16 model as teacher. + +| Checkpoint | Deployed precision | MMLU accuracy ↑ | WikiText-2 perplexity ↓ | +|---|---|---:|---:| +| Teacher (BF16) | BF16 | [74.94%](results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json) | [13.485](results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json) | +| PTQ (NVFP4) | NVFP4 weights / BF16 activations | [73.59%](results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json) | [13.883](results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json) | +| PTQ → QAD (NVFP4) | NVFP4 weights / BF16 activations | [73.76%](results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json) | [13.684](results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json) | + +QAD improves both metrics over PTQ but remains below the BF16 teacher. The quantized checkpoints use Marlin's W4A16 fallback on H100; these results do not measure native NVFP4 W4A4 execution on Blackwell. KV cache is BF16 for all checkpoints. diff --git a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml new file mode 100644 index 00000000000..8d3a7398fc9 --- /dev/null +++ b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml @@ -0,0 +1,36 @@ +# Shared benchmark recipe for nemo-evaluator-launcher==0.2.6. +# All 57 MMLU subjects, test split, 5-shot log-likelihood accuracy. No sample limit by default. +evaluation: + nemo_evaluator_config: + config: + params: + parallelism: 2 + request_timeout: 300 + max_retries: 2 + temperature: null + top_p: null + max_new_tokens: null + task: mmlu + target: + api_endpoint: + api_key_name: null + adapter_config: + use_reasoning: false + use_request_logging: true + use_response_logging: true + max_logged_requests: 2 + max_logged_responses: 2 + tasks: + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + task: ${evaluation.nemo_evaluator_config.config.params.task} + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 diff --git a/examples/llm_qat/evaluation/nel_configs/wikitext.yaml b/examples/llm_qat/evaluation/nel_configs/wikitext.yaml new file mode 100644 index 00000000000..b31f03a4ad4 --- /dev/null +++ b/examples/llm_qat/evaluation/nel_configs/wikitext.yaml @@ -0,0 +1,36 @@ +# Shared benchmark recipe for nemo-evaluator-launcher==0.2.6. +# WikiText-2 test split, zero-shot rolling log-likelihood perplexity. No sample limit by default. +evaluation: + nemo_evaluator_config: + config: + params: + parallelism: 2 + request_timeout: 300 + max_retries: 2 + temperature: null + top_p: null + max_new_tokens: null + task: wikitext + target: + api_endpoint: + api_key_name: null + adapter_config: + use_reasoning: false + use_request_logging: true + use_response_logging: true + max_logged_requests: 2 + max_logged_responses: 2 + tasks: + - name: lm-evaluation-harness.wikitext + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + task: ${evaluation.nemo_evaluator_config.config.params.task} + extra: + num_fewshot: 0 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json new file mode 100644 index 00000000000..b151d52a0af --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json @@ -0,0 +1,4030 @@ +{ + "results": { + "mmlu": { + "acc,none": 0.7358638370602478, + "acc_stderr,none": 0.003529928328054296, + "alias": "mmlu" + }, + "mmlu_humanities": { + "acc,none": 0.6463336875664187, + "acc_stderr,none": 0.006627653147654618, + "alias": " - humanities" + }, + "mmlu_formal_logic": { + "alias": " - formal_logic", + "acc,none": 0.6666666666666666, + "acc_stderr,none": 0.04216370213557839 + }, + "mmlu_high_school_european_history": { + "alias": " - high_school_european_history", + "acc,none": 0.8545454545454545, + "acc_stderr,none": 0.02753019635506653 + }, + "mmlu_high_school_us_history": { + "alias": " - high_school_us_history", + "acc,none": 0.8529411764705882, + "acc_stderr,none": 0.024857478080250486 + }, + "mmlu_high_school_world_history": { + "alias": " - high_school_world_history", + "acc,none": 0.8565400843881856, + "acc_stderr,none": 0.02281829182101699 + }, + "mmlu_international_law": { + "alias": " - international_law", + "acc,none": 0.7355371900826446, + "acc_stderr,none": 0.04026187527591209 + }, + "mmlu_jurisprudence": { + "alias": " - jurisprudence", + "acc,none": 0.8333333333333334, + "acc_stderr,none": 0.03602814176392641 + }, + "mmlu_logical_fallacies": { + "alias": " - logical_fallacies", + "acc,none": 0.803680981595092, + "acc_stderr,none": 0.031207970394709197 + }, + "mmlu_moral_disputes": { + "alias": " - moral_disputes", + "acc,none": 0.7225433526011561, + "acc_stderr,none": 0.024105712607754307 + }, + "mmlu_moral_scenarios": { + "alias": " - moral_scenarios", + "acc,none": 0.5385474860335195, + "acc_stderr,none": 0.01667273126755234 + }, + "mmlu_philosophy": { + "alias": " - philosophy", + "acc,none": 0.7427652733118971, + "acc_stderr,none": 0.0248261712892509 + }, + "mmlu_prehistory": { + "alias": " - prehistory", + "acc,none": 0.8271604938271605, + "acc_stderr,none": 0.02103851777015737 + }, + "mmlu_professional_law": { + "alias": " - professional_law", + "acc,none": 0.49022164276401564, + "acc_stderr,none": 0.012767793787729296 + }, + "mmlu_world_religions": { + "alias": " - world_religions", + "acc,none": 0.8538011695906432, + "acc_stderr,none": 0.02709729011807083 + }, + "mmlu_other": { + "acc,none": 0.7740585774058577, + "acc_stderr,none": 0.00721392999736686, + "alias": " - other" + }, + "mmlu_business_ethics": { + "alias": " - business_ethics", + "acc,none": 0.77, + "acc_stderr,none": 0.042295258468165065 + }, + "mmlu_clinical_knowledge": { + "alias": " - clinical_knowledge", + "acc,none": 0.8037735849056604, + "acc_stderr,none": 0.0244423881311008 + }, + "mmlu_college_medicine": { + "alias": " - college_medicine", + "acc,none": 0.7803468208092486, + "acc_stderr,none": 0.03156809362703173 + }, + "mmlu_global_facts": { + "alias": " - global_facts", + "acc,none": 0.44, + "acc_stderr,none": 0.049888765156985884 + }, + "mmlu_human_aging": { + "alias": " - human_aging", + "acc,none": 0.7533632286995515, + "acc_stderr,none": 0.028930413120910853 + }, + "mmlu_management": { + "alias": " - management", + "acc,none": 0.8932038834951457, + "acc_stderr,none": 0.030581088928331335 + }, + "mmlu_marketing": { + "alias": " - marketing", + "acc,none": 0.9273504273504274, + "acc_stderr,none": 0.017004368568132394 + }, + "mmlu_medical_genetics": { + "alias": " - medical_genetics", + "acc,none": 0.82, + "acc_stderr,none": 0.03861229196653691 + }, + "mmlu_miscellaneous": { + "alias": " - miscellaneous", + "acc,none": 0.8378033205619413, + "acc_stderr,none": 0.01318222261672081 + }, + "mmlu_nutrition": { + "alias": " - nutrition", + "acc,none": 0.8169934640522876, + "acc_stderr,none": 0.022140767512880986 + }, + "mmlu_professional_accounting": { + "alias": " - professional_accounting", + "acc,none": 0.5851063829787234, + "acc_stderr,none": 0.02939223658461248 + }, + "mmlu_professional_medicine": { + "alias": " - professional_medicine", + "acc,none": 0.8051470588235294, + "acc_stderr,none": 0.024060599423487473 + }, + "mmlu_virology": { + "alias": " - virology", + "acc,none": 0.5240963855421686, + "acc_stderr,none": 0.03887971849597267 + }, + "mmlu_social_sciences": { + "acc,none": 0.8290542736431589, + "acc_stderr,none": 0.00666849172124334, + "alias": " - social sciences" + }, + "mmlu_econometrics": { + "alias": " - econometrics", + "acc,none": 0.6491228070175439, + "acc_stderr,none": 0.044895393502707 + }, + "mmlu_high_school_geography": { + "alias": " - high_school_geography", + "acc,none": 0.8636363636363636, + "acc_stderr,none": 0.02445015597318985 + }, + "mmlu_high_school_government_and_politics": { + "alias": " - high_school_government_and_politics", + "acc,none": 0.9222797927461139, + "acc_stderr,none": 0.01932180555722318 + }, + "mmlu_high_school_macroeconomics": { + "alias": " - high_school_macroeconomics", + "acc,none": 0.7948717948717948, + "acc_stderr,none": 0.02047323317355194 + }, + "mmlu_high_school_microeconomics": { + "alias": " - high_school_microeconomics", + "acc,none": 0.9159663865546218, + "acc_stderr,none": 0.01802154889879824 + }, + "mmlu_high_school_psychology": { + "alias": " - high_school_psychology", + "acc,none": 0.9045871559633027, + "acc_stderr,none": 0.012595899282335761 + }, + "mmlu_human_sexuality": { + "alias": " - human_sexuality", + "acc,none": 0.8854961832061069, + "acc_stderr,none": 0.027927473753597453 + }, + "mmlu_professional_psychology": { + "alias": " - professional_psychology", + "acc,none": 0.7630718954248366, + "acc_stderr,none": 0.017201662169789703 + }, + "mmlu_public_relations": { + "alias": " - public_relations", + "acc,none": 0.6818181818181818, + "acc_stderr,none": 0.044612721759105065 + }, + "mmlu_security_studies": { + "alias": " - security_studies", + "acc,none": 0.8, + "acc_stderr,none": 0.02560737598657919 + }, + "mmlu_sociology": { + "alias": " - sociology", + "acc,none": 0.845771144278607, + "acc_stderr,none": 0.025538433368578303 + }, + "mmlu_us_foreign_policy": { + "alias": " - us_foreign_policy", + "acc,none": 0.83, + "acc_stderr,none": 0.03775251680686369 + }, + "mmlu_stem": { + "acc,none": 0.7408816999682841, + "acc_stderr,none": 0.007511790624977365, + "alias": " - stem" + }, + "mmlu_abstract_algebra": { + "alias": " - abstract_algebra", + "acc,none": 0.56, + "acc_stderr,none": 0.049888765156985884 + }, + "mmlu_anatomy": { + "alias": " - anatomy", + "acc,none": 0.6888888888888889, + "acc_stderr,none": 0.039992628766177166 + }, + "mmlu_astronomy": { + "alias": " - astronomy", + "acc,none": 0.8881578947368421, + "acc_stderr,none": 0.025648341251693615 + }, + "mmlu_college_biology": { + "alias": " - college_biology", + "acc,none": 0.8680555555555556, + "acc_stderr,none": 0.02830096838204442 + }, + "mmlu_college_chemistry": { + "alias": " - college_chemistry", + "acc,none": 0.59, + "acc_stderr,none": 0.04943110704237104 + }, + "mmlu_college_computer_science": { + "alias": " - college_computer_science", + "acc,none": 0.72, + "acc_stderr,none": 0.045126085985421296 + }, + "mmlu_college_mathematics": { + "alias": " - college_mathematics", + "acc,none": 0.59, + "acc_stderr,none": 0.04943110704237104 + }, + "mmlu_college_physics": { + "alias": " - college_physics", + "acc,none": 0.5980392156862745, + "acc_stderr,none": 0.04878608714466993 + }, + "mmlu_computer_security": { + "alias": " - computer_security", + "acc,none": 0.81, + "acc_stderr,none": 0.039427724440366255 + }, + "mmlu_conceptual_physics": { + "alias": " - conceptual_physics", + "acc,none": 0.851063829787234, + "acc_stderr,none": 0.02327411784801041 + }, + "mmlu_electrical_engineering": { + "alias": " - electrical_engineering", + "acc,none": 0.8068965517241379, + "acc_stderr,none": 0.03289445522127399 + }, + "mmlu_elementary_mathematics": { + "alias": " - elementary_mathematics", + "acc,none": 0.753968253968254, + "acc_stderr,none": 0.022182037202948424 + }, + "mmlu_high_school_biology": { + "alias": " - high_school_biology", + "acc,none": 0.9129032258064517, + "acc_stderr,none": 0.016041100741696738 + }, + "mmlu_high_school_chemistry": { + "alias": " - high_school_chemistry", + "acc,none": 0.729064039408867, + "acc_stderr,none": 0.03127090713297701 + }, + "mmlu_high_school_computer_science": { + "alias": " - high_school_computer_science", + "acc,none": 0.86, + "acc_stderr,none": 0.0348735088019777 + }, + "mmlu_high_school_mathematics": { + "alias": " - high_school_mathematics", + "acc,none": 0.5, + "acc_stderr,none": 0.030485538042484616 + }, + "mmlu_high_school_physics": { + "alias": " - high_school_physics", + "acc,none": 0.695364238410596, + "acc_stderr,none": 0.03757949922943339 + }, + "mmlu_high_school_statistics": { + "alias": " - high_school_statistics", + "acc,none": 0.7592592592592593, + "acc_stderr,none": 0.029157522184605607 + }, + "mmlu_machine_learning": { + "alias": " - machine_learning", + "acc,none": 0.6428571428571429, + "acc_stderr,none": 0.045479609997643805 + } + }, + "groups": { + "mmlu": { + "acc,none": 0.7358638370602478, + "acc_stderr,none": 0.003529928328054296, + "alias": "mmlu" + }, + "mmlu_humanities": { + "acc,none": 0.6463336875664187, + "acc_stderr,none": 0.006627653147654618, + "alias": " - humanities" + }, + "mmlu_other": { + "acc,none": 0.7740585774058577, + "acc_stderr,none": 0.00721392999736686, + "alias": " - other" + }, + "mmlu_social_sciences": { + "acc,none": 0.8290542736431589, + "acc_stderr,none": 0.00666849172124334, + "alias": " - social sciences" + }, + "mmlu_stem": { + "acc,none": 0.7408816999682841, + "acc_stderr,none": 0.007511790624977365, + "alias": " - stem" + } + }, + "group_subtasks": { + "mmlu_humanities": [ + "mmlu_moral_scenarios", + "mmlu_high_school_us_history", + "mmlu_formal_logic", + "mmlu_international_law", + "mmlu_jurisprudence", + "mmlu_logical_fallacies", + "mmlu_philosophy", + "mmlu_prehistory", + "mmlu_high_school_european_history", + "mmlu_professional_law", + "mmlu_world_religions", + "mmlu_moral_disputes", + "mmlu_high_school_world_history" + ], + "mmlu_social_sciences": [ + "mmlu_public_relations", + "mmlu_human_sexuality", + "mmlu_us_foreign_policy", + "mmlu_high_school_microeconomics", + "mmlu_high_school_macroeconomics", + "mmlu_high_school_psychology", + "mmlu_econometrics", + "mmlu_sociology", + "mmlu_professional_psychology", + "mmlu_high_school_government_and_politics", + "mmlu_security_studies", + "mmlu_high_school_geography" + ], + "mmlu_other": [ + "mmlu_business_ethics", + "mmlu_marketing", + "mmlu_human_aging", + "mmlu_college_medicine", + "mmlu_medical_genetics", + "mmlu_professional_accounting", + "mmlu_professional_medicine", + "mmlu_global_facts", + "mmlu_miscellaneous", + "mmlu_nutrition", + "mmlu_virology", + "mmlu_clinical_knowledge", + "mmlu_management" + ], + "mmlu_stem": [ + "mmlu_high_school_computer_science", + "mmlu_college_biology", + "mmlu_high_school_physics", + "mmlu_college_physics", + "mmlu_elementary_mathematics", + "mmlu_college_mathematics", + "mmlu_machine_learning", + "mmlu_high_school_statistics", + "mmlu_college_chemistry", + "mmlu_anatomy", + "mmlu_conceptual_physics", + "mmlu_high_school_mathematics", + "mmlu_abstract_algebra", + "mmlu_computer_security", + "mmlu_college_computer_science", + "mmlu_astronomy", + "mmlu_high_school_biology", + "mmlu_high_school_chemistry", + "mmlu_electrical_engineering" + ], + "mmlu": [ + "mmlu_stem", + "mmlu_other", + "mmlu_social_sciences", + "mmlu_humanities" + ] + }, + "configs": { + "mmlu_abstract_algebra": { + "task": "mmlu_abstract_algebra", + "task_alias": "abstract_algebra", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "abstract_algebra", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about abstract algebra.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_anatomy": { + "task": "mmlu_anatomy", + "task_alias": "anatomy", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "anatomy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about anatomy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_astronomy": { + "task": "mmlu_astronomy", + "task_alias": "astronomy", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "astronomy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about astronomy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_business_ethics": { + "task": "mmlu_business_ethics", + "task_alias": "business_ethics", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "business_ethics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about business ethics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_clinical_knowledge": { + "task": "mmlu_clinical_knowledge", + "task_alias": "clinical_knowledge", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "clinical_knowledge", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about clinical knowledge.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_biology": { + "task": "mmlu_college_biology", + "task_alias": "college_biology", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_biology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college biology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_chemistry": { + "task": "mmlu_college_chemistry", + "task_alias": "college_chemistry", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_chemistry", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college chemistry.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_computer_science": { + "task": "mmlu_college_computer_science", + "task_alias": "college_computer_science", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_computer_science", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college computer science.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_mathematics": { + "task": "mmlu_college_mathematics", + "task_alias": "college_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_medicine": { + "task": "mmlu_college_medicine", + "task_alias": "college_medicine", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_medicine", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college medicine.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_physics": { + "task": "mmlu_college_physics", + "task_alias": "college_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_computer_security": { + "task": "mmlu_computer_security", + "task_alias": "computer_security", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "computer_security", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about computer security.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_conceptual_physics": { + "task": "mmlu_conceptual_physics", + "task_alias": "conceptual_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "conceptual_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about conceptual physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_econometrics": { + "task": "mmlu_econometrics", + "task_alias": "econometrics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "econometrics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about econometrics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_electrical_engineering": { + "task": "mmlu_electrical_engineering", + "task_alias": "electrical_engineering", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "electrical_engineering", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about electrical engineering.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_elementary_mathematics": { + "task": "mmlu_elementary_mathematics", + "task_alias": "elementary_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "elementary_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about elementary mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_formal_logic": { + "task": "mmlu_formal_logic", + "task_alias": "formal_logic", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "formal_logic", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about formal logic.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_global_facts": { + "task": "mmlu_global_facts", + "task_alias": "global_facts", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "global_facts", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about global facts.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_biology": { + "task": "mmlu_high_school_biology", + "task_alias": "high_school_biology", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_biology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school biology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_chemistry": { + "task": "mmlu_high_school_chemistry", + "task_alias": "high_school_chemistry", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_chemistry", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school chemistry.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_computer_science": { + "task": "mmlu_high_school_computer_science", + "task_alias": "high_school_computer_science", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_computer_science", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school computer science.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_european_history": { + "task": "mmlu_high_school_european_history", + "task_alias": "high_school_european_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_european_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school european history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_geography": { + "task": "mmlu_high_school_geography", + "task_alias": "high_school_geography", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_geography", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school geography.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_government_and_politics": { + "task": "mmlu_high_school_government_and_politics", + "task_alias": "high_school_government_and_politics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_government_and_politics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school government and politics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_macroeconomics": { + "task": "mmlu_high_school_macroeconomics", + "task_alias": "high_school_macroeconomics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_macroeconomics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school macroeconomics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_mathematics": { + "task": "mmlu_high_school_mathematics", + "task_alias": "high_school_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_microeconomics": { + "task": "mmlu_high_school_microeconomics", + "task_alias": "high_school_microeconomics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_microeconomics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school microeconomics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_physics": { + "task": "mmlu_high_school_physics", + "task_alias": "high_school_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_psychology": { + "task": "mmlu_high_school_psychology", + "task_alias": "high_school_psychology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_psychology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school psychology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_statistics": { + "task": "mmlu_high_school_statistics", + "task_alias": "high_school_statistics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_statistics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school statistics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_us_history": { + "task": "mmlu_high_school_us_history", + "task_alias": "high_school_us_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_us_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school us history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_world_history": { + "task": "mmlu_high_school_world_history", + "task_alias": "high_school_world_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_world_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school world history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_human_aging": { + "task": "mmlu_human_aging", + "task_alias": "human_aging", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "human_aging", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about human aging.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_human_sexuality": { + "task": "mmlu_human_sexuality", + "task_alias": "human_sexuality", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "human_sexuality", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about human sexuality.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_international_law": { + "task": "mmlu_international_law", + "task_alias": "international_law", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "international_law", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about international law.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_jurisprudence": { + "task": "mmlu_jurisprudence", + "task_alias": "jurisprudence", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "jurisprudence", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about jurisprudence.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_logical_fallacies": { + "task": "mmlu_logical_fallacies", + "task_alias": "logical_fallacies", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "logical_fallacies", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about logical fallacies.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_machine_learning": { + "task": "mmlu_machine_learning", + "task_alias": "machine_learning", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "machine_learning", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about machine learning.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_management": { + "task": "mmlu_management", + "task_alias": "management", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "management", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about management.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_marketing": { + "task": "mmlu_marketing", + "task_alias": "marketing", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "marketing", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about marketing.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_medical_genetics": { + "task": "mmlu_medical_genetics", + "task_alias": "medical_genetics", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "medical_genetics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about medical genetics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_miscellaneous": { + "task": "mmlu_miscellaneous", + "task_alias": "miscellaneous", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "miscellaneous", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about miscellaneous.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_moral_disputes": { + "task": "mmlu_moral_disputes", + "task_alias": "moral_disputes", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "moral_disputes", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about moral disputes.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_moral_scenarios": { + "task": "mmlu_moral_scenarios", + "task_alias": "moral_scenarios", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "moral_scenarios", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about moral scenarios.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_nutrition": { + "task": "mmlu_nutrition", + "task_alias": "nutrition", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "nutrition", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about nutrition.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_philosophy": { + "task": "mmlu_philosophy", + "task_alias": "philosophy", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "philosophy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about philosophy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_prehistory": { + "task": "mmlu_prehistory", + "task_alias": "prehistory", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "prehistory", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about prehistory.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_accounting": { + "task": "mmlu_professional_accounting", + "task_alias": "professional_accounting", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_accounting", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional accounting.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_law": { + "task": "mmlu_professional_law", + "task_alias": "professional_law", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_law", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional law.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_medicine": { + "task": "mmlu_professional_medicine", + "task_alias": "professional_medicine", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_medicine", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional medicine.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_psychology": { + "task": "mmlu_professional_psychology", + "task_alias": "professional_psychology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_psychology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional psychology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_public_relations": { + "task": "mmlu_public_relations", + "task_alias": "public_relations", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "public_relations", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about public relations.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_security_studies": { + "task": "mmlu_security_studies", + "task_alias": "security_studies", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "security_studies", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about security studies.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_sociology": { + "task": "mmlu_sociology", + "task_alias": "sociology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "sociology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about sociology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_us_foreign_policy": { + "task": "mmlu_us_foreign_policy", + "task_alias": "us_foreign_policy", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "us_foreign_policy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about us foreign policy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_virology": { + "task": "mmlu_virology", + "task_alias": "virology", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "virology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about virology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_world_religions": { + "task": "mmlu_world_religions", + "task_alias": "world_religions", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "world_religions", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about world religions.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18304/v1/completions", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + } + }, + "versions": { + "mmlu": 2, + "mmlu_abstract_algebra": 1.0, + "mmlu_anatomy": 1.0, + "mmlu_astronomy": 1.0, + "mmlu_business_ethics": 1.0, + "mmlu_clinical_knowledge": 1.0, + "mmlu_college_biology": 1.0, + "mmlu_college_chemistry": 1.0, + "mmlu_college_computer_science": 1.0, + "mmlu_college_mathematics": 1.0, + "mmlu_college_medicine": 1.0, + "mmlu_college_physics": 1.0, + "mmlu_computer_security": 1.0, + "mmlu_conceptual_physics": 1.0, + "mmlu_econometrics": 1.0, + "mmlu_electrical_engineering": 1.0, + "mmlu_elementary_mathematics": 1.0, + "mmlu_formal_logic": 1.0, + "mmlu_global_facts": 1.0, + "mmlu_high_school_biology": 1.0, + "mmlu_high_school_chemistry": 1.0, + "mmlu_high_school_computer_science": 1.0, + "mmlu_high_school_european_history": 1.0, + "mmlu_high_school_geography": 1.0, + "mmlu_high_school_government_and_politics": 1.0, + "mmlu_high_school_macroeconomics": 1.0, + "mmlu_high_school_mathematics": 1.0, + "mmlu_high_school_microeconomics": 1.0, + "mmlu_high_school_physics": 1.0, + "mmlu_high_school_psychology": 1.0, + "mmlu_high_school_statistics": 1.0, + "mmlu_high_school_us_history": 1.0, + "mmlu_high_school_world_history": 1.0, + "mmlu_human_aging": 1.0, + "mmlu_human_sexuality": 1.0, + "mmlu_humanities": 2, + "mmlu_international_law": 1.0, + "mmlu_jurisprudence": 1.0, + "mmlu_logical_fallacies": 1.0, + "mmlu_machine_learning": 1.0, + "mmlu_management": 1.0, + "mmlu_marketing": 1.0, + "mmlu_medical_genetics": 1.0, + "mmlu_miscellaneous": 1.0, + "mmlu_moral_disputes": 1.0, + "mmlu_moral_scenarios": 1.0, + "mmlu_nutrition": 1.0, + "mmlu_other": 2, + "mmlu_philosophy": 1.0, + "mmlu_prehistory": 1.0, + "mmlu_professional_accounting": 1.0, + "mmlu_professional_law": 1.0, + "mmlu_professional_medicine": 1.0, + "mmlu_professional_psychology": 1.0, + "mmlu_public_relations": 1.0, + "mmlu_security_studies": 1.0, + "mmlu_social_sciences": 2, + "mmlu_sociology": 1.0, + "mmlu_stem": 2, + "mmlu_us_foreign_policy": 1.0, + "mmlu_virology": 1.0, + "mmlu_world_religions": 1.0 + }, + "n-shot": { + "mmlu_abstract_algebra": 5, + "mmlu_anatomy": 5, + "mmlu_astronomy": 5, + "mmlu_business_ethics": 5, + "mmlu_clinical_knowledge": 5, + "mmlu_college_biology": 5, + "mmlu_college_chemistry": 5, + "mmlu_college_computer_science": 5, + "mmlu_college_mathematics": 5, + "mmlu_college_medicine": 5, + "mmlu_college_physics": 5, + "mmlu_computer_security": 5, + "mmlu_conceptual_physics": 5, + "mmlu_econometrics": 5, + "mmlu_electrical_engineering": 5, + "mmlu_elementary_mathematics": 5, + "mmlu_formal_logic": 5, + "mmlu_global_facts": 5, + "mmlu_high_school_biology": 5, + "mmlu_high_school_chemistry": 5, + "mmlu_high_school_computer_science": 5, + "mmlu_high_school_european_history": 5, + "mmlu_high_school_geography": 5, + "mmlu_high_school_government_and_politics": 5, + "mmlu_high_school_macroeconomics": 5, + "mmlu_high_school_mathematics": 5, + "mmlu_high_school_microeconomics": 5, + "mmlu_high_school_physics": 5, + "mmlu_high_school_psychology": 5, + "mmlu_high_school_statistics": 5, + "mmlu_high_school_us_history": 5, + "mmlu_high_school_world_history": 5, + "mmlu_human_aging": 5, + "mmlu_human_sexuality": 5, + "mmlu_international_law": 5, + "mmlu_jurisprudence": 5, + "mmlu_logical_fallacies": 5, + "mmlu_machine_learning": 5, + "mmlu_management": 5, + "mmlu_marketing": 5, + "mmlu_medical_genetics": 5, + "mmlu_miscellaneous": 5, + "mmlu_moral_disputes": 5, + "mmlu_moral_scenarios": 5, + "mmlu_nutrition": 5, + "mmlu_philosophy": 5, + "mmlu_prehistory": 5, + "mmlu_professional_accounting": 5, + "mmlu_professional_law": 5, + "mmlu_professional_medicine": 5, + "mmlu_professional_psychology": 5, + "mmlu_public_relations": 5, + "mmlu_security_studies": 5, + "mmlu_sociology": 5, + "mmlu_us_foreign_policy": 5, + "mmlu_virology": 5, + "mmlu_world_religions": 5 + }, + "higher_is_better": { + "mmlu": { + "acc": true + }, + "mmlu_abstract_algebra": { + "acc": true + }, + "mmlu_anatomy": { + "acc": true + }, + "mmlu_astronomy": { + "acc": true + }, + "mmlu_business_ethics": { + "acc": true + }, + "mmlu_clinical_knowledge": { + "acc": true + }, + "mmlu_college_biology": { + "acc": true + }, + "mmlu_college_chemistry": { + "acc": true + }, + "mmlu_college_computer_science": { + "acc": true + }, + "mmlu_college_mathematics": { + "acc": true + }, + "mmlu_college_medicine": { + "acc": true + }, + "mmlu_college_physics": { + "acc": true + }, + "mmlu_computer_security": { + "acc": true + }, + "mmlu_conceptual_physics": { + "acc": true + }, + "mmlu_econometrics": { + "acc": true + }, + "mmlu_electrical_engineering": { + "acc": true + }, + "mmlu_elementary_mathematics": { + "acc": true + }, + "mmlu_formal_logic": { + "acc": true + }, + "mmlu_global_facts": { + "acc": true + }, + "mmlu_high_school_biology": { + "acc": true + }, + "mmlu_high_school_chemistry": { + "acc": true + }, + "mmlu_high_school_computer_science": { + "acc": true + }, + "mmlu_high_school_european_history": { + "acc": true + }, + "mmlu_high_school_geography": { + "acc": true + }, + "mmlu_high_school_government_and_politics": { + "acc": true + }, + "mmlu_high_school_macroeconomics": { + "acc": true + }, + "mmlu_high_school_mathematics": { + "acc": true + }, + "mmlu_high_school_microeconomics": { + "acc": true + }, + "mmlu_high_school_physics": { + "acc": true + }, + "mmlu_high_school_psychology": { + "acc": true + }, + "mmlu_high_school_statistics": { + "acc": true + }, + "mmlu_high_school_us_history": { + "acc": true + }, + "mmlu_high_school_world_history": { + "acc": true + }, + "mmlu_human_aging": { + "acc": true + }, + "mmlu_human_sexuality": { + "acc": true + }, + "mmlu_humanities": { + "acc": true + }, + "mmlu_international_law": { + "acc": true + }, + "mmlu_jurisprudence": { + "acc": true + }, + "mmlu_logical_fallacies": { + "acc": true + }, + "mmlu_machine_learning": { + "acc": true + }, + "mmlu_management": { + "acc": true + }, + "mmlu_marketing": { + "acc": true + }, + "mmlu_medical_genetics": { + "acc": true + }, + "mmlu_miscellaneous": { + "acc": true + }, + "mmlu_moral_disputes": { + "acc": true + }, + "mmlu_moral_scenarios": { + "acc": true + }, + "mmlu_nutrition": { + "acc": true + }, + "mmlu_other": { + "acc": true + }, + "mmlu_philosophy": { + "acc": true + }, + "mmlu_prehistory": { + "acc": true + }, + "mmlu_professional_accounting": { + "acc": true + }, + "mmlu_professional_law": { + "acc": true + }, + "mmlu_professional_medicine": { + "acc": true + }, + "mmlu_professional_psychology": { + "acc": true + }, + "mmlu_public_relations": { + "acc": true + }, + "mmlu_security_studies": { + "acc": true + }, + "mmlu_social_sciences": { + "acc": true + }, + "mmlu_sociology": { + "acc": true + }, + "mmlu_stem": { + "acc": true + }, + "mmlu_us_foreign_policy": { + "acc": true + }, + "mmlu_virology": { + "acc": true + }, + "mmlu_world_religions": { + "acc": true + } + }, + "n-samples": { + "mmlu_high_school_computer_science": { + "original": 100, + "effective": 100 + }, + "mmlu_college_biology": { + "original": 144, + "effective": 144 + }, + "mmlu_high_school_physics": { + "original": 151, + "effective": 151 + }, + "mmlu_college_physics": { + "original": 102, + "effective": 102 + }, + "mmlu_elementary_mathematics": { + "original": 378, + "effective": 378 + }, + "mmlu_college_mathematics": { + "original": 100, + "effective": 100 + }, + "mmlu_machine_learning": { + "original": 112, + "effective": 112 + }, + "mmlu_high_school_statistics": { + "original": 216, + "effective": 216 + }, + "mmlu_college_chemistry": { + "original": 100, + "effective": 100 + }, + "mmlu_anatomy": { + "original": 135, + "effective": 135 + }, + "mmlu_conceptual_physics": { + "original": 235, + "effective": 235 + }, + "mmlu_high_school_mathematics": { + "original": 270, + "effective": 270 + }, + "mmlu_abstract_algebra": { + "original": 100, + "effective": 100 + }, + "mmlu_computer_security": { + "original": 100, + "effective": 100 + }, + "mmlu_college_computer_science": { + "original": 100, + "effective": 100 + }, + "mmlu_astronomy": { + "original": 152, + "effective": 152 + }, + "mmlu_high_school_biology": { + "original": 310, + "effective": 310 + }, + "mmlu_high_school_chemistry": { + "original": 203, + "effective": 203 + }, + "mmlu_electrical_engineering": { + "original": 145, + "effective": 145 + }, + "mmlu_business_ethics": { + "original": 100, + "effective": 100 + }, + "mmlu_marketing": { + "original": 234, + "effective": 234 + }, + "mmlu_human_aging": { + "original": 223, + "effective": 223 + }, + "mmlu_college_medicine": { + "original": 173, + "effective": 173 + }, + "mmlu_medical_genetics": { + "original": 100, + "effective": 100 + }, + "mmlu_professional_accounting": { + "original": 282, + "effective": 282 + }, + "mmlu_professional_medicine": { + "original": 272, + "effective": 272 + }, + "mmlu_global_facts": { + "original": 100, + "effective": 100 + }, + "mmlu_miscellaneous": { + "original": 783, + "effective": 783 + }, + "mmlu_nutrition": { + "original": 306, + "effective": 306 + }, + "mmlu_virology": { + "original": 166, + "effective": 166 + }, + "mmlu_clinical_knowledge": { + "original": 265, + "effective": 265 + }, + "mmlu_management": { + "original": 103, + "effective": 103 + }, + "mmlu_public_relations": { + "original": 110, + "effective": 110 + }, + "mmlu_human_sexuality": { + "original": 131, + "effective": 131 + }, + "mmlu_us_foreign_policy": { + "original": 100, + "effective": 100 + }, + "mmlu_high_school_microeconomics": { + "original": 238, + "effective": 238 + }, + "mmlu_high_school_macroeconomics": { + "original": 390, + "effective": 390 + }, + "mmlu_high_school_psychology": { + "original": 545, + "effective": 545 + }, + "mmlu_econometrics": { + "original": 114, + "effective": 114 + }, + "mmlu_sociology": { + "original": 201, + "effective": 201 + }, + "mmlu_professional_psychology": { + "original": 612, + "effective": 612 + }, + "mmlu_high_school_government_and_politics": { + "original": 193, + "effective": 193 + }, + "mmlu_security_studies": { + "original": 245, + "effective": 245 + }, + "mmlu_high_school_geography": { + "original": 198, + "effective": 198 + }, + "mmlu_moral_scenarios": { + "original": 895, + "effective": 895 + }, + "mmlu_high_school_us_history": { + "original": 204, + "effective": 204 + }, + "mmlu_formal_logic": { + "original": 126, + "effective": 126 + }, + "mmlu_international_law": { + "original": 121, + "effective": 121 + }, + "mmlu_jurisprudence": { + "original": 108, + "effective": 108 + }, + "mmlu_logical_fallacies": { + "original": 163, + "effective": 163 + }, + "mmlu_philosophy": { + "original": 311, + "effective": 311 + }, + "mmlu_prehistory": { + "original": 324, + "effective": 324 + }, + "mmlu_high_school_european_history": { + "original": 165, + "effective": 165 + }, + "mmlu_professional_law": { + "original": 1534, + "effective": 1534 + }, + "mmlu_world_religions": { + "original": 171, + "effective": 171 + }, + "mmlu_moral_disputes": { + "original": 346, + "effective": 346 + }, + "mmlu_high_school_world_history": { + "original": 237, + "effective": 237 + } + }, + "config": { + "model": "local-completions", + "model_args": "base_url=http://127.0.0.1:18304/v1/completions,model=qwen3-8b-ptq,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768", + "batch_size": 1, + "batch_sizes": [], + "device": null, + "use_cache": "/results/lm_cache", + "limit": null, + "downsampling_ratio": null, + "bootstrap_iters": 100000, + "gen_kwargs": { + "temperature": 1e-07, + "top_p": 0.9999999 + }, + "random_seed": 42, + "numpy_seed": 42, + "torch_seed": 42, + "fewshot_seed": 42 + }, + "git_hash": null, + "date": 1791476867.980376, + "pretty_env_info": "name 'get_pretty_env_info' is not defined", + "transformers_version": "4.50.3", + "lm_eval_version": "No package metadata was found for lm_eval", + "upper_git_hash": null, + "tokenizer_pad_token": [ + "<|endoftext|>", + "151643" + ], + "tokenizer_eos_token": [ + "<|im_end|>", + "151645" + ], + "tokenizer_bos_token": [ + null, + "None" + ], + "eot_token_id": 151645, + "max_length": 32767, + "task_hashes": { + "mmlu_high_school_computer_science": "2db28b8942d29e448cb587b398788f3cba6693cfe20772102b5bb1907999abea", + "mmlu_college_biology": "14cbae605287fc3c9ff25c8f2b2175619e036287a2e217767382b5f48f044943", + "mmlu_high_school_physics": "7780579caacd56f607968e81d5256d7d386095ab1d93b672047c5a5d8444ec1d", + "mmlu_college_physics": "ffe4456084dbfeb89e20d4b6138b97bbf415fd619691dbc7b38bfe5934b30e10", + "mmlu_elementary_mathematics": "da1d3600f010ce76c00a62b991b7606ee7df0b2e82413ee1f1977440ddd8550d", + "mmlu_college_mathematics": "33a226a6b1a6492f38387286d516c0941f5d8debbdfaa054b11fcda1cdb6dc1a", + "mmlu_machine_learning": "d02c202c0ad4c729890962d842ea70cad80faaae66a51a4c35c3685783ca06c1", + "mmlu_high_school_statistics": "0598ccf63c944eeab0ba3ad2dcfbcaebddf4c5d124174071beb52e8a73f19e86", + "mmlu_college_chemistry": "20b91eb7e582696bf39216d1757ac477675fe6de640150d7ae29993234af04dc", + "mmlu_anatomy": "9578a460fa25b65077e6da48a36e49d1e6b094807d65c9e5bf7e07ad0fbb6296", + "mmlu_conceptual_physics": "d9a7c4fd2d286215624dffc2aaec8e0295a6353daf2300d3461537e3fb5425eb", + "mmlu_high_school_mathematics": "313c7a073f5b7339d283ad977ea74584374b6e0281898dfd89ff4bdfd2fb7b03", + "mmlu_abstract_algebra": "0e2f49943ac30da4eb2d7d2f5db73abb69da69d6988007197e6bdf138bfb3369", + "mmlu_computer_security": "e122c4bc113d812e7e083e316b358cc7e16a0d1c2e6506480caebcf9d1e27ecd", + "mmlu_college_computer_science": "c2355a53d39d8bf1fbec4ea4b0f2b50c3fe104f2b36556ecae1038b0c315816a", + "mmlu_astronomy": "83e572d3948bb6dbfb66031ab4dcb5b70054a5bc110de1a0fbc324940f4c04da", + "mmlu_high_school_biology": "cfa833fafa27f991b540ece0e0583e1ebedfcaa50d030891935b000b7757c1cb", + "mmlu_high_school_chemistry": "7b7e216abbacc50a1613fc9e02f3f3dda2709059394b121e43f3ad534cadcf80", + "mmlu_electrical_engineering": "58857d23657299553045d3db7b7b0de43f9a078a2fd53902d0e59251ac497e9c", + "mmlu_business_ethics": "a0e93295030eaac8a5dd303dd1514b9eb25ae1807f55d000b876c0e8ddd5e1c8", + "mmlu_marketing": "2018f5181b9a272ca8a4f65858e0d12a1bf87792934fc483022e9d1bc7e05997", + "mmlu_human_aging": "88802f2005a5c3c719ace0291d92034ef3c90ee1cf7488d5195f8c545c899d4d", + "mmlu_college_medicine": "bc04947722ee05b4d5051df09f0e50de44d4f49df44f5ff909dbe3264cbc2c53", + "mmlu_medical_genetics": "8819bc558f8142680673b743cf8e3681ea6856ece1e3895c24fc0f77a60da7c4", + "mmlu_professional_accounting": "654d1dafcb0ff52e603404c1158924e92c5067cda9df2429eb0777c99c648995", + "mmlu_professional_medicine": "ba4af62ac5d26864b9e6339c6cbd16cbabcb5fbcf00366c02dd15da73c28daf9", + "mmlu_global_facts": "856fc8218f2f89f3e9abbe96070c530689a228cf5c9269c58bd03b4f0aa833b1", + "mmlu_miscellaneous": "22548428bd482f372a4fba10d2a0c0a939faddd7e9781701a74358145211a660", + "mmlu_nutrition": "468a945d5f4f35a2b1c974d7180d159d3e80a0611fa9160afee572df0dd69432", + "mmlu_virology": "67042eb89ab0d2a45f1d254dbb9da9de152fa1098d7740a05ac7ff8495b26534", + "mmlu_clinical_knowledge": "a3dba4354de80aa98c7db318923aedb8b78c02014a2be97c4071453ba619b8dc", + "mmlu_management": "6345661bb012d1a72c5d34f39e692cf47c131807a3bdfff7bf07ee9c5149d29e", + "mmlu_public_relations": "355105913812b28cc46607bf6710a92dcb346c9d35f489eb84e388de03390b32", + "mmlu_human_sexuality": "9b20e343f77622c458fe394d257924eb8895d07ddd98b1e3dbaf366563f77cbb", + "mmlu_us_foreign_policy": "1c1d39c546896fb1b9a513c5db8233ea4066c3ec79d8b85c2ab7944fa79ca8c5", + "mmlu_high_school_microeconomics": "705c7c5543f9987bdf14f7b39ddae1910e9097d0f491bb98a536140175d94732", + "mmlu_high_school_macroeconomics": "50ee63c5ac58c48f651464b5fd6cbbe0c8fa6002dd480fa9634ba630d9dc0417", + "mmlu_high_school_psychology": "e62d1ebd73c032fd0c3b99eaf20e7b77b13ae49e4fc506d7e69f9ae15e485a4c", + "mmlu_econometrics": "905a5e83b9a8173ef7345d14ac535c91c78c34b076314a2deb52cb2e6138502c", + "mmlu_sociology": "65917e6a3b0027bdb12703440e17afc8f155c15087a9df249e0061d84381e0d2", + "mmlu_professional_psychology": "70ec543a94dccdb5e82f3ac5f006de88a1352fb86b2a2c68d496529cdd1f5e0c", + "mmlu_high_school_government_and_politics": "bd59fed7fa1dcf3e39a65a7ea71cc81573c4fb28bcd942cbc8d8d9afbe0e9d4c", + "mmlu_security_studies": "e7c1c4bd8669cc22b7a907cd3584c5caf80b82772d9741a3680563b4fb564084", + "mmlu_high_school_geography": "30d201a5bfcbaec8d0e74f58ddd23bd58204b1d99b9211e6a531d8b6e84eab24", + "mmlu_moral_scenarios": "83a877f6af97d85f853c38196827d751dc8f33db4253da17965fde0c5e3554e1", + "mmlu_high_school_us_history": "b5fdc459ac365a794c663a17cf8a34a702ea87eaa1a8b40fe49e8196ef88232c", + "mmlu_formal_logic": "646ee3f6abf70a7d0efd03cdc0fbebb9f2e8f40e3b5563e53a9d0a7a291bc2a7", + "mmlu_international_law": "144d9e8693486d00f5cac911f3fbfbacdba9ac7c10340d50061262d9c3fdff9d", + "mmlu_jurisprudence": "325cb19fa404379425d7add71b29e2e4cd3428c08a25f5cc718fd87e79ccd2bb", + "mmlu_logical_fallacies": "f4aa9887846d578f13ca7420b65369c344053cabe092607a16237dfeef07a1ba", + "mmlu_philosophy": "c61151b7b50a37e90635e8b219018a35533a65a2a684542ead156b9f1020ffc1", + "mmlu_prehistory": "4aa1d25e8211294e34dead164bbde415d42feb245a4716a2471d929f81846f04", + "mmlu_high_school_european_history": "8c8ff9bcf9924cd457cc5b67c5f6a21925c09aadba42e646c088f2b99ba4e0e2", + "mmlu_professional_law": "0e0e2459e1f45df128cf8460ebdea4da6cc99a9fbd9cb99fb5ebeee78575800e", + "mmlu_world_religions": "8ad90daa76079af018b36cb86143468d22e34a9cd3273a3c48ce4c4648f886dc", + "mmlu_moral_disputes": "4d3e2d32e558de3c5eb0369e9a20280fda45fe04eeedf3356ad820e352293720", + "mmlu_high_school_world_history": "59ebbb27916baa0f2c6fa8b064ecbb042b43f6d492bbe18f2a78a63d40bf3dd0" + }, + "model_source": "local-completions", + "model_name": "qwen3-8b-ptq", + "model_name_sanitized": "qwen3-8b-ptq", + "system_instruction": null, + "system_instruction_sha": null, + "fewshot_as_multiturn": false, + "chat_template": null, + "chat_template_sha": null, + "start_time": 5792961.753521479, + "end_time": 5796646.816937493, + "total_evaluation_time_seconds": "3685.0634160144255" +} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml new file mode 100644 index 00000000000..63b378d1add --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml @@ -0,0 +1,58 @@ +defaults: +- execution: slurm/default +- deployment: vllm +- ../../../nel_configs/mmlu@_global_ +- _self_ +execution: + hostname: ${oc.env:NEL_SLURM_HOST} + username: ${oc.env:SLURM_USER} + account: ${oc.env:NEL_SLURM_ACCOUNT} + partition: ${oc.env:NEL_SLURM_PARTITION} + output_dir: ${oc.env:NEL_OUTPUT_DIR} + num_nodes: 1 + ntasks_per_node: 1 + gres: gpu:1 + walltime: 04:00:00 + sbatch_extra_flags: + exclusive: false + mem: 64G + mounts: + mount_home: false + deployment: + /path/to/hf-cache: /hf-cache + /path/to/checkpoint: /checkpoint + evaluation: + /path/to/hf-cache: /hf-cache + /path/to/checkpoint: /checkpoint +deployment: + image: vllm/vllm-openai:v0.28.0-cu129 + checkpoint_path: ${oc.env:QWEN3_MODEL_CHECKPOINT} + served_model_name: qwen3-8b-ptq + tensor_parallel_size: 1 + data_parallel_size: 1 + env_vars: + HF_HOME: lit:/hf-cache + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} + --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization + 0.85 --seed 42 --kv-cache-dtype auto + tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + port: 18104 +evaluation: + tasks: + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" + nemo_evaluator_config: + config: + params: + parallelism: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json new file mode 100644 index 00000000000..34de5729c40 --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json @@ -0,0 +1,136 @@ +{ + "results": { + "wikitext": { + "alias": "wikitext", + "word_perplexity,none": 13.88340951298615, + "word_perplexity_stderr,none": "N/A", + "byte_perplexity,none": 1.6355073044598794, + "byte_perplexity_stderr,none": "N/A", + "bits_per_byte,none": 0.7097382027618786, + "bits_per_byte_stderr,none": "N/A" + } + }, + "group_subtasks": { + "wikitext": [] + }, + "configs": { + "wikitext": { + "task": "wikitext", + "dataset_path": "EleutherAI/wikitext_document_level", + "dataset_name": "wikitext-2-raw-v1", + "dataset_kwargs": { + "trust_remote_code": true + }, + "training_split": "train", + "validation_split": "validation", + "test_split": "test", + "doc_to_text": "", + "doc_to_target": "def wikitext_detokenizer(doc):\n string = doc[\"page\"]\n # contractions\n string = string.replace(\"s '\", \"s'\")\n string = re.sub(r\"/' [0-9]/\", r\"/'[0-9]/\", string)\n # number separators\n string = string.replace(\" @-@ \", \"-\")\n string = string.replace(\" @,@ \", \",\")\n string = string.replace(\" @.@ \", \".\")\n # punctuation\n string = string.replace(\" : \", \": \")\n string = string.replace(\" ; \", \"; \")\n string = string.replace(\" . \", \". \")\n string = string.replace(\" ! \", \"! \")\n string = string.replace(\" ? \", \"? \")\n string = string.replace(\" , \", \", \")\n # double brackets\n string = re.sub(r\"\\(\\s*([^\\)]*?)\\s*\\)\", r\"(\\1)\", string)\n string = re.sub(r\"\\[\\s*([^\\]]*?)\\s*\\]\", r\"[\\1]\", string)\n string = re.sub(r\"{\\s*([^}]*?)\\s*}\", r\"{\\1}\", string)\n string = re.sub(r\"\\\"\\s*([^\\\"]*?)\\s*\\\"\", r'\"\\1\"', string)\n string = re.sub(r\"'\\s*([^']*?)\\s*'\", r\"'\\1'\", string)\n # miscellaneous\n string = string.replace(\"= = = =\", \"====\")\n string = string.replace(\"= = =\", \"===\")\n string = string.replace(\"= =\", \"==\")\n string = string.replace(\" \" + chr(176) + \" \", chr(176))\n string = string.replace(\" \\n\", \"\\n\")\n string = string.replace(\"\\n \", \"\\n\")\n string = string.replace(\" N \", \" 1 \")\n string = string.replace(\" 's\", \"'s\")\n\n return string\n", + "unsafe_code": false, + "process_results": "def process_results(doc, results):\n (loglikelihood,) = results\n # IMPORTANT: wikitext counts number of words in *original doc before detokenization*\n _words = len(re.split(r\"\\s+\", doc[\"page\"]))\n _bytes = len(doc[\"page\"].encode(\"utf-8\"))\n return {\n \"word_perplexity\": (loglikelihood, _words),\n \"byte_perplexity\": (loglikelihood, _bytes),\n \"bits_per_byte\": (loglikelihood, _bytes),\n }\n", + "description": "", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "num_fewshot": 0, + "metric_list": [ + { + "metric": "word_perplexity" + }, + { + "metric": "byte_perplexity" + }, + { + "metric": "bits_per_byte" + } + ], + "output_type": "loglikelihood_rolling", + "repeats": 1, + "should_decontaminate": true, + "doc_to_decontamination_query": "{{page}}", + "metadata": { + "version": 2.0, + "base_url": "http://localhost:3825", + "model": "qwen3-8b-ptq", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 2, + "timeout": 300, + "max_retries": 2, + "stream": false + } + } + }, + "versions": { + "wikitext": 2.0 + }, + "n-shot": { + "wikitext": 0 + }, + "higher_is_better": { + "wikitext": { + "word_perplexity": false, + "byte_perplexity": false, + "bits_per_byte": false + } + }, + "n-samples": { + "wikitext": { + "original": 62, + "effective": 62 + } + }, + "config": { + "model": "local-completions", + "model_args": "base_url=http://localhost:3825,model=qwen3-8b-ptq,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=2,timeout=300,max_retries=2,stream=False", + "batch_size": 1, + "batch_sizes": [], + "device": null, + "use_cache": "/results/lm_cache", + "limit": null, + "downsampling_ratio": null, + "bootstrap_iters": 100000, + "gen_kwargs": { + "temperature": 1e-07, + "top_p": 0.9999999 + }, + "random_seed": 42, + "numpy_seed": 42, + "torch_seed": 42, + "fewshot_seed": 42 + }, + "git_hash": null, + "date": 1791469370.721785, + "pretty_env_info": "name 'get_pretty_env_info' is not defined", + "transformers_version": "4.50.3", + "lm_eval_version": "No package metadata was found for lm_eval", + "upper_git_hash": null, + "tokenizer_pad_token": [ + "<|endoftext|>", + "151643" + ], + "tokenizer_eos_token": [ + "<|im_end|>", + "151645" + ], + "tokenizer_bos_token": [ + null, + "None" + ], + "eot_token_id": 151645, + "max_length": 2047, + "task_hashes": { + "wikitext": "233109c41076287b5ba5a338ab7b085728acacf36d237a7ab99f34fcda161641" + }, + "model_source": "local-completions", + "model_name": "qwen3-8b-ptq", + "model_name_sanitized": "qwen3-8b-ptq", + "system_instruction": null, + "system_instruction_sha": null, + "fewshot_as_multiturn": false, + "chat_template": null, + "chat_template_sha": null, + "start_time": 309861.529678691, + "end_time": 309889.04976299, + "total_evaluation_time_seconds": "27.520084298972506" +} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml new file mode 100644 index 00000000000..1e6ee3aa8ce --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml @@ -0,0 +1,41 @@ +# Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. +defaults: +- execution: slurm/default +- deployment: vllm +- ../../../nel_configs/wikitext@_global_ +- _self_ +execution: + hostname: ${oc.env:NEL_SLURM_HOST} + username: ${oc.env:SLURM_USER} + account: ${oc.env:NEL_SLURM_ACCOUNT} + partition: ${oc.env:NEL_SLURM_PARTITION} + output_dir: ${oc.env:NEL_OUTPUT_DIR} + num_nodes: 1 + ntasks_per_node: 1 + gres: gpu:1 + walltime: 04:00:00 + sbatch_extra_flags: + exclusive: false + mem: 64G + mounts: + mount_home: false + deployment: + /path/to/hf-cache: /hf-cache + /path/to/checkpoint: /checkpoint + evaluation: + /path/to/hf-cache: /hf-cache + /path/to/checkpoint: /checkpoint +deployment: + image: vllm/vllm-openai:v0.28.0-cu129 + checkpoint_path: ${oc.env:QWEN3_MODEL_CHECKPOINT} + served_model_name: qwen3-8b-ptq + tensor_parallel_size: 1 + data_parallel_size: 1 + env_vars: + HF_HOME: lit:/hf-cache + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} + --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization + 0.85 --seed 42 --kv-cache-dtype auto + tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + port: 18106 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json new file mode 100644 index 00000000000..6a67505239b --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json @@ -0,0 +1,4030 @@ +{ + "results": { + "mmlu": { + "acc,none": 0.7375729952998148, + "acc_stderr,none": 0.0035282530148200222, + "alias": "mmlu" + }, + "mmlu_humanities": { + "acc,none": 0.6512221041445271, + "acc_stderr,none": 0.006616358365310683, + "alias": " - humanities" + }, + "mmlu_formal_logic": { + "alias": " - formal_logic", + "acc,none": 0.6031746031746031, + "acc_stderr,none": 0.04375888492727054 + }, + "mmlu_high_school_european_history": { + "alias": " - high_school_european_history", + "acc,none": 0.8727272727272727, + "acc_stderr,none": 0.026024657651656246 + }, + "mmlu_high_school_us_history": { + "alias": " - high_school_us_history", + "acc,none": 0.8382352941176471, + "acc_stderr,none": 0.025845017986926906 + }, + "mmlu_high_school_world_history": { + "alias": " - high_school_world_history", + "acc,none": 0.8565400843881856, + "acc_stderr,none": 0.02281829182101699 + }, + "mmlu_international_law": { + "alias": " - international_law", + "acc,none": 0.7355371900826446, + "acc_stderr,none": 0.04026187527591209 + }, + "mmlu_jurisprudence": { + "alias": " - jurisprudence", + "acc,none": 0.7962962962962963, + "acc_stderr,none": 0.038935425188248496 + }, + "mmlu_logical_fallacies": { + "alias": " - logical_fallacies", + "acc,none": 0.8220858895705522, + "acc_stderr,none": 0.03004735765580667 + }, + "mmlu_moral_disputes": { + "alias": " - moral_disputes", + "acc,none": 0.7398843930635838, + "acc_stderr,none": 0.02361867831006933 + }, + "mmlu_moral_scenarios": { + "alias": " - moral_scenarios", + "acc,none": 0.5385474860335195, + "acc_stderr,none": 0.01667273126755234 + }, + "mmlu_philosophy": { + "alias": " - philosophy", + "acc,none": 0.7684887459807074, + "acc_stderr,none": 0.023956532766639154 + }, + "mmlu_prehistory": { + "alias": " - prehistory", + "acc,none": 0.8302469135802469, + "acc_stderr,none": 0.020888690414093802 + }, + "mmlu_professional_law": { + "alias": " - professional_law", + "acc,none": 0.5026075619295959, + "acc_stderr,none": 0.012770062445433189 + }, + "mmlu_world_religions": { + "alias": " - world_religions", + "acc,none": 0.8421052631578947, + "acc_stderr,none": 0.027966785859160903 + }, + "mmlu_other": { + "acc,none": 0.7685870614740907, + "acc_stderr,none": 0.007283169591179256, + "alias": " - other" + }, + "mmlu_business_ethics": { + "alias": " - business_ethics", + "acc,none": 0.79, + "acc_stderr,none": 0.040936018074033236 + }, + "mmlu_clinical_knowledge": { + "alias": " - clinical_knowledge", + "acc,none": 0.7962264150943397, + "acc_stderr,none": 0.024790784501775447 + }, + "mmlu_college_medicine": { + "alias": " - college_medicine", + "acc,none": 0.7803468208092486, + "acc_stderr,none": 0.03156809362703173 + }, + "mmlu_global_facts": { + "alias": " - global_facts", + "acc,none": 0.43, + "acc_stderr,none": 0.049756985195624305 + }, + "mmlu_human_aging": { + "alias": " - human_aging", + "acc,none": 0.7399103139013453, + "acc_stderr,none": 0.029442495585857424 + }, + "mmlu_management": { + "alias": " - management", + "acc,none": 0.8737864077669902, + "acc_stderr,none": 0.03288180278808631 + }, + "mmlu_marketing": { + "alias": " - marketing", + "acc,none": 0.9188034188034188, + "acc_stderr,none": 0.017893784904018484 + }, + "mmlu_medical_genetics": { + "alias": " - medical_genetics", + "acc,none": 0.82, + "acc_stderr,none": 0.03861229196653691 + }, + "mmlu_miscellaneous": { + "alias": " - miscellaneous", + "acc,none": 0.8339719029374202, + "acc_stderr,none": 0.013306478243066229 + }, + "mmlu_nutrition": { + "alias": " - nutrition", + "acc,none": 0.8006535947712419, + "acc_stderr,none": 0.022875816993464054 + }, + "mmlu_professional_accounting": { + "alias": " - professional_accounting", + "acc,none": 0.5709219858156028, + "acc_stderr,none": 0.029525914302558586 + }, + "mmlu_professional_medicine": { + "alias": " - professional_medicine", + "acc,none": 0.8088235294117647, + "acc_stderr,none": 0.023886881922440366 + }, + "mmlu_virology": { + "alias": " - virology", + "acc,none": 0.536144578313253, + "acc_stderr,none": 0.038823108508905954 + }, + "mmlu_social_sciences": { + "acc,none": 0.8323041923951902, + "acc_stderr,none": 0.006626384400414149, + "alias": " - social sciences" + }, + "mmlu_econometrics": { + "alias": " - econometrics", + "acc,none": 0.6842105263157895, + "acc_stderr,none": 0.04372748290278002 + }, + "mmlu_high_school_geography": { + "alias": " - high_school_geography", + "acc,none": 0.8686868686868687, + "acc_stderr,none": 0.02406315641682249 + }, + "mmlu_high_school_government_and_politics": { + "alias": " - high_school_government_and_politics", + "acc,none": 0.9326424870466321, + "acc_stderr,none": 0.01808839383907894 + }, + "mmlu_high_school_macroeconomics": { + "alias": " - high_school_macroeconomics", + "acc,none": 0.8, + "acc_stderr,none": 0.020280805062535757 + }, + "mmlu_high_school_microeconomics": { + "alias": " - high_school_microeconomics", + "acc,none": 0.9201680672268907, + "acc_stderr,none": 0.017605474225715036 + }, + "mmlu_high_school_psychology": { + "alias": " - high_school_psychology", + "acc,none": 0.908256880733945, + "acc_stderr,none": 0.012376323409137066 + }, + "mmlu_human_sexuality": { + "alias": " - human_sexuality", + "acc,none": 0.8473282442748091, + "acc_stderr,none": 0.031545216720054704 + }, + "mmlu_professional_psychology": { + "alias": " - professional_psychology", + "acc,none": 0.761437908496732, + "acc_stderr,none": 0.017242385828779655 + }, + "mmlu_public_relations": { + "alias": " - public_relations", + "acc,none": 0.7, + "acc_stderr,none": 0.04389311454644288 + }, + "mmlu_security_studies": { + "alias": " - security_studies", + "acc,none": 0.8040816326530612, + "acc_stderr,none": 0.025409301953225705 + }, + "mmlu_sociology": { + "alias": " - sociology", + "acc,none": 0.8407960199004975, + "acc_stderr,none": 0.025870646766169153 + }, + "mmlu_us_foreign_policy": { + "alias": " - us_foreign_policy", + "acc,none": 0.85, + "acc_stderr,none": 0.03588702812826367 + }, + "mmlu_stem": { + "acc,none": 0.743418966064066, + "acc_stderr,none": 0.007488923716325649, + "alias": " - stem" + }, + "mmlu_abstract_algebra": { + "alias": " - abstract_algebra", + "acc,none": 0.56, + "acc_stderr,none": 0.049888765156985884 + }, + "mmlu_anatomy": { + "alias": " - anatomy", + "acc,none": 0.6888888888888889, + "acc_stderr,none": 0.039992628766177166 + }, + "mmlu_astronomy": { + "alias": " - astronomy", + "acc,none": 0.9013157894736842, + "acc_stderr,none": 0.024270227737522694 + }, + "mmlu_college_biology": { + "alias": " - college_biology", + "acc,none": 0.875, + "acc_stderr,none": 0.02765610492929436 + }, + "mmlu_college_chemistry": { + "alias": " - college_chemistry", + "acc,none": 0.56, + "acc_stderr,none": 0.049888765156985884 + }, + "mmlu_college_computer_science": { + "alias": " - college_computer_science", + "acc,none": 0.68, + "acc_stderr,none": 0.046882617226215076 + }, + "mmlu_college_mathematics": { + "alias": " - college_mathematics", + "acc,none": 0.58, + "acc_stderr,none": 0.04960449637488582 + }, + "mmlu_college_physics": { + "alias": " - college_physics", + "acc,none": 0.6372549019607843, + "acc_stderr,none": 0.047840607041056527 + }, + "mmlu_computer_security": { + "alias": " - computer_security", + "acc,none": 0.8, + "acc_stderr,none": 0.04020151261036849 + }, + "mmlu_conceptual_physics": { + "alias": " - conceptual_physics", + "acc,none": 0.8468085106382979, + "acc_stderr,none": 0.023545179061675213 + }, + "mmlu_electrical_engineering": { + "alias": " - electrical_engineering", + "acc,none": 0.8275862068965517, + "acc_stderr,none": 0.03147830790259575 + }, + "mmlu_elementary_mathematics": { + "alias": " - elementary_mathematics", + "acc,none": 0.7619047619047619, + "acc_stderr,none": 0.0219358780811847 + }, + "mmlu_high_school_biology": { + "alias": " - high_school_biology", + "acc,none": 0.9161290322580645, + "acc_stderr,none": 0.015769027496775664 + }, + "mmlu_high_school_chemistry": { + "alias": " - high_school_chemistry", + "acc,none": 0.7192118226600985, + "acc_stderr,none": 0.031618563353586135 + }, + "mmlu_high_school_computer_science": { + "alias": " - high_school_computer_science", + "acc,none": 0.86, + "acc_stderr,none": 0.0348735088019777 + }, + "mmlu_high_school_mathematics": { + "alias": " - high_school_mathematics", + "acc,none": 0.5185185185185185, + "acc_stderr,none": 0.03046462171889531 + }, + "mmlu_high_school_physics": { + "alias": " - high_school_physics", + "acc,none": 0.7019867549668874, + "acc_stderr,none": 0.03734535676787201 + }, + "mmlu_high_school_statistics": { + "alias": " - high_school_statistics", + "acc,none": 0.7546296296296297, + "acc_stderr,none": 0.02934666509437301 + }, + "mmlu_machine_learning": { + "alias": " - machine_learning", + "acc,none": 0.6517857142857143, + "acc_stderr,none": 0.04521829902833581 + } + }, + "groups": { + "mmlu": { + "acc,none": 0.7375729952998148, + "acc_stderr,none": 0.0035282530148200222, + "alias": "mmlu" + }, + "mmlu_humanities": { + "acc,none": 0.6512221041445271, + "acc_stderr,none": 0.006616358365310683, + "alias": " - humanities" + }, + "mmlu_other": { + "acc,none": 0.7685870614740907, + "acc_stderr,none": 0.007283169591179256, + "alias": " - other" + }, + "mmlu_social_sciences": { + "acc,none": 0.8323041923951902, + "acc_stderr,none": 0.006626384400414149, + "alias": " - social sciences" + }, + "mmlu_stem": { + "acc,none": 0.743418966064066, + "acc_stderr,none": 0.007488923716325649, + "alias": " - stem" + } + }, + "group_subtasks": { + "mmlu_humanities": [ + "mmlu_professional_law", + "mmlu_world_religions", + "mmlu_logical_fallacies", + "mmlu_formal_logic", + "mmlu_moral_scenarios", + "mmlu_high_school_european_history", + "mmlu_high_school_us_history", + "mmlu_prehistory", + "mmlu_international_law", + "mmlu_philosophy", + "mmlu_jurisprudence", + "mmlu_high_school_world_history", + "mmlu_moral_disputes" + ], + "mmlu_social_sciences": [ + "mmlu_us_foreign_policy", + "mmlu_econometrics", + "mmlu_human_sexuality", + "mmlu_public_relations", + "mmlu_high_school_government_and_politics", + "mmlu_high_school_psychology", + "mmlu_high_school_macroeconomics", + "mmlu_sociology", + "mmlu_high_school_geography", + "mmlu_professional_psychology", + "mmlu_high_school_microeconomics", + "mmlu_security_studies" + ], + "mmlu_other": [ + "mmlu_professional_medicine", + "mmlu_professional_accounting", + "mmlu_global_facts", + "mmlu_nutrition", + "mmlu_marketing", + "mmlu_management", + "mmlu_human_aging", + "mmlu_medical_genetics", + "mmlu_virology", + "mmlu_clinical_knowledge", + "mmlu_miscellaneous", + "mmlu_college_medicine", + "mmlu_business_ethics" + ], + "mmlu_stem": [ + "mmlu_conceptual_physics", + "mmlu_college_mathematics", + "mmlu_high_school_statistics", + "mmlu_elementary_mathematics", + "mmlu_high_school_physics", + "mmlu_electrical_engineering", + "mmlu_college_chemistry", + "mmlu_high_school_mathematics", + "mmlu_abstract_algebra", + "mmlu_astronomy", + "mmlu_machine_learning", + "mmlu_college_computer_science", + "mmlu_high_school_chemistry", + "mmlu_college_physics", + "mmlu_college_biology", + "mmlu_anatomy", + "mmlu_high_school_biology", + "mmlu_computer_security", + "mmlu_high_school_computer_science" + ], + "mmlu": [ + "mmlu_stem", + "mmlu_other", + "mmlu_social_sciences", + "mmlu_humanities" + ] + }, + "configs": { + "mmlu_abstract_algebra": { + "task": "mmlu_abstract_algebra", + "task_alias": "abstract_algebra", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "abstract_algebra", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about abstract algebra.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_anatomy": { + "task": "mmlu_anatomy", + "task_alias": "anatomy", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "anatomy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about anatomy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_astronomy": { + "task": "mmlu_astronomy", + "task_alias": "astronomy", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "astronomy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about astronomy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_business_ethics": { + "task": "mmlu_business_ethics", + "task_alias": "business_ethics", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "business_ethics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about business ethics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_clinical_knowledge": { + "task": "mmlu_clinical_knowledge", + "task_alias": "clinical_knowledge", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "clinical_knowledge", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about clinical knowledge.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_biology": { + "task": "mmlu_college_biology", + "task_alias": "college_biology", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_biology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college biology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_chemistry": { + "task": "mmlu_college_chemistry", + "task_alias": "college_chemistry", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_chemistry", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college chemistry.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_computer_science": { + "task": "mmlu_college_computer_science", + "task_alias": "college_computer_science", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_computer_science", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college computer science.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_mathematics": { + "task": "mmlu_college_mathematics", + "task_alias": "college_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_medicine": { + "task": "mmlu_college_medicine", + "task_alias": "college_medicine", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_medicine", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college medicine.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_physics": { + "task": "mmlu_college_physics", + "task_alias": "college_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_computer_security": { + "task": "mmlu_computer_security", + "task_alias": "computer_security", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "computer_security", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about computer security.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_conceptual_physics": { + "task": "mmlu_conceptual_physics", + "task_alias": "conceptual_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "conceptual_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about conceptual physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_econometrics": { + "task": "mmlu_econometrics", + "task_alias": "econometrics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "econometrics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about econometrics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_electrical_engineering": { + "task": "mmlu_electrical_engineering", + "task_alias": "electrical_engineering", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "electrical_engineering", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about electrical engineering.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_elementary_mathematics": { + "task": "mmlu_elementary_mathematics", + "task_alias": "elementary_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "elementary_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about elementary mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_formal_logic": { + "task": "mmlu_formal_logic", + "task_alias": "formal_logic", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "formal_logic", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about formal logic.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_global_facts": { + "task": "mmlu_global_facts", + "task_alias": "global_facts", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "global_facts", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about global facts.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_biology": { + "task": "mmlu_high_school_biology", + "task_alias": "high_school_biology", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_biology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school biology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_chemistry": { + "task": "mmlu_high_school_chemistry", + "task_alias": "high_school_chemistry", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_chemistry", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school chemistry.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_computer_science": { + "task": "mmlu_high_school_computer_science", + "task_alias": "high_school_computer_science", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_computer_science", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school computer science.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_european_history": { + "task": "mmlu_high_school_european_history", + "task_alias": "high_school_european_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_european_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school european history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_geography": { + "task": "mmlu_high_school_geography", + "task_alias": "high_school_geography", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_geography", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school geography.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_government_and_politics": { + "task": "mmlu_high_school_government_and_politics", + "task_alias": "high_school_government_and_politics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_government_and_politics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school government and politics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_macroeconomics": { + "task": "mmlu_high_school_macroeconomics", + "task_alias": "high_school_macroeconomics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_macroeconomics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school macroeconomics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_mathematics": { + "task": "mmlu_high_school_mathematics", + "task_alias": "high_school_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_microeconomics": { + "task": "mmlu_high_school_microeconomics", + "task_alias": "high_school_microeconomics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_microeconomics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school microeconomics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_physics": { + "task": "mmlu_high_school_physics", + "task_alias": "high_school_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_psychology": { + "task": "mmlu_high_school_psychology", + "task_alias": "high_school_psychology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_psychology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school psychology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_statistics": { + "task": "mmlu_high_school_statistics", + "task_alias": "high_school_statistics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_statistics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school statistics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_us_history": { + "task": "mmlu_high_school_us_history", + "task_alias": "high_school_us_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_us_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school us history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_world_history": { + "task": "mmlu_high_school_world_history", + "task_alias": "high_school_world_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_world_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school world history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_human_aging": { + "task": "mmlu_human_aging", + "task_alias": "human_aging", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "human_aging", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about human aging.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_human_sexuality": { + "task": "mmlu_human_sexuality", + "task_alias": "human_sexuality", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "human_sexuality", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about human sexuality.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_international_law": { + "task": "mmlu_international_law", + "task_alias": "international_law", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "international_law", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about international law.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_jurisprudence": { + "task": "mmlu_jurisprudence", + "task_alias": "jurisprudence", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "jurisprudence", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about jurisprudence.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_logical_fallacies": { + "task": "mmlu_logical_fallacies", + "task_alias": "logical_fallacies", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "logical_fallacies", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about logical fallacies.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_machine_learning": { + "task": "mmlu_machine_learning", + "task_alias": "machine_learning", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "machine_learning", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about machine learning.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_management": { + "task": "mmlu_management", + "task_alias": "management", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "management", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about management.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_marketing": { + "task": "mmlu_marketing", + "task_alias": "marketing", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "marketing", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about marketing.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_medical_genetics": { + "task": "mmlu_medical_genetics", + "task_alias": "medical_genetics", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "medical_genetics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about medical genetics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_miscellaneous": { + "task": "mmlu_miscellaneous", + "task_alias": "miscellaneous", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "miscellaneous", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about miscellaneous.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_moral_disputes": { + "task": "mmlu_moral_disputes", + "task_alias": "moral_disputes", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "moral_disputes", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about moral disputes.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_moral_scenarios": { + "task": "mmlu_moral_scenarios", + "task_alias": "moral_scenarios", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "moral_scenarios", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about moral scenarios.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_nutrition": { + "task": "mmlu_nutrition", + "task_alias": "nutrition", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "nutrition", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about nutrition.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_philosophy": { + "task": "mmlu_philosophy", + "task_alias": "philosophy", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "philosophy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about philosophy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_prehistory": { + "task": "mmlu_prehistory", + "task_alias": "prehistory", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "prehistory", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about prehistory.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_accounting": { + "task": "mmlu_professional_accounting", + "task_alias": "professional_accounting", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_accounting", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional accounting.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_law": { + "task": "mmlu_professional_law", + "task_alias": "professional_law", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_law", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional law.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_medicine": { + "task": "mmlu_professional_medicine", + "task_alias": "professional_medicine", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_medicine", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional medicine.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_psychology": { + "task": "mmlu_professional_psychology", + "task_alias": "professional_psychology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_psychology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional psychology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_public_relations": { + "task": "mmlu_public_relations", + "task_alias": "public_relations", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "public_relations", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about public relations.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_security_studies": { + "task": "mmlu_security_studies", + "task_alias": "security_studies", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "security_studies", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about security studies.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_sociology": { + "task": "mmlu_sociology", + "task_alias": "sociology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "sociology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about sociology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_us_foreign_policy": { + "task": "mmlu_us_foreign_policy", + "task_alias": "us_foreign_policy", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "us_foreign_policy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about us foreign policy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_virology": { + "task": "mmlu_virology", + "task_alias": "virology", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "virology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about virology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_world_religions": { + "task": "mmlu_world_religions", + "task_alias": "world_religions", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "world_religions", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about world religions.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18308/v1/completions", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + } + }, + "versions": { + "mmlu": 2, + "mmlu_abstract_algebra": 1.0, + "mmlu_anatomy": 1.0, + "mmlu_astronomy": 1.0, + "mmlu_business_ethics": 1.0, + "mmlu_clinical_knowledge": 1.0, + "mmlu_college_biology": 1.0, + "mmlu_college_chemistry": 1.0, + "mmlu_college_computer_science": 1.0, + "mmlu_college_mathematics": 1.0, + "mmlu_college_medicine": 1.0, + "mmlu_college_physics": 1.0, + "mmlu_computer_security": 1.0, + "mmlu_conceptual_physics": 1.0, + "mmlu_econometrics": 1.0, + "mmlu_electrical_engineering": 1.0, + "mmlu_elementary_mathematics": 1.0, + "mmlu_formal_logic": 1.0, + "mmlu_global_facts": 1.0, + "mmlu_high_school_biology": 1.0, + "mmlu_high_school_chemistry": 1.0, + "mmlu_high_school_computer_science": 1.0, + "mmlu_high_school_european_history": 1.0, + "mmlu_high_school_geography": 1.0, + "mmlu_high_school_government_and_politics": 1.0, + "mmlu_high_school_macroeconomics": 1.0, + "mmlu_high_school_mathematics": 1.0, + "mmlu_high_school_microeconomics": 1.0, + "mmlu_high_school_physics": 1.0, + "mmlu_high_school_psychology": 1.0, + "mmlu_high_school_statistics": 1.0, + "mmlu_high_school_us_history": 1.0, + "mmlu_high_school_world_history": 1.0, + "mmlu_human_aging": 1.0, + "mmlu_human_sexuality": 1.0, + "mmlu_humanities": 2, + "mmlu_international_law": 1.0, + "mmlu_jurisprudence": 1.0, + "mmlu_logical_fallacies": 1.0, + "mmlu_machine_learning": 1.0, + "mmlu_management": 1.0, + "mmlu_marketing": 1.0, + "mmlu_medical_genetics": 1.0, + "mmlu_miscellaneous": 1.0, + "mmlu_moral_disputes": 1.0, + "mmlu_moral_scenarios": 1.0, + "mmlu_nutrition": 1.0, + "mmlu_other": 2, + "mmlu_philosophy": 1.0, + "mmlu_prehistory": 1.0, + "mmlu_professional_accounting": 1.0, + "mmlu_professional_law": 1.0, + "mmlu_professional_medicine": 1.0, + "mmlu_professional_psychology": 1.0, + "mmlu_public_relations": 1.0, + "mmlu_security_studies": 1.0, + "mmlu_social_sciences": 2, + "mmlu_sociology": 1.0, + "mmlu_stem": 2, + "mmlu_us_foreign_policy": 1.0, + "mmlu_virology": 1.0, + "mmlu_world_religions": 1.0 + }, + "n-shot": { + "mmlu_abstract_algebra": 5, + "mmlu_anatomy": 5, + "mmlu_astronomy": 5, + "mmlu_business_ethics": 5, + "mmlu_clinical_knowledge": 5, + "mmlu_college_biology": 5, + "mmlu_college_chemistry": 5, + "mmlu_college_computer_science": 5, + "mmlu_college_mathematics": 5, + "mmlu_college_medicine": 5, + "mmlu_college_physics": 5, + "mmlu_computer_security": 5, + "mmlu_conceptual_physics": 5, + "mmlu_econometrics": 5, + "mmlu_electrical_engineering": 5, + "mmlu_elementary_mathematics": 5, + "mmlu_formal_logic": 5, + "mmlu_global_facts": 5, + "mmlu_high_school_biology": 5, + "mmlu_high_school_chemistry": 5, + "mmlu_high_school_computer_science": 5, + "mmlu_high_school_european_history": 5, + "mmlu_high_school_geography": 5, + "mmlu_high_school_government_and_politics": 5, + "mmlu_high_school_macroeconomics": 5, + "mmlu_high_school_mathematics": 5, + "mmlu_high_school_microeconomics": 5, + "mmlu_high_school_physics": 5, + "mmlu_high_school_psychology": 5, + "mmlu_high_school_statistics": 5, + "mmlu_high_school_us_history": 5, + "mmlu_high_school_world_history": 5, + "mmlu_human_aging": 5, + "mmlu_human_sexuality": 5, + "mmlu_international_law": 5, + "mmlu_jurisprudence": 5, + "mmlu_logical_fallacies": 5, + "mmlu_machine_learning": 5, + "mmlu_management": 5, + "mmlu_marketing": 5, + "mmlu_medical_genetics": 5, + "mmlu_miscellaneous": 5, + "mmlu_moral_disputes": 5, + "mmlu_moral_scenarios": 5, + "mmlu_nutrition": 5, + "mmlu_philosophy": 5, + "mmlu_prehistory": 5, + "mmlu_professional_accounting": 5, + "mmlu_professional_law": 5, + "mmlu_professional_medicine": 5, + "mmlu_professional_psychology": 5, + "mmlu_public_relations": 5, + "mmlu_security_studies": 5, + "mmlu_sociology": 5, + "mmlu_us_foreign_policy": 5, + "mmlu_virology": 5, + "mmlu_world_religions": 5 + }, + "higher_is_better": { + "mmlu": { + "acc": true + }, + "mmlu_abstract_algebra": { + "acc": true + }, + "mmlu_anatomy": { + "acc": true + }, + "mmlu_astronomy": { + "acc": true + }, + "mmlu_business_ethics": { + "acc": true + }, + "mmlu_clinical_knowledge": { + "acc": true + }, + "mmlu_college_biology": { + "acc": true + }, + "mmlu_college_chemistry": { + "acc": true + }, + "mmlu_college_computer_science": { + "acc": true + }, + "mmlu_college_mathematics": { + "acc": true + }, + "mmlu_college_medicine": { + "acc": true + }, + "mmlu_college_physics": { + "acc": true + }, + "mmlu_computer_security": { + "acc": true + }, + "mmlu_conceptual_physics": { + "acc": true + }, + "mmlu_econometrics": { + "acc": true + }, + "mmlu_electrical_engineering": { + "acc": true + }, + "mmlu_elementary_mathematics": { + "acc": true + }, + "mmlu_formal_logic": { + "acc": true + }, + "mmlu_global_facts": { + "acc": true + }, + "mmlu_high_school_biology": { + "acc": true + }, + "mmlu_high_school_chemistry": { + "acc": true + }, + "mmlu_high_school_computer_science": { + "acc": true + }, + "mmlu_high_school_european_history": { + "acc": true + }, + "mmlu_high_school_geography": { + "acc": true + }, + "mmlu_high_school_government_and_politics": { + "acc": true + }, + "mmlu_high_school_macroeconomics": { + "acc": true + }, + "mmlu_high_school_mathematics": { + "acc": true + }, + "mmlu_high_school_microeconomics": { + "acc": true + }, + "mmlu_high_school_physics": { + "acc": true + }, + "mmlu_high_school_psychology": { + "acc": true + }, + "mmlu_high_school_statistics": { + "acc": true + }, + "mmlu_high_school_us_history": { + "acc": true + }, + "mmlu_high_school_world_history": { + "acc": true + }, + "mmlu_human_aging": { + "acc": true + }, + "mmlu_human_sexuality": { + "acc": true + }, + "mmlu_humanities": { + "acc": true + }, + "mmlu_international_law": { + "acc": true + }, + "mmlu_jurisprudence": { + "acc": true + }, + "mmlu_logical_fallacies": { + "acc": true + }, + "mmlu_machine_learning": { + "acc": true + }, + "mmlu_management": { + "acc": true + }, + "mmlu_marketing": { + "acc": true + }, + "mmlu_medical_genetics": { + "acc": true + }, + "mmlu_miscellaneous": { + "acc": true + }, + "mmlu_moral_disputes": { + "acc": true + }, + "mmlu_moral_scenarios": { + "acc": true + }, + "mmlu_nutrition": { + "acc": true + }, + "mmlu_other": { + "acc": true + }, + "mmlu_philosophy": { + "acc": true + }, + "mmlu_prehistory": { + "acc": true + }, + "mmlu_professional_accounting": { + "acc": true + }, + "mmlu_professional_law": { + "acc": true + }, + "mmlu_professional_medicine": { + "acc": true + }, + "mmlu_professional_psychology": { + "acc": true + }, + "mmlu_public_relations": { + "acc": true + }, + "mmlu_security_studies": { + "acc": true + }, + "mmlu_social_sciences": { + "acc": true + }, + "mmlu_sociology": { + "acc": true + }, + "mmlu_stem": { + "acc": true + }, + "mmlu_us_foreign_policy": { + "acc": true + }, + "mmlu_virology": { + "acc": true + }, + "mmlu_world_religions": { + "acc": true + } + }, + "n-samples": { + "mmlu_conceptual_physics": { + "original": 235, + "effective": 235 + }, + "mmlu_college_mathematics": { + "original": 100, + "effective": 100 + }, + "mmlu_high_school_statistics": { + "original": 216, + "effective": 216 + }, + "mmlu_elementary_mathematics": { + "original": 378, + "effective": 378 + }, + "mmlu_high_school_physics": { + "original": 151, + "effective": 151 + }, + "mmlu_electrical_engineering": { + "original": 145, + "effective": 145 + }, + "mmlu_college_chemistry": { + "original": 100, + "effective": 100 + }, + "mmlu_high_school_mathematics": { + "original": 270, + "effective": 270 + }, + "mmlu_abstract_algebra": { + "original": 100, + "effective": 100 + }, + "mmlu_astronomy": { + "original": 152, + "effective": 152 + }, + "mmlu_machine_learning": { + "original": 112, + "effective": 112 + }, + "mmlu_college_computer_science": { + "original": 100, + "effective": 100 + }, + "mmlu_high_school_chemistry": { + "original": 203, + "effective": 203 + }, + "mmlu_college_physics": { + "original": 102, + "effective": 102 + }, + "mmlu_college_biology": { + "original": 144, + "effective": 144 + }, + "mmlu_anatomy": { + "original": 135, + "effective": 135 + }, + "mmlu_high_school_biology": { + "original": 310, + "effective": 310 + }, + "mmlu_computer_security": { + "original": 100, + "effective": 100 + }, + "mmlu_high_school_computer_science": { + "original": 100, + "effective": 100 + }, + "mmlu_professional_medicine": { + "original": 272, + "effective": 272 + }, + "mmlu_professional_accounting": { + "original": 282, + "effective": 282 + }, + "mmlu_global_facts": { + "original": 100, + "effective": 100 + }, + "mmlu_nutrition": { + "original": 306, + "effective": 306 + }, + "mmlu_marketing": { + "original": 234, + "effective": 234 + }, + "mmlu_management": { + "original": 103, + "effective": 103 + }, + "mmlu_human_aging": { + "original": 223, + "effective": 223 + }, + "mmlu_medical_genetics": { + "original": 100, + "effective": 100 + }, + "mmlu_virology": { + "original": 166, + "effective": 166 + }, + "mmlu_clinical_knowledge": { + "original": 265, + "effective": 265 + }, + "mmlu_miscellaneous": { + "original": 783, + "effective": 783 + }, + "mmlu_college_medicine": { + "original": 173, + "effective": 173 + }, + "mmlu_business_ethics": { + "original": 100, + "effective": 100 + }, + "mmlu_us_foreign_policy": { + "original": 100, + "effective": 100 + }, + "mmlu_econometrics": { + "original": 114, + "effective": 114 + }, + "mmlu_human_sexuality": { + "original": 131, + "effective": 131 + }, + "mmlu_public_relations": { + "original": 110, + "effective": 110 + }, + "mmlu_high_school_government_and_politics": { + "original": 193, + "effective": 193 + }, + "mmlu_high_school_psychology": { + "original": 545, + "effective": 545 + }, + "mmlu_high_school_macroeconomics": { + "original": 390, + "effective": 390 + }, + "mmlu_sociology": { + "original": 201, + "effective": 201 + }, + "mmlu_high_school_geography": { + "original": 198, + "effective": 198 + }, + "mmlu_professional_psychology": { + "original": 612, + "effective": 612 + }, + "mmlu_high_school_microeconomics": { + "original": 238, + "effective": 238 + }, + "mmlu_security_studies": { + "original": 245, + "effective": 245 + }, + "mmlu_professional_law": { + "original": 1534, + "effective": 1534 + }, + "mmlu_world_religions": { + "original": 171, + "effective": 171 + }, + "mmlu_logical_fallacies": { + "original": 163, + "effective": 163 + }, + "mmlu_formal_logic": { + "original": 126, + "effective": 126 + }, + "mmlu_moral_scenarios": { + "original": 895, + "effective": 895 + }, + "mmlu_high_school_european_history": { + "original": 165, + "effective": 165 + }, + "mmlu_high_school_us_history": { + "original": 204, + "effective": 204 + }, + "mmlu_prehistory": { + "original": 324, + "effective": 324 + }, + "mmlu_international_law": { + "original": 121, + "effective": 121 + }, + "mmlu_philosophy": { + "original": 311, + "effective": 311 + }, + "mmlu_jurisprudence": { + "original": 108, + "effective": 108 + }, + "mmlu_high_school_world_history": { + "original": 237, + "effective": 237 + }, + "mmlu_moral_disputes": { + "original": 346, + "effective": 346 + } + }, + "config": { + "model": "local-completions", + "model_args": "base_url=http://127.0.0.1:18308/v1/completions,model=qwen3-8b-ptq-then-qad,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768", + "batch_size": 1, + "batch_sizes": [], + "device": null, + "use_cache": "/results/lm_cache", + "limit": null, + "downsampling_ratio": null, + "bootstrap_iters": 100000, + "gen_kwargs": { + "temperature": 1e-07, + "top_p": 0.9999999 + }, + "random_seed": 42, + "numpy_seed": 42, + "torch_seed": 42, + "fewshot_seed": 42 + }, + "git_hash": null, + "date": 1791480725.774736, + "pretty_env_info": "name 'get_pretty_env_info' is not defined", + "transformers_version": "4.50.3", + "lm_eval_version": "No package metadata was found for lm_eval", + "upper_git_hash": null, + "tokenizer_pad_token": [ + "<|endoftext|>", + "151643" + ], + "tokenizer_eos_token": [ + "<|im_end|>", + "151645" + ], + "tokenizer_bos_token": [ + null, + "None" + ], + "eot_token_id": 151645, + "max_length": 32767, + "task_hashes": { + "mmlu_conceptual_physics": "d9a7c4fd2d286215624dffc2aaec8e0295a6353daf2300d3461537e3fb5425eb", + "mmlu_college_mathematics": "33a226a6b1a6492f38387286d516c0941f5d8debbdfaa054b11fcda1cdb6dc1a", + "mmlu_high_school_statistics": "0598ccf63c944eeab0ba3ad2dcfbcaebddf4c5d124174071beb52e8a73f19e86", + "mmlu_elementary_mathematics": "da1d3600f010ce76c00a62b991b7606ee7df0b2e82413ee1f1977440ddd8550d", + "mmlu_high_school_physics": "7780579caacd56f607968e81d5256d7d386095ab1d93b672047c5a5d8444ec1d", + "mmlu_electrical_engineering": "58857d23657299553045d3db7b7b0de43f9a078a2fd53902d0e59251ac497e9c", + "mmlu_college_chemistry": "20b91eb7e582696bf39216d1757ac477675fe6de640150d7ae29993234af04dc", + "mmlu_high_school_mathematics": "313c7a073f5b7339d283ad977ea74584374b6e0281898dfd89ff4bdfd2fb7b03", + "mmlu_abstract_algebra": "0e2f49943ac30da4eb2d7d2f5db73abb69da69d6988007197e6bdf138bfb3369", + "mmlu_astronomy": "83e572d3948bb6dbfb66031ab4dcb5b70054a5bc110de1a0fbc324940f4c04da", + "mmlu_machine_learning": "d02c202c0ad4c729890962d842ea70cad80faaae66a51a4c35c3685783ca06c1", + "mmlu_college_computer_science": "c2355a53d39d8bf1fbec4ea4b0f2b50c3fe104f2b36556ecae1038b0c315816a", + "mmlu_high_school_chemistry": "7b7e216abbacc50a1613fc9e02f3f3dda2709059394b121e43f3ad534cadcf80", + "mmlu_college_physics": "ffe4456084dbfeb89e20d4b6138b97bbf415fd619691dbc7b38bfe5934b30e10", + "mmlu_college_biology": "14cbae605287fc3c9ff25c8f2b2175619e036287a2e217767382b5f48f044943", + "mmlu_anatomy": "9578a460fa25b65077e6da48a36e49d1e6b094807d65c9e5bf7e07ad0fbb6296", + "mmlu_high_school_biology": "cfa833fafa27f991b540ece0e0583e1ebedfcaa50d030891935b000b7757c1cb", + "mmlu_computer_security": "e122c4bc113d812e7e083e316b358cc7e16a0d1c2e6506480caebcf9d1e27ecd", + "mmlu_high_school_computer_science": "2db28b8942d29e448cb587b398788f3cba6693cfe20772102b5bb1907999abea", + "mmlu_professional_medicine": "ba4af62ac5d26864b9e6339c6cbd16cbabcb5fbcf00366c02dd15da73c28daf9", + "mmlu_professional_accounting": "654d1dafcb0ff52e603404c1158924e92c5067cda9df2429eb0777c99c648995", + "mmlu_global_facts": "856fc8218f2f89f3e9abbe96070c530689a228cf5c9269c58bd03b4f0aa833b1", + "mmlu_nutrition": "468a945d5f4f35a2b1c974d7180d159d3e80a0611fa9160afee572df0dd69432", + "mmlu_marketing": "2018f5181b9a272ca8a4f65858e0d12a1bf87792934fc483022e9d1bc7e05997", + "mmlu_management": "6345661bb012d1a72c5d34f39e692cf47c131807a3bdfff7bf07ee9c5149d29e", + "mmlu_human_aging": "88802f2005a5c3c719ace0291d92034ef3c90ee1cf7488d5195f8c545c899d4d", + "mmlu_medical_genetics": "8819bc558f8142680673b743cf8e3681ea6856ece1e3895c24fc0f77a60da7c4", + "mmlu_virology": "67042eb89ab0d2a45f1d254dbb9da9de152fa1098d7740a05ac7ff8495b26534", + "mmlu_clinical_knowledge": "a3dba4354de80aa98c7db318923aedb8b78c02014a2be97c4071453ba619b8dc", + "mmlu_miscellaneous": "22548428bd482f372a4fba10d2a0c0a939faddd7e9781701a74358145211a660", + "mmlu_college_medicine": "bc04947722ee05b4d5051df09f0e50de44d4f49df44f5ff909dbe3264cbc2c53", + "mmlu_business_ethics": "a0e93295030eaac8a5dd303dd1514b9eb25ae1807f55d000b876c0e8ddd5e1c8", + "mmlu_us_foreign_policy": "1c1d39c546896fb1b9a513c5db8233ea4066c3ec79d8b85c2ab7944fa79ca8c5", + "mmlu_econometrics": "905a5e83b9a8173ef7345d14ac535c91c78c34b076314a2deb52cb2e6138502c", + "mmlu_human_sexuality": "9b20e343f77622c458fe394d257924eb8895d07ddd98b1e3dbaf366563f77cbb", + "mmlu_public_relations": "355105913812b28cc46607bf6710a92dcb346c9d35f489eb84e388de03390b32", + "mmlu_high_school_government_and_politics": "bd59fed7fa1dcf3e39a65a7ea71cc81573c4fb28bcd942cbc8d8d9afbe0e9d4c", + "mmlu_high_school_psychology": "e62d1ebd73c032fd0c3b99eaf20e7b77b13ae49e4fc506d7e69f9ae15e485a4c", + "mmlu_high_school_macroeconomics": "50ee63c5ac58c48f651464b5fd6cbbe0c8fa6002dd480fa9634ba630d9dc0417", + "mmlu_sociology": "65917e6a3b0027bdb12703440e17afc8f155c15087a9df249e0061d84381e0d2", + "mmlu_high_school_geography": "30d201a5bfcbaec8d0e74f58ddd23bd58204b1d99b9211e6a531d8b6e84eab24", + "mmlu_professional_psychology": "70ec543a94dccdb5e82f3ac5f006de88a1352fb86b2a2c68d496529cdd1f5e0c", + "mmlu_high_school_microeconomics": "705c7c5543f9987bdf14f7b39ddae1910e9097d0f491bb98a536140175d94732", + "mmlu_security_studies": "e7c1c4bd8669cc22b7a907cd3584c5caf80b82772d9741a3680563b4fb564084", + "mmlu_professional_law": "0e0e2459e1f45df128cf8460ebdea4da6cc99a9fbd9cb99fb5ebeee78575800e", + "mmlu_world_religions": "8ad90daa76079af018b36cb86143468d22e34a9cd3273a3c48ce4c4648f886dc", + "mmlu_logical_fallacies": "f4aa9887846d578f13ca7420b65369c344053cabe092607a16237dfeef07a1ba", + "mmlu_formal_logic": "646ee3f6abf70a7d0efd03cdc0fbebb9f2e8f40e3b5563e53a9d0a7a291bc2a7", + "mmlu_moral_scenarios": "83a877f6af97d85f853c38196827d751dc8f33db4253da17965fde0c5e3554e1", + "mmlu_high_school_european_history": "8c8ff9bcf9924cd457cc5b67c5f6a21925c09aadba42e646c088f2b99ba4e0e2", + "mmlu_high_school_us_history": "b5fdc459ac365a794c663a17cf8a34a702ea87eaa1a8b40fe49e8196ef88232c", + "mmlu_prehistory": "4aa1d25e8211294e34dead164bbde415d42feb245a4716a2471d929f81846f04", + "mmlu_international_law": "144d9e8693486d00f5cac911f3fbfbacdba9ac7c10340d50061262d9c3fdff9d", + "mmlu_philosophy": "c61151b7b50a37e90635e8b219018a35533a65a2a684542ead156b9f1020ffc1", + "mmlu_jurisprudence": "325cb19fa404379425d7add71b29e2e4cd3428c08a25f5cc718fd87e79ccd2bb", + "mmlu_high_school_world_history": "59ebbb27916baa0f2c6fa8b064ecbb042b43f6d492bbe18f2a78a63d40bf3dd0", + "mmlu_moral_disputes": "4d3e2d32e558de3c5eb0369e9a20280fda45fe04eeedf3356ad820e352293720" + }, + "model_source": "local-completions", + "model_name": "qwen3-8b-ptq-then-qad", + "model_name_sanitized": "qwen3-8b-ptq-then-qad", + "system_instruction": null, + "system_instruction_sha": null, + "fewshot_as_multiturn": false, + "chat_template": null, + "chat_template_sha": null, + "start_time": 9127034.591303237, + "end_time": 9130813.532724572, + "total_evaluation_time_seconds": "3778.941421335563" +} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml new file mode 100644 index 00000000000..f86d4020b40 --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml @@ -0,0 +1,58 @@ +defaults: +- execution: slurm/default +- deployment: vllm +- ../../../nel_configs/mmlu@_global_ +- _self_ +execution: + hostname: ${oc.env:NEL_SLURM_HOST} + username: ${oc.env:SLURM_USER} + account: ${oc.env:NEL_SLURM_ACCOUNT} + partition: ${oc.env:NEL_SLURM_PARTITION} + output_dir: ${oc.env:NEL_OUTPUT_DIR} + num_nodes: 1 + ntasks_per_node: 1 + gres: gpu:1 + walltime: 04:00:00 + sbatch_extra_flags: + exclusive: false + mem: 64G + mounts: + mount_home: false + deployment: + /path/to/hf-cache: /hf-cache + /path/to/checkpoint: /checkpoint + evaluation: + /path/to/hf-cache: /hf-cache + /path/to/checkpoint: /checkpoint +deployment: + image: vllm/vllm-openai:v0.28.0-cu129 + checkpoint_path: ${oc.env:QWEN3_MODEL_CHECKPOINT} + served_model_name: qwen3-8b-ptq-then-qad + tensor_parallel_size: 1 + data_parallel_size: 1 + env_vars: + HF_HOME: lit:/hf-cache + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} + --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization + 0.85 --seed 42 --kv-cache-dtype auto + tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + port: 18108 +evaluation: + tasks: + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" + nemo_evaluator_config: + config: + params: + parallelism: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json new file mode 100644 index 00000000000..82333ad4cc2 --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json @@ -0,0 +1,136 @@ +{ + "results": { + "wikitext": { + "alias": "wikitext", + "word_perplexity,none": 13.68410924315444, + "word_perplexity_stderr,none": "N/A", + "byte_perplexity,none": 1.6310909243257237, + "byte_perplexity_stderr,none": "N/A", + "bits_per_byte,none": 0.7058372065854064, + "bits_per_byte_stderr,none": "N/A" + } + }, + "group_subtasks": { + "wikitext": [] + }, + "configs": { + "wikitext": { + "task": "wikitext", + "dataset_path": "EleutherAI/wikitext_document_level", + "dataset_name": "wikitext-2-raw-v1", + "dataset_kwargs": { + "trust_remote_code": true + }, + "training_split": "train", + "validation_split": "validation", + "test_split": "test", + "doc_to_text": "", + "doc_to_target": "def wikitext_detokenizer(doc):\n string = doc[\"page\"]\n # contractions\n string = string.replace(\"s '\", \"s'\")\n string = re.sub(r\"/' [0-9]/\", r\"/'[0-9]/\", string)\n # number separators\n string = string.replace(\" @-@ \", \"-\")\n string = string.replace(\" @,@ \", \",\")\n string = string.replace(\" @.@ \", \".\")\n # punctuation\n string = string.replace(\" : \", \": \")\n string = string.replace(\" ; \", \"; \")\n string = string.replace(\" . \", \". \")\n string = string.replace(\" ! \", \"! \")\n string = string.replace(\" ? \", \"? \")\n string = string.replace(\" , \", \", \")\n # double brackets\n string = re.sub(r\"\\(\\s*([^\\)]*?)\\s*\\)\", r\"(\\1)\", string)\n string = re.sub(r\"\\[\\s*([^\\]]*?)\\s*\\]\", r\"[\\1]\", string)\n string = re.sub(r\"{\\s*([^}]*?)\\s*}\", r\"{\\1}\", string)\n string = re.sub(r\"\\\"\\s*([^\\\"]*?)\\s*\\\"\", r'\"\\1\"', string)\n string = re.sub(r\"'\\s*([^']*?)\\s*'\", r\"'\\1'\", string)\n # miscellaneous\n string = string.replace(\"= = = =\", \"====\")\n string = string.replace(\"= = =\", \"===\")\n string = string.replace(\"= =\", \"==\")\n string = string.replace(\" \" + chr(176) + \" \", chr(176))\n string = string.replace(\" \\n\", \"\\n\")\n string = string.replace(\"\\n \", \"\\n\")\n string = string.replace(\" N \", \" 1 \")\n string = string.replace(\" 's\", \"'s\")\n\n return string\n", + "unsafe_code": false, + "process_results": "def process_results(doc, results):\n (loglikelihood,) = results\n # IMPORTANT: wikitext counts number of words in *original doc before detokenization*\n _words = len(re.split(r\"\\s+\", doc[\"page\"]))\n _bytes = len(doc[\"page\"].encode(\"utf-8\"))\n return {\n \"word_perplexity\": (loglikelihood, _words),\n \"byte_perplexity\": (loglikelihood, _bytes),\n \"bits_per_byte\": (loglikelihood, _bytes),\n }\n", + "description": "", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "num_fewshot": 0, + "metric_list": [ + { + "metric": "word_perplexity" + }, + { + "metric": "byte_perplexity" + }, + { + "metric": "bits_per_byte" + } + ], + "output_type": "loglikelihood_rolling", + "repeats": 1, + "should_decontaminate": true, + "doc_to_decontamination_query": "{{page}}", + "metadata": { + "version": 2.0, + "base_url": "http://localhost:3826", + "model": "qwen3-8b-ptq-then-qad", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 2, + "timeout": 300, + "max_retries": 2, + "stream": false + } + } + }, + "versions": { + "wikitext": 2.0 + }, + "n-shot": { + "wikitext": 0 + }, + "higher_is_better": { + "wikitext": { + "word_perplexity": false, + "byte_perplexity": false, + "bits_per_byte": false + } + }, + "n-samples": { + "wikitext": { + "original": 62, + "effective": 62 + } + }, + "config": { + "model": "local-completions", + "model_args": "base_url=http://localhost:3826,model=qwen3-8b-ptq-then-qad,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=2,timeout=300,max_retries=2,stream=False", + "batch_size": 1, + "batch_sizes": [], + "device": null, + "use_cache": "/results/lm_cache", + "limit": null, + "downsampling_ratio": null, + "bootstrap_iters": 100000, + "gen_kwargs": { + "temperature": 1e-07, + "top_p": 0.9999999 + }, + "random_seed": 42, + "numpy_seed": 42, + "torch_seed": 42, + "fewshot_seed": 42 + }, + "git_hash": null, + "date": 1791469379.5896866, + "pretty_env_info": "name 'get_pretty_env_info' is not defined", + "transformers_version": "4.50.3", + "lm_eval_version": "No package metadata was found for lm_eval", + "upper_git_hash": null, + "tokenizer_pad_token": [ + "<|endoftext|>", + "151643" + ], + "tokenizer_eos_token": [ + "<|im_end|>", + "151645" + ], + "tokenizer_bos_token": [ + null, + "None" + ], + "eot_token_id": 151645, + "max_length": 2047, + "task_hashes": { + "wikitext": "233109c41076287b5ba5a338ab7b085728acacf36d237a7ab99f34fcda161641" + }, + "model_source": "local-completions", + "model_name": "qwen3-8b-ptq-then-qad", + "model_name_sanitized": "qwen3-8b-ptq-then-qad", + "system_instruction": null, + "system_instruction_sha": null, + "fewshot_as_multiturn": false, + "chat_template": null, + "chat_template_sha": null, + "start_time": 1557654.864151966, + "end_time": 1557680.726953163, + "total_evaluation_time_seconds": "25.862801196984947" +} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml new file mode 100644 index 00000000000..13f58862318 --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml @@ -0,0 +1,41 @@ +# Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. +defaults: +- execution: slurm/default +- deployment: vllm +- ../../../nel_configs/wikitext@_global_ +- _self_ +execution: + hostname: ${oc.env:NEL_SLURM_HOST} + username: ${oc.env:SLURM_USER} + account: ${oc.env:NEL_SLURM_ACCOUNT} + partition: ${oc.env:NEL_SLURM_PARTITION} + output_dir: ${oc.env:NEL_OUTPUT_DIR} + num_nodes: 1 + ntasks_per_node: 1 + gres: gpu:1 + walltime: 04:00:00 + sbatch_extra_flags: + exclusive: false + mem: 64G + mounts: + mount_home: false + deployment: + /path/to/hf-cache: /hf-cache + /path/to/checkpoint: /checkpoint + evaluation: + /path/to/hf-cache: /hf-cache + /path/to/checkpoint: /checkpoint +deployment: + image: vllm/vllm-openai:v0.28.0-cu129 + checkpoint_path: ${oc.env:QWEN3_MODEL_CHECKPOINT} + served_model_name: qwen3-8b-ptq-then-qad + tensor_parallel_size: 1 + data_parallel_size: 1 + env_vars: + HF_HOME: lit:/hf-cache + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} + --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization + 0.85 --seed 42 --kv-cache-dtype auto + tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + port: 18110 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json new file mode 100644 index 00000000000..e2342ced11b --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json @@ -0,0 +1,4030 @@ +{ + "results": { + "mmlu": { + "acc,none": 0.7493946731234867, + "acc_stderr,none": 0.0034779802227510367, + "alias": "mmlu" + }, + "mmlu_humanities": { + "acc,none": 0.6641870350690755, + "acc_stderr,none": 0.006552018414272379, + "alias": " - humanities" + }, + "mmlu_formal_logic": { + "alias": " - formal_logic", + "acc,none": 0.6746031746031746, + "acc_stderr,none": 0.04190596438871138 + }, + "mmlu_high_school_european_history": { + "alias": " - high_school_european_history", + "acc,none": 0.8545454545454545, + "acc_stderr,none": 0.02753019635506653 + }, + "mmlu_high_school_us_history": { + "alias": " - high_school_us_history", + "acc,none": 0.8627450980392157, + "acc_stderr,none": 0.024152225962801553 + }, + "mmlu_high_school_world_history": { + "alias": " - high_school_world_history", + "acc,none": 0.8649789029535865, + "acc_stderr,none": 0.022245776632003705 + }, + "mmlu_international_law": { + "alias": " - international_law", + "acc,none": 0.7355371900826446, + "acc_stderr,none": 0.04026187527591209 + }, + "mmlu_jurisprudence": { + "alias": " - jurisprudence", + "acc,none": 0.8148148148148148, + "acc_stderr,none": 0.03755265865037181 + }, + "mmlu_logical_fallacies": { + "alias": " - logical_fallacies", + "acc,none": 0.8220858895705522, + "acc_stderr,none": 0.03004735765580667 + }, + "mmlu_moral_disputes": { + "alias": " - moral_disputes", + "acc,none": 0.7601156069364162, + "acc_stderr,none": 0.02298959254312351 + }, + "mmlu_moral_scenarios": { + "alias": " - moral_scenarios", + "acc,none": 0.5541899441340782, + "acc_stderr,none": 0.016623998513333176 + }, + "mmlu_philosophy": { + "alias": " - philosophy", + "acc,none": 0.7781350482315113, + "acc_stderr,none": 0.023598858292862998 + }, + "mmlu_prehistory": { + "alias": " - prehistory", + "acc,none": 0.8395061728395061, + "acc_stderr,none": 0.02042395535477805 + }, + "mmlu_professional_law": { + "alias": " - professional_law", + "acc,none": 0.5117340286831812, + "acc_stderr,none": 0.012766719019686833 + }, + "mmlu_world_religions": { + "alias": " - world_religions", + "acc,none": 0.8713450292397661, + "acc_stderr,none": 0.02567934272327688 + }, + "mmlu_other": { + "acc,none": 0.784036047634374, + "acc_stderr,none": 0.007130383176974831, + "alias": " - other" + }, + "mmlu_business_ethics": { + "alias": " - business_ethics", + "acc,none": 0.78, + "acc_stderr,none": 0.041633319989322654 + }, + "mmlu_clinical_knowledge": { + "alias": " - clinical_knowledge", + "acc,none": 0.7849056603773585, + "acc_stderr,none": 0.02528839450289141 + }, + "mmlu_college_medicine": { + "alias": " - college_medicine", + "acc,none": 0.791907514450867, + "acc_stderr,none": 0.030952890217749857 + }, + "mmlu_global_facts": { + "alias": " - global_facts", + "acc,none": 0.52, + "acc_stderr,none": 0.05021167315686783 + }, + "mmlu_human_aging": { + "alias": " - human_aging", + "acc,none": 0.7443946188340808, + "acc_stderr,none": 0.029275891003969906 + }, + "mmlu_management": { + "alias": " - management", + "acc,none": 0.8737864077669902, + "acc_stderr,none": 0.03288180278808631 + }, + "mmlu_marketing": { + "alias": " - marketing", + "acc,none": 0.9358974358974359, + "acc_stderr,none": 0.016046261631673164 + }, + "mmlu_medical_genetics": { + "alias": " - medical_genetics", + "acc,none": 0.83, + "acc_stderr,none": 0.03775251680686369 + }, + "mmlu_miscellaneous": { + "alias": " - miscellaneous", + "acc,none": 0.8582375478927203, + "acc_stderr,none": 0.012473289071271968 + }, + "mmlu_nutrition": { + "alias": " - nutrition", + "acc,none": 0.8104575163398693, + "acc_stderr,none": 0.022442358263336185 + }, + "mmlu_professional_accounting": { + "alias": " - professional_accounting", + "acc,none": 0.5921985815602837, + "acc_stderr,none": 0.02931601177634362 + }, + "mmlu_professional_medicine": { + "alias": " - professional_medicine", + "acc,none": 0.8198529411764706, + "acc_stderr,none": 0.0233451636165449 + }, + "mmlu_virology": { + "alias": " - virology", + "acc,none": 0.5602409638554217, + "acc_stderr,none": 0.0386413992369912 + }, + "mmlu_social_sciences": { + "acc,none": 0.8394540136496588, + "acc_stderr,none": 0.006505444273959088, + "alias": " - social sciences" + }, + "mmlu_econometrics": { + "alias": " - econometrics", + "acc,none": 0.6578947368421053, + "acc_stderr,none": 0.044629175353369355 + }, + "mmlu_high_school_geography": { + "alias": " - high_school_geography", + "acc,none": 0.8686868686868687, + "acc_stderr,none": 0.02406315641682249 + }, + "mmlu_high_school_government_and_politics": { + "alias": " - high_school_government_and_politics", + "acc,none": 0.9326424870466321, + "acc_stderr,none": 0.01808839383907894 + }, + "mmlu_high_school_macroeconomics": { + "alias": " - high_school_macroeconomics", + "acc,none": 0.8102564102564103, + "acc_stderr,none": 0.019880165406588844 + }, + "mmlu_high_school_microeconomics": { + "alias": " - high_school_microeconomics", + "acc,none": 0.9369747899159664, + "acc_stderr,none": 0.015785085223670888 + }, + "mmlu_high_school_psychology": { + "alias": " - high_school_psychology", + "acc,none": 0.9119266055045872, + "acc_stderr,none": 0.012150743719481636 + }, + "mmlu_human_sexuality": { + "alias": " - human_sexuality", + "acc,none": 0.8320610687022901, + "acc_stderr,none": 0.0327854853734314 + }, + "mmlu_professional_psychology": { + "alias": " - professional_psychology", + "acc,none": 0.7859477124183006, + "acc_stderr,none": 0.01659342966232898 + }, + "mmlu_public_relations": { + "alias": " - public_relations", + "acc,none": 0.7, + "acc_stderr,none": 0.04389311454644288 + }, + "mmlu_security_studies": { + "alias": " - security_studies", + "acc,none": 0.7959183673469388, + "acc_stderr,none": 0.02580128347509046 + }, + "mmlu_sociology": { + "alias": " - sociology", + "acc,none": 0.8656716417910447, + "acc_stderr,none": 0.024112678240900763 + }, + "mmlu_us_foreign_policy": { + "alias": " - us_foreign_policy", + "acc,none": 0.84, + "acc_stderr,none": 0.03684529491774706 + }, + "mmlu_stem": { + "acc,none": 0.7545195052331113, + "acc_stderr,none": 0.0073915569536853855, + "alias": " - stem" + }, + "mmlu_abstract_algebra": { + "alias": " - abstract_algebra", + "acc,none": 0.55, + "acc_stderr,none": 0.05 + }, + "mmlu_anatomy": { + "alias": " - anatomy", + "acc,none": 0.7037037037037037, + "acc_stderr,none": 0.03944624162501111 + }, + "mmlu_astronomy": { + "alias": " - astronomy", + "acc,none": 0.9013157894736842, + "acc_stderr,none": 0.024270227737522694 + }, + "mmlu_college_biology": { + "alias": " - college_biology", + "acc,none": 0.875, + "acc_stderr,none": 0.02765610492929436 + }, + "mmlu_college_chemistry": { + "alias": " - college_chemistry", + "acc,none": 0.56, + "acc_stderr,none": 0.049888765156985884 + }, + "mmlu_college_computer_science": { + "alias": " - college_computer_science", + "acc,none": 0.73, + "acc_stderr,none": 0.04461960433384737 + }, + "mmlu_college_mathematics": { + "alias": " - college_mathematics", + "acc,none": 0.58, + "acc_stderr,none": 0.04960449637488582 + }, + "mmlu_college_physics": { + "alias": " - college_physics", + "acc,none": 0.6470588235294118, + "acc_stderr,none": 0.04755129616062946 + }, + "mmlu_computer_security": { + "alias": " - computer_security", + "acc,none": 0.81, + "acc_stderr,none": 0.039427724440366255 + }, + "mmlu_conceptual_physics": { + "alias": " - conceptual_physics", + "acc,none": 0.8468085106382979, + "acc_stderr,none": 0.023545179061675213 + }, + "mmlu_electrical_engineering": { + "alias": " - electrical_engineering", + "acc,none": 0.8068965517241379, + "acc_stderr,none": 0.03289445522127399 + }, + "mmlu_elementary_mathematics": { + "alias": " - elementary_mathematics", + "acc,none": 0.7883597883597884, + "acc_stderr,none": 0.02103733150526295 + }, + "mmlu_high_school_biology": { + "alias": " - high_school_biology", + "acc,none": 0.9096774193548387, + "acc_stderr,none": 0.016306570644488316 + }, + "mmlu_high_school_chemistry": { + "alias": " - high_school_chemistry", + "acc,none": 0.7586206896551724, + "acc_stderr,none": 0.030108330718011642 + }, + "mmlu_high_school_computer_science": { + "alias": " - high_school_computer_science", + "acc,none": 0.86, + "acc_stderr,none": 0.0348735088019777 + }, + "mmlu_high_school_mathematics": { + "alias": " - high_school_mathematics", + "acc,none": 0.5370370370370371, + "acc_stderr,none": 0.030401786406101534 + }, + "mmlu_high_school_physics": { + "alias": " - high_school_physics", + "acc,none": 0.7284768211920529, + "acc_stderr,none": 0.03631329803969657 + }, + "mmlu_high_school_statistics": { + "alias": " - high_school_statistics", + "acc,none": 0.7824074074074074, + "acc_stderr,none": 0.028139689444859672 + }, + "mmlu_machine_learning": { + "alias": " - machine_learning", + "acc,none": 0.6428571428571429, + "acc_stderr,none": 0.045479609997643805 + } + }, + "groups": { + "mmlu": { + "acc,none": 0.7493946731234867, + "acc_stderr,none": 0.0034779802227510367, + "alias": "mmlu" + }, + "mmlu_humanities": { + "acc,none": 0.6641870350690755, + "acc_stderr,none": 0.006552018414272379, + "alias": " - humanities" + }, + "mmlu_other": { + "acc,none": 0.784036047634374, + "acc_stderr,none": 0.007130383176974831, + "alias": " - other" + }, + "mmlu_social_sciences": { + "acc,none": 0.8394540136496588, + "acc_stderr,none": 0.006505444273959088, + "alias": " - social sciences" + }, + "mmlu_stem": { + "acc,none": 0.7545195052331113, + "acc_stderr,none": 0.0073915569536853855, + "alias": " - stem" + } + }, + "group_subtasks": { + "mmlu_humanities": [ + "mmlu_formal_logic", + "mmlu_logical_fallacies", + "mmlu_jurisprudence", + "mmlu_professional_law", + "mmlu_high_school_european_history", + "mmlu_moral_scenarios", + "mmlu_moral_disputes", + "mmlu_international_law", + "mmlu_world_religions", + "mmlu_high_school_us_history", + "mmlu_prehistory", + "mmlu_philosophy", + "mmlu_high_school_world_history" + ], + "mmlu_social_sciences": [ + "mmlu_sociology", + "mmlu_econometrics", + "mmlu_high_school_government_and_politics", + "mmlu_security_studies", + "mmlu_high_school_microeconomics", + "mmlu_human_sexuality", + "mmlu_public_relations", + "mmlu_us_foreign_policy", + "mmlu_high_school_geography", + "mmlu_high_school_macroeconomics", + "mmlu_high_school_psychology", + "mmlu_professional_psychology" + ], + "mmlu_other": [ + "mmlu_management", + "mmlu_nutrition", + "mmlu_business_ethics", + "mmlu_college_medicine", + "mmlu_global_facts", + "mmlu_virology", + "mmlu_human_aging", + "mmlu_marketing", + "mmlu_professional_medicine", + "mmlu_professional_accounting", + "mmlu_medical_genetics", + "mmlu_miscellaneous", + "mmlu_clinical_knowledge" + ], + "mmlu_stem": [ + "mmlu_college_mathematics", + "mmlu_college_biology", + "mmlu_anatomy", + "mmlu_electrical_engineering", + "mmlu_high_school_chemistry", + "mmlu_college_computer_science", + "mmlu_high_school_statistics", + "mmlu_abstract_algebra", + "mmlu_college_chemistry", + "mmlu_elementary_mathematics", + "mmlu_conceptual_physics", + "mmlu_high_school_mathematics", + "mmlu_college_physics", + "mmlu_machine_learning", + "mmlu_high_school_computer_science", + "mmlu_high_school_biology", + "mmlu_computer_security", + "mmlu_astronomy", + "mmlu_high_school_physics" + ], + "mmlu": [ + "mmlu_stem", + "mmlu_other", + "mmlu_social_sciences", + "mmlu_humanities" + ] + }, + "configs": { + "mmlu_abstract_algebra": { + "task": "mmlu_abstract_algebra", + "task_alias": "abstract_algebra", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "abstract_algebra", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about abstract algebra.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_anatomy": { + "task": "mmlu_anatomy", + "task_alias": "anatomy", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "anatomy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about anatomy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_astronomy": { + "task": "mmlu_astronomy", + "task_alias": "astronomy", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "astronomy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about astronomy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_business_ethics": { + "task": "mmlu_business_ethics", + "task_alias": "business_ethics", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "business_ethics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about business ethics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_clinical_knowledge": { + "task": "mmlu_clinical_knowledge", + "task_alias": "clinical_knowledge", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "clinical_knowledge", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about clinical knowledge.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_biology": { + "task": "mmlu_college_biology", + "task_alias": "college_biology", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_biology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college biology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_chemistry": { + "task": "mmlu_college_chemistry", + "task_alias": "college_chemistry", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_chemistry", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college chemistry.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_computer_science": { + "task": "mmlu_college_computer_science", + "task_alias": "college_computer_science", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_computer_science", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college computer science.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_mathematics": { + "task": "mmlu_college_mathematics", + "task_alias": "college_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_medicine": { + "task": "mmlu_college_medicine", + "task_alias": "college_medicine", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_medicine", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college medicine.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_college_physics": { + "task": "mmlu_college_physics", + "task_alias": "college_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "college_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about college physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_computer_security": { + "task": "mmlu_computer_security", + "task_alias": "computer_security", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "computer_security", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about computer security.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_conceptual_physics": { + "task": "mmlu_conceptual_physics", + "task_alias": "conceptual_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "conceptual_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about conceptual physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_econometrics": { + "task": "mmlu_econometrics", + "task_alias": "econometrics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "econometrics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about econometrics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_electrical_engineering": { + "task": "mmlu_electrical_engineering", + "task_alias": "electrical_engineering", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "electrical_engineering", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about electrical engineering.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_elementary_mathematics": { + "task": "mmlu_elementary_mathematics", + "task_alias": "elementary_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "elementary_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about elementary mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_formal_logic": { + "task": "mmlu_formal_logic", + "task_alias": "formal_logic", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "formal_logic", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about formal logic.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_global_facts": { + "task": "mmlu_global_facts", + "task_alias": "global_facts", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "global_facts", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about global facts.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_biology": { + "task": "mmlu_high_school_biology", + "task_alias": "high_school_biology", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_biology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school biology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_chemistry": { + "task": "mmlu_high_school_chemistry", + "task_alias": "high_school_chemistry", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_chemistry", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school chemistry.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_computer_science": { + "task": "mmlu_high_school_computer_science", + "task_alias": "high_school_computer_science", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_computer_science", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school computer science.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_european_history": { + "task": "mmlu_high_school_european_history", + "task_alias": "high_school_european_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_european_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school european history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_geography": { + "task": "mmlu_high_school_geography", + "task_alias": "high_school_geography", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_geography", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school geography.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_government_and_politics": { + "task": "mmlu_high_school_government_and_politics", + "task_alias": "high_school_government_and_politics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_government_and_politics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school government and politics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_macroeconomics": { + "task": "mmlu_high_school_macroeconomics", + "task_alias": "high_school_macroeconomics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_macroeconomics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school macroeconomics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_mathematics": { + "task": "mmlu_high_school_mathematics", + "task_alias": "high_school_mathematics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_mathematics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school mathematics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_microeconomics": { + "task": "mmlu_high_school_microeconomics", + "task_alias": "high_school_microeconomics", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_microeconomics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school microeconomics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_physics": { + "task": "mmlu_high_school_physics", + "task_alias": "high_school_physics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_physics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school physics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_psychology": { + "task": "mmlu_high_school_psychology", + "task_alias": "high_school_psychology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_psychology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school psychology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_statistics": { + "task": "mmlu_high_school_statistics", + "task_alias": "high_school_statistics", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_statistics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school statistics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_us_history": { + "task": "mmlu_high_school_us_history", + "task_alias": "high_school_us_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_us_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school us history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_high_school_world_history": { + "task": "mmlu_high_school_world_history", + "task_alias": "high_school_world_history", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "high_school_world_history", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about high school world history.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_human_aging": { + "task": "mmlu_human_aging", + "task_alias": "human_aging", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "human_aging", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about human aging.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_human_sexuality": { + "task": "mmlu_human_sexuality", + "task_alias": "human_sexuality", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "human_sexuality", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about human sexuality.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_international_law": { + "task": "mmlu_international_law", + "task_alias": "international_law", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "international_law", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about international law.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_jurisprudence": { + "task": "mmlu_jurisprudence", + "task_alias": "jurisprudence", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "jurisprudence", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about jurisprudence.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_logical_fallacies": { + "task": "mmlu_logical_fallacies", + "task_alias": "logical_fallacies", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "logical_fallacies", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about logical fallacies.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_machine_learning": { + "task": "mmlu_machine_learning", + "task_alias": "machine_learning", + "tag": "mmlu_stem_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "machine_learning", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about machine learning.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_management": { + "task": "mmlu_management", + "task_alias": "management", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "management", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about management.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_marketing": { + "task": "mmlu_marketing", + "task_alias": "marketing", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "marketing", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about marketing.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_medical_genetics": { + "task": "mmlu_medical_genetics", + "task_alias": "medical_genetics", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "medical_genetics", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about medical genetics.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_miscellaneous": { + "task": "mmlu_miscellaneous", + "task_alias": "miscellaneous", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "miscellaneous", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about miscellaneous.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_moral_disputes": { + "task": "mmlu_moral_disputes", + "task_alias": "moral_disputes", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "moral_disputes", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about moral disputes.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_moral_scenarios": { + "task": "mmlu_moral_scenarios", + "task_alias": "moral_scenarios", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "moral_scenarios", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about moral scenarios.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_nutrition": { + "task": "mmlu_nutrition", + "task_alias": "nutrition", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "nutrition", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about nutrition.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_philosophy": { + "task": "mmlu_philosophy", + "task_alias": "philosophy", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "philosophy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about philosophy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_prehistory": { + "task": "mmlu_prehistory", + "task_alias": "prehistory", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "prehistory", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about prehistory.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_accounting": { + "task": "mmlu_professional_accounting", + "task_alias": "professional_accounting", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_accounting", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional accounting.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_law": { + "task": "mmlu_professional_law", + "task_alias": "professional_law", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_law", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional law.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_medicine": { + "task": "mmlu_professional_medicine", + "task_alias": "professional_medicine", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_medicine", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional medicine.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_professional_psychology": { + "task": "mmlu_professional_psychology", + "task_alias": "professional_psychology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "professional_psychology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about professional psychology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_public_relations": { + "task": "mmlu_public_relations", + "task_alias": "public_relations", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "public_relations", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about public relations.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_security_studies": { + "task": "mmlu_security_studies", + "task_alias": "security_studies", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "security_studies", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about security studies.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_sociology": { + "task": "mmlu_sociology", + "task_alias": "sociology", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "sociology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about sociology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_us_foreign_policy": { + "task": "mmlu_us_foreign_policy", + "task_alias": "us_foreign_policy", + "tag": "mmlu_social_sciences_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "us_foreign_policy", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about us foreign policy.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_virology": { + "task": "mmlu_virology", + "task_alias": "virology", + "tag": "mmlu_other_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "virology", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about virology.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + }, + "mmlu_world_religions": { + "task": "mmlu_world_religions", + "task_alias": "world_religions", + "tag": "mmlu_humanities_tasks", + "dataset_path": "cais/mmlu", + "dataset_name": "world_religions", + "dataset_kwargs": { + "trust_remote_code": true + }, + "test_split": "test", + "fewshot_split": "dev", + "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", + "doc_to_target": "answer", + "unsafe_code": false, + "doc_to_choice": [ + "A", + "B", + "C", + "D" + ], + "description": "The following are multiple choice questions (with answers) about world religions.\n\n", + "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "fewshot_config": { + "sampler": "first_n" + }, + "num_fewshot": 5, + "metric_list": [ + { + "metric": "acc", + "aggregation": "mean", + "higher_is_better": true + } + ], + "output_type": "multiple_choice", + "repeats": 1, + "should_decontaminate": false, + "metadata": { + "version": 1.0, + "base_url": "http://127.0.0.1:18300/v1/completions", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 1, + "timeout": 300, + "max_retries": 2, + "stream": false, + "max_length": 32768 + } + } + }, + "versions": { + "mmlu": 2, + "mmlu_abstract_algebra": 1.0, + "mmlu_anatomy": 1.0, + "mmlu_astronomy": 1.0, + "mmlu_business_ethics": 1.0, + "mmlu_clinical_knowledge": 1.0, + "mmlu_college_biology": 1.0, + "mmlu_college_chemistry": 1.0, + "mmlu_college_computer_science": 1.0, + "mmlu_college_mathematics": 1.0, + "mmlu_college_medicine": 1.0, + "mmlu_college_physics": 1.0, + "mmlu_computer_security": 1.0, + "mmlu_conceptual_physics": 1.0, + "mmlu_econometrics": 1.0, + "mmlu_electrical_engineering": 1.0, + "mmlu_elementary_mathematics": 1.0, + "mmlu_formal_logic": 1.0, + "mmlu_global_facts": 1.0, + "mmlu_high_school_biology": 1.0, + "mmlu_high_school_chemistry": 1.0, + "mmlu_high_school_computer_science": 1.0, + "mmlu_high_school_european_history": 1.0, + "mmlu_high_school_geography": 1.0, + "mmlu_high_school_government_and_politics": 1.0, + "mmlu_high_school_macroeconomics": 1.0, + "mmlu_high_school_mathematics": 1.0, + "mmlu_high_school_microeconomics": 1.0, + "mmlu_high_school_physics": 1.0, + "mmlu_high_school_psychology": 1.0, + "mmlu_high_school_statistics": 1.0, + "mmlu_high_school_us_history": 1.0, + "mmlu_high_school_world_history": 1.0, + "mmlu_human_aging": 1.0, + "mmlu_human_sexuality": 1.0, + "mmlu_humanities": 2, + "mmlu_international_law": 1.0, + "mmlu_jurisprudence": 1.0, + "mmlu_logical_fallacies": 1.0, + "mmlu_machine_learning": 1.0, + "mmlu_management": 1.0, + "mmlu_marketing": 1.0, + "mmlu_medical_genetics": 1.0, + "mmlu_miscellaneous": 1.0, + "mmlu_moral_disputes": 1.0, + "mmlu_moral_scenarios": 1.0, + "mmlu_nutrition": 1.0, + "mmlu_other": 2, + "mmlu_philosophy": 1.0, + "mmlu_prehistory": 1.0, + "mmlu_professional_accounting": 1.0, + "mmlu_professional_law": 1.0, + "mmlu_professional_medicine": 1.0, + "mmlu_professional_psychology": 1.0, + "mmlu_public_relations": 1.0, + "mmlu_security_studies": 1.0, + "mmlu_social_sciences": 2, + "mmlu_sociology": 1.0, + "mmlu_stem": 2, + "mmlu_us_foreign_policy": 1.0, + "mmlu_virology": 1.0, + "mmlu_world_religions": 1.0 + }, + "n-shot": { + "mmlu_abstract_algebra": 5, + "mmlu_anatomy": 5, + "mmlu_astronomy": 5, + "mmlu_business_ethics": 5, + "mmlu_clinical_knowledge": 5, + "mmlu_college_biology": 5, + "mmlu_college_chemistry": 5, + "mmlu_college_computer_science": 5, + "mmlu_college_mathematics": 5, + "mmlu_college_medicine": 5, + "mmlu_college_physics": 5, + "mmlu_computer_security": 5, + "mmlu_conceptual_physics": 5, + "mmlu_econometrics": 5, + "mmlu_electrical_engineering": 5, + "mmlu_elementary_mathematics": 5, + "mmlu_formal_logic": 5, + "mmlu_global_facts": 5, + "mmlu_high_school_biology": 5, + "mmlu_high_school_chemistry": 5, + "mmlu_high_school_computer_science": 5, + "mmlu_high_school_european_history": 5, + "mmlu_high_school_geography": 5, + "mmlu_high_school_government_and_politics": 5, + "mmlu_high_school_macroeconomics": 5, + "mmlu_high_school_mathematics": 5, + "mmlu_high_school_microeconomics": 5, + "mmlu_high_school_physics": 5, + "mmlu_high_school_psychology": 5, + "mmlu_high_school_statistics": 5, + "mmlu_high_school_us_history": 5, + "mmlu_high_school_world_history": 5, + "mmlu_human_aging": 5, + "mmlu_human_sexuality": 5, + "mmlu_international_law": 5, + "mmlu_jurisprudence": 5, + "mmlu_logical_fallacies": 5, + "mmlu_machine_learning": 5, + "mmlu_management": 5, + "mmlu_marketing": 5, + "mmlu_medical_genetics": 5, + "mmlu_miscellaneous": 5, + "mmlu_moral_disputes": 5, + "mmlu_moral_scenarios": 5, + "mmlu_nutrition": 5, + "mmlu_philosophy": 5, + "mmlu_prehistory": 5, + "mmlu_professional_accounting": 5, + "mmlu_professional_law": 5, + "mmlu_professional_medicine": 5, + "mmlu_professional_psychology": 5, + "mmlu_public_relations": 5, + "mmlu_security_studies": 5, + "mmlu_sociology": 5, + "mmlu_us_foreign_policy": 5, + "mmlu_virology": 5, + "mmlu_world_religions": 5 + }, + "higher_is_better": { + "mmlu": { + "acc": true + }, + "mmlu_abstract_algebra": { + "acc": true + }, + "mmlu_anatomy": { + "acc": true + }, + "mmlu_astronomy": { + "acc": true + }, + "mmlu_business_ethics": { + "acc": true + }, + "mmlu_clinical_knowledge": { + "acc": true + }, + "mmlu_college_biology": { + "acc": true + }, + "mmlu_college_chemistry": { + "acc": true + }, + "mmlu_college_computer_science": { + "acc": true + }, + "mmlu_college_mathematics": { + "acc": true + }, + "mmlu_college_medicine": { + "acc": true + }, + "mmlu_college_physics": { + "acc": true + }, + "mmlu_computer_security": { + "acc": true + }, + "mmlu_conceptual_physics": { + "acc": true + }, + "mmlu_econometrics": { + "acc": true + }, + "mmlu_electrical_engineering": { + "acc": true + }, + "mmlu_elementary_mathematics": { + "acc": true + }, + "mmlu_formal_logic": { + "acc": true + }, + "mmlu_global_facts": { + "acc": true + }, + "mmlu_high_school_biology": { + "acc": true + }, + "mmlu_high_school_chemistry": { + "acc": true + }, + "mmlu_high_school_computer_science": { + "acc": true + }, + "mmlu_high_school_european_history": { + "acc": true + }, + "mmlu_high_school_geography": { + "acc": true + }, + "mmlu_high_school_government_and_politics": { + "acc": true + }, + "mmlu_high_school_macroeconomics": { + "acc": true + }, + "mmlu_high_school_mathematics": { + "acc": true + }, + "mmlu_high_school_microeconomics": { + "acc": true + }, + "mmlu_high_school_physics": { + "acc": true + }, + "mmlu_high_school_psychology": { + "acc": true + }, + "mmlu_high_school_statistics": { + "acc": true + }, + "mmlu_high_school_us_history": { + "acc": true + }, + "mmlu_high_school_world_history": { + "acc": true + }, + "mmlu_human_aging": { + "acc": true + }, + "mmlu_human_sexuality": { + "acc": true + }, + "mmlu_humanities": { + "acc": true + }, + "mmlu_international_law": { + "acc": true + }, + "mmlu_jurisprudence": { + "acc": true + }, + "mmlu_logical_fallacies": { + "acc": true + }, + "mmlu_machine_learning": { + "acc": true + }, + "mmlu_management": { + "acc": true + }, + "mmlu_marketing": { + "acc": true + }, + "mmlu_medical_genetics": { + "acc": true + }, + "mmlu_miscellaneous": { + "acc": true + }, + "mmlu_moral_disputes": { + "acc": true + }, + "mmlu_moral_scenarios": { + "acc": true + }, + "mmlu_nutrition": { + "acc": true + }, + "mmlu_other": { + "acc": true + }, + "mmlu_philosophy": { + "acc": true + }, + "mmlu_prehistory": { + "acc": true + }, + "mmlu_professional_accounting": { + "acc": true + }, + "mmlu_professional_law": { + "acc": true + }, + "mmlu_professional_medicine": { + "acc": true + }, + "mmlu_professional_psychology": { + "acc": true + }, + "mmlu_public_relations": { + "acc": true + }, + "mmlu_security_studies": { + "acc": true + }, + "mmlu_social_sciences": { + "acc": true + }, + "mmlu_sociology": { + "acc": true + }, + "mmlu_stem": { + "acc": true + }, + "mmlu_us_foreign_policy": { + "acc": true + }, + "mmlu_virology": { + "acc": true + }, + "mmlu_world_religions": { + "acc": true + } + }, + "n-samples": { + "mmlu_college_mathematics": { + "original": 100, + "effective": 100 + }, + "mmlu_college_biology": { + "original": 144, + "effective": 144 + }, + "mmlu_anatomy": { + "original": 135, + "effective": 135 + }, + "mmlu_electrical_engineering": { + "original": 145, + "effective": 145 + }, + "mmlu_high_school_chemistry": { + "original": 203, + "effective": 203 + }, + "mmlu_college_computer_science": { + "original": 100, + "effective": 100 + }, + "mmlu_high_school_statistics": { + "original": 216, + "effective": 216 + }, + "mmlu_abstract_algebra": { + "original": 100, + "effective": 100 + }, + "mmlu_college_chemistry": { + "original": 100, + "effective": 100 + }, + "mmlu_elementary_mathematics": { + "original": 378, + "effective": 378 + }, + "mmlu_conceptual_physics": { + "original": 235, + "effective": 235 + }, + "mmlu_high_school_mathematics": { + "original": 270, + "effective": 270 + }, + "mmlu_college_physics": { + "original": 102, + "effective": 102 + }, + "mmlu_machine_learning": { + "original": 112, + "effective": 112 + }, + "mmlu_high_school_computer_science": { + "original": 100, + "effective": 100 + }, + "mmlu_high_school_biology": { + "original": 310, + "effective": 310 + }, + "mmlu_computer_security": { + "original": 100, + "effective": 100 + }, + "mmlu_astronomy": { + "original": 152, + "effective": 152 + }, + "mmlu_high_school_physics": { + "original": 151, + "effective": 151 + }, + "mmlu_management": { + "original": 103, + "effective": 103 + }, + "mmlu_nutrition": { + "original": 306, + "effective": 306 + }, + "mmlu_business_ethics": { + "original": 100, + "effective": 100 + }, + "mmlu_college_medicine": { + "original": 173, + "effective": 173 + }, + "mmlu_global_facts": { + "original": 100, + "effective": 100 + }, + "mmlu_virology": { + "original": 166, + "effective": 166 + }, + "mmlu_human_aging": { + "original": 223, + "effective": 223 + }, + "mmlu_marketing": { + "original": 234, + "effective": 234 + }, + "mmlu_professional_medicine": { + "original": 272, + "effective": 272 + }, + "mmlu_professional_accounting": { + "original": 282, + "effective": 282 + }, + "mmlu_medical_genetics": { + "original": 100, + "effective": 100 + }, + "mmlu_miscellaneous": { + "original": 783, + "effective": 783 + }, + "mmlu_clinical_knowledge": { + "original": 265, + "effective": 265 + }, + "mmlu_sociology": { + "original": 201, + "effective": 201 + }, + "mmlu_econometrics": { + "original": 114, + "effective": 114 + }, + "mmlu_high_school_government_and_politics": { + "original": 193, + "effective": 193 + }, + "mmlu_security_studies": { + "original": 245, + "effective": 245 + }, + "mmlu_high_school_microeconomics": { + "original": 238, + "effective": 238 + }, + "mmlu_human_sexuality": { + "original": 131, + "effective": 131 + }, + "mmlu_public_relations": { + "original": 110, + "effective": 110 + }, + "mmlu_us_foreign_policy": { + "original": 100, + "effective": 100 + }, + "mmlu_high_school_geography": { + "original": 198, + "effective": 198 + }, + "mmlu_high_school_macroeconomics": { + "original": 390, + "effective": 390 + }, + "mmlu_high_school_psychology": { + "original": 545, + "effective": 545 + }, + "mmlu_professional_psychology": { + "original": 612, + "effective": 612 + }, + "mmlu_formal_logic": { + "original": 126, + "effective": 126 + }, + "mmlu_logical_fallacies": { + "original": 163, + "effective": 163 + }, + "mmlu_jurisprudence": { + "original": 108, + "effective": 108 + }, + "mmlu_professional_law": { + "original": 1534, + "effective": 1534 + }, + "mmlu_high_school_european_history": { + "original": 165, + "effective": 165 + }, + "mmlu_moral_scenarios": { + "original": 895, + "effective": 895 + }, + "mmlu_moral_disputes": { + "original": 346, + "effective": 346 + }, + "mmlu_international_law": { + "original": 121, + "effective": 121 + }, + "mmlu_world_religions": { + "original": 171, + "effective": 171 + }, + "mmlu_high_school_us_history": { + "original": 204, + "effective": 204 + }, + "mmlu_prehistory": { + "original": 324, + "effective": 324 + }, + "mmlu_philosophy": { + "original": 311, + "effective": 311 + }, + "mmlu_high_school_world_history": { + "original": 237, + "effective": 237 + } + }, + "config": { + "model": "local-completions", + "model_args": "base_url=http://127.0.0.1:18300/v1/completions,model=qwen3-8b-teacher,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768", + "batch_size": 1, + "batch_sizes": [], + "device": null, + "use_cache": "/results/lm_cache", + "limit": null, + "downsampling_ratio": null, + "bootstrap_iters": 100000, + "gen_kwargs": { + "temperature": 1e-07, + "top_p": 0.9999999 + }, + "random_seed": 42, + "numpy_seed": 42, + "torch_seed": 42, + "fewshot_seed": 42 + }, + "git_hash": null, + "date": 1791474058.866205, + "pretty_env_info": "name 'get_pretty_env_info' is not defined", + "transformers_version": "4.50.3", + "lm_eval_version": "No package metadata was found for lm_eval", + "upper_git_hash": null, + "tokenizer_pad_token": [ + "<|endoftext|>", + "151643" + ], + "tokenizer_eos_token": [ + "<|im_end|>", + "151645" + ], + "tokenizer_bos_token": [ + null, + "None" + ], + "eot_token_id": 151645, + "max_length": 32767, + "task_hashes": { + "mmlu_college_mathematics": "33a226a6b1a6492f38387286d516c0941f5d8debbdfaa054b11fcda1cdb6dc1a", + "mmlu_college_biology": "14cbae605287fc3c9ff25c8f2b2175619e036287a2e217767382b5f48f044943", + "mmlu_anatomy": "9578a460fa25b65077e6da48a36e49d1e6b094807d65c9e5bf7e07ad0fbb6296", + "mmlu_electrical_engineering": "58857d23657299553045d3db7b7b0de43f9a078a2fd53902d0e59251ac497e9c", + "mmlu_high_school_chemistry": "7b7e216abbacc50a1613fc9e02f3f3dda2709059394b121e43f3ad534cadcf80", + "mmlu_college_computer_science": "c2355a53d39d8bf1fbec4ea4b0f2b50c3fe104f2b36556ecae1038b0c315816a", + "mmlu_high_school_statistics": "0598ccf63c944eeab0ba3ad2dcfbcaebddf4c5d124174071beb52e8a73f19e86", + "mmlu_abstract_algebra": "0e2f49943ac30da4eb2d7d2f5db73abb69da69d6988007197e6bdf138bfb3369", + "mmlu_college_chemistry": "20b91eb7e582696bf39216d1757ac477675fe6de640150d7ae29993234af04dc", + "mmlu_elementary_mathematics": "da1d3600f010ce76c00a62b991b7606ee7df0b2e82413ee1f1977440ddd8550d", + "mmlu_conceptual_physics": "d9a7c4fd2d286215624dffc2aaec8e0295a6353daf2300d3461537e3fb5425eb", + "mmlu_high_school_mathematics": "313c7a073f5b7339d283ad977ea74584374b6e0281898dfd89ff4bdfd2fb7b03", + "mmlu_college_physics": "ffe4456084dbfeb89e20d4b6138b97bbf415fd619691dbc7b38bfe5934b30e10", + "mmlu_machine_learning": "d02c202c0ad4c729890962d842ea70cad80faaae66a51a4c35c3685783ca06c1", + "mmlu_high_school_computer_science": "2db28b8942d29e448cb587b398788f3cba6693cfe20772102b5bb1907999abea", + "mmlu_high_school_biology": "cfa833fafa27f991b540ece0e0583e1ebedfcaa50d030891935b000b7757c1cb", + "mmlu_computer_security": "e122c4bc113d812e7e083e316b358cc7e16a0d1c2e6506480caebcf9d1e27ecd", + "mmlu_astronomy": "83e572d3948bb6dbfb66031ab4dcb5b70054a5bc110de1a0fbc324940f4c04da", + "mmlu_high_school_physics": "7780579caacd56f607968e81d5256d7d386095ab1d93b672047c5a5d8444ec1d", + "mmlu_management": "6345661bb012d1a72c5d34f39e692cf47c131807a3bdfff7bf07ee9c5149d29e", + "mmlu_nutrition": "468a945d5f4f35a2b1c974d7180d159d3e80a0611fa9160afee572df0dd69432", + "mmlu_business_ethics": "a0e93295030eaac8a5dd303dd1514b9eb25ae1807f55d000b876c0e8ddd5e1c8", + "mmlu_college_medicine": "bc04947722ee05b4d5051df09f0e50de44d4f49df44f5ff909dbe3264cbc2c53", + "mmlu_global_facts": "856fc8218f2f89f3e9abbe96070c530689a228cf5c9269c58bd03b4f0aa833b1", + "mmlu_virology": "67042eb89ab0d2a45f1d254dbb9da9de152fa1098d7740a05ac7ff8495b26534", + "mmlu_human_aging": "88802f2005a5c3c719ace0291d92034ef3c90ee1cf7488d5195f8c545c899d4d", + "mmlu_marketing": "2018f5181b9a272ca8a4f65858e0d12a1bf87792934fc483022e9d1bc7e05997", + "mmlu_professional_medicine": "ba4af62ac5d26864b9e6339c6cbd16cbabcb5fbcf00366c02dd15da73c28daf9", + "mmlu_professional_accounting": "654d1dafcb0ff52e603404c1158924e92c5067cda9df2429eb0777c99c648995", + "mmlu_medical_genetics": "8819bc558f8142680673b743cf8e3681ea6856ece1e3895c24fc0f77a60da7c4", + "mmlu_miscellaneous": "22548428bd482f372a4fba10d2a0c0a939faddd7e9781701a74358145211a660", + "mmlu_clinical_knowledge": "a3dba4354de80aa98c7db318923aedb8b78c02014a2be97c4071453ba619b8dc", + "mmlu_sociology": "65917e6a3b0027bdb12703440e17afc8f155c15087a9df249e0061d84381e0d2", + "mmlu_econometrics": "905a5e83b9a8173ef7345d14ac535c91c78c34b076314a2deb52cb2e6138502c", + "mmlu_high_school_government_and_politics": "bd59fed7fa1dcf3e39a65a7ea71cc81573c4fb28bcd942cbc8d8d9afbe0e9d4c", + "mmlu_security_studies": "e7c1c4bd8669cc22b7a907cd3584c5caf80b82772d9741a3680563b4fb564084", + "mmlu_high_school_microeconomics": "705c7c5543f9987bdf14f7b39ddae1910e9097d0f491bb98a536140175d94732", + "mmlu_human_sexuality": "9b20e343f77622c458fe394d257924eb8895d07ddd98b1e3dbaf366563f77cbb", + "mmlu_public_relations": "355105913812b28cc46607bf6710a92dcb346c9d35f489eb84e388de03390b32", + "mmlu_us_foreign_policy": "1c1d39c546896fb1b9a513c5db8233ea4066c3ec79d8b85c2ab7944fa79ca8c5", + "mmlu_high_school_geography": "30d201a5bfcbaec8d0e74f58ddd23bd58204b1d99b9211e6a531d8b6e84eab24", + "mmlu_high_school_macroeconomics": "50ee63c5ac58c48f651464b5fd6cbbe0c8fa6002dd480fa9634ba630d9dc0417", + "mmlu_high_school_psychology": "e62d1ebd73c032fd0c3b99eaf20e7b77b13ae49e4fc506d7e69f9ae15e485a4c", + "mmlu_professional_psychology": "70ec543a94dccdb5e82f3ac5f006de88a1352fb86b2a2c68d496529cdd1f5e0c", + "mmlu_formal_logic": "646ee3f6abf70a7d0efd03cdc0fbebb9f2e8f40e3b5563e53a9d0a7a291bc2a7", + "mmlu_logical_fallacies": "f4aa9887846d578f13ca7420b65369c344053cabe092607a16237dfeef07a1ba", + "mmlu_jurisprudence": "325cb19fa404379425d7add71b29e2e4cd3428c08a25f5cc718fd87e79ccd2bb", + "mmlu_professional_law": "0e0e2459e1f45df128cf8460ebdea4da6cc99a9fbd9cb99fb5ebeee78575800e", + "mmlu_high_school_european_history": "8c8ff9bcf9924cd457cc5b67c5f6a21925c09aadba42e646c088f2b99ba4e0e2", + "mmlu_moral_scenarios": "83a877f6af97d85f853c38196827d751dc8f33db4253da17965fde0c5e3554e1", + "mmlu_moral_disputes": "4d3e2d32e558de3c5eb0369e9a20280fda45fe04eeedf3356ad820e352293720", + "mmlu_international_law": "144d9e8693486d00f5cac911f3fbfbacdba9ac7c10340d50061262d9c3fdff9d", + "mmlu_world_religions": "8ad90daa76079af018b36cb86143468d22e34a9cd3273a3c48ce4c4648f886dc", + "mmlu_high_school_us_history": "b5fdc459ac365a794c663a17cf8a34a702ea87eaa1a8b40fe49e8196ef88232c", + "mmlu_prehistory": "4aa1d25e8211294e34dead164bbde415d42feb245a4716a2471d929f81846f04", + "mmlu_philosophy": "c61151b7b50a37e90635e8b219018a35533a65a2a684542ead156b9f1020ffc1", + "mmlu_high_school_world_history": "59ebbb27916baa0f2c6fa8b064ecbb042b43f6d492bbe18f2a78a63d40bf3dd0" + }, + "model_source": "local-completions", + "model_name": "qwen3-8b-teacher", + "model_name_sanitized": "qwen3-8b-teacher", + "system_instruction": null, + "system_instruction_sha": null, + "fewshot_as_multiturn": false, + "chat_template": null, + "chat_template_sha": null, + "start_time": 1506881.406094961, + "end_time": 1509498.154492089, + "total_evaluation_time_seconds": "2616.748397127958" +} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml new file mode 100644 index 00000000000..50e0077b769 --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml @@ -0,0 +1,57 @@ +defaults: +- execution: slurm/default +- deployment: vllm +- ../../../nel_configs/mmlu@_global_ +- _self_ +execution: + hostname: ${oc.env:NEL_SLURM_HOST} + username: ${oc.env:SLURM_USER} + account: ${oc.env:NEL_SLURM_ACCOUNT} + partition: ${oc.env:NEL_SLURM_PARTITION} + output_dir: ${oc.env:NEL_OUTPUT_DIR} + num_nodes: 1 + ntasks_per_node: 1 + gres: gpu:1 + walltime: 04:00:00 + sbatch_extra_flags: + exclusive: false + mem: 64G + mounts: + mount_home: false + deployment: + /path/to/hf-cache: /hf-cache + evaluation: + /path/to/hf-cache: /hf-cache +deployment: + image: vllm/vllm-openai:v0.28.0-cu129 + checkpoint_path: ${oc.env:QWEN3_HF_CACHE}/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + served_model_name: qwen3-8b-teacher + tensor_parallel_size: 1 + data_parallel_size: 1 + env_vars: + HF_HOME: lit:/hf-cache + command: vllm serve /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} + --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len + 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype + auto + tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + port: 18100 +evaluation: + tasks: + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" + nemo_evaluator_config: + config: + params: + parallelism: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json new file mode 100644 index 00000000000..cd67a2972b2 --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json @@ -0,0 +1,136 @@ +{ + "results": { + "wikitext": { + "alias": "wikitext", + "word_perplexity,none": 13.484664501311268, + "word_perplexity_stderr,none": "N/A", + "byte_perplexity,none": 1.6266186785114045, + "byte_perplexity_stderr,none": "N/A", + "bits_per_byte,none": 0.7018760856152558, + "bits_per_byte_stderr,none": "N/A" + } + }, + "group_subtasks": { + "wikitext": [] + }, + "configs": { + "wikitext": { + "task": "wikitext", + "dataset_path": "EleutherAI/wikitext_document_level", + "dataset_name": "wikitext-2-raw-v1", + "dataset_kwargs": { + "trust_remote_code": true + }, + "training_split": "train", + "validation_split": "validation", + "test_split": "test", + "doc_to_text": "", + "doc_to_target": "def wikitext_detokenizer(doc):\n string = doc[\"page\"]\n # contractions\n string = string.replace(\"s '\", \"s'\")\n string = re.sub(r\"/' [0-9]/\", r\"/'[0-9]/\", string)\n # number separators\n string = string.replace(\" @-@ \", \"-\")\n string = string.replace(\" @,@ \", \",\")\n string = string.replace(\" @.@ \", \".\")\n # punctuation\n string = string.replace(\" : \", \": \")\n string = string.replace(\" ; \", \"; \")\n string = string.replace(\" . \", \". \")\n string = string.replace(\" ! \", \"! \")\n string = string.replace(\" ? \", \"? \")\n string = string.replace(\" , \", \", \")\n # double brackets\n string = re.sub(r\"\\(\\s*([^\\)]*?)\\s*\\)\", r\"(\\1)\", string)\n string = re.sub(r\"\\[\\s*([^\\]]*?)\\s*\\]\", r\"[\\1]\", string)\n string = re.sub(r\"{\\s*([^}]*?)\\s*}\", r\"{\\1}\", string)\n string = re.sub(r\"\\\"\\s*([^\\\"]*?)\\s*\\\"\", r'\"\\1\"', string)\n string = re.sub(r\"'\\s*([^']*?)\\s*'\", r\"'\\1'\", string)\n # miscellaneous\n string = string.replace(\"= = = =\", \"====\")\n string = string.replace(\"= = =\", \"===\")\n string = string.replace(\"= =\", \"==\")\n string = string.replace(\" \" + chr(176) + \" \", chr(176))\n string = string.replace(\" \\n\", \"\\n\")\n string = string.replace(\"\\n \", \"\\n\")\n string = string.replace(\" N \", \" 1 \")\n string = string.replace(\" 's\", \"'s\")\n\n return string\n", + "unsafe_code": false, + "process_results": "def process_results(doc, results):\n (loglikelihood,) = results\n # IMPORTANT: wikitext counts number of words in *original doc before detokenization*\n _words = len(re.split(r\"\\s+\", doc[\"page\"]))\n _bytes = len(doc[\"page\"].encode(\"utf-8\"))\n return {\n \"word_perplexity\": (loglikelihood, _words),\n \"byte_perplexity\": (loglikelihood, _bytes),\n \"bits_per_byte\": (loglikelihood, _bytes),\n }\n", + "description": "", + "target_delimiter": " ", + "fewshot_delimiter": "\n\n", + "num_fewshot": 0, + "metric_list": [ + { + "metric": "word_perplexity" + }, + { + "metric": "byte_perplexity" + }, + { + "metric": "bits_per_byte" + } + ], + "output_type": "loglikelihood_rolling", + "repeats": 1, + "should_decontaminate": true, + "doc_to_decontamination_query": "{{page}}", + "metadata": { + "version": 2.0, + "base_url": "http://localhost:3826", + "model": "qwen3-8b-teacher", + "tokenized_requests": true, + "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", + "tokenizer_backend": "huggingface", + "num_concurrent": 2, + "timeout": 300, + "max_retries": 2, + "stream": false + } + } + }, + "versions": { + "wikitext": 2.0 + }, + "n-shot": { + "wikitext": 0 + }, + "higher_is_better": { + "wikitext": { + "word_perplexity": false, + "byte_perplexity": false, + "bits_per_byte": false + } + }, + "n-samples": { + "wikitext": { + "original": 62, + "effective": 62 + } + }, + "config": { + "model": "local-completions", + "model_args": "base_url=http://localhost:3826,model=qwen3-8b-teacher,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=2,timeout=300,max_retries=2,stream=False", + "batch_size": 1, + "batch_sizes": [], + "device": null, + "use_cache": "/results/lm_cache", + "limit": null, + "downsampling_ratio": null, + "bootstrap_iters": 100000, + "gen_kwargs": { + "temperature": 1e-07, + "top_p": 0.9999999 + }, + "random_seed": 42, + "numpy_seed": 42, + "torch_seed": 42, + "fewshot_seed": 42 + }, + "git_hash": null, + "date": 1791469340.7379298, + "pretty_env_info": "name 'get_pretty_env_info' is not defined", + "transformers_version": "4.50.3", + "lm_eval_version": "No package metadata was found for lm_eval", + "upper_git_hash": null, + "tokenizer_pad_token": [ + "<|endoftext|>", + "151643" + ], + "tokenizer_eos_token": [ + "<|im_end|>", + "151645" + ], + "tokenizer_bos_token": [ + null, + "None" + ], + "eot_token_id": 151645, + "max_length": 2047, + "task_hashes": { + "wikitext": "233109c41076287b5ba5a338ab7b085728acacf36d237a7ab99f34fcda161641" + }, + "model_source": "local-completions", + "model_name": "qwen3-8b-teacher", + "model_name_sanitized": "qwen3-8b-teacher", + "system_instruction": null, + "system_instruction_sha": null, + "fewshot_as_multiturn": false, + "chat_template": null, + "chat_template_sha": null, + "start_time": 309831.550427795, + "end_time": 309851.959472321, + "total_evaluation_time_seconds": "20.409044525993522" +} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml new file mode 100644 index 00000000000..967d72af982 --- /dev/null +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml @@ -0,0 +1,40 @@ +# Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. +defaults: +- execution: slurm/default +- deployment: vllm +- ../../../nel_configs/wikitext@_global_ +- _self_ +execution: + hostname: ${oc.env:NEL_SLURM_HOST} + username: ${oc.env:SLURM_USER} + account: ${oc.env:NEL_SLURM_ACCOUNT} + partition: ${oc.env:NEL_SLURM_PARTITION} + output_dir: ${oc.env:NEL_OUTPUT_DIR} + num_nodes: 1 + ntasks_per_node: 1 + gres: gpu:1 + walltime: 04:00:00 + sbatch_extra_flags: + exclusive: false + mem: 64G + mounts: + mount_home: false + deployment: + /path/to/hf-cache: /hf-cache + evaluation: + /path/to/hf-cache: /hf-cache +deployment: + image: vllm/vllm-openai:v0.28.0-cu129 + checkpoint_path: ${oc.env:QWEN3_HF_CACHE}/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + served_model_name: qwen3-8b-teacher + tensor_parallel_size: 1 + data_parallel_size: 1 + env_vars: + HF_HOME: lit:/hf-cache + command: vllm serve /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} + --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len + 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype + auto + tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + port: 18102 From 62b315996a5d940746f08cfadce2ef1d3ce92e32 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Mon, 12 Oct 2026 10:28:20 -0700 Subject: [PATCH 02/19] fix formatting Signed-off-by: Daniel Korzekwa --- .../llm_qat/evaluation/nel_configs/mmlu.yaml | 34 ++++++++-------- .../evaluation/nel_configs/wikitext.yaml | 34 ++++++++-------- .../mmlu_full/nel_config.yaml | 38 +++++++++--------- .../wikitext_full/nel_config.yaml | 14 +++---- .../mmlu_full/nel_config.yaml | 38 +++++++++--------- .../wikitext_full/nel_config.yaml | 14 +++---- .../mmlu_full/nel_config.yaml | 40 +++++++++---------- .../wikitext_full/nel_config.yaml | 16 ++++---- 8 files changed, 108 insertions(+), 120 deletions(-) diff --git a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml index 8d3a7398fc9..0651f8f377d 100644 --- a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml +++ b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml @@ -7,13 +7,13 @@ evaluation: parallelism: 2 request_timeout: 300 max_retries: 2 - temperature: null - top_p: null - max_new_tokens: null + temperature: + top_p: + max_new_tokens: task: mmlu target: api_endpoint: - api_key_name: null + api_key_name: adapter_config: use_reasoning: false use_request_logging: true @@ -21,16 +21,16 @@ evaluation: max_logged_requests: 2 max_logged_responses: 2 tasks: - - name: lm-evaluation-harness.mmlu - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - task: ${evaluation.nemo_evaluator_config.config.params.task} - extra: - num_fewshot: 5 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + task: ${evaluation.nemo_evaluator_config.config.params.task} + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 diff --git a/examples/llm_qat/evaluation/nel_configs/wikitext.yaml b/examples/llm_qat/evaluation/nel_configs/wikitext.yaml index b31f03a4ad4..455a7aeffac 100644 --- a/examples/llm_qat/evaluation/nel_configs/wikitext.yaml +++ b/examples/llm_qat/evaluation/nel_configs/wikitext.yaml @@ -7,13 +7,13 @@ evaluation: parallelism: 2 request_timeout: 300 max_retries: 2 - temperature: null - top_p: null - max_new_tokens: null + temperature: + top_p: + max_new_tokens: task: wikitext target: api_endpoint: - api_key_name: null + api_key_name: adapter_config: use_reasoning: false use_request_logging: true @@ -21,16 +21,16 @@ evaluation: max_logged_requests: 2 max_logged_responses: 2 tasks: - - name: lm-evaluation-harness.wikitext - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - task: ${evaluation.nemo_evaluator_config.config.params.task} - extra: - num_fewshot: 0 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 + - name: lm-evaluation-harness.wikitext + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + task: ${evaluation.nemo_evaluator_config.config.params.task} + extra: + num_fewshot: 0 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml index 63b378d1add..1ac23d2ed40 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml @@ -1,8 +1,8 @@ defaults: -- execution: slurm/default -- deployment: vllm -- ../../../nel_configs/mmlu@_global_ -- _self_ + - execution: slurm/default + - deployment: vllm + - ../../../nel_configs/mmlu@_global_ + - _self_ execution: hostname: ${oc.env:NEL_SLURM_HOST} username: ${oc.env:SLURM_USER} @@ -32,26 +32,24 @@ deployment: data_parallel_size: 1 env_vars: HF_HOME: lit:/hf-cache - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} - --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization - 0.85 --seed 42 --kv-cache-dtype auto + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 port: 18104 evaluation: tasks: - - name: lm-evaluation-harness.mmlu - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - extra: - num_fewshot: 5 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" nemo_evaluator_config: config: params: diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml index 1e6ee3aa8ce..33f31c4e269 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml @@ -1,9 +1,9 @@ # Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: -- execution: slurm/default -- deployment: vllm -- ../../../nel_configs/wikitext@_global_ -- _self_ + - execution: slurm/default + - deployment: vllm + - ../../../nel_configs/wikitext@_global_ + - _self_ execution: hostname: ${oc.env:NEL_SLURM_HOST} username: ${oc.env:SLURM_USER} @@ -33,9 +33,7 @@ deployment: data_parallel_size: 1 env_vars: HF_HOME: lit:/hf-cache - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} - --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization - 0.85 --seed 42 --kv-cache-dtype auto + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 port: 18106 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml index f86d4020b40..0615a9dba11 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml @@ -1,8 +1,8 @@ defaults: -- execution: slurm/default -- deployment: vllm -- ../../../nel_configs/mmlu@_global_ -- _self_ + - execution: slurm/default + - deployment: vllm + - ../../../nel_configs/mmlu@_global_ + - _self_ execution: hostname: ${oc.env:NEL_SLURM_HOST} username: ${oc.env:SLURM_USER} @@ -32,26 +32,24 @@ deployment: data_parallel_size: 1 env_vars: HF_HOME: lit:/hf-cache - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} - --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization - 0.85 --seed 42 --kv-cache-dtype auto + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 port: 18108 evaluation: tasks: - - name: lm-evaluation-harness.mmlu - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - extra: - num_fewshot: 5 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" nemo_evaluator_config: config: params: diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml index 13f58862318..bf394beddd6 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml @@ -1,9 +1,9 @@ # Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: -- execution: slurm/default -- deployment: vllm -- ../../../nel_configs/wikitext@_global_ -- _self_ + - execution: slurm/default + - deployment: vllm + - ../../../nel_configs/wikitext@_global_ + - _self_ execution: hostname: ${oc.env:NEL_SLURM_HOST} username: ${oc.env:SLURM_USER} @@ -33,9 +33,7 @@ deployment: data_parallel_size: 1 env_vars: HF_HOME: lit:/hf-cache - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} - --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization - 0.85 --seed 42 --kv-cache-dtype auto + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 port: 18110 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml index 50e0077b769..60aa86cddc7 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml @@ -1,8 +1,8 @@ defaults: -- execution: slurm/default -- deployment: vllm -- ../../../nel_configs/mmlu@_global_ -- _self_ + - execution: slurm/default + - deployment: vllm + - ../../../nel_configs/mmlu@_global_ + - _self_ execution: hostname: ${oc.env:NEL_SLURM_HOST} username: ${oc.env:SLURM_USER} @@ -30,27 +30,25 @@ deployment: data_parallel_size: 1 env_vars: HF_HOME: lit:/hf-cache - command: vllm serve /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 - --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} - --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len - 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype - auto + command: vllm serve /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 --served-model-name ${deployment.served_model_name} + --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization + 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 port: 18100 evaluation: tasks: - - name: lm-evaluation-harness.mmlu - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - extra: - num_fewshot: 5 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" nemo_evaluator_config: config: params: diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml index 967d72af982..3d0da2e88bb 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml @@ -1,9 +1,9 @@ # Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: -- execution: slurm/default -- deployment: vllm -- ../../../nel_configs/wikitext@_global_ -- _self_ + - execution: slurm/default + - deployment: vllm + - ../../../nel_configs/wikitext@_global_ + - _self_ execution: hostname: ${oc.env:NEL_SLURM_HOST} username: ${oc.env:SLURM_USER} @@ -31,10 +31,8 @@ deployment: data_parallel_size: 1 env_vars: HF_HOME: lit:/hf-cache - command: vllm serve /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 - --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} - --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len - 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype - auto + command: vllm serve /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 --served-model-name ${deployment.served_model_name} + --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization + 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 port: 18102 From b7e93cc289a89036345109b98e21b11e45ba7530 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 00:24:27 -0700 Subject: [PATCH 03/19] improving text clarity Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index 29a948e49b8..f97fc53a2f6 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -1,9 +1,9 @@ # Evaluation -NeMo Evaluator Launcher (NEL) configurations and evaluation artifacts for the [Hugging Face QAT/QAD tutorial](../README.md). +NeMo Evaluator Launcher (NEL) configurations and evaluation results for the [Hugging Face QAT/QAD tutorial](../README.md). - `nel_configs/`: shared benchmark recipes. -- `results///`: each evaluation's `nel_config.yaml` and original `lm_eval_results_*.json` generated by lm-evaluation-harness. Per-example samples are not stored here. +- `results///`: each evaluation's config `nel_config.yaml` and original results `lm_eval_results_*.json` generated by lm-evaluation-harness. Per-example samples are not stored here. ## Results From f59839602d4b298938a9ddbcdbcb327f8ba98691 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 00:27:00 -0700 Subject: [PATCH 04/19] Improve eval results formatting Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index f97fc53a2f6..f2fcf7e71da 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -1,10 +1,5 @@ # Evaluation -NeMo Evaluator Launcher (NEL) configurations and evaluation results for the [Hugging Face QAT/QAD tutorial](../README.md). - -- `nel_configs/`: shared benchmark recipes. -- `results///`: each evaluation's config `nel_config.yaml` and original results `lm_eval_results_*.json` generated by lm-evaluation-harness. Per-example samples are not stored here. - ## Results Qwen3-8B was evaluated using NEL with vLLM on one H100 per evaluation. QAD starts from the PTQ checkpoint and uses the BF16 model as teacher. @@ -16,3 +11,10 @@ Qwen3-8B was evaluated using NEL with vLLM on one H100 per evaluation. QAD start | PTQ → QAD (NVFP4) | NVFP4 weights / BF16 activations | [73.76%](results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json) | [13.684](results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json) | QAD improves both metrics over PTQ but remains below the BF16 teacher. The quantized checkpoints use Marlin's W4A16 fallback on H100; these results do not measure native NVFP4 W4A4 execution on Blackwell. KV cache is BF16 for all checkpoints. + +## File structure + +NeMo Evaluator Launcher (NEL) configurations and evaluation results for the [Hugging Face QAT/QAD tutorial](../README.md). + +- `nel_configs/`: shared benchmark recipes. +- `results///`: each evaluation's config `nel_config.yaml` and original results `lm_eval_results_*.json` generated by lm-evaluation-harness. Per-example samples are not stored here. From 8e22e305654a15ed483f169b9d75694324aefbb8 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 00:30:07 -0700 Subject: [PATCH 05/19] Remove raw lm_eval results Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 8 +- ...al_results_2026-10-08T10-29-11.704256.json | 4030 ----------------- ...al_results_2026-10-08T07-23-16.887993.json | 136 - ...al_results_2026-10-08T11-35-03.360012.json | 4030 ----------------- ...al_results_2026-10-08T07-23-24.097523.json | 136 - ...al_results_2026-10-08T09-24-34.263592.json | 4030 ----------------- ...al_results_2026-10-08T07-22-39.797863.json | 136 - 7 files changed, 4 insertions(+), 12502 deletions(-) delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index f2fcf7e71da..f76bbdb0df0 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -6,9 +6,9 @@ Qwen3-8B was evaluated using NEL with vLLM on one H100 per evaluation. QAD start | Checkpoint | Deployed precision | MMLU accuracy ↑ | WikiText-2 perplexity ↓ | |---|---|---:|---:| -| Teacher (BF16) | BF16 | [74.94%](results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json) | [13.485](results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json) | -| PTQ (NVFP4) | NVFP4 weights / BF16 activations | [73.59%](results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json) | [13.883](results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json) | -| PTQ → QAD (NVFP4) | NVFP4 weights / BF16 activations | [73.76%](results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json) | [13.684](results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json) | +| Teacher (BF16) | BF16 | 74.94% | 13.485 | +| PTQ (NVFP4) | NVFP4 weights / BF16 activations | 73.59% | 13.883 | +| PTQ → QAD (NVFP4) | NVFP4 weights / BF16 activations | 73.76% | 13.684 | QAD improves both metrics over PTQ but remains below the BF16 teacher. The quantized checkpoints use Marlin's W4A16 fallback on H100; these results do not measure native NVFP4 W4A4 execution on Blackwell. KV cache is BF16 for all checkpoints. @@ -17,4 +17,4 @@ QAD improves both metrics over PTQ but remains below the BF16 teacher. The quant NeMo Evaluator Launcher (NEL) configurations and evaluation results for the [Hugging Face QAT/QAD tutorial](../README.md). - `nel_configs/`: shared benchmark recipes. -- `results///`: each evaluation's config `nel_config.yaml` and original results `lm_eval_results_*.json` generated by lm-evaluation-harness. Per-example samples are not stored here. +- `results///`: each evaluation's config `nel_config.yaml`. diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json deleted file mode 100644 index b151d52a0af..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/lm_eval_results_2026-10-08T10-29-11.704256.json +++ /dev/null @@ -1,4030 +0,0 @@ -{ - "results": { - "mmlu": { - "acc,none": 0.7358638370602478, - "acc_stderr,none": 0.003529928328054296, - "alias": "mmlu" - }, - "mmlu_humanities": { - "acc,none": 0.6463336875664187, - "acc_stderr,none": 0.006627653147654618, - "alias": " - humanities" - }, - "mmlu_formal_logic": { - "alias": " - formal_logic", - "acc,none": 0.6666666666666666, - "acc_stderr,none": 0.04216370213557839 - }, - "mmlu_high_school_european_history": { - "alias": " - high_school_european_history", - "acc,none": 0.8545454545454545, - "acc_stderr,none": 0.02753019635506653 - }, - "mmlu_high_school_us_history": { - "alias": " - high_school_us_history", - "acc,none": 0.8529411764705882, - "acc_stderr,none": 0.024857478080250486 - }, - "mmlu_high_school_world_history": { - "alias": " - high_school_world_history", - "acc,none": 0.8565400843881856, - "acc_stderr,none": 0.02281829182101699 - }, - "mmlu_international_law": { - "alias": " - international_law", - "acc,none": 0.7355371900826446, - "acc_stderr,none": 0.04026187527591209 - }, - "mmlu_jurisprudence": { - "alias": " - jurisprudence", - "acc,none": 0.8333333333333334, - "acc_stderr,none": 0.03602814176392641 - }, - "mmlu_logical_fallacies": { - "alias": " - logical_fallacies", - "acc,none": 0.803680981595092, - "acc_stderr,none": 0.031207970394709197 - }, - "mmlu_moral_disputes": { - "alias": " - moral_disputes", - "acc,none": 0.7225433526011561, - "acc_stderr,none": 0.024105712607754307 - }, - "mmlu_moral_scenarios": { - "alias": " - moral_scenarios", - "acc,none": 0.5385474860335195, - "acc_stderr,none": 0.01667273126755234 - }, - "mmlu_philosophy": { - "alias": " - philosophy", - "acc,none": 0.7427652733118971, - "acc_stderr,none": 0.0248261712892509 - }, - "mmlu_prehistory": { - "alias": " - prehistory", - "acc,none": 0.8271604938271605, - "acc_stderr,none": 0.02103851777015737 - }, - "mmlu_professional_law": { - "alias": " - professional_law", - "acc,none": 0.49022164276401564, - "acc_stderr,none": 0.012767793787729296 - }, - "mmlu_world_religions": { - "alias": " - world_religions", - "acc,none": 0.8538011695906432, - "acc_stderr,none": 0.02709729011807083 - }, - "mmlu_other": { - "acc,none": 0.7740585774058577, - "acc_stderr,none": 0.00721392999736686, - "alias": " - other" - }, - "mmlu_business_ethics": { - "alias": " - business_ethics", - "acc,none": 0.77, - "acc_stderr,none": 0.042295258468165065 - }, - "mmlu_clinical_knowledge": { - "alias": " - clinical_knowledge", - "acc,none": 0.8037735849056604, - "acc_stderr,none": 0.0244423881311008 - }, - "mmlu_college_medicine": { - "alias": " - college_medicine", - "acc,none": 0.7803468208092486, - "acc_stderr,none": 0.03156809362703173 - }, - "mmlu_global_facts": { - "alias": " - global_facts", - "acc,none": 0.44, - "acc_stderr,none": 0.049888765156985884 - }, - "mmlu_human_aging": { - "alias": " - human_aging", - "acc,none": 0.7533632286995515, - "acc_stderr,none": 0.028930413120910853 - }, - "mmlu_management": { - "alias": " - management", - "acc,none": 0.8932038834951457, - "acc_stderr,none": 0.030581088928331335 - }, - "mmlu_marketing": { - "alias": " - marketing", - "acc,none": 0.9273504273504274, - "acc_stderr,none": 0.017004368568132394 - }, - "mmlu_medical_genetics": { - "alias": " - medical_genetics", - "acc,none": 0.82, - "acc_stderr,none": 0.03861229196653691 - }, - "mmlu_miscellaneous": { - "alias": " - miscellaneous", - "acc,none": 0.8378033205619413, - "acc_stderr,none": 0.01318222261672081 - }, - "mmlu_nutrition": { - "alias": " - nutrition", - "acc,none": 0.8169934640522876, - "acc_stderr,none": 0.022140767512880986 - }, - "mmlu_professional_accounting": { - "alias": " - professional_accounting", - "acc,none": 0.5851063829787234, - "acc_stderr,none": 0.02939223658461248 - }, - "mmlu_professional_medicine": { - "alias": " - professional_medicine", - "acc,none": 0.8051470588235294, - "acc_stderr,none": 0.024060599423487473 - }, - "mmlu_virology": { - "alias": " - virology", - "acc,none": 0.5240963855421686, - "acc_stderr,none": 0.03887971849597267 - }, - "mmlu_social_sciences": { - "acc,none": 0.8290542736431589, - "acc_stderr,none": 0.00666849172124334, - "alias": " - social sciences" - }, - "mmlu_econometrics": { - "alias": " - econometrics", - "acc,none": 0.6491228070175439, - "acc_stderr,none": 0.044895393502707 - }, - "mmlu_high_school_geography": { - "alias": " - high_school_geography", - "acc,none": 0.8636363636363636, - "acc_stderr,none": 0.02445015597318985 - }, - "mmlu_high_school_government_and_politics": { - "alias": " - high_school_government_and_politics", - "acc,none": 0.9222797927461139, - "acc_stderr,none": 0.01932180555722318 - }, - "mmlu_high_school_macroeconomics": { - "alias": " - high_school_macroeconomics", - "acc,none": 0.7948717948717948, - "acc_stderr,none": 0.02047323317355194 - }, - "mmlu_high_school_microeconomics": { - "alias": " - high_school_microeconomics", - "acc,none": 0.9159663865546218, - "acc_stderr,none": 0.01802154889879824 - }, - "mmlu_high_school_psychology": { - "alias": " - high_school_psychology", - "acc,none": 0.9045871559633027, - "acc_stderr,none": 0.012595899282335761 - }, - "mmlu_human_sexuality": { - "alias": " - human_sexuality", - "acc,none": 0.8854961832061069, - "acc_stderr,none": 0.027927473753597453 - }, - "mmlu_professional_psychology": { - "alias": " - professional_psychology", - "acc,none": 0.7630718954248366, - "acc_stderr,none": 0.017201662169789703 - }, - "mmlu_public_relations": { - "alias": " - public_relations", - "acc,none": 0.6818181818181818, - "acc_stderr,none": 0.044612721759105065 - }, - "mmlu_security_studies": { - "alias": " - security_studies", - "acc,none": 0.8, - "acc_stderr,none": 0.02560737598657919 - }, - "mmlu_sociology": { - "alias": " - sociology", - "acc,none": 0.845771144278607, - "acc_stderr,none": 0.025538433368578303 - }, - "mmlu_us_foreign_policy": { - "alias": " - us_foreign_policy", - "acc,none": 0.83, - "acc_stderr,none": 0.03775251680686369 - }, - "mmlu_stem": { - "acc,none": 0.7408816999682841, - "acc_stderr,none": 0.007511790624977365, - "alias": " - stem" - }, - "mmlu_abstract_algebra": { - "alias": " - abstract_algebra", - "acc,none": 0.56, - "acc_stderr,none": 0.049888765156985884 - }, - "mmlu_anatomy": { - "alias": " - anatomy", - "acc,none": 0.6888888888888889, - "acc_stderr,none": 0.039992628766177166 - }, - "mmlu_astronomy": { - "alias": " - astronomy", - "acc,none": 0.8881578947368421, - "acc_stderr,none": 0.025648341251693615 - }, - "mmlu_college_biology": { - "alias": " - college_biology", - "acc,none": 0.8680555555555556, - "acc_stderr,none": 0.02830096838204442 - }, - "mmlu_college_chemistry": { - "alias": " - college_chemistry", - "acc,none": 0.59, - "acc_stderr,none": 0.04943110704237104 - }, - "mmlu_college_computer_science": { - "alias": " - college_computer_science", - "acc,none": 0.72, - "acc_stderr,none": 0.045126085985421296 - }, - "mmlu_college_mathematics": { - "alias": " - college_mathematics", - "acc,none": 0.59, - "acc_stderr,none": 0.04943110704237104 - }, - "mmlu_college_physics": { - "alias": " - college_physics", - "acc,none": 0.5980392156862745, - "acc_stderr,none": 0.04878608714466993 - }, - "mmlu_computer_security": { - "alias": " - computer_security", - "acc,none": 0.81, - "acc_stderr,none": 0.039427724440366255 - }, - "mmlu_conceptual_physics": { - "alias": " - conceptual_physics", - "acc,none": 0.851063829787234, - "acc_stderr,none": 0.02327411784801041 - }, - "mmlu_electrical_engineering": { - "alias": " - electrical_engineering", - "acc,none": 0.8068965517241379, - "acc_stderr,none": 0.03289445522127399 - }, - "mmlu_elementary_mathematics": { - "alias": " - elementary_mathematics", - "acc,none": 0.753968253968254, - "acc_stderr,none": 0.022182037202948424 - }, - "mmlu_high_school_biology": { - "alias": " - high_school_biology", - "acc,none": 0.9129032258064517, - "acc_stderr,none": 0.016041100741696738 - }, - "mmlu_high_school_chemistry": { - "alias": " - high_school_chemistry", - "acc,none": 0.729064039408867, - "acc_stderr,none": 0.03127090713297701 - }, - "mmlu_high_school_computer_science": { - "alias": " - high_school_computer_science", - "acc,none": 0.86, - "acc_stderr,none": 0.0348735088019777 - }, - "mmlu_high_school_mathematics": { - "alias": " - high_school_mathematics", - "acc,none": 0.5, - "acc_stderr,none": 0.030485538042484616 - }, - "mmlu_high_school_physics": { - "alias": " - high_school_physics", - "acc,none": 0.695364238410596, - "acc_stderr,none": 0.03757949922943339 - }, - "mmlu_high_school_statistics": { - "alias": " - high_school_statistics", - "acc,none": 0.7592592592592593, - "acc_stderr,none": 0.029157522184605607 - }, - "mmlu_machine_learning": { - "alias": " - machine_learning", - "acc,none": 0.6428571428571429, - "acc_stderr,none": 0.045479609997643805 - } - }, - "groups": { - "mmlu": { - "acc,none": 0.7358638370602478, - "acc_stderr,none": 0.003529928328054296, - "alias": "mmlu" - }, - "mmlu_humanities": { - "acc,none": 0.6463336875664187, - "acc_stderr,none": 0.006627653147654618, - "alias": " - humanities" - }, - "mmlu_other": { - "acc,none": 0.7740585774058577, - "acc_stderr,none": 0.00721392999736686, - "alias": " - other" - }, - "mmlu_social_sciences": { - "acc,none": 0.8290542736431589, - "acc_stderr,none": 0.00666849172124334, - "alias": " - social sciences" - }, - "mmlu_stem": { - "acc,none": 0.7408816999682841, - "acc_stderr,none": 0.007511790624977365, - "alias": " - stem" - } - }, - "group_subtasks": { - "mmlu_humanities": [ - "mmlu_moral_scenarios", - "mmlu_high_school_us_history", - "mmlu_formal_logic", - "mmlu_international_law", - "mmlu_jurisprudence", - "mmlu_logical_fallacies", - "mmlu_philosophy", - "mmlu_prehistory", - "mmlu_high_school_european_history", - "mmlu_professional_law", - "mmlu_world_religions", - "mmlu_moral_disputes", - "mmlu_high_school_world_history" - ], - "mmlu_social_sciences": [ - "mmlu_public_relations", - "mmlu_human_sexuality", - "mmlu_us_foreign_policy", - "mmlu_high_school_microeconomics", - "mmlu_high_school_macroeconomics", - "mmlu_high_school_psychology", - "mmlu_econometrics", - "mmlu_sociology", - "mmlu_professional_psychology", - "mmlu_high_school_government_and_politics", - "mmlu_security_studies", - "mmlu_high_school_geography" - ], - "mmlu_other": [ - "mmlu_business_ethics", - "mmlu_marketing", - "mmlu_human_aging", - "mmlu_college_medicine", - "mmlu_medical_genetics", - "mmlu_professional_accounting", - "mmlu_professional_medicine", - "mmlu_global_facts", - "mmlu_miscellaneous", - "mmlu_nutrition", - "mmlu_virology", - "mmlu_clinical_knowledge", - "mmlu_management" - ], - "mmlu_stem": [ - "mmlu_high_school_computer_science", - "mmlu_college_biology", - "mmlu_high_school_physics", - "mmlu_college_physics", - "mmlu_elementary_mathematics", - "mmlu_college_mathematics", - "mmlu_machine_learning", - "mmlu_high_school_statistics", - "mmlu_college_chemistry", - "mmlu_anatomy", - "mmlu_conceptual_physics", - "mmlu_high_school_mathematics", - "mmlu_abstract_algebra", - "mmlu_computer_security", - "mmlu_college_computer_science", - "mmlu_astronomy", - "mmlu_high_school_biology", - "mmlu_high_school_chemistry", - "mmlu_electrical_engineering" - ], - "mmlu": [ - "mmlu_stem", - "mmlu_other", - "mmlu_social_sciences", - "mmlu_humanities" - ] - }, - "configs": { - "mmlu_abstract_algebra": { - "task": "mmlu_abstract_algebra", - "task_alias": "abstract_algebra", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "abstract_algebra", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about abstract algebra.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_anatomy": { - "task": "mmlu_anatomy", - "task_alias": "anatomy", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "anatomy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about anatomy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_astronomy": { - "task": "mmlu_astronomy", - "task_alias": "astronomy", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "astronomy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about astronomy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_business_ethics": { - "task": "mmlu_business_ethics", - "task_alias": "business_ethics", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "business_ethics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about business ethics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_clinical_knowledge": { - "task": "mmlu_clinical_knowledge", - "task_alias": "clinical_knowledge", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "clinical_knowledge", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about clinical knowledge.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_biology": { - "task": "mmlu_college_biology", - "task_alias": "college_biology", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_biology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college biology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_chemistry": { - "task": "mmlu_college_chemistry", - "task_alias": "college_chemistry", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_chemistry", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college chemistry.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_computer_science": { - "task": "mmlu_college_computer_science", - "task_alias": "college_computer_science", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_computer_science", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college computer science.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_mathematics": { - "task": "mmlu_college_mathematics", - "task_alias": "college_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_medicine": { - "task": "mmlu_college_medicine", - "task_alias": "college_medicine", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_medicine", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college medicine.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_physics": { - "task": "mmlu_college_physics", - "task_alias": "college_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_computer_security": { - "task": "mmlu_computer_security", - "task_alias": "computer_security", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "computer_security", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about computer security.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_conceptual_physics": { - "task": "mmlu_conceptual_physics", - "task_alias": "conceptual_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "conceptual_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about conceptual physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_econometrics": { - "task": "mmlu_econometrics", - "task_alias": "econometrics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "econometrics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about econometrics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_electrical_engineering": { - "task": "mmlu_electrical_engineering", - "task_alias": "electrical_engineering", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "electrical_engineering", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about electrical engineering.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_elementary_mathematics": { - "task": "mmlu_elementary_mathematics", - "task_alias": "elementary_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "elementary_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about elementary mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_formal_logic": { - "task": "mmlu_formal_logic", - "task_alias": "formal_logic", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "formal_logic", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about formal logic.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_global_facts": { - "task": "mmlu_global_facts", - "task_alias": "global_facts", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "global_facts", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about global facts.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_biology": { - "task": "mmlu_high_school_biology", - "task_alias": "high_school_biology", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_biology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school biology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_chemistry": { - "task": "mmlu_high_school_chemistry", - "task_alias": "high_school_chemistry", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_chemistry", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school chemistry.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_computer_science": { - "task": "mmlu_high_school_computer_science", - "task_alias": "high_school_computer_science", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_computer_science", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school computer science.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_european_history": { - "task": "mmlu_high_school_european_history", - "task_alias": "high_school_european_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_european_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school european history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_geography": { - "task": "mmlu_high_school_geography", - "task_alias": "high_school_geography", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_geography", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school geography.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_government_and_politics": { - "task": "mmlu_high_school_government_and_politics", - "task_alias": "high_school_government_and_politics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_government_and_politics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school government and politics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_macroeconomics": { - "task": "mmlu_high_school_macroeconomics", - "task_alias": "high_school_macroeconomics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_macroeconomics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school macroeconomics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_mathematics": { - "task": "mmlu_high_school_mathematics", - "task_alias": "high_school_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_microeconomics": { - "task": "mmlu_high_school_microeconomics", - "task_alias": "high_school_microeconomics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_microeconomics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school microeconomics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_physics": { - "task": "mmlu_high_school_physics", - "task_alias": "high_school_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_psychology": { - "task": "mmlu_high_school_psychology", - "task_alias": "high_school_psychology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_psychology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school psychology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_statistics": { - "task": "mmlu_high_school_statistics", - "task_alias": "high_school_statistics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_statistics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school statistics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_us_history": { - "task": "mmlu_high_school_us_history", - "task_alias": "high_school_us_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_us_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school us history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_world_history": { - "task": "mmlu_high_school_world_history", - "task_alias": "high_school_world_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_world_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school world history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_human_aging": { - "task": "mmlu_human_aging", - "task_alias": "human_aging", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "human_aging", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about human aging.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_human_sexuality": { - "task": "mmlu_human_sexuality", - "task_alias": "human_sexuality", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "human_sexuality", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about human sexuality.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_international_law": { - "task": "mmlu_international_law", - "task_alias": "international_law", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "international_law", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about international law.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_jurisprudence": { - "task": "mmlu_jurisprudence", - "task_alias": "jurisprudence", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "jurisprudence", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about jurisprudence.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_logical_fallacies": { - "task": "mmlu_logical_fallacies", - "task_alias": "logical_fallacies", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "logical_fallacies", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about logical fallacies.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_machine_learning": { - "task": "mmlu_machine_learning", - "task_alias": "machine_learning", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "machine_learning", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about machine learning.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_management": { - "task": "mmlu_management", - "task_alias": "management", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "management", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about management.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_marketing": { - "task": "mmlu_marketing", - "task_alias": "marketing", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "marketing", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about marketing.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_medical_genetics": { - "task": "mmlu_medical_genetics", - "task_alias": "medical_genetics", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "medical_genetics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about medical genetics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_miscellaneous": { - "task": "mmlu_miscellaneous", - "task_alias": "miscellaneous", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "miscellaneous", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about miscellaneous.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_moral_disputes": { - "task": "mmlu_moral_disputes", - "task_alias": "moral_disputes", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "moral_disputes", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about moral disputes.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_moral_scenarios": { - "task": "mmlu_moral_scenarios", - "task_alias": "moral_scenarios", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "moral_scenarios", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about moral scenarios.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_nutrition": { - "task": "mmlu_nutrition", - "task_alias": "nutrition", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "nutrition", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about nutrition.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_philosophy": { - "task": "mmlu_philosophy", - "task_alias": "philosophy", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "philosophy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about philosophy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_prehistory": { - "task": "mmlu_prehistory", - "task_alias": "prehistory", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "prehistory", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about prehistory.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_accounting": { - "task": "mmlu_professional_accounting", - "task_alias": "professional_accounting", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_accounting", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional accounting.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_law": { - "task": "mmlu_professional_law", - "task_alias": "professional_law", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_law", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional law.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_medicine": { - "task": "mmlu_professional_medicine", - "task_alias": "professional_medicine", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_medicine", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional medicine.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_psychology": { - "task": "mmlu_professional_psychology", - "task_alias": "professional_psychology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_psychology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional psychology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_public_relations": { - "task": "mmlu_public_relations", - "task_alias": "public_relations", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "public_relations", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about public relations.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_security_studies": { - "task": "mmlu_security_studies", - "task_alias": "security_studies", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "security_studies", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about security studies.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_sociology": { - "task": "mmlu_sociology", - "task_alias": "sociology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "sociology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about sociology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_us_foreign_policy": { - "task": "mmlu_us_foreign_policy", - "task_alias": "us_foreign_policy", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "us_foreign_policy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about us foreign policy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_virology": { - "task": "mmlu_virology", - "task_alias": "virology", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "virology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about virology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_world_religions": { - "task": "mmlu_world_religions", - "task_alias": "world_religions", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "world_religions", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about world religions.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18304/v1/completions", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - } - }, - "versions": { - "mmlu": 2, - "mmlu_abstract_algebra": 1.0, - "mmlu_anatomy": 1.0, - "mmlu_astronomy": 1.0, - "mmlu_business_ethics": 1.0, - "mmlu_clinical_knowledge": 1.0, - "mmlu_college_biology": 1.0, - "mmlu_college_chemistry": 1.0, - "mmlu_college_computer_science": 1.0, - "mmlu_college_mathematics": 1.0, - "mmlu_college_medicine": 1.0, - "mmlu_college_physics": 1.0, - "mmlu_computer_security": 1.0, - "mmlu_conceptual_physics": 1.0, - "mmlu_econometrics": 1.0, - "mmlu_electrical_engineering": 1.0, - "mmlu_elementary_mathematics": 1.0, - "mmlu_formal_logic": 1.0, - "mmlu_global_facts": 1.0, - "mmlu_high_school_biology": 1.0, - "mmlu_high_school_chemistry": 1.0, - "mmlu_high_school_computer_science": 1.0, - "mmlu_high_school_european_history": 1.0, - "mmlu_high_school_geography": 1.0, - "mmlu_high_school_government_and_politics": 1.0, - "mmlu_high_school_macroeconomics": 1.0, - "mmlu_high_school_mathematics": 1.0, - "mmlu_high_school_microeconomics": 1.0, - "mmlu_high_school_physics": 1.0, - "mmlu_high_school_psychology": 1.0, - "mmlu_high_school_statistics": 1.0, - "mmlu_high_school_us_history": 1.0, - "mmlu_high_school_world_history": 1.0, - "mmlu_human_aging": 1.0, - "mmlu_human_sexuality": 1.0, - "mmlu_humanities": 2, - "mmlu_international_law": 1.0, - "mmlu_jurisprudence": 1.0, - "mmlu_logical_fallacies": 1.0, - "mmlu_machine_learning": 1.0, - "mmlu_management": 1.0, - "mmlu_marketing": 1.0, - "mmlu_medical_genetics": 1.0, - "mmlu_miscellaneous": 1.0, - "mmlu_moral_disputes": 1.0, - "mmlu_moral_scenarios": 1.0, - "mmlu_nutrition": 1.0, - "mmlu_other": 2, - "mmlu_philosophy": 1.0, - "mmlu_prehistory": 1.0, - "mmlu_professional_accounting": 1.0, - "mmlu_professional_law": 1.0, - "mmlu_professional_medicine": 1.0, - "mmlu_professional_psychology": 1.0, - "mmlu_public_relations": 1.0, - "mmlu_security_studies": 1.0, - "mmlu_social_sciences": 2, - "mmlu_sociology": 1.0, - "mmlu_stem": 2, - "mmlu_us_foreign_policy": 1.0, - "mmlu_virology": 1.0, - "mmlu_world_religions": 1.0 - }, - "n-shot": { - "mmlu_abstract_algebra": 5, - "mmlu_anatomy": 5, - "mmlu_astronomy": 5, - "mmlu_business_ethics": 5, - "mmlu_clinical_knowledge": 5, - "mmlu_college_biology": 5, - "mmlu_college_chemistry": 5, - "mmlu_college_computer_science": 5, - "mmlu_college_mathematics": 5, - "mmlu_college_medicine": 5, - "mmlu_college_physics": 5, - "mmlu_computer_security": 5, - "mmlu_conceptual_physics": 5, - "mmlu_econometrics": 5, - "mmlu_electrical_engineering": 5, - "mmlu_elementary_mathematics": 5, - "mmlu_formal_logic": 5, - "mmlu_global_facts": 5, - "mmlu_high_school_biology": 5, - "mmlu_high_school_chemistry": 5, - "mmlu_high_school_computer_science": 5, - "mmlu_high_school_european_history": 5, - "mmlu_high_school_geography": 5, - "mmlu_high_school_government_and_politics": 5, - "mmlu_high_school_macroeconomics": 5, - "mmlu_high_school_mathematics": 5, - "mmlu_high_school_microeconomics": 5, - "mmlu_high_school_physics": 5, - "mmlu_high_school_psychology": 5, - "mmlu_high_school_statistics": 5, - "mmlu_high_school_us_history": 5, - "mmlu_high_school_world_history": 5, - "mmlu_human_aging": 5, - "mmlu_human_sexuality": 5, - "mmlu_international_law": 5, - "mmlu_jurisprudence": 5, - "mmlu_logical_fallacies": 5, - "mmlu_machine_learning": 5, - "mmlu_management": 5, - "mmlu_marketing": 5, - "mmlu_medical_genetics": 5, - "mmlu_miscellaneous": 5, - "mmlu_moral_disputes": 5, - "mmlu_moral_scenarios": 5, - "mmlu_nutrition": 5, - "mmlu_philosophy": 5, - "mmlu_prehistory": 5, - "mmlu_professional_accounting": 5, - "mmlu_professional_law": 5, - "mmlu_professional_medicine": 5, - "mmlu_professional_psychology": 5, - "mmlu_public_relations": 5, - "mmlu_security_studies": 5, - "mmlu_sociology": 5, - "mmlu_us_foreign_policy": 5, - "mmlu_virology": 5, - "mmlu_world_religions": 5 - }, - "higher_is_better": { - "mmlu": { - "acc": true - }, - "mmlu_abstract_algebra": { - "acc": true - }, - "mmlu_anatomy": { - "acc": true - }, - "mmlu_astronomy": { - "acc": true - }, - "mmlu_business_ethics": { - "acc": true - }, - "mmlu_clinical_knowledge": { - "acc": true - }, - "mmlu_college_biology": { - "acc": true - }, - "mmlu_college_chemistry": { - "acc": true - }, - "mmlu_college_computer_science": { - "acc": true - }, - "mmlu_college_mathematics": { - "acc": true - }, - "mmlu_college_medicine": { - "acc": true - }, - "mmlu_college_physics": { - "acc": true - }, - "mmlu_computer_security": { - "acc": true - }, - "mmlu_conceptual_physics": { - "acc": true - }, - "mmlu_econometrics": { - "acc": true - }, - "mmlu_electrical_engineering": { - "acc": true - }, - "mmlu_elementary_mathematics": { - "acc": true - }, - "mmlu_formal_logic": { - "acc": true - }, - "mmlu_global_facts": { - "acc": true - }, - "mmlu_high_school_biology": { - "acc": true - }, - "mmlu_high_school_chemistry": { - "acc": true - }, - "mmlu_high_school_computer_science": { - "acc": true - }, - "mmlu_high_school_european_history": { - "acc": true - }, - "mmlu_high_school_geography": { - "acc": true - }, - "mmlu_high_school_government_and_politics": { - "acc": true - }, - "mmlu_high_school_macroeconomics": { - "acc": true - }, - "mmlu_high_school_mathematics": { - "acc": true - }, - "mmlu_high_school_microeconomics": { - "acc": true - }, - "mmlu_high_school_physics": { - "acc": true - }, - "mmlu_high_school_psychology": { - "acc": true - }, - "mmlu_high_school_statistics": { - "acc": true - }, - "mmlu_high_school_us_history": { - "acc": true - }, - "mmlu_high_school_world_history": { - "acc": true - }, - "mmlu_human_aging": { - "acc": true - }, - "mmlu_human_sexuality": { - "acc": true - }, - "mmlu_humanities": { - "acc": true - }, - "mmlu_international_law": { - "acc": true - }, - "mmlu_jurisprudence": { - "acc": true - }, - "mmlu_logical_fallacies": { - "acc": true - }, - "mmlu_machine_learning": { - "acc": true - }, - "mmlu_management": { - "acc": true - }, - "mmlu_marketing": { - "acc": true - }, - "mmlu_medical_genetics": { - "acc": true - }, - "mmlu_miscellaneous": { - "acc": true - }, - "mmlu_moral_disputes": { - "acc": true - }, - "mmlu_moral_scenarios": { - "acc": true - }, - "mmlu_nutrition": { - "acc": true - }, - "mmlu_other": { - "acc": true - }, - "mmlu_philosophy": { - "acc": true - }, - "mmlu_prehistory": { - "acc": true - }, - "mmlu_professional_accounting": { - "acc": true - }, - "mmlu_professional_law": { - "acc": true - }, - "mmlu_professional_medicine": { - "acc": true - }, - "mmlu_professional_psychology": { - "acc": true - }, - "mmlu_public_relations": { - "acc": true - }, - "mmlu_security_studies": { - "acc": true - }, - "mmlu_social_sciences": { - "acc": true - }, - "mmlu_sociology": { - "acc": true - }, - "mmlu_stem": { - "acc": true - }, - "mmlu_us_foreign_policy": { - "acc": true - }, - "mmlu_virology": { - "acc": true - }, - "mmlu_world_religions": { - "acc": true - } - }, - "n-samples": { - "mmlu_high_school_computer_science": { - "original": 100, - "effective": 100 - }, - "mmlu_college_biology": { - "original": 144, - "effective": 144 - }, - "mmlu_high_school_physics": { - "original": 151, - "effective": 151 - }, - "mmlu_college_physics": { - "original": 102, - "effective": 102 - }, - "mmlu_elementary_mathematics": { - "original": 378, - "effective": 378 - }, - "mmlu_college_mathematics": { - "original": 100, - "effective": 100 - }, - "mmlu_machine_learning": { - "original": 112, - "effective": 112 - }, - "mmlu_high_school_statistics": { - "original": 216, - "effective": 216 - }, - "mmlu_college_chemistry": { - "original": 100, - "effective": 100 - }, - "mmlu_anatomy": { - "original": 135, - "effective": 135 - }, - "mmlu_conceptual_physics": { - "original": 235, - "effective": 235 - }, - "mmlu_high_school_mathematics": { - "original": 270, - "effective": 270 - }, - "mmlu_abstract_algebra": { - "original": 100, - "effective": 100 - }, - "mmlu_computer_security": { - "original": 100, - "effective": 100 - }, - "mmlu_college_computer_science": { - "original": 100, - "effective": 100 - }, - "mmlu_astronomy": { - "original": 152, - "effective": 152 - }, - "mmlu_high_school_biology": { - "original": 310, - "effective": 310 - }, - "mmlu_high_school_chemistry": { - "original": 203, - "effective": 203 - }, - "mmlu_electrical_engineering": { - "original": 145, - "effective": 145 - }, - "mmlu_business_ethics": { - "original": 100, - "effective": 100 - }, - "mmlu_marketing": { - "original": 234, - "effective": 234 - }, - "mmlu_human_aging": { - "original": 223, - "effective": 223 - }, - "mmlu_college_medicine": { - "original": 173, - "effective": 173 - }, - "mmlu_medical_genetics": { - "original": 100, - "effective": 100 - }, - "mmlu_professional_accounting": { - "original": 282, - "effective": 282 - }, - "mmlu_professional_medicine": { - "original": 272, - "effective": 272 - }, - "mmlu_global_facts": { - "original": 100, - "effective": 100 - }, - "mmlu_miscellaneous": { - "original": 783, - "effective": 783 - }, - "mmlu_nutrition": { - "original": 306, - "effective": 306 - }, - "mmlu_virology": { - "original": 166, - "effective": 166 - }, - "mmlu_clinical_knowledge": { - "original": 265, - "effective": 265 - }, - "mmlu_management": { - "original": 103, - "effective": 103 - }, - "mmlu_public_relations": { - "original": 110, - "effective": 110 - }, - "mmlu_human_sexuality": { - "original": 131, - "effective": 131 - }, - "mmlu_us_foreign_policy": { - "original": 100, - "effective": 100 - }, - "mmlu_high_school_microeconomics": { - "original": 238, - "effective": 238 - }, - "mmlu_high_school_macroeconomics": { - "original": 390, - "effective": 390 - }, - "mmlu_high_school_psychology": { - "original": 545, - "effective": 545 - }, - "mmlu_econometrics": { - "original": 114, - "effective": 114 - }, - "mmlu_sociology": { - "original": 201, - "effective": 201 - }, - "mmlu_professional_psychology": { - "original": 612, - "effective": 612 - }, - "mmlu_high_school_government_and_politics": { - "original": 193, - "effective": 193 - }, - "mmlu_security_studies": { - "original": 245, - "effective": 245 - }, - "mmlu_high_school_geography": { - "original": 198, - "effective": 198 - }, - "mmlu_moral_scenarios": { - "original": 895, - "effective": 895 - }, - "mmlu_high_school_us_history": { - "original": 204, - "effective": 204 - }, - "mmlu_formal_logic": { - "original": 126, - "effective": 126 - }, - "mmlu_international_law": { - "original": 121, - "effective": 121 - }, - "mmlu_jurisprudence": { - "original": 108, - "effective": 108 - }, - "mmlu_logical_fallacies": { - "original": 163, - "effective": 163 - }, - "mmlu_philosophy": { - "original": 311, - "effective": 311 - }, - "mmlu_prehistory": { - "original": 324, - "effective": 324 - }, - "mmlu_high_school_european_history": { - "original": 165, - "effective": 165 - }, - "mmlu_professional_law": { - "original": 1534, - "effective": 1534 - }, - "mmlu_world_religions": { - "original": 171, - "effective": 171 - }, - "mmlu_moral_disputes": { - "original": 346, - "effective": 346 - }, - "mmlu_high_school_world_history": { - "original": 237, - "effective": 237 - } - }, - "config": { - "model": "local-completions", - "model_args": "base_url=http://127.0.0.1:18304/v1/completions,model=qwen3-8b-ptq,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768", - "batch_size": 1, - "batch_sizes": [], - "device": null, - "use_cache": "/results/lm_cache", - "limit": null, - "downsampling_ratio": null, - "bootstrap_iters": 100000, - "gen_kwargs": { - "temperature": 1e-07, - "top_p": 0.9999999 - }, - "random_seed": 42, - "numpy_seed": 42, - "torch_seed": 42, - "fewshot_seed": 42 - }, - "git_hash": null, - "date": 1791476867.980376, - "pretty_env_info": "name 'get_pretty_env_info' is not defined", - "transformers_version": "4.50.3", - "lm_eval_version": "No package metadata was found for lm_eval", - "upper_git_hash": null, - "tokenizer_pad_token": [ - "<|endoftext|>", - "151643" - ], - "tokenizer_eos_token": [ - "<|im_end|>", - "151645" - ], - "tokenizer_bos_token": [ - null, - "None" - ], - "eot_token_id": 151645, - "max_length": 32767, - "task_hashes": { - "mmlu_high_school_computer_science": "2db28b8942d29e448cb587b398788f3cba6693cfe20772102b5bb1907999abea", - "mmlu_college_biology": "14cbae605287fc3c9ff25c8f2b2175619e036287a2e217767382b5f48f044943", - "mmlu_high_school_physics": "7780579caacd56f607968e81d5256d7d386095ab1d93b672047c5a5d8444ec1d", - "mmlu_college_physics": "ffe4456084dbfeb89e20d4b6138b97bbf415fd619691dbc7b38bfe5934b30e10", - "mmlu_elementary_mathematics": "da1d3600f010ce76c00a62b991b7606ee7df0b2e82413ee1f1977440ddd8550d", - "mmlu_college_mathematics": "33a226a6b1a6492f38387286d516c0941f5d8debbdfaa054b11fcda1cdb6dc1a", - "mmlu_machine_learning": "d02c202c0ad4c729890962d842ea70cad80faaae66a51a4c35c3685783ca06c1", - "mmlu_high_school_statistics": "0598ccf63c944eeab0ba3ad2dcfbcaebddf4c5d124174071beb52e8a73f19e86", - "mmlu_college_chemistry": "20b91eb7e582696bf39216d1757ac477675fe6de640150d7ae29993234af04dc", - "mmlu_anatomy": "9578a460fa25b65077e6da48a36e49d1e6b094807d65c9e5bf7e07ad0fbb6296", - "mmlu_conceptual_physics": "d9a7c4fd2d286215624dffc2aaec8e0295a6353daf2300d3461537e3fb5425eb", - "mmlu_high_school_mathematics": "313c7a073f5b7339d283ad977ea74584374b6e0281898dfd89ff4bdfd2fb7b03", - "mmlu_abstract_algebra": "0e2f49943ac30da4eb2d7d2f5db73abb69da69d6988007197e6bdf138bfb3369", - "mmlu_computer_security": "e122c4bc113d812e7e083e316b358cc7e16a0d1c2e6506480caebcf9d1e27ecd", - "mmlu_college_computer_science": "c2355a53d39d8bf1fbec4ea4b0f2b50c3fe104f2b36556ecae1038b0c315816a", - "mmlu_astronomy": "83e572d3948bb6dbfb66031ab4dcb5b70054a5bc110de1a0fbc324940f4c04da", - "mmlu_high_school_biology": "cfa833fafa27f991b540ece0e0583e1ebedfcaa50d030891935b000b7757c1cb", - "mmlu_high_school_chemistry": "7b7e216abbacc50a1613fc9e02f3f3dda2709059394b121e43f3ad534cadcf80", - "mmlu_electrical_engineering": "58857d23657299553045d3db7b7b0de43f9a078a2fd53902d0e59251ac497e9c", - "mmlu_business_ethics": "a0e93295030eaac8a5dd303dd1514b9eb25ae1807f55d000b876c0e8ddd5e1c8", - "mmlu_marketing": "2018f5181b9a272ca8a4f65858e0d12a1bf87792934fc483022e9d1bc7e05997", - "mmlu_human_aging": "88802f2005a5c3c719ace0291d92034ef3c90ee1cf7488d5195f8c545c899d4d", - "mmlu_college_medicine": "bc04947722ee05b4d5051df09f0e50de44d4f49df44f5ff909dbe3264cbc2c53", - "mmlu_medical_genetics": "8819bc558f8142680673b743cf8e3681ea6856ece1e3895c24fc0f77a60da7c4", - "mmlu_professional_accounting": "654d1dafcb0ff52e603404c1158924e92c5067cda9df2429eb0777c99c648995", - "mmlu_professional_medicine": "ba4af62ac5d26864b9e6339c6cbd16cbabcb5fbcf00366c02dd15da73c28daf9", - "mmlu_global_facts": "856fc8218f2f89f3e9abbe96070c530689a228cf5c9269c58bd03b4f0aa833b1", - "mmlu_miscellaneous": "22548428bd482f372a4fba10d2a0c0a939faddd7e9781701a74358145211a660", - "mmlu_nutrition": "468a945d5f4f35a2b1c974d7180d159d3e80a0611fa9160afee572df0dd69432", - "mmlu_virology": "67042eb89ab0d2a45f1d254dbb9da9de152fa1098d7740a05ac7ff8495b26534", - "mmlu_clinical_knowledge": "a3dba4354de80aa98c7db318923aedb8b78c02014a2be97c4071453ba619b8dc", - "mmlu_management": "6345661bb012d1a72c5d34f39e692cf47c131807a3bdfff7bf07ee9c5149d29e", - "mmlu_public_relations": "355105913812b28cc46607bf6710a92dcb346c9d35f489eb84e388de03390b32", - "mmlu_human_sexuality": "9b20e343f77622c458fe394d257924eb8895d07ddd98b1e3dbaf366563f77cbb", - "mmlu_us_foreign_policy": "1c1d39c546896fb1b9a513c5db8233ea4066c3ec79d8b85c2ab7944fa79ca8c5", - "mmlu_high_school_microeconomics": "705c7c5543f9987bdf14f7b39ddae1910e9097d0f491bb98a536140175d94732", - "mmlu_high_school_macroeconomics": "50ee63c5ac58c48f651464b5fd6cbbe0c8fa6002dd480fa9634ba630d9dc0417", - "mmlu_high_school_psychology": "e62d1ebd73c032fd0c3b99eaf20e7b77b13ae49e4fc506d7e69f9ae15e485a4c", - "mmlu_econometrics": "905a5e83b9a8173ef7345d14ac535c91c78c34b076314a2deb52cb2e6138502c", - "mmlu_sociology": "65917e6a3b0027bdb12703440e17afc8f155c15087a9df249e0061d84381e0d2", - "mmlu_professional_psychology": "70ec543a94dccdb5e82f3ac5f006de88a1352fb86b2a2c68d496529cdd1f5e0c", - "mmlu_high_school_government_and_politics": "bd59fed7fa1dcf3e39a65a7ea71cc81573c4fb28bcd942cbc8d8d9afbe0e9d4c", - "mmlu_security_studies": "e7c1c4bd8669cc22b7a907cd3584c5caf80b82772d9741a3680563b4fb564084", - "mmlu_high_school_geography": "30d201a5bfcbaec8d0e74f58ddd23bd58204b1d99b9211e6a531d8b6e84eab24", - "mmlu_moral_scenarios": "83a877f6af97d85f853c38196827d751dc8f33db4253da17965fde0c5e3554e1", - "mmlu_high_school_us_history": "b5fdc459ac365a794c663a17cf8a34a702ea87eaa1a8b40fe49e8196ef88232c", - "mmlu_formal_logic": "646ee3f6abf70a7d0efd03cdc0fbebb9f2e8f40e3b5563e53a9d0a7a291bc2a7", - "mmlu_international_law": "144d9e8693486d00f5cac911f3fbfbacdba9ac7c10340d50061262d9c3fdff9d", - "mmlu_jurisprudence": "325cb19fa404379425d7add71b29e2e4cd3428c08a25f5cc718fd87e79ccd2bb", - "mmlu_logical_fallacies": "f4aa9887846d578f13ca7420b65369c344053cabe092607a16237dfeef07a1ba", - "mmlu_philosophy": "c61151b7b50a37e90635e8b219018a35533a65a2a684542ead156b9f1020ffc1", - "mmlu_prehistory": "4aa1d25e8211294e34dead164bbde415d42feb245a4716a2471d929f81846f04", - "mmlu_high_school_european_history": "8c8ff9bcf9924cd457cc5b67c5f6a21925c09aadba42e646c088f2b99ba4e0e2", - "mmlu_professional_law": "0e0e2459e1f45df128cf8460ebdea4da6cc99a9fbd9cb99fb5ebeee78575800e", - "mmlu_world_religions": "8ad90daa76079af018b36cb86143468d22e34a9cd3273a3c48ce4c4648f886dc", - "mmlu_moral_disputes": "4d3e2d32e558de3c5eb0369e9a20280fda45fe04eeedf3356ad820e352293720", - "mmlu_high_school_world_history": "59ebbb27916baa0f2c6fa8b064ecbb042b43f6d492bbe18f2a78a63d40bf3dd0" - }, - "model_source": "local-completions", - "model_name": "qwen3-8b-ptq", - "model_name_sanitized": "qwen3-8b-ptq", - "system_instruction": null, - "system_instruction_sha": null, - "fewshot_as_multiturn": false, - "chat_template": null, - "chat_template_sha": null, - "start_time": 5792961.753521479, - "end_time": 5796646.816937493, - "total_evaluation_time_seconds": "3685.0634160144255" -} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json deleted file mode 100644 index 34de5729c40..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-16.887993.json +++ /dev/null @@ -1,136 +0,0 @@ -{ - "results": { - "wikitext": { - "alias": "wikitext", - "word_perplexity,none": 13.88340951298615, - "word_perplexity_stderr,none": "N/A", - "byte_perplexity,none": 1.6355073044598794, - "byte_perplexity_stderr,none": "N/A", - "bits_per_byte,none": 0.7097382027618786, - "bits_per_byte_stderr,none": "N/A" - } - }, - "group_subtasks": { - "wikitext": [] - }, - "configs": { - "wikitext": { - "task": "wikitext", - "dataset_path": "EleutherAI/wikitext_document_level", - "dataset_name": "wikitext-2-raw-v1", - "dataset_kwargs": { - "trust_remote_code": true - }, - "training_split": "train", - "validation_split": "validation", - "test_split": "test", - "doc_to_text": "", - "doc_to_target": "def wikitext_detokenizer(doc):\n string = doc[\"page\"]\n # contractions\n string = string.replace(\"s '\", \"s'\")\n string = re.sub(r\"/' [0-9]/\", r\"/'[0-9]/\", string)\n # number separators\n string = string.replace(\" @-@ \", \"-\")\n string = string.replace(\" @,@ \", \",\")\n string = string.replace(\" @.@ \", \".\")\n # punctuation\n string = string.replace(\" : \", \": \")\n string = string.replace(\" ; \", \"; \")\n string = string.replace(\" . \", \". \")\n string = string.replace(\" ! \", \"! \")\n string = string.replace(\" ? \", \"? \")\n string = string.replace(\" , \", \", \")\n # double brackets\n string = re.sub(r\"\\(\\s*([^\\)]*?)\\s*\\)\", r\"(\\1)\", string)\n string = re.sub(r\"\\[\\s*([^\\]]*?)\\s*\\]\", r\"[\\1]\", string)\n string = re.sub(r\"{\\s*([^}]*?)\\s*}\", r\"{\\1}\", string)\n string = re.sub(r\"\\\"\\s*([^\\\"]*?)\\s*\\\"\", r'\"\\1\"', string)\n string = re.sub(r\"'\\s*([^']*?)\\s*'\", r\"'\\1'\", string)\n # miscellaneous\n string = string.replace(\"= = = =\", \"====\")\n string = string.replace(\"= = =\", \"===\")\n string = string.replace(\"= =\", \"==\")\n string = string.replace(\" \" + chr(176) + \" \", chr(176))\n string = string.replace(\" \\n\", \"\\n\")\n string = string.replace(\"\\n \", \"\\n\")\n string = string.replace(\" N \", \" 1 \")\n string = string.replace(\" 's\", \"'s\")\n\n return string\n", - "unsafe_code": false, - "process_results": "def process_results(doc, results):\n (loglikelihood,) = results\n # IMPORTANT: wikitext counts number of words in *original doc before detokenization*\n _words = len(re.split(r\"\\s+\", doc[\"page\"]))\n _bytes = len(doc[\"page\"].encode(\"utf-8\"))\n return {\n \"word_perplexity\": (loglikelihood, _words),\n \"byte_perplexity\": (loglikelihood, _bytes),\n \"bits_per_byte\": (loglikelihood, _bytes),\n }\n", - "description": "", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "num_fewshot": 0, - "metric_list": [ - { - "metric": "word_perplexity" - }, - { - "metric": "byte_perplexity" - }, - { - "metric": "bits_per_byte" - } - ], - "output_type": "loglikelihood_rolling", - "repeats": 1, - "should_decontaminate": true, - "doc_to_decontamination_query": "{{page}}", - "metadata": { - "version": 2.0, - "base_url": "http://localhost:3825", - "model": "qwen3-8b-ptq", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 2, - "timeout": 300, - "max_retries": 2, - "stream": false - } - } - }, - "versions": { - "wikitext": 2.0 - }, - "n-shot": { - "wikitext": 0 - }, - "higher_is_better": { - "wikitext": { - "word_perplexity": false, - "byte_perplexity": false, - "bits_per_byte": false - } - }, - "n-samples": { - "wikitext": { - "original": 62, - "effective": 62 - } - }, - "config": { - "model": "local-completions", - "model_args": "base_url=http://localhost:3825,model=qwen3-8b-ptq,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=2,timeout=300,max_retries=2,stream=False", - "batch_size": 1, - "batch_sizes": [], - "device": null, - "use_cache": "/results/lm_cache", - "limit": null, - "downsampling_ratio": null, - "bootstrap_iters": 100000, - "gen_kwargs": { - "temperature": 1e-07, - "top_p": 0.9999999 - }, - "random_seed": 42, - "numpy_seed": 42, - "torch_seed": 42, - "fewshot_seed": 42 - }, - "git_hash": null, - "date": 1791469370.721785, - "pretty_env_info": "name 'get_pretty_env_info' is not defined", - "transformers_version": "4.50.3", - "lm_eval_version": "No package metadata was found for lm_eval", - "upper_git_hash": null, - "tokenizer_pad_token": [ - "<|endoftext|>", - "151643" - ], - "tokenizer_eos_token": [ - "<|im_end|>", - "151645" - ], - "tokenizer_bos_token": [ - null, - "None" - ], - "eot_token_id": 151645, - "max_length": 2047, - "task_hashes": { - "wikitext": "233109c41076287b5ba5a338ab7b085728acacf36d237a7ab99f34fcda161641" - }, - "model_source": "local-completions", - "model_name": "qwen3-8b-ptq", - "model_name_sanitized": "qwen3-8b-ptq", - "system_instruction": null, - "system_instruction_sha": null, - "fewshot_as_multiturn": false, - "chat_template": null, - "chat_template_sha": null, - "start_time": 309861.529678691, - "end_time": 309889.04976299, - "total_evaluation_time_seconds": "27.520084298972506" -} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json deleted file mode 100644 index 6a67505239b..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/lm_eval_results_2026-10-08T11-35-03.360012.json +++ /dev/null @@ -1,4030 +0,0 @@ -{ - "results": { - "mmlu": { - "acc,none": 0.7375729952998148, - "acc_stderr,none": 0.0035282530148200222, - "alias": "mmlu" - }, - "mmlu_humanities": { - "acc,none": 0.6512221041445271, - "acc_stderr,none": 0.006616358365310683, - "alias": " - humanities" - }, - "mmlu_formal_logic": { - "alias": " - formal_logic", - "acc,none": 0.6031746031746031, - "acc_stderr,none": 0.04375888492727054 - }, - "mmlu_high_school_european_history": { - "alias": " - high_school_european_history", - "acc,none": 0.8727272727272727, - "acc_stderr,none": 0.026024657651656246 - }, - "mmlu_high_school_us_history": { - "alias": " - high_school_us_history", - "acc,none": 0.8382352941176471, - "acc_stderr,none": 0.025845017986926906 - }, - "mmlu_high_school_world_history": { - "alias": " - high_school_world_history", - "acc,none": 0.8565400843881856, - "acc_stderr,none": 0.02281829182101699 - }, - "mmlu_international_law": { - "alias": " - international_law", - "acc,none": 0.7355371900826446, - "acc_stderr,none": 0.04026187527591209 - }, - "mmlu_jurisprudence": { - "alias": " - jurisprudence", - "acc,none": 0.7962962962962963, - "acc_stderr,none": 0.038935425188248496 - }, - "mmlu_logical_fallacies": { - "alias": " - logical_fallacies", - "acc,none": 0.8220858895705522, - "acc_stderr,none": 0.03004735765580667 - }, - "mmlu_moral_disputes": { - "alias": " - moral_disputes", - "acc,none": 0.7398843930635838, - "acc_stderr,none": 0.02361867831006933 - }, - "mmlu_moral_scenarios": { - "alias": " - moral_scenarios", - "acc,none": 0.5385474860335195, - "acc_stderr,none": 0.01667273126755234 - }, - "mmlu_philosophy": { - "alias": " - philosophy", - "acc,none": 0.7684887459807074, - "acc_stderr,none": 0.023956532766639154 - }, - "mmlu_prehistory": { - "alias": " - prehistory", - "acc,none": 0.8302469135802469, - "acc_stderr,none": 0.020888690414093802 - }, - "mmlu_professional_law": { - "alias": " - professional_law", - "acc,none": 0.5026075619295959, - "acc_stderr,none": 0.012770062445433189 - }, - "mmlu_world_religions": { - "alias": " - world_religions", - "acc,none": 0.8421052631578947, - "acc_stderr,none": 0.027966785859160903 - }, - "mmlu_other": { - "acc,none": 0.7685870614740907, - "acc_stderr,none": 0.007283169591179256, - "alias": " - other" - }, - "mmlu_business_ethics": { - "alias": " - business_ethics", - "acc,none": 0.79, - "acc_stderr,none": 0.040936018074033236 - }, - "mmlu_clinical_knowledge": { - "alias": " - clinical_knowledge", - "acc,none": 0.7962264150943397, - "acc_stderr,none": 0.024790784501775447 - }, - "mmlu_college_medicine": { - "alias": " - college_medicine", - "acc,none": 0.7803468208092486, - "acc_stderr,none": 0.03156809362703173 - }, - "mmlu_global_facts": { - "alias": " - global_facts", - "acc,none": 0.43, - "acc_stderr,none": 0.049756985195624305 - }, - "mmlu_human_aging": { - "alias": " - human_aging", - "acc,none": 0.7399103139013453, - "acc_stderr,none": 0.029442495585857424 - }, - "mmlu_management": { - "alias": " - management", - "acc,none": 0.8737864077669902, - "acc_stderr,none": 0.03288180278808631 - }, - "mmlu_marketing": { - "alias": " - marketing", - "acc,none": 0.9188034188034188, - "acc_stderr,none": 0.017893784904018484 - }, - "mmlu_medical_genetics": { - "alias": " - medical_genetics", - "acc,none": 0.82, - "acc_stderr,none": 0.03861229196653691 - }, - "mmlu_miscellaneous": { - "alias": " - miscellaneous", - "acc,none": 0.8339719029374202, - "acc_stderr,none": 0.013306478243066229 - }, - "mmlu_nutrition": { - "alias": " - nutrition", - "acc,none": 0.8006535947712419, - "acc_stderr,none": 0.022875816993464054 - }, - "mmlu_professional_accounting": { - "alias": " - professional_accounting", - "acc,none": 0.5709219858156028, - "acc_stderr,none": 0.029525914302558586 - }, - "mmlu_professional_medicine": { - "alias": " - professional_medicine", - "acc,none": 0.8088235294117647, - "acc_stderr,none": 0.023886881922440366 - }, - "mmlu_virology": { - "alias": " - virology", - "acc,none": 0.536144578313253, - "acc_stderr,none": 0.038823108508905954 - }, - "mmlu_social_sciences": { - "acc,none": 0.8323041923951902, - "acc_stderr,none": 0.006626384400414149, - "alias": " - social sciences" - }, - "mmlu_econometrics": { - "alias": " - econometrics", - "acc,none": 0.6842105263157895, - "acc_stderr,none": 0.04372748290278002 - }, - "mmlu_high_school_geography": { - "alias": " - high_school_geography", - "acc,none": 0.8686868686868687, - "acc_stderr,none": 0.02406315641682249 - }, - "mmlu_high_school_government_and_politics": { - "alias": " - high_school_government_and_politics", - "acc,none": 0.9326424870466321, - "acc_stderr,none": 0.01808839383907894 - }, - "mmlu_high_school_macroeconomics": { - "alias": " - high_school_macroeconomics", - "acc,none": 0.8, - "acc_stderr,none": 0.020280805062535757 - }, - "mmlu_high_school_microeconomics": { - "alias": " - high_school_microeconomics", - "acc,none": 0.9201680672268907, - "acc_stderr,none": 0.017605474225715036 - }, - "mmlu_high_school_psychology": { - "alias": " - high_school_psychology", - "acc,none": 0.908256880733945, - "acc_stderr,none": 0.012376323409137066 - }, - "mmlu_human_sexuality": { - "alias": " - human_sexuality", - "acc,none": 0.8473282442748091, - "acc_stderr,none": 0.031545216720054704 - }, - "mmlu_professional_psychology": { - "alias": " - professional_psychology", - "acc,none": 0.761437908496732, - "acc_stderr,none": 0.017242385828779655 - }, - "mmlu_public_relations": { - "alias": " - public_relations", - "acc,none": 0.7, - "acc_stderr,none": 0.04389311454644288 - }, - "mmlu_security_studies": { - "alias": " - security_studies", - "acc,none": 0.8040816326530612, - "acc_stderr,none": 0.025409301953225705 - }, - "mmlu_sociology": { - "alias": " - sociology", - "acc,none": 0.8407960199004975, - "acc_stderr,none": 0.025870646766169153 - }, - "mmlu_us_foreign_policy": { - "alias": " - us_foreign_policy", - "acc,none": 0.85, - "acc_stderr,none": 0.03588702812826367 - }, - "mmlu_stem": { - "acc,none": 0.743418966064066, - "acc_stderr,none": 0.007488923716325649, - "alias": " - stem" - }, - "mmlu_abstract_algebra": { - "alias": " - abstract_algebra", - "acc,none": 0.56, - "acc_stderr,none": 0.049888765156985884 - }, - "mmlu_anatomy": { - "alias": " - anatomy", - "acc,none": 0.6888888888888889, - "acc_stderr,none": 0.039992628766177166 - }, - "mmlu_astronomy": { - "alias": " - astronomy", - "acc,none": 0.9013157894736842, - "acc_stderr,none": 0.024270227737522694 - }, - "mmlu_college_biology": { - "alias": " - college_biology", - "acc,none": 0.875, - "acc_stderr,none": 0.02765610492929436 - }, - "mmlu_college_chemistry": { - "alias": " - college_chemistry", - "acc,none": 0.56, - "acc_stderr,none": 0.049888765156985884 - }, - "mmlu_college_computer_science": { - "alias": " - college_computer_science", - "acc,none": 0.68, - "acc_stderr,none": 0.046882617226215076 - }, - "mmlu_college_mathematics": { - "alias": " - college_mathematics", - "acc,none": 0.58, - "acc_stderr,none": 0.04960449637488582 - }, - "mmlu_college_physics": { - "alias": " - college_physics", - "acc,none": 0.6372549019607843, - "acc_stderr,none": 0.047840607041056527 - }, - "mmlu_computer_security": { - "alias": " - computer_security", - "acc,none": 0.8, - "acc_stderr,none": 0.04020151261036849 - }, - "mmlu_conceptual_physics": { - "alias": " - conceptual_physics", - "acc,none": 0.8468085106382979, - "acc_stderr,none": 0.023545179061675213 - }, - "mmlu_electrical_engineering": { - "alias": " - electrical_engineering", - "acc,none": 0.8275862068965517, - "acc_stderr,none": 0.03147830790259575 - }, - "mmlu_elementary_mathematics": { - "alias": " - elementary_mathematics", - "acc,none": 0.7619047619047619, - "acc_stderr,none": 0.0219358780811847 - }, - "mmlu_high_school_biology": { - "alias": " - high_school_biology", - "acc,none": 0.9161290322580645, - "acc_stderr,none": 0.015769027496775664 - }, - "mmlu_high_school_chemistry": { - "alias": " - high_school_chemistry", - "acc,none": 0.7192118226600985, - "acc_stderr,none": 0.031618563353586135 - }, - "mmlu_high_school_computer_science": { - "alias": " - high_school_computer_science", - "acc,none": 0.86, - "acc_stderr,none": 0.0348735088019777 - }, - "mmlu_high_school_mathematics": { - "alias": " - high_school_mathematics", - "acc,none": 0.5185185185185185, - "acc_stderr,none": 0.03046462171889531 - }, - "mmlu_high_school_physics": { - "alias": " - high_school_physics", - "acc,none": 0.7019867549668874, - "acc_stderr,none": 0.03734535676787201 - }, - "mmlu_high_school_statistics": { - "alias": " - high_school_statistics", - "acc,none": 0.7546296296296297, - "acc_stderr,none": 0.02934666509437301 - }, - "mmlu_machine_learning": { - "alias": " - machine_learning", - "acc,none": 0.6517857142857143, - "acc_stderr,none": 0.04521829902833581 - } - }, - "groups": { - "mmlu": { - "acc,none": 0.7375729952998148, - "acc_stderr,none": 0.0035282530148200222, - "alias": "mmlu" - }, - "mmlu_humanities": { - "acc,none": 0.6512221041445271, - "acc_stderr,none": 0.006616358365310683, - "alias": " - humanities" - }, - "mmlu_other": { - "acc,none": 0.7685870614740907, - "acc_stderr,none": 0.007283169591179256, - "alias": " - other" - }, - "mmlu_social_sciences": { - "acc,none": 0.8323041923951902, - "acc_stderr,none": 0.006626384400414149, - "alias": " - social sciences" - }, - "mmlu_stem": { - "acc,none": 0.743418966064066, - "acc_stderr,none": 0.007488923716325649, - "alias": " - stem" - } - }, - "group_subtasks": { - "mmlu_humanities": [ - "mmlu_professional_law", - "mmlu_world_religions", - "mmlu_logical_fallacies", - "mmlu_formal_logic", - "mmlu_moral_scenarios", - "mmlu_high_school_european_history", - "mmlu_high_school_us_history", - "mmlu_prehistory", - "mmlu_international_law", - "mmlu_philosophy", - "mmlu_jurisprudence", - "mmlu_high_school_world_history", - "mmlu_moral_disputes" - ], - "mmlu_social_sciences": [ - "mmlu_us_foreign_policy", - "mmlu_econometrics", - "mmlu_human_sexuality", - "mmlu_public_relations", - "mmlu_high_school_government_and_politics", - "mmlu_high_school_psychology", - "mmlu_high_school_macroeconomics", - "mmlu_sociology", - "mmlu_high_school_geography", - "mmlu_professional_psychology", - "mmlu_high_school_microeconomics", - "mmlu_security_studies" - ], - "mmlu_other": [ - "mmlu_professional_medicine", - "mmlu_professional_accounting", - "mmlu_global_facts", - "mmlu_nutrition", - "mmlu_marketing", - "mmlu_management", - "mmlu_human_aging", - "mmlu_medical_genetics", - "mmlu_virology", - "mmlu_clinical_knowledge", - "mmlu_miscellaneous", - "mmlu_college_medicine", - "mmlu_business_ethics" - ], - "mmlu_stem": [ - "mmlu_conceptual_physics", - "mmlu_college_mathematics", - "mmlu_high_school_statistics", - "mmlu_elementary_mathematics", - "mmlu_high_school_physics", - "mmlu_electrical_engineering", - "mmlu_college_chemistry", - "mmlu_high_school_mathematics", - "mmlu_abstract_algebra", - "mmlu_astronomy", - "mmlu_machine_learning", - "mmlu_college_computer_science", - "mmlu_high_school_chemistry", - "mmlu_college_physics", - "mmlu_college_biology", - "mmlu_anatomy", - "mmlu_high_school_biology", - "mmlu_computer_security", - "mmlu_high_school_computer_science" - ], - "mmlu": [ - "mmlu_stem", - "mmlu_other", - "mmlu_social_sciences", - "mmlu_humanities" - ] - }, - "configs": { - "mmlu_abstract_algebra": { - "task": "mmlu_abstract_algebra", - "task_alias": "abstract_algebra", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "abstract_algebra", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about abstract algebra.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_anatomy": { - "task": "mmlu_anatomy", - "task_alias": "anatomy", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "anatomy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about anatomy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_astronomy": { - "task": "mmlu_astronomy", - "task_alias": "astronomy", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "astronomy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about astronomy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_business_ethics": { - "task": "mmlu_business_ethics", - "task_alias": "business_ethics", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "business_ethics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about business ethics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_clinical_knowledge": { - "task": "mmlu_clinical_knowledge", - "task_alias": "clinical_knowledge", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "clinical_knowledge", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about clinical knowledge.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_biology": { - "task": "mmlu_college_biology", - "task_alias": "college_biology", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_biology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college biology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_chemistry": { - "task": "mmlu_college_chemistry", - "task_alias": "college_chemistry", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_chemistry", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college chemistry.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_computer_science": { - "task": "mmlu_college_computer_science", - "task_alias": "college_computer_science", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_computer_science", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college computer science.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_mathematics": { - "task": "mmlu_college_mathematics", - "task_alias": "college_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_medicine": { - "task": "mmlu_college_medicine", - "task_alias": "college_medicine", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_medicine", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college medicine.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_physics": { - "task": "mmlu_college_physics", - "task_alias": "college_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_computer_security": { - "task": "mmlu_computer_security", - "task_alias": "computer_security", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "computer_security", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about computer security.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_conceptual_physics": { - "task": "mmlu_conceptual_physics", - "task_alias": "conceptual_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "conceptual_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about conceptual physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_econometrics": { - "task": "mmlu_econometrics", - "task_alias": "econometrics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "econometrics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about econometrics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_electrical_engineering": { - "task": "mmlu_electrical_engineering", - "task_alias": "electrical_engineering", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "electrical_engineering", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about electrical engineering.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_elementary_mathematics": { - "task": "mmlu_elementary_mathematics", - "task_alias": "elementary_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "elementary_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about elementary mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_formal_logic": { - "task": "mmlu_formal_logic", - "task_alias": "formal_logic", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "formal_logic", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about formal logic.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_global_facts": { - "task": "mmlu_global_facts", - "task_alias": "global_facts", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "global_facts", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about global facts.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_biology": { - "task": "mmlu_high_school_biology", - "task_alias": "high_school_biology", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_biology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school biology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_chemistry": { - "task": "mmlu_high_school_chemistry", - "task_alias": "high_school_chemistry", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_chemistry", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school chemistry.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_computer_science": { - "task": "mmlu_high_school_computer_science", - "task_alias": "high_school_computer_science", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_computer_science", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school computer science.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_european_history": { - "task": "mmlu_high_school_european_history", - "task_alias": "high_school_european_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_european_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school european history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_geography": { - "task": "mmlu_high_school_geography", - "task_alias": "high_school_geography", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_geography", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school geography.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_government_and_politics": { - "task": "mmlu_high_school_government_and_politics", - "task_alias": "high_school_government_and_politics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_government_and_politics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school government and politics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_macroeconomics": { - "task": "mmlu_high_school_macroeconomics", - "task_alias": "high_school_macroeconomics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_macroeconomics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school macroeconomics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_mathematics": { - "task": "mmlu_high_school_mathematics", - "task_alias": "high_school_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_microeconomics": { - "task": "mmlu_high_school_microeconomics", - "task_alias": "high_school_microeconomics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_microeconomics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school microeconomics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_physics": { - "task": "mmlu_high_school_physics", - "task_alias": "high_school_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_psychology": { - "task": "mmlu_high_school_psychology", - "task_alias": "high_school_psychology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_psychology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school psychology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_statistics": { - "task": "mmlu_high_school_statistics", - "task_alias": "high_school_statistics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_statistics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school statistics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_us_history": { - "task": "mmlu_high_school_us_history", - "task_alias": "high_school_us_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_us_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school us history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_world_history": { - "task": "mmlu_high_school_world_history", - "task_alias": "high_school_world_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_world_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school world history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_human_aging": { - "task": "mmlu_human_aging", - "task_alias": "human_aging", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "human_aging", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about human aging.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_human_sexuality": { - "task": "mmlu_human_sexuality", - "task_alias": "human_sexuality", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "human_sexuality", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about human sexuality.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_international_law": { - "task": "mmlu_international_law", - "task_alias": "international_law", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "international_law", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about international law.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_jurisprudence": { - "task": "mmlu_jurisprudence", - "task_alias": "jurisprudence", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "jurisprudence", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about jurisprudence.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_logical_fallacies": { - "task": "mmlu_logical_fallacies", - "task_alias": "logical_fallacies", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "logical_fallacies", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about logical fallacies.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_machine_learning": { - "task": "mmlu_machine_learning", - "task_alias": "machine_learning", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "machine_learning", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about machine learning.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_management": { - "task": "mmlu_management", - "task_alias": "management", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "management", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about management.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_marketing": { - "task": "mmlu_marketing", - "task_alias": "marketing", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "marketing", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about marketing.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_medical_genetics": { - "task": "mmlu_medical_genetics", - "task_alias": "medical_genetics", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "medical_genetics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about medical genetics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_miscellaneous": { - "task": "mmlu_miscellaneous", - "task_alias": "miscellaneous", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "miscellaneous", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about miscellaneous.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_moral_disputes": { - "task": "mmlu_moral_disputes", - "task_alias": "moral_disputes", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "moral_disputes", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about moral disputes.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_moral_scenarios": { - "task": "mmlu_moral_scenarios", - "task_alias": "moral_scenarios", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "moral_scenarios", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about moral scenarios.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_nutrition": { - "task": "mmlu_nutrition", - "task_alias": "nutrition", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "nutrition", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about nutrition.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_philosophy": { - "task": "mmlu_philosophy", - "task_alias": "philosophy", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "philosophy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about philosophy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_prehistory": { - "task": "mmlu_prehistory", - "task_alias": "prehistory", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "prehistory", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about prehistory.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_accounting": { - "task": "mmlu_professional_accounting", - "task_alias": "professional_accounting", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_accounting", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional accounting.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_law": { - "task": "mmlu_professional_law", - "task_alias": "professional_law", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_law", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional law.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_medicine": { - "task": "mmlu_professional_medicine", - "task_alias": "professional_medicine", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_medicine", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional medicine.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_psychology": { - "task": "mmlu_professional_psychology", - "task_alias": "professional_psychology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_psychology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional psychology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_public_relations": { - "task": "mmlu_public_relations", - "task_alias": "public_relations", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "public_relations", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about public relations.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_security_studies": { - "task": "mmlu_security_studies", - "task_alias": "security_studies", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "security_studies", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about security studies.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_sociology": { - "task": "mmlu_sociology", - "task_alias": "sociology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "sociology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about sociology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_us_foreign_policy": { - "task": "mmlu_us_foreign_policy", - "task_alias": "us_foreign_policy", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "us_foreign_policy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about us foreign policy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_virology": { - "task": "mmlu_virology", - "task_alias": "virology", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "virology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about virology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_world_religions": { - "task": "mmlu_world_religions", - "task_alias": "world_religions", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "world_religions", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about world religions.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18308/v1/completions", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - } - }, - "versions": { - "mmlu": 2, - "mmlu_abstract_algebra": 1.0, - "mmlu_anatomy": 1.0, - "mmlu_astronomy": 1.0, - "mmlu_business_ethics": 1.0, - "mmlu_clinical_knowledge": 1.0, - "mmlu_college_biology": 1.0, - "mmlu_college_chemistry": 1.0, - "mmlu_college_computer_science": 1.0, - "mmlu_college_mathematics": 1.0, - "mmlu_college_medicine": 1.0, - "mmlu_college_physics": 1.0, - "mmlu_computer_security": 1.0, - "mmlu_conceptual_physics": 1.0, - "mmlu_econometrics": 1.0, - "mmlu_electrical_engineering": 1.0, - "mmlu_elementary_mathematics": 1.0, - "mmlu_formal_logic": 1.0, - "mmlu_global_facts": 1.0, - "mmlu_high_school_biology": 1.0, - "mmlu_high_school_chemistry": 1.0, - "mmlu_high_school_computer_science": 1.0, - "mmlu_high_school_european_history": 1.0, - "mmlu_high_school_geography": 1.0, - "mmlu_high_school_government_and_politics": 1.0, - "mmlu_high_school_macroeconomics": 1.0, - "mmlu_high_school_mathematics": 1.0, - "mmlu_high_school_microeconomics": 1.0, - "mmlu_high_school_physics": 1.0, - "mmlu_high_school_psychology": 1.0, - "mmlu_high_school_statistics": 1.0, - "mmlu_high_school_us_history": 1.0, - "mmlu_high_school_world_history": 1.0, - "mmlu_human_aging": 1.0, - "mmlu_human_sexuality": 1.0, - "mmlu_humanities": 2, - "mmlu_international_law": 1.0, - "mmlu_jurisprudence": 1.0, - "mmlu_logical_fallacies": 1.0, - "mmlu_machine_learning": 1.0, - "mmlu_management": 1.0, - "mmlu_marketing": 1.0, - "mmlu_medical_genetics": 1.0, - "mmlu_miscellaneous": 1.0, - "mmlu_moral_disputes": 1.0, - "mmlu_moral_scenarios": 1.0, - "mmlu_nutrition": 1.0, - "mmlu_other": 2, - "mmlu_philosophy": 1.0, - "mmlu_prehistory": 1.0, - "mmlu_professional_accounting": 1.0, - "mmlu_professional_law": 1.0, - "mmlu_professional_medicine": 1.0, - "mmlu_professional_psychology": 1.0, - "mmlu_public_relations": 1.0, - "mmlu_security_studies": 1.0, - "mmlu_social_sciences": 2, - "mmlu_sociology": 1.0, - "mmlu_stem": 2, - "mmlu_us_foreign_policy": 1.0, - "mmlu_virology": 1.0, - "mmlu_world_religions": 1.0 - }, - "n-shot": { - "mmlu_abstract_algebra": 5, - "mmlu_anatomy": 5, - "mmlu_astronomy": 5, - "mmlu_business_ethics": 5, - "mmlu_clinical_knowledge": 5, - "mmlu_college_biology": 5, - "mmlu_college_chemistry": 5, - "mmlu_college_computer_science": 5, - "mmlu_college_mathematics": 5, - "mmlu_college_medicine": 5, - "mmlu_college_physics": 5, - "mmlu_computer_security": 5, - "mmlu_conceptual_physics": 5, - "mmlu_econometrics": 5, - "mmlu_electrical_engineering": 5, - "mmlu_elementary_mathematics": 5, - "mmlu_formal_logic": 5, - "mmlu_global_facts": 5, - "mmlu_high_school_biology": 5, - "mmlu_high_school_chemistry": 5, - "mmlu_high_school_computer_science": 5, - "mmlu_high_school_european_history": 5, - "mmlu_high_school_geography": 5, - "mmlu_high_school_government_and_politics": 5, - "mmlu_high_school_macroeconomics": 5, - "mmlu_high_school_mathematics": 5, - "mmlu_high_school_microeconomics": 5, - "mmlu_high_school_physics": 5, - "mmlu_high_school_psychology": 5, - "mmlu_high_school_statistics": 5, - "mmlu_high_school_us_history": 5, - "mmlu_high_school_world_history": 5, - "mmlu_human_aging": 5, - "mmlu_human_sexuality": 5, - "mmlu_international_law": 5, - "mmlu_jurisprudence": 5, - "mmlu_logical_fallacies": 5, - "mmlu_machine_learning": 5, - "mmlu_management": 5, - "mmlu_marketing": 5, - "mmlu_medical_genetics": 5, - "mmlu_miscellaneous": 5, - "mmlu_moral_disputes": 5, - "mmlu_moral_scenarios": 5, - "mmlu_nutrition": 5, - "mmlu_philosophy": 5, - "mmlu_prehistory": 5, - "mmlu_professional_accounting": 5, - "mmlu_professional_law": 5, - "mmlu_professional_medicine": 5, - "mmlu_professional_psychology": 5, - "mmlu_public_relations": 5, - "mmlu_security_studies": 5, - "mmlu_sociology": 5, - "mmlu_us_foreign_policy": 5, - "mmlu_virology": 5, - "mmlu_world_religions": 5 - }, - "higher_is_better": { - "mmlu": { - "acc": true - }, - "mmlu_abstract_algebra": { - "acc": true - }, - "mmlu_anatomy": { - "acc": true - }, - "mmlu_astronomy": { - "acc": true - }, - "mmlu_business_ethics": { - "acc": true - }, - "mmlu_clinical_knowledge": { - "acc": true - }, - "mmlu_college_biology": { - "acc": true - }, - "mmlu_college_chemistry": { - "acc": true - }, - "mmlu_college_computer_science": { - "acc": true - }, - "mmlu_college_mathematics": { - "acc": true - }, - "mmlu_college_medicine": { - "acc": true - }, - "mmlu_college_physics": { - "acc": true - }, - "mmlu_computer_security": { - "acc": true - }, - "mmlu_conceptual_physics": { - "acc": true - }, - "mmlu_econometrics": { - "acc": true - }, - "mmlu_electrical_engineering": { - "acc": true - }, - "mmlu_elementary_mathematics": { - "acc": true - }, - "mmlu_formal_logic": { - "acc": true - }, - "mmlu_global_facts": { - "acc": true - }, - "mmlu_high_school_biology": { - "acc": true - }, - "mmlu_high_school_chemistry": { - "acc": true - }, - "mmlu_high_school_computer_science": { - "acc": true - }, - "mmlu_high_school_european_history": { - "acc": true - }, - "mmlu_high_school_geography": { - "acc": true - }, - "mmlu_high_school_government_and_politics": { - "acc": true - }, - "mmlu_high_school_macroeconomics": { - "acc": true - }, - "mmlu_high_school_mathematics": { - "acc": true - }, - "mmlu_high_school_microeconomics": { - "acc": true - }, - "mmlu_high_school_physics": { - "acc": true - }, - "mmlu_high_school_psychology": { - "acc": true - }, - "mmlu_high_school_statistics": { - "acc": true - }, - "mmlu_high_school_us_history": { - "acc": true - }, - "mmlu_high_school_world_history": { - "acc": true - }, - "mmlu_human_aging": { - "acc": true - }, - "mmlu_human_sexuality": { - "acc": true - }, - "mmlu_humanities": { - "acc": true - }, - "mmlu_international_law": { - "acc": true - }, - "mmlu_jurisprudence": { - "acc": true - }, - "mmlu_logical_fallacies": { - "acc": true - }, - "mmlu_machine_learning": { - "acc": true - }, - "mmlu_management": { - "acc": true - }, - "mmlu_marketing": { - "acc": true - }, - "mmlu_medical_genetics": { - "acc": true - }, - "mmlu_miscellaneous": { - "acc": true - }, - "mmlu_moral_disputes": { - "acc": true - }, - "mmlu_moral_scenarios": { - "acc": true - }, - "mmlu_nutrition": { - "acc": true - }, - "mmlu_other": { - "acc": true - }, - "mmlu_philosophy": { - "acc": true - }, - "mmlu_prehistory": { - "acc": true - }, - "mmlu_professional_accounting": { - "acc": true - }, - "mmlu_professional_law": { - "acc": true - }, - "mmlu_professional_medicine": { - "acc": true - }, - "mmlu_professional_psychology": { - "acc": true - }, - "mmlu_public_relations": { - "acc": true - }, - "mmlu_security_studies": { - "acc": true - }, - "mmlu_social_sciences": { - "acc": true - }, - "mmlu_sociology": { - "acc": true - }, - "mmlu_stem": { - "acc": true - }, - "mmlu_us_foreign_policy": { - "acc": true - }, - "mmlu_virology": { - "acc": true - }, - "mmlu_world_religions": { - "acc": true - } - }, - "n-samples": { - "mmlu_conceptual_physics": { - "original": 235, - "effective": 235 - }, - "mmlu_college_mathematics": { - "original": 100, - "effective": 100 - }, - "mmlu_high_school_statistics": { - "original": 216, - "effective": 216 - }, - "mmlu_elementary_mathematics": { - "original": 378, - "effective": 378 - }, - "mmlu_high_school_physics": { - "original": 151, - "effective": 151 - }, - "mmlu_electrical_engineering": { - "original": 145, - "effective": 145 - }, - "mmlu_college_chemistry": { - "original": 100, - "effective": 100 - }, - "mmlu_high_school_mathematics": { - "original": 270, - "effective": 270 - }, - "mmlu_abstract_algebra": { - "original": 100, - "effective": 100 - }, - "mmlu_astronomy": { - "original": 152, - "effective": 152 - }, - "mmlu_machine_learning": { - "original": 112, - "effective": 112 - }, - "mmlu_college_computer_science": { - "original": 100, - "effective": 100 - }, - "mmlu_high_school_chemistry": { - "original": 203, - "effective": 203 - }, - "mmlu_college_physics": { - "original": 102, - "effective": 102 - }, - "mmlu_college_biology": { - "original": 144, - "effective": 144 - }, - "mmlu_anatomy": { - "original": 135, - "effective": 135 - }, - "mmlu_high_school_biology": { - "original": 310, - "effective": 310 - }, - "mmlu_computer_security": { - "original": 100, - "effective": 100 - }, - "mmlu_high_school_computer_science": { - "original": 100, - "effective": 100 - }, - "mmlu_professional_medicine": { - "original": 272, - "effective": 272 - }, - "mmlu_professional_accounting": { - "original": 282, - "effective": 282 - }, - "mmlu_global_facts": { - "original": 100, - "effective": 100 - }, - "mmlu_nutrition": { - "original": 306, - "effective": 306 - }, - "mmlu_marketing": { - "original": 234, - "effective": 234 - }, - "mmlu_management": { - "original": 103, - "effective": 103 - }, - "mmlu_human_aging": { - "original": 223, - "effective": 223 - }, - "mmlu_medical_genetics": { - "original": 100, - "effective": 100 - }, - "mmlu_virology": { - "original": 166, - "effective": 166 - }, - "mmlu_clinical_knowledge": { - "original": 265, - "effective": 265 - }, - "mmlu_miscellaneous": { - "original": 783, - "effective": 783 - }, - "mmlu_college_medicine": { - "original": 173, - "effective": 173 - }, - "mmlu_business_ethics": { - "original": 100, - "effective": 100 - }, - "mmlu_us_foreign_policy": { - "original": 100, - "effective": 100 - }, - "mmlu_econometrics": { - "original": 114, - "effective": 114 - }, - "mmlu_human_sexuality": { - "original": 131, - "effective": 131 - }, - "mmlu_public_relations": { - "original": 110, - "effective": 110 - }, - "mmlu_high_school_government_and_politics": { - "original": 193, - "effective": 193 - }, - "mmlu_high_school_psychology": { - "original": 545, - "effective": 545 - }, - "mmlu_high_school_macroeconomics": { - "original": 390, - "effective": 390 - }, - "mmlu_sociology": { - "original": 201, - "effective": 201 - }, - "mmlu_high_school_geography": { - "original": 198, - "effective": 198 - }, - "mmlu_professional_psychology": { - "original": 612, - "effective": 612 - }, - "mmlu_high_school_microeconomics": { - "original": 238, - "effective": 238 - }, - "mmlu_security_studies": { - "original": 245, - "effective": 245 - }, - "mmlu_professional_law": { - "original": 1534, - "effective": 1534 - }, - "mmlu_world_religions": { - "original": 171, - "effective": 171 - }, - "mmlu_logical_fallacies": { - "original": 163, - "effective": 163 - }, - "mmlu_formal_logic": { - "original": 126, - "effective": 126 - }, - "mmlu_moral_scenarios": { - "original": 895, - "effective": 895 - }, - "mmlu_high_school_european_history": { - "original": 165, - "effective": 165 - }, - "mmlu_high_school_us_history": { - "original": 204, - "effective": 204 - }, - "mmlu_prehistory": { - "original": 324, - "effective": 324 - }, - "mmlu_international_law": { - "original": 121, - "effective": 121 - }, - "mmlu_philosophy": { - "original": 311, - "effective": 311 - }, - "mmlu_jurisprudence": { - "original": 108, - "effective": 108 - }, - "mmlu_high_school_world_history": { - "original": 237, - "effective": 237 - }, - "mmlu_moral_disputes": { - "original": 346, - "effective": 346 - } - }, - "config": { - "model": "local-completions", - "model_args": "base_url=http://127.0.0.1:18308/v1/completions,model=qwen3-8b-ptq-then-qad,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768", - "batch_size": 1, - "batch_sizes": [], - "device": null, - "use_cache": "/results/lm_cache", - "limit": null, - "downsampling_ratio": null, - "bootstrap_iters": 100000, - "gen_kwargs": { - "temperature": 1e-07, - "top_p": 0.9999999 - }, - "random_seed": 42, - "numpy_seed": 42, - "torch_seed": 42, - "fewshot_seed": 42 - }, - "git_hash": null, - "date": 1791480725.774736, - "pretty_env_info": "name 'get_pretty_env_info' is not defined", - "transformers_version": "4.50.3", - "lm_eval_version": "No package metadata was found for lm_eval", - "upper_git_hash": null, - "tokenizer_pad_token": [ - "<|endoftext|>", - "151643" - ], - "tokenizer_eos_token": [ - "<|im_end|>", - "151645" - ], - "tokenizer_bos_token": [ - null, - "None" - ], - "eot_token_id": 151645, - "max_length": 32767, - "task_hashes": { - "mmlu_conceptual_physics": "d9a7c4fd2d286215624dffc2aaec8e0295a6353daf2300d3461537e3fb5425eb", - "mmlu_college_mathematics": "33a226a6b1a6492f38387286d516c0941f5d8debbdfaa054b11fcda1cdb6dc1a", - "mmlu_high_school_statistics": "0598ccf63c944eeab0ba3ad2dcfbcaebddf4c5d124174071beb52e8a73f19e86", - "mmlu_elementary_mathematics": "da1d3600f010ce76c00a62b991b7606ee7df0b2e82413ee1f1977440ddd8550d", - "mmlu_high_school_physics": "7780579caacd56f607968e81d5256d7d386095ab1d93b672047c5a5d8444ec1d", - "mmlu_electrical_engineering": "58857d23657299553045d3db7b7b0de43f9a078a2fd53902d0e59251ac497e9c", - "mmlu_college_chemistry": "20b91eb7e582696bf39216d1757ac477675fe6de640150d7ae29993234af04dc", - "mmlu_high_school_mathematics": "313c7a073f5b7339d283ad977ea74584374b6e0281898dfd89ff4bdfd2fb7b03", - "mmlu_abstract_algebra": "0e2f49943ac30da4eb2d7d2f5db73abb69da69d6988007197e6bdf138bfb3369", - "mmlu_astronomy": "83e572d3948bb6dbfb66031ab4dcb5b70054a5bc110de1a0fbc324940f4c04da", - "mmlu_machine_learning": "d02c202c0ad4c729890962d842ea70cad80faaae66a51a4c35c3685783ca06c1", - "mmlu_college_computer_science": "c2355a53d39d8bf1fbec4ea4b0f2b50c3fe104f2b36556ecae1038b0c315816a", - "mmlu_high_school_chemistry": "7b7e216abbacc50a1613fc9e02f3f3dda2709059394b121e43f3ad534cadcf80", - "mmlu_college_physics": "ffe4456084dbfeb89e20d4b6138b97bbf415fd619691dbc7b38bfe5934b30e10", - "mmlu_college_biology": "14cbae605287fc3c9ff25c8f2b2175619e036287a2e217767382b5f48f044943", - "mmlu_anatomy": "9578a460fa25b65077e6da48a36e49d1e6b094807d65c9e5bf7e07ad0fbb6296", - "mmlu_high_school_biology": "cfa833fafa27f991b540ece0e0583e1ebedfcaa50d030891935b000b7757c1cb", - "mmlu_computer_security": "e122c4bc113d812e7e083e316b358cc7e16a0d1c2e6506480caebcf9d1e27ecd", - "mmlu_high_school_computer_science": "2db28b8942d29e448cb587b398788f3cba6693cfe20772102b5bb1907999abea", - "mmlu_professional_medicine": "ba4af62ac5d26864b9e6339c6cbd16cbabcb5fbcf00366c02dd15da73c28daf9", - "mmlu_professional_accounting": "654d1dafcb0ff52e603404c1158924e92c5067cda9df2429eb0777c99c648995", - "mmlu_global_facts": "856fc8218f2f89f3e9abbe96070c530689a228cf5c9269c58bd03b4f0aa833b1", - "mmlu_nutrition": "468a945d5f4f35a2b1c974d7180d159d3e80a0611fa9160afee572df0dd69432", - "mmlu_marketing": "2018f5181b9a272ca8a4f65858e0d12a1bf87792934fc483022e9d1bc7e05997", - "mmlu_management": "6345661bb012d1a72c5d34f39e692cf47c131807a3bdfff7bf07ee9c5149d29e", - "mmlu_human_aging": "88802f2005a5c3c719ace0291d92034ef3c90ee1cf7488d5195f8c545c899d4d", - "mmlu_medical_genetics": "8819bc558f8142680673b743cf8e3681ea6856ece1e3895c24fc0f77a60da7c4", - "mmlu_virology": "67042eb89ab0d2a45f1d254dbb9da9de152fa1098d7740a05ac7ff8495b26534", - "mmlu_clinical_knowledge": "a3dba4354de80aa98c7db318923aedb8b78c02014a2be97c4071453ba619b8dc", - "mmlu_miscellaneous": "22548428bd482f372a4fba10d2a0c0a939faddd7e9781701a74358145211a660", - "mmlu_college_medicine": "bc04947722ee05b4d5051df09f0e50de44d4f49df44f5ff909dbe3264cbc2c53", - "mmlu_business_ethics": "a0e93295030eaac8a5dd303dd1514b9eb25ae1807f55d000b876c0e8ddd5e1c8", - "mmlu_us_foreign_policy": "1c1d39c546896fb1b9a513c5db8233ea4066c3ec79d8b85c2ab7944fa79ca8c5", - "mmlu_econometrics": "905a5e83b9a8173ef7345d14ac535c91c78c34b076314a2deb52cb2e6138502c", - "mmlu_human_sexuality": "9b20e343f77622c458fe394d257924eb8895d07ddd98b1e3dbaf366563f77cbb", - "mmlu_public_relations": "355105913812b28cc46607bf6710a92dcb346c9d35f489eb84e388de03390b32", - "mmlu_high_school_government_and_politics": "bd59fed7fa1dcf3e39a65a7ea71cc81573c4fb28bcd942cbc8d8d9afbe0e9d4c", - "mmlu_high_school_psychology": "e62d1ebd73c032fd0c3b99eaf20e7b77b13ae49e4fc506d7e69f9ae15e485a4c", - "mmlu_high_school_macroeconomics": "50ee63c5ac58c48f651464b5fd6cbbe0c8fa6002dd480fa9634ba630d9dc0417", - "mmlu_sociology": "65917e6a3b0027bdb12703440e17afc8f155c15087a9df249e0061d84381e0d2", - "mmlu_high_school_geography": "30d201a5bfcbaec8d0e74f58ddd23bd58204b1d99b9211e6a531d8b6e84eab24", - "mmlu_professional_psychology": "70ec543a94dccdb5e82f3ac5f006de88a1352fb86b2a2c68d496529cdd1f5e0c", - "mmlu_high_school_microeconomics": "705c7c5543f9987bdf14f7b39ddae1910e9097d0f491bb98a536140175d94732", - "mmlu_security_studies": "e7c1c4bd8669cc22b7a907cd3584c5caf80b82772d9741a3680563b4fb564084", - "mmlu_professional_law": "0e0e2459e1f45df128cf8460ebdea4da6cc99a9fbd9cb99fb5ebeee78575800e", - "mmlu_world_religions": "8ad90daa76079af018b36cb86143468d22e34a9cd3273a3c48ce4c4648f886dc", - "mmlu_logical_fallacies": "f4aa9887846d578f13ca7420b65369c344053cabe092607a16237dfeef07a1ba", - "mmlu_formal_logic": "646ee3f6abf70a7d0efd03cdc0fbebb9f2e8f40e3b5563e53a9d0a7a291bc2a7", - "mmlu_moral_scenarios": "83a877f6af97d85f853c38196827d751dc8f33db4253da17965fde0c5e3554e1", - "mmlu_high_school_european_history": "8c8ff9bcf9924cd457cc5b67c5f6a21925c09aadba42e646c088f2b99ba4e0e2", - "mmlu_high_school_us_history": "b5fdc459ac365a794c663a17cf8a34a702ea87eaa1a8b40fe49e8196ef88232c", - "mmlu_prehistory": "4aa1d25e8211294e34dead164bbde415d42feb245a4716a2471d929f81846f04", - "mmlu_international_law": "144d9e8693486d00f5cac911f3fbfbacdba9ac7c10340d50061262d9c3fdff9d", - "mmlu_philosophy": "c61151b7b50a37e90635e8b219018a35533a65a2a684542ead156b9f1020ffc1", - "mmlu_jurisprudence": "325cb19fa404379425d7add71b29e2e4cd3428c08a25f5cc718fd87e79ccd2bb", - "mmlu_high_school_world_history": "59ebbb27916baa0f2c6fa8b064ecbb042b43f6d492bbe18f2a78a63d40bf3dd0", - "mmlu_moral_disputes": "4d3e2d32e558de3c5eb0369e9a20280fda45fe04eeedf3356ad820e352293720" - }, - "model_source": "local-completions", - "model_name": "qwen3-8b-ptq-then-qad", - "model_name_sanitized": "qwen3-8b-ptq-then-qad", - "system_instruction": null, - "system_instruction_sha": null, - "fewshot_as_multiturn": false, - "chat_template": null, - "chat_template_sha": null, - "start_time": 9127034.591303237, - "end_time": 9130813.532724572, - "total_evaluation_time_seconds": "3778.941421335563" -} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json deleted file mode 100644 index 82333ad4cc2..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/lm_eval_results_2026-10-08T07-23-24.097523.json +++ /dev/null @@ -1,136 +0,0 @@ -{ - "results": { - "wikitext": { - "alias": "wikitext", - "word_perplexity,none": 13.68410924315444, - "word_perplexity_stderr,none": "N/A", - "byte_perplexity,none": 1.6310909243257237, - "byte_perplexity_stderr,none": "N/A", - "bits_per_byte,none": 0.7058372065854064, - "bits_per_byte_stderr,none": "N/A" - } - }, - "group_subtasks": { - "wikitext": [] - }, - "configs": { - "wikitext": { - "task": "wikitext", - "dataset_path": "EleutherAI/wikitext_document_level", - "dataset_name": "wikitext-2-raw-v1", - "dataset_kwargs": { - "trust_remote_code": true - }, - "training_split": "train", - "validation_split": "validation", - "test_split": "test", - "doc_to_text": "", - "doc_to_target": "def wikitext_detokenizer(doc):\n string = doc[\"page\"]\n # contractions\n string = string.replace(\"s '\", \"s'\")\n string = re.sub(r\"/' [0-9]/\", r\"/'[0-9]/\", string)\n # number separators\n string = string.replace(\" @-@ \", \"-\")\n string = string.replace(\" @,@ \", \",\")\n string = string.replace(\" @.@ \", \".\")\n # punctuation\n string = string.replace(\" : \", \": \")\n string = string.replace(\" ; \", \"; \")\n string = string.replace(\" . \", \". \")\n string = string.replace(\" ! \", \"! \")\n string = string.replace(\" ? \", \"? \")\n string = string.replace(\" , \", \", \")\n # double brackets\n string = re.sub(r\"\\(\\s*([^\\)]*?)\\s*\\)\", r\"(\\1)\", string)\n string = re.sub(r\"\\[\\s*([^\\]]*?)\\s*\\]\", r\"[\\1]\", string)\n string = re.sub(r\"{\\s*([^}]*?)\\s*}\", r\"{\\1}\", string)\n string = re.sub(r\"\\\"\\s*([^\\\"]*?)\\s*\\\"\", r'\"\\1\"', string)\n string = re.sub(r\"'\\s*([^']*?)\\s*'\", r\"'\\1'\", string)\n # miscellaneous\n string = string.replace(\"= = = =\", \"====\")\n string = string.replace(\"= = =\", \"===\")\n string = string.replace(\"= =\", \"==\")\n string = string.replace(\" \" + chr(176) + \" \", chr(176))\n string = string.replace(\" \\n\", \"\\n\")\n string = string.replace(\"\\n \", \"\\n\")\n string = string.replace(\" N \", \" 1 \")\n string = string.replace(\" 's\", \"'s\")\n\n return string\n", - "unsafe_code": false, - "process_results": "def process_results(doc, results):\n (loglikelihood,) = results\n # IMPORTANT: wikitext counts number of words in *original doc before detokenization*\n _words = len(re.split(r\"\\s+\", doc[\"page\"]))\n _bytes = len(doc[\"page\"].encode(\"utf-8\"))\n return {\n \"word_perplexity\": (loglikelihood, _words),\n \"byte_perplexity\": (loglikelihood, _bytes),\n \"bits_per_byte\": (loglikelihood, _bytes),\n }\n", - "description": "", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "num_fewshot": 0, - "metric_list": [ - { - "metric": "word_perplexity" - }, - { - "metric": "byte_perplexity" - }, - { - "metric": "bits_per_byte" - } - ], - "output_type": "loglikelihood_rolling", - "repeats": 1, - "should_decontaminate": true, - "doc_to_decontamination_query": "{{page}}", - "metadata": { - "version": 2.0, - "base_url": "http://localhost:3826", - "model": "qwen3-8b-ptq-then-qad", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 2, - "timeout": 300, - "max_retries": 2, - "stream": false - } - } - }, - "versions": { - "wikitext": 2.0 - }, - "n-shot": { - "wikitext": 0 - }, - "higher_is_better": { - "wikitext": { - "word_perplexity": false, - "byte_perplexity": false, - "bits_per_byte": false - } - }, - "n-samples": { - "wikitext": { - "original": 62, - "effective": 62 - } - }, - "config": { - "model": "local-completions", - "model_args": "base_url=http://localhost:3826,model=qwen3-8b-ptq-then-qad,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=2,timeout=300,max_retries=2,stream=False", - "batch_size": 1, - "batch_sizes": [], - "device": null, - "use_cache": "/results/lm_cache", - "limit": null, - "downsampling_ratio": null, - "bootstrap_iters": 100000, - "gen_kwargs": { - "temperature": 1e-07, - "top_p": 0.9999999 - }, - "random_seed": 42, - "numpy_seed": 42, - "torch_seed": 42, - "fewshot_seed": 42 - }, - "git_hash": null, - "date": 1791469379.5896866, - "pretty_env_info": "name 'get_pretty_env_info' is not defined", - "transformers_version": "4.50.3", - "lm_eval_version": "No package metadata was found for lm_eval", - "upper_git_hash": null, - "tokenizer_pad_token": [ - "<|endoftext|>", - "151643" - ], - "tokenizer_eos_token": [ - "<|im_end|>", - "151645" - ], - "tokenizer_bos_token": [ - null, - "None" - ], - "eot_token_id": 151645, - "max_length": 2047, - "task_hashes": { - "wikitext": "233109c41076287b5ba5a338ab7b085728acacf36d237a7ab99f34fcda161641" - }, - "model_source": "local-completions", - "model_name": "qwen3-8b-ptq-then-qad", - "model_name_sanitized": "qwen3-8b-ptq-then-qad", - "system_instruction": null, - "system_instruction_sha": null, - "fewshot_as_multiturn": false, - "chat_template": null, - "chat_template_sha": null, - "start_time": 1557654.864151966, - "end_time": 1557680.726953163, - "total_evaluation_time_seconds": "25.862801196984947" -} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json deleted file mode 100644 index e2342ced11b..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/lm_eval_results_2026-10-08T09-24-34.263592.json +++ /dev/null @@ -1,4030 +0,0 @@ -{ - "results": { - "mmlu": { - "acc,none": 0.7493946731234867, - "acc_stderr,none": 0.0034779802227510367, - "alias": "mmlu" - }, - "mmlu_humanities": { - "acc,none": 0.6641870350690755, - "acc_stderr,none": 0.006552018414272379, - "alias": " - humanities" - }, - "mmlu_formal_logic": { - "alias": " - formal_logic", - "acc,none": 0.6746031746031746, - "acc_stderr,none": 0.04190596438871138 - }, - "mmlu_high_school_european_history": { - "alias": " - high_school_european_history", - "acc,none": 0.8545454545454545, - "acc_stderr,none": 0.02753019635506653 - }, - "mmlu_high_school_us_history": { - "alias": " - high_school_us_history", - "acc,none": 0.8627450980392157, - "acc_stderr,none": 0.024152225962801553 - }, - "mmlu_high_school_world_history": { - "alias": " - high_school_world_history", - "acc,none": 0.8649789029535865, - "acc_stderr,none": 0.022245776632003705 - }, - "mmlu_international_law": { - "alias": " - international_law", - "acc,none": 0.7355371900826446, - "acc_stderr,none": 0.04026187527591209 - }, - "mmlu_jurisprudence": { - "alias": " - jurisprudence", - "acc,none": 0.8148148148148148, - "acc_stderr,none": 0.03755265865037181 - }, - "mmlu_logical_fallacies": { - "alias": " - logical_fallacies", - "acc,none": 0.8220858895705522, - "acc_stderr,none": 0.03004735765580667 - }, - "mmlu_moral_disputes": { - "alias": " - moral_disputes", - "acc,none": 0.7601156069364162, - "acc_stderr,none": 0.02298959254312351 - }, - "mmlu_moral_scenarios": { - "alias": " - moral_scenarios", - "acc,none": 0.5541899441340782, - "acc_stderr,none": 0.016623998513333176 - }, - "mmlu_philosophy": { - "alias": " - philosophy", - "acc,none": 0.7781350482315113, - "acc_stderr,none": 0.023598858292862998 - }, - "mmlu_prehistory": { - "alias": " - prehistory", - "acc,none": 0.8395061728395061, - "acc_stderr,none": 0.02042395535477805 - }, - "mmlu_professional_law": { - "alias": " - professional_law", - "acc,none": 0.5117340286831812, - "acc_stderr,none": 0.012766719019686833 - }, - "mmlu_world_religions": { - "alias": " - world_religions", - "acc,none": 0.8713450292397661, - "acc_stderr,none": 0.02567934272327688 - }, - "mmlu_other": { - "acc,none": 0.784036047634374, - "acc_stderr,none": 0.007130383176974831, - "alias": " - other" - }, - "mmlu_business_ethics": { - "alias": " - business_ethics", - "acc,none": 0.78, - "acc_stderr,none": 0.041633319989322654 - }, - "mmlu_clinical_knowledge": { - "alias": " - clinical_knowledge", - "acc,none": 0.7849056603773585, - "acc_stderr,none": 0.02528839450289141 - }, - "mmlu_college_medicine": { - "alias": " - college_medicine", - "acc,none": 0.791907514450867, - "acc_stderr,none": 0.030952890217749857 - }, - "mmlu_global_facts": { - "alias": " - global_facts", - "acc,none": 0.52, - "acc_stderr,none": 0.05021167315686783 - }, - "mmlu_human_aging": { - "alias": " - human_aging", - "acc,none": 0.7443946188340808, - "acc_stderr,none": 0.029275891003969906 - }, - "mmlu_management": { - "alias": " - management", - "acc,none": 0.8737864077669902, - "acc_stderr,none": 0.03288180278808631 - }, - "mmlu_marketing": { - "alias": " - marketing", - "acc,none": 0.9358974358974359, - "acc_stderr,none": 0.016046261631673164 - }, - "mmlu_medical_genetics": { - "alias": " - medical_genetics", - "acc,none": 0.83, - "acc_stderr,none": 0.03775251680686369 - }, - "mmlu_miscellaneous": { - "alias": " - miscellaneous", - "acc,none": 0.8582375478927203, - "acc_stderr,none": 0.012473289071271968 - }, - "mmlu_nutrition": { - "alias": " - nutrition", - "acc,none": 0.8104575163398693, - "acc_stderr,none": 0.022442358263336185 - }, - "mmlu_professional_accounting": { - "alias": " - professional_accounting", - "acc,none": 0.5921985815602837, - "acc_stderr,none": 0.02931601177634362 - }, - "mmlu_professional_medicine": { - "alias": " - professional_medicine", - "acc,none": 0.8198529411764706, - "acc_stderr,none": 0.0233451636165449 - }, - "mmlu_virology": { - "alias": " - virology", - "acc,none": 0.5602409638554217, - "acc_stderr,none": 0.0386413992369912 - }, - "mmlu_social_sciences": { - "acc,none": 0.8394540136496588, - "acc_stderr,none": 0.006505444273959088, - "alias": " - social sciences" - }, - "mmlu_econometrics": { - "alias": " - econometrics", - "acc,none": 0.6578947368421053, - "acc_stderr,none": 0.044629175353369355 - }, - "mmlu_high_school_geography": { - "alias": " - high_school_geography", - "acc,none": 0.8686868686868687, - "acc_stderr,none": 0.02406315641682249 - }, - "mmlu_high_school_government_and_politics": { - "alias": " - high_school_government_and_politics", - "acc,none": 0.9326424870466321, - "acc_stderr,none": 0.01808839383907894 - }, - "mmlu_high_school_macroeconomics": { - "alias": " - high_school_macroeconomics", - "acc,none": 0.8102564102564103, - "acc_stderr,none": 0.019880165406588844 - }, - "mmlu_high_school_microeconomics": { - "alias": " - high_school_microeconomics", - "acc,none": 0.9369747899159664, - "acc_stderr,none": 0.015785085223670888 - }, - "mmlu_high_school_psychology": { - "alias": " - high_school_psychology", - "acc,none": 0.9119266055045872, - "acc_stderr,none": 0.012150743719481636 - }, - "mmlu_human_sexuality": { - "alias": " - human_sexuality", - "acc,none": 0.8320610687022901, - "acc_stderr,none": 0.0327854853734314 - }, - "mmlu_professional_psychology": { - "alias": " - professional_psychology", - "acc,none": 0.7859477124183006, - "acc_stderr,none": 0.01659342966232898 - }, - "mmlu_public_relations": { - "alias": " - public_relations", - "acc,none": 0.7, - "acc_stderr,none": 0.04389311454644288 - }, - "mmlu_security_studies": { - "alias": " - security_studies", - "acc,none": 0.7959183673469388, - "acc_stderr,none": 0.02580128347509046 - }, - "mmlu_sociology": { - "alias": " - sociology", - "acc,none": 0.8656716417910447, - "acc_stderr,none": 0.024112678240900763 - }, - "mmlu_us_foreign_policy": { - "alias": " - us_foreign_policy", - "acc,none": 0.84, - "acc_stderr,none": 0.03684529491774706 - }, - "mmlu_stem": { - "acc,none": 0.7545195052331113, - "acc_stderr,none": 0.0073915569536853855, - "alias": " - stem" - }, - "mmlu_abstract_algebra": { - "alias": " - abstract_algebra", - "acc,none": 0.55, - "acc_stderr,none": 0.05 - }, - "mmlu_anatomy": { - "alias": " - anatomy", - "acc,none": 0.7037037037037037, - "acc_stderr,none": 0.03944624162501111 - }, - "mmlu_astronomy": { - "alias": " - astronomy", - "acc,none": 0.9013157894736842, - "acc_stderr,none": 0.024270227737522694 - }, - "mmlu_college_biology": { - "alias": " - college_biology", - "acc,none": 0.875, - "acc_stderr,none": 0.02765610492929436 - }, - "mmlu_college_chemistry": { - "alias": " - college_chemistry", - "acc,none": 0.56, - "acc_stderr,none": 0.049888765156985884 - }, - "mmlu_college_computer_science": { - "alias": " - college_computer_science", - "acc,none": 0.73, - "acc_stderr,none": 0.04461960433384737 - }, - "mmlu_college_mathematics": { - "alias": " - college_mathematics", - "acc,none": 0.58, - "acc_stderr,none": 0.04960449637488582 - }, - "mmlu_college_physics": { - "alias": " - college_physics", - "acc,none": 0.6470588235294118, - "acc_stderr,none": 0.04755129616062946 - }, - "mmlu_computer_security": { - "alias": " - computer_security", - "acc,none": 0.81, - "acc_stderr,none": 0.039427724440366255 - }, - "mmlu_conceptual_physics": { - "alias": " - conceptual_physics", - "acc,none": 0.8468085106382979, - "acc_stderr,none": 0.023545179061675213 - }, - "mmlu_electrical_engineering": { - "alias": " - electrical_engineering", - "acc,none": 0.8068965517241379, - "acc_stderr,none": 0.03289445522127399 - }, - "mmlu_elementary_mathematics": { - "alias": " - elementary_mathematics", - "acc,none": 0.7883597883597884, - "acc_stderr,none": 0.02103733150526295 - }, - "mmlu_high_school_biology": { - "alias": " - high_school_biology", - "acc,none": 0.9096774193548387, - "acc_stderr,none": 0.016306570644488316 - }, - "mmlu_high_school_chemistry": { - "alias": " - high_school_chemistry", - "acc,none": 0.7586206896551724, - "acc_stderr,none": 0.030108330718011642 - }, - "mmlu_high_school_computer_science": { - "alias": " - high_school_computer_science", - "acc,none": 0.86, - "acc_stderr,none": 0.0348735088019777 - }, - "mmlu_high_school_mathematics": { - "alias": " - high_school_mathematics", - "acc,none": 0.5370370370370371, - "acc_stderr,none": 0.030401786406101534 - }, - "mmlu_high_school_physics": { - "alias": " - high_school_physics", - "acc,none": 0.7284768211920529, - "acc_stderr,none": 0.03631329803969657 - }, - "mmlu_high_school_statistics": { - "alias": " - high_school_statistics", - "acc,none": 0.7824074074074074, - "acc_stderr,none": 0.028139689444859672 - }, - "mmlu_machine_learning": { - "alias": " - machine_learning", - "acc,none": 0.6428571428571429, - "acc_stderr,none": 0.045479609997643805 - } - }, - "groups": { - "mmlu": { - "acc,none": 0.7493946731234867, - "acc_stderr,none": 0.0034779802227510367, - "alias": "mmlu" - }, - "mmlu_humanities": { - "acc,none": 0.6641870350690755, - "acc_stderr,none": 0.006552018414272379, - "alias": " - humanities" - }, - "mmlu_other": { - "acc,none": 0.784036047634374, - "acc_stderr,none": 0.007130383176974831, - "alias": " - other" - }, - "mmlu_social_sciences": { - "acc,none": 0.8394540136496588, - "acc_stderr,none": 0.006505444273959088, - "alias": " - social sciences" - }, - "mmlu_stem": { - "acc,none": 0.7545195052331113, - "acc_stderr,none": 0.0073915569536853855, - "alias": " - stem" - } - }, - "group_subtasks": { - "mmlu_humanities": [ - "mmlu_formal_logic", - "mmlu_logical_fallacies", - "mmlu_jurisprudence", - "mmlu_professional_law", - "mmlu_high_school_european_history", - "mmlu_moral_scenarios", - "mmlu_moral_disputes", - "mmlu_international_law", - "mmlu_world_religions", - "mmlu_high_school_us_history", - "mmlu_prehistory", - "mmlu_philosophy", - "mmlu_high_school_world_history" - ], - "mmlu_social_sciences": [ - "mmlu_sociology", - "mmlu_econometrics", - "mmlu_high_school_government_and_politics", - "mmlu_security_studies", - "mmlu_high_school_microeconomics", - "mmlu_human_sexuality", - "mmlu_public_relations", - "mmlu_us_foreign_policy", - "mmlu_high_school_geography", - "mmlu_high_school_macroeconomics", - "mmlu_high_school_psychology", - "mmlu_professional_psychology" - ], - "mmlu_other": [ - "mmlu_management", - "mmlu_nutrition", - "mmlu_business_ethics", - "mmlu_college_medicine", - "mmlu_global_facts", - "mmlu_virology", - "mmlu_human_aging", - "mmlu_marketing", - "mmlu_professional_medicine", - "mmlu_professional_accounting", - "mmlu_medical_genetics", - "mmlu_miscellaneous", - "mmlu_clinical_knowledge" - ], - "mmlu_stem": [ - "mmlu_college_mathematics", - "mmlu_college_biology", - "mmlu_anatomy", - "mmlu_electrical_engineering", - "mmlu_high_school_chemistry", - "mmlu_college_computer_science", - "mmlu_high_school_statistics", - "mmlu_abstract_algebra", - "mmlu_college_chemistry", - "mmlu_elementary_mathematics", - "mmlu_conceptual_physics", - "mmlu_high_school_mathematics", - "mmlu_college_physics", - "mmlu_machine_learning", - "mmlu_high_school_computer_science", - "mmlu_high_school_biology", - "mmlu_computer_security", - "mmlu_astronomy", - "mmlu_high_school_physics" - ], - "mmlu": [ - "mmlu_stem", - "mmlu_other", - "mmlu_social_sciences", - "mmlu_humanities" - ] - }, - "configs": { - "mmlu_abstract_algebra": { - "task": "mmlu_abstract_algebra", - "task_alias": "abstract_algebra", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "abstract_algebra", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about abstract algebra.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_anatomy": { - "task": "mmlu_anatomy", - "task_alias": "anatomy", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "anatomy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about anatomy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_astronomy": { - "task": "mmlu_astronomy", - "task_alias": "astronomy", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "astronomy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about astronomy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_business_ethics": { - "task": "mmlu_business_ethics", - "task_alias": "business_ethics", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "business_ethics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about business ethics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_clinical_knowledge": { - "task": "mmlu_clinical_knowledge", - "task_alias": "clinical_knowledge", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "clinical_knowledge", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about clinical knowledge.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_biology": { - "task": "mmlu_college_biology", - "task_alias": "college_biology", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_biology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college biology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_chemistry": { - "task": "mmlu_college_chemistry", - "task_alias": "college_chemistry", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_chemistry", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college chemistry.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_computer_science": { - "task": "mmlu_college_computer_science", - "task_alias": "college_computer_science", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_computer_science", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college computer science.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_mathematics": { - "task": "mmlu_college_mathematics", - "task_alias": "college_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_medicine": { - "task": "mmlu_college_medicine", - "task_alias": "college_medicine", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_medicine", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college medicine.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_college_physics": { - "task": "mmlu_college_physics", - "task_alias": "college_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "college_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about college physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_computer_security": { - "task": "mmlu_computer_security", - "task_alias": "computer_security", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "computer_security", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about computer security.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_conceptual_physics": { - "task": "mmlu_conceptual_physics", - "task_alias": "conceptual_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "conceptual_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about conceptual physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_econometrics": { - "task": "mmlu_econometrics", - "task_alias": "econometrics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "econometrics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about econometrics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_electrical_engineering": { - "task": "mmlu_electrical_engineering", - "task_alias": "electrical_engineering", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "electrical_engineering", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about electrical engineering.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_elementary_mathematics": { - "task": "mmlu_elementary_mathematics", - "task_alias": "elementary_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "elementary_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about elementary mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_formal_logic": { - "task": "mmlu_formal_logic", - "task_alias": "formal_logic", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "formal_logic", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about formal logic.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_global_facts": { - "task": "mmlu_global_facts", - "task_alias": "global_facts", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "global_facts", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about global facts.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_biology": { - "task": "mmlu_high_school_biology", - "task_alias": "high_school_biology", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_biology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school biology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_chemistry": { - "task": "mmlu_high_school_chemistry", - "task_alias": "high_school_chemistry", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_chemistry", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school chemistry.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_computer_science": { - "task": "mmlu_high_school_computer_science", - "task_alias": "high_school_computer_science", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_computer_science", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school computer science.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_european_history": { - "task": "mmlu_high_school_european_history", - "task_alias": "high_school_european_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_european_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school european history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_geography": { - "task": "mmlu_high_school_geography", - "task_alias": "high_school_geography", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_geography", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school geography.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_government_and_politics": { - "task": "mmlu_high_school_government_and_politics", - "task_alias": "high_school_government_and_politics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_government_and_politics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school government and politics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_macroeconomics": { - "task": "mmlu_high_school_macroeconomics", - "task_alias": "high_school_macroeconomics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_macroeconomics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school macroeconomics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_mathematics": { - "task": "mmlu_high_school_mathematics", - "task_alias": "high_school_mathematics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_mathematics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school mathematics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_microeconomics": { - "task": "mmlu_high_school_microeconomics", - "task_alias": "high_school_microeconomics", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_microeconomics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school microeconomics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_physics": { - "task": "mmlu_high_school_physics", - "task_alias": "high_school_physics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_physics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school physics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_psychology": { - "task": "mmlu_high_school_psychology", - "task_alias": "high_school_psychology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_psychology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school psychology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_statistics": { - "task": "mmlu_high_school_statistics", - "task_alias": "high_school_statistics", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_statistics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school statistics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_us_history": { - "task": "mmlu_high_school_us_history", - "task_alias": "high_school_us_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_us_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school us history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_high_school_world_history": { - "task": "mmlu_high_school_world_history", - "task_alias": "high_school_world_history", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "high_school_world_history", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about high school world history.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_human_aging": { - "task": "mmlu_human_aging", - "task_alias": "human_aging", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "human_aging", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about human aging.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_human_sexuality": { - "task": "mmlu_human_sexuality", - "task_alias": "human_sexuality", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "human_sexuality", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about human sexuality.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_international_law": { - "task": "mmlu_international_law", - "task_alias": "international_law", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "international_law", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about international law.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_jurisprudence": { - "task": "mmlu_jurisprudence", - "task_alias": "jurisprudence", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "jurisprudence", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about jurisprudence.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_logical_fallacies": { - "task": "mmlu_logical_fallacies", - "task_alias": "logical_fallacies", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "logical_fallacies", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about logical fallacies.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_machine_learning": { - "task": "mmlu_machine_learning", - "task_alias": "machine_learning", - "tag": "mmlu_stem_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "machine_learning", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about machine learning.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_management": { - "task": "mmlu_management", - "task_alias": "management", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "management", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about management.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_marketing": { - "task": "mmlu_marketing", - "task_alias": "marketing", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "marketing", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about marketing.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_medical_genetics": { - "task": "mmlu_medical_genetics", - "task_alias": "medical_genetics", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "medical_genetics", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about medical genetics.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_miscellaneous": { - "task": "mmlu_miscellaneous", - "task_alias": "miscellaneous", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "miscellaneous", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about miscellaneous.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_moral_disputes": { - "task": "mmlu_moral_disputes", - "task_alias": "moral_disputes", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "moral_disputes", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about moral disputes.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_moral_scenarios": { - "task": "mmlu_moral_scenarios", - "task_alias": "moral_scenarios", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "moral_scenarios", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about moral scenarios.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_nutrition": { - "task": "mmlu_nutrition", - "task_alias": "nutrition", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "nutrition", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about nutrition.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_philosophy": { - "task": "mmlu_philosophy", - "task_alias": "philosophy", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "philosophy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about philosophy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_prehistory": { - "task": "mmlu_prehistory", - "task_alias": "prehistory", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "prehistory", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about prehistory.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_accounting": { - "task": "mmlu_professional_accounting", - "task_alias": "professional_accounting", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_accounting", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional accounting.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_law": { - "task": "mmlu_professional_law", - "task_alias": "professional_law", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_law", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional law.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_medicine": { - "task": "mmlu_professional_medicine", - "task_alias": "professional_medicine", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_medicine", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional medicine.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_professional_psychology": { - "task": "mmlu_professional_psychology", - "task_alias": "professional_psychology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "professional_psychology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about professional psychology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_public_relations": { - "task": "mmlu_public_relations", - "task_alias": "public_relations", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "public_relations", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about public relations.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_security_studies": { - "task": "mmlu_security_studies", - "task_alias": "security_studies", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "security_studies", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about security studies.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_sociology": { - "task": "mmlu_sociology", - "task_alias": "sociology", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "sociology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about sociology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_us_foreign_policy": { - "task": "mmlu_us_foreign_policy", - "task_alias": "us_foreign_policy", - "tag": "mmlu_social_sciences_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "us_foreign_policy", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about us foreign policy.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_virology": { - "task": "mmlu_virology", - "task_alias": "virology", - "tag": "mmlu_other_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "virology", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about virology.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - }, - "mmlu_world_religions": { - "task": "mmlu_world_religions", - "task_alias": "world_religions", - "tag": "mmlu_humanities_tasks", - "dataset_path": "cais/mmlu", - "dataset_name": "world_religions", - "dataset_kwargs": { - "trust_remote_code": true - }, - "test_split": "test", - "fewshot_split": "dev", - "doc_to_text": "{{question.strip()}}\nA. {{choices[0]}}\nB. {{choices[1]}}\nC. {{choices[2]}}\nD. {{choices[3]}}\nAnswer:", - "doc_to_target": "answer", - "unsafe_code": false, - "doc_to_choice": [ - "A", - "B", - "C", - "D" - ], - "description": "The following are multiple choice questions (with answers) about world religions.\n\n", - "instruction": "Below is a multi-choice question. You must reply with only a single letter (either A, B, C or D)", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "fewshot_config": { - "sampler": "first_n" - }, - "num_fewshot": 5, - "metric_list": [ - { - "metric": "acc", - "aggregation": "mean", - "higher_is_better": true - } - ], - "output_type": "multiple_choice", - "repeats": 1, - "should_decontaminate": false, - "metadata": { - "version": 1.0, - "base_url": "http://127.0.0.1:18300/v1/completions", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 1, - "timeout": 300, - "max_retries": 2, - "stream": false, - "max_length": 32768 - } - } - }, - "versions": { - "mmlu": 2, - "mmlu_abstract_algebra": 1.0, - "mmlu_anatomy": 1.0, - "mmlu_astronomy": 1.0, - "mmlu_business_ethics": 1.0, - "mmlu_clinical_knowledge": 1.0, - "mmlu_college_biology": 1.0, - "mmlu_college_chemistry": 1.0, - "mmlu_college_computer_science": 1.0, - "mmlu_college_mathematics": 1.0, - "mmlu_college_medicine": 1.0, - "mmlu_college_physics": 1.0, - "mmlu_computer_security": 1.0, - "mmlu_conceptual_physics": 1.0, - "mmlu_econometrics": 1.0, - "mmlu_electrical_engineering": 1.0, - "mmlu_elementary_mathematics": 1.0, - "mmlu_formal_logic": 1.0, - "mmlu_global_facts": 1.0, - "mmlu_high_school_biology": 1.0, - "mmlu_high_school_chemistry": 1.0, - "mmlu_high_school_computer_science": 1.0, - "mmlu_high_school_european_history": 1.0, - "mmlu_high_school_geography": 1.0, - "mmlu_high_school_government_and_politics": 1.0, - "mmlu_high_school_macroeconomics": 1.0, - "mmlu_high_school_mathematics": 1.0, - "mmlu_high_school_microeconomics": 1.0, - "mmlu_high_school_physics": 1.0, - "mmlu_high_school_psychology": 1.0, - "mmlu_high_school_statistics": 1.0, - "mmlu_high_school_us_history": 1.0, - "mmlu_high_school_world_history": 1.0, - "mmlu_human_aging": 1.0, - "mmlu_human_sexuality": 1.0, - "mmlu_humanities": 2, - "mmlu_international_law": 1.0, - "mmlu_jurisprudence": 1.0, - "mmlu_logical_fallacies": 1.0, - "mmlu_machine_learning": 1.0, - "mmlu_management": 1.0, - "mmlu_marketing": 1.0, - "mmlu_medical_genetics": 1.0, - "mmlu_miscellaneous": 1.0, - "mmlu_moral_disputes": 1.0, - "mmlu_moral_scenarios": 1.0, - "mmlu_nutrition": 1.0, - "mmlu_other": 2, - "mmlu_philosophy": 1.0, - "mmlu_prehistory": 1.0, - "mmlu_professional_accounting": 1.0, - "mmlu_professional_law": 1.0, - "mmlu_professional_medicine": 1.0, - "mmlu_professional_psychology": 1.0, - "mmlu_public_relations": 1.0, - "mmlu_security_studies": 1.0, - "mmlu_social_sciences": 2, - "mmlu_sociology": 1.0, - "mmlu_stem": 2, - "mmlu_us_foreign_policy": 1.0, - "mmlu_virology": 1.0, - "mmlu_world_religions": 1.0 - }, - "n-shot": { - "mmlu_abstract_algebra": 5, - "mmlu_anatomy": 5, - "mmlu_astronomy": 5, - "mmlu_business_ethics": 5, - "mmlu_clinical_knowledge": 5, - "mmlu_college_biology": 5, - "mmlu_college_chemistry": 5, - "mmlu_college_computer_science": 5, - "mmlu_college_mathematics": 5, - "mmlu_college_medicine": 5, - "mmlu_college_physics": 5, - "mmlu_computer_security": 5, - "mmlu_conceptual_physics": 5, - "mmlu_econometrics": 5, - "mmlu_electrical_engineering": 5, - "mmlu_elementary_mathematics": 5, - "mmlu_formal_logic": 5, - "mmlu_global_facts": 5, - "mmlu_high_school_biology": 5, - "mmlu_high_school_chemistry": 5, - "mmlu_high_school_computer_science": 5, - "mmlu_high_school_european_history": 5, - "mmlu_high_school_geography": 5, - "mmlu_high_school_government_and_politics": 5, - "mmlu_high_school_macroeconomics": 5, - "mmlu_high_school_mathematics": 5, - "mmlu_high_school_microeconomics": 5, - "mmlu_high_school_physics": 5, - "mmlu_high_school_psychology": 5, - "mmlu_high_school_statistics": 5, - "mmlu_high_school_us_history": 5, - "mmlu_high_school_world_history": 5, - "mmlu_human_aging": 5, - "mmlu_human_sexuality": 5, - "mmlu_international_law": 5, - "mmlu_jurisprudence": 5, - "mmlu_logical_fallacies": 5, - "mmlu_machine_learning": 5, - "mmlu_management": 5, - "mmlu_marketing": 5, - "mmlu_medical_genetics": 5, - "mmlu_miscellaneous": 5, - "mmlu_moral_disputes": 5, - "mmlu_moral_scenarios": 5, - "mmlu_nutrition": 5, - "mmlu_philosophy": 5, - "mmlu_prehistory": 5, - "mmlu_professional_accounting": 5, - "mmlu_professional_law": 5, - "mmlu_professional_medicine": 5, - "mmlu_professional_psychology": 5, - "mmlu_public_relations": 5, - "mmlu_security_studies": 5, - "mmlu_sociology": 5, - "mmlu_us_foreign_policy": 5, - "mmlu_virology": 5, - "mmlu_world_religions": 5 - }, - "higher_is_better": { - "mmlu": { - "acc": true - }, - "mmlu_abstract_algebra": { - "acc": true - }, - "mmlu_anatomy": { - "acc": true - }, - "mmlu_astronomy": { - "acc": true - }, - "mmlu_business_ethics": { - "acc": true - }, - "mmlu_clinical_knowledge": { - "acc": true - }, - "mmlu_college_biology": { - "acc": true - }, - "mmlu_college_chemistry": { - "acc": true - }, - "mmlu_college_computer_science": { - "acc": true - }, - "mmlu_college_mathematics": { - "acc": true - }, - "mmlu_college_medicine": { - "acc": true - }, - "mmlu_college_physics": { - "acc": true - }, - "mmlu_computer_security": { - "acc": true - }, - "mmlu_conceptual_physics": { - "acc": true - }, - "mmlu_econometrics": { - "acc": true - }, - "mmlu_electrical_engineering": { - "acc": true - }, - "mmlu_elementary_mathematics": { - "acc": true - }, - "mmlu_formal_logic": { - "acc": true - }, - "mmlu_global_facts": { - "acc": true - }, - "mmlu_high_school_biology": { - "acc": true - }, - "mmlu_high_school_chemistry": { - "acc": true - }, - "mmlu_high_school_computer_science": { - "acc": true - }, - "mmlu_high_school_european_history": { - "acc": true - }, - "mmlu_high_school_geography": { - "acc": true - }, - "mmlu_high_school_government_and_politics": { - "acc": true - }, - "mmlu_high_school_macroeconomics": { - "acc": true - }, - "mmlu_high_school_mathematics": { - "acc": true - }, - "mmlu_high_school_microeconomics": { - "acc": true - }, - "mmlu_high_school_physics": { - "acc": true - }, - "mmlu_high_school_psychology": { - "acc": true - }, - "mmlu_high_school_statistics": { - "acc": true - }, - "mmlu_high_school_us_history": { - "acc": true - }, - "mmlu_high_school_world_history": { - "acc": true - }, - "mmlu_human_aging": { - "acc": true - }, - "mmlu_human_sexuality": { - "acc": true - }, - "mmlu_humanities": { - "acc": true - }, - "mmlu_international_law": { - "acc": true - }, - "mmlu_jurisprudence": { - "acc": true - }, - "mmlu_logical_fallacies": { - "acc": true - }, - "mmlu_machine_learning": { - "acc": true - }, - "mmlu_management": { - "acc": true - }, - "mmlu_marketing": { - "acc": true - }, - "mmlu_medical_genetics": { - "acc": true - }, - "mmlu_miscellaneous": { - "acc": true - }, - "mmlu_moral_disputes": { - "acc": true - }, - "mmlu_moral_scenarios": { - "acc": true - }, - "mmlu_nutrition": { - "acc": true - }, - "mmlu_other": { - "acc": true - }, - "mmlu_philosophy": { - "acc": true - }, - "mmlu_prehistory": { - "acc": true - }, - "mmlu_professional_accounting": { - "acc": true - }, - "mmlu_professional_law": { - "acc": true - }, - "mmlu_professional_medicine": { - "acc": true - }, - "mmlu_professional_psychology": { - "acc": true - }, - "mmlu_public_relations": { - "acc": true - }, - "mmlu_security_studies": { - "acc": true - }, - "mmlu_social_sciences": { - "acc": true - }, - "mmlu_sociology": { - "acc": true - }, - "mmlu_stem": { - "acc": true - }, - "mmlu_us_foreign_policy": { - "acc": true - }, - "mmlu_virology": { - "acc": true - }, - "mmlu_world_religions": { - "acc": true - } - }, - "n-samples": { - "mmlu_college_mathematics": { - "original": 100, - "effective": 100 - }, - "mmlu_college_biology": { - "original": 144, - "effective": 144 - }, - "mmlu_anatomy": { - "original": 135, - "effective": 135 - }, - "mmlu_electrical_engineering": { - "original": 145, - "effective": 145 - }, - "mmlu_high_school_chemistry": { - "original": 203, - "effective": 203 - }, - "mmlu_college_computer_science": { - "original": 100, - "effective": 100 - }, - "mmlu_high_school_statistics": { - "original": 216, - "effective": 216 - }, - "mmlu_abstract_algebra": { - "original": 100, - "effective": 100 - }, - "mmlu_college_chemistry": { - "original": 100, - "effective": 100 - }, - "mmlu_elementary_mathematics": { - "original": 378, - "effective": 378 - }, - "mmlu_conceptual_physics": { - "original": 235, - "effective": 235 - }, - "mmlu_high_school_mathematics": { - "original": 270, - "effective": 270 - }, - "mmlu_college_physics": { - "original": 102, - "effective": 102 - }, - "mmlu_machine_learning": { - "original": 112, - "effective": 112 - }, - "mmlu_high_school_computer_science": { - "original": 100, - "effective": 100 - }, - "mmlu_high_school_biology": { - "original": 310, - "effective": 310 - }, - "mmlu_computer_security": { - "original": 100, - "effective": 100 - }, - "mmlu_astronomy": { - "original": 152, - "effective": 152 - }, - "mmlu_high_school_physics": { - "original": 151, - "effective": 151 - }, - "mmlu_management": { - "original": 103, - "effective": 103 - }, - "mmlu_nutrition": { - "original": 306, - "effective": 306 - }, - "mmlu_business_ethics": { - "original": 100, - "effective": 100 - }, - "mmlu_college_medicine": { - "original": 173, - "effective": 173 - }, - "mmlu_global_facts": { - "original": 100, - "effective": 100 - }, - "mmlu_virology": { - "original": 166, - "effective": 166 - }, - "mmlu_human_aging": { - "original": 223, - "effective": 223 - }, - "mmlu_marketing": { - "original": 234, - "effective": 234 - }, - "mmlu_professional_medicine": { - "original": 272, - "effective": 272 - }, - "mmlu_professional_accounting": { - "original": 282, - "effective": 282 - }, - "mmlu_medical_genetics": { - "original": 100, - "effective": 100 - }, - "mmlu_miscellaneous": { - "original": 783, - "effective": 783 - }, - "mmlu_clinical_knowledge": { - "original": 265, - "effective": 265 - }, - "mmlu_sociology": { - "original": 201, - "effective": 201 - }, - "mmlu_econometrics": { - "original": 114, - "effective": 114 - }, - "mmlu_high_school_government_and_politics": { - "original": 193, - "effective": 193 - }, - "mmlu_security_studies": { - "original": 245, - "effective": 245 - }, - "mmlu_high_school_microeconomics": { - "original": 238, - "effective": 238 - }, - "mmlu_human_sexuality": { - "original": 131, - "effective": 131 - }, - "mmlu_public_relations": { - "original": 110, - "effective": 110 - }, - "mmlu_us_foreign_policy": { - "original": 100, - "effective": 100 - }, - "mmlu_high_school_geography": { - "original": 198, - "effective": 198 - }, - "mmlu_high_school_macroeconomics": { - "original": 390, - "effective": 390 - }, - "mmlu_high_school_psychology": { - "original": 545, - "effective": 545 - }, - "mmlu_professional_psychology": { - "original": 612, - "effective": 612 - }, - "mmlu_formal_logic": { - "original": 126, - "effective": 126 - }, - "mmlu_logical_fallacies": { - "original": 163, - "effective": 163 - }, - "mmlu_jurisprudence": { - "original": 108, - "effective": 108 - }, - "mmlu_professional_law": { - "original": 1534, - "effective": 1534 - }, - "mmlu_high_school_european_history": { - "original": 165, - "effective": 165 - }, - "mmlu_moral_scenarios": { - "original": 895, - "effective": 895 - }, - "mmlu_moral_disputes": { - "original": 346, - "effective": 346 - }, - "mmlu_international_law": { - "original": 121, - "effective": 121 - }, - "mmlu_world_religions": { - "original": 171, - "effective": 171 - }, - "mmlu_high_school_us_history": { - "original": 204, - "effective": 204 - }, - "mmlu_prehistory": { - "original": 324, - "effective": 324 - }, - "mmlu_philosophy": { - "original": 311, - "effective": 311 - }, - "mmlu_high_school_world_history": { - "original": 237, - "effective": 237 - } - }, - "config": { - "model": "local-completions", - "model_args": "base_url=http://127.0.0.1:18300/v1/completions,model=qwen3-8b-teacher,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768", - "batch_size": 1, - "batch_sizes": [], - "device": null, - "use_cache": "/results/lm_cache", - "limit": null, - "downsampling_ratio": null, - "bootstrap_iters": 100000, - "gen_kwargs": { - "temperature": 1e-07, - "top_p": 0.9999999 - }, - "random_seed": 42, - "numpy_seed": 42, - "torch_seed": 42, - "fewshot_seed": 42 - }, - "git_hash": null, - "date": 1791474058.866205, - "pretty_env_info": "name 'get_pretty_env_info' is not defined", - "transformers_version": "4.50.3", - "lm_eval_version": "No package metadata was found for lm_eval", - "upper_git_hash": null, - "tokenizer_pad_token": [ - "<|endoftext|>", - "151643" - ], - "tokenizer_eos_token": [ - "<|im_end|>", - "151645" - ], - "tokenizer_bos_token": [ - null, - "None" - ], - "eot_token_id": 151645, - "max_length": 32767, - "task_hashes": { - "mmlu_college_mathematics": "33a226a6b1a6492f38387286d516c0941f5d8debbdfaa054b11fcda1cdb6dc1a", - "mmlu_college_biology": "14cbae605287fc3c9ff25c8f2b2175619e036287a2e217767382b5f48f044943", - "mmlu_anatomy": "9578a460fa25b65077e6da48a36e49d1e6b094807d65c9e5bf7e07ad0fbb6296", - "mmlu_electrical_engineering": "58857d23657299553045d3db7b7b0de43f9a078a2fd53902d0e59251ac497e9c", - "mmlu_high_school_chemistry": "7b7e216abbacc50a1613fc9e02f3f3dda2709059394b121e43f3ad534cadcf80", - "mmlu_college_computer_science": "c2355a53d39d8bf1fbec4ea4b0f2b50c3fe104f2b36556ecae1038b0c315816a", - "mmlu_high_school_statistics": "0598ccf63c944eeab0ba3ad2dcfbcaebddf4c5d124174071beb52e8a73f19e86", - "mmlu_abstract_algebra": "0e2f49943ac30da4eb2d7d2f5db73abb69da69d6988007197e6bdf138bfb3369", - "mmlu_college_chemistry": "20b91eb7e582696bf39216d1757ac477675fe6de640150d7ae29993234af04dc", - "mmlu_elementary_mathematics": "da1d3600f010ce76c00a62b991b7606ee7df0b2e82413ee1f1977440ddd8550d", - "mmlu_conceptual_physics": "d9a7c4fd2d286215624dffc2aaec8e0295a6353daf2300d3461537e3fb5425eb", - "mmlu_high_school_mathematics": "313c7a073f5b7339d283ad977ea74584374b6e0281898dfd89ff4bdfd2fb7b03", - "mmlu_college_physics": "ffe4456084dbfeb89e20d4b6138b97bbf415fd619691dbc7b38bfe5934b30e10", - "mmlu_machine_learning": "d02c202c0ad4c729890962d842ea70cad80faaae66a51a4c35c3685783ca06c1", - "mmlu_high_school_computer_science": "2db28b8942d29e448cb587b398788f3cba6693cfe20772102b5bb1907999abea", - "mmlu_high_school_biology": "cfa833fafa27f991b540ece0e0583e1ebedfcaa50d030891935b000b7757c1cb", - "mmlu_computer_security": "e122c4bc113d812e7e083e316b358cc7e16a0d1c2e6506480caebcf9d1e27ecd", - "mmlu_astronomy": "83e572d3948bb6dbfb66031ab4dcb5b70054a5bc110de1a0fbc324940f4c04da", - "mmlu_high_school_physics": "7780579caacd56f607968e81d5256d7d386095ab1d93b672047c5a5d8444ec1d", - "mmlu_management": "6345661bb012d1a72c5d34f39e692cf47c131807a3bdfff7bf07ee9c5149d29e", - "mmlu_nutrition": "468a945d5f4f35a2b1c974d7180d159d3e80a0611fa9160afee572df0dd69432", - "mmlu_business_ethics": "a0e93295030eaac8a5dd303dd1514b9eb25ae1807f55d000b876c0e8ddd5e1c8", - "mmlu_college_medicine": "bc04947722ee05b4d5051df09f0e50de44d4f49df44f5ff909dbe3264cbc2c53", - "mmlu_global_facts": "856fc8218f2f89f3e9abbe96070c530689a228cf5c9269c58bd03b4f0aa833b1", - "mmlu_virology": "67042eb89ab0d2a45f1d254dbb9da9de152fa1098d7740a05ac7ff8495b26534", - "mmlu_human_aging": "88802f2005a5c3c719ace0291d92034ef3c90ee1cf7488d5195f8c545c899d4d", - "mmlu_marketing": "2018f5181b9a272ca8a4f65858e0d12a1bf87792934fc483022e9d1bc7e05997", - "mmlu_professional_medicine": "ba4af62ac5d26864b9e6339c6cbd16cbabcb5fbcf00366c02dd15da73c28daf9", - "mmlu_professional_accounting": "654d1dafcb0ff52e603404c1158924e92c5067cda9df2429eb0777c99c648995", - "mmlu_medical_genetics": "8819bc558f8142680673b743cf8e3681ea6856ece1e3895c24fc0f77a60da7c4", - "mmlu_miscellaneous": "22548428bd482f372a4fba10d2a0c0a939faddd7e9781701a74358145211a660", - "mmlu_clinical_knowledge": "a3dba4354de80aa98c7db318923aedb8b78c02014a2be97c4071453ba619b8dc", - "mmlu_sociology": "65917e6a3b0027bdb12703440e17afc8f155c15087a9df249e0061d84381e0d2", - "mmlu_econometrics": "905a5e83b9a8173ef7345d14ac535c91c78c34b076314a2deb52cb2e6138502c", - "mmlu_high_school_government_and_politics": "bd59fed7fa1dcf3e39a65a7ea71cc81573c4fb28bcd942cbc8d8d9afbe0e9d4c", - "mmlu_security_studies": "e7c1c4bd8669cc22b7a907cd3584c5caf80b82772d9741a3680563b4fb564084", - "mmlu_high_school_microeconomics": "705c7c5543f9987bdf14f7b39ddae1910e9097d0f491bb98a536140175d94732", - "mmlu_human_sexuality": "9b20e343f77622c458fe394d257924eb8895d07ddd98b1e3dbaf366563f77cbb", - "mmlu_public_relations": "355105913812b28cc46607bf6710a92dcb346c9d35f489eb84e388de03390b32", - "mmlu_us_foreign_policy": "1c1d39c546896fb1b9a513c5db8233ea4066c3ec79d8b85c2ab7944fa79ca8c5", - "mmlu_high_school_geography": "30d201a5bfcbaec8d0e74f58ddd23bd58204b1d99b9211e6a531d8b6e84eab24", - "mmlu_high_school_macroeconomics": "50ee63c5ac58c48f651464b5fd6cbbe0c8fa6002dd480fa9634ba630d9dc0417", - "mmlu_high_school_psychology": "e62d1ebd73c032fd0c3b99eaf20e7b77b13ae49e4fc506d7e69f9ae15e485a4c", - "mmlu_professional_psychology": "70ec543a94dccdb5e82f3ac5f006de88a1352fb86b2a2c68d496529cdd1f5e0c", - "mmlu_formal_logic": "646ee3f6abf70a7d0efd03cdc0fbebb9f2e8f40e3b5563e53a9d0a7a291bc2a7", - "mmlu_logical_fallacies": "f4aa9887846d578f13ca7420b65369c344053cabe092607a16237dfeef07a1ba", - "mmlu_jurisprudence": "325cb19fa404379425d7add71b29e2e4cd3428c08a25f5cc718fd87e79ccd2bb", - "mmlu_professional_law": "0e0e2459e1f45df128cf8460ebdea4da6cc99a9fbd9cb99fb5ebeee78575800e", - "mmlu_high_school_european_history": "8c8ff9bcf9924cd457cc5b67c5f6a21925c09aadba42e646c088f2b99ba4e0e2", - "mmlu_moral_scenarios": "83a877f6af97d85f853c38196827d751dc8f33db4253da17965fde0c5e3554e1", - "mmlu_moral_disputes": "4d3e2d32e558de3c5eb0369e9a20280fda45fe04eeedf3356ad820e352293720", - "mmlu_international_law": "144d9e8693486d00f5cac911f3fbfbacdba9ac7c10340d50061262d9c3fdff9d", - "mmlu_world_religions": "8ad90daa76079af018b36cb86143468d22e34a9cd3273a3c48ce4c4648f886dc", - "mmlu_high_school_us_history": "b5fdc459ac365a794c663a17cf8a34a702ea87eaa1a8b40fe49e8196ef88232c", - "mmlu_prehistory": "4aa1d25e8211294e34dead164bbde415d42feb245a4716a2471d929f81846f04", - "mmlu_philosophy": "c61151b7b50a37e90635e8b219018a35533a65a2a684542ead156b9f1020ffc1", - "mmlu_high_school_world_history": "59ebbb27916baa0f2c6fa8b064ecbb042b43f6d492bbe18f2a78a63d40bf3dd0" - }, - "model_source": "local-completions", - "model_name": "qwen3-8b-teacher", - "model_name_sanitized": "qwen3-8b-teacher", - "system_instruction": null, - "system_instruction_sha": null, - "fewshot_as_multiturn": false, - "chat_template": null, - "chat_template_sha": null, - "start_time": 1506881.406094961, - "end_time": 1509498.154492089, - "total_evaluation_time_seconds": "2616.748397127958" -} \ No newline at end of file diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json deleted file mode 100644 index cd67a2972b2..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/lm_eval_results_2026-10-08T07-22-39.797863.json +++ /dev/null @@ -1,136 +0,0 @@ -{ - "results": { - "wikitext": { - "alias": "wikitext", - "word_perplexity,none": 13.484664501311268, - "word_perplexity_stderr,none": "N/A", - "byte_perplexity,none": 1.6266186785114045, - "byte_perplexity_stderr,none": "N/A", - "bits_per_byte,none": 0.7018760856152558, - "bits_per_byte_stderr,none": "N/A" - } - }, - "group_subtasks": { - "wikitext": [] - }, - "configs": { - "wikitext": { - "task": "wikitext", - "dataset_path": "EleutherAI/wikitext_document_level", - "dataset_name": "wikitext-2-raw-v1", - "dataset_kwargs": { - "trust_remote_code": true - }, - "training_split": "train", - "validation_split": "validation", - "test_split": "test", - "doc_to_text": "", - "doc_to_target": "def wikitext_detokenizer(doc):\n string = doc[\"page\"]\n # contractions\n string = string.replace(\"s '\", \"s'\")\n string = re.sub(r\"/' [0-9]/\", r\"/'[0-9]/\", string)\n # number separators\n string = string.replace(\" @-@ \", \"-\")\n string = string.replace(\" @,@ \", \",\")\n string = string.replace(\" @.@ \", \".\")\n # punctuation\n string = string.replace(\" : \", \": \")\n string = string.replace(\" ; \", \"; \")\n string = string.replace(\" . \", \". \")\n string = string.replace(\" ! \", \"! \")\n string = string.replace(\" ? \", \"? \")\n string = string.replace(\" , \", \", \")\n # double brackets\n string = re.sub(r\"\\(\\s*([^\\)]*?)\\s*\\)\", r\"(\\1)\", string)\n string = re.sub(r\"\\[\\s*([^\\]]*?)\\s*\\]\", r\"[\\1]\", string)\n string = re.sub(r\"{\\s*([^}]*?)\\s*}\", r\"{\\1}\", string)\n string = re.sub(r\"\\\"\\s*([^\\\"]*?)\\s*\\\"\", r'\"\\1\"', string)\n string = re.sub(r\"'\\s*([^']*?)\\s*'\", r\"'\\1'\", string)\n # miscellaneous\n string = string.replace(\"= = = =\", \"====\")\n string = string.replace(\"= = =\", \"===\")\n string = string.replace(\"= =\", \"==\")\n string = string.replace(\" \" + chr(176) + \" \", chr(176))\n string = string.replace(\" \\n\", \"\\n\")\n string = string.replace(\"\\n \", \"\\n\")\n string = string.replace(\" N \", \" 1 \")\n string = string.replace(\" 's\", \"'s\")\n\n return string\n", - "unsafe_code": false, - "process_results": "def process_results(doc, results):\n (loglikelihood,) = results\n # IMPORTANT: wikitext counts number of words in *original doc before detokenization*\n _words = len(re.split(r\"\\s+\", doc[\"page\"]))\n _bytes = len(doc[\"page\"].encode(\"utf-8\"))\n return {\n \"word_perplexity\": (loglikelihood, _words),\n \"byte_perplexity\": (loglikelihood, _bytes),\n \"bits_per_byte\": (loglikelihood, _bytes),\n }\n", - "description": "", - "target_delimiter": " ", - "fewshot_delimiter": "\n\n", - "num_fewshot": 0, - "metric_list": [ - { - "metric": "word_perplexity" - }, - { - "metric": "byte_perplexity" - }, - { - "metric": "bits_per_byte" - } - ], - "output_type": "loglikelihood_rolling", - "repeats": 1, - "should_decontaminate": true, - "doc_to_decontamination_query": "{{page}}", - "metadata": { - "version": 2.0, - "base_url": "http://localhost:3826", - "model": "qwen3-8b-teacher", - "tokenized_requests": true, - "tokenizer": "/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218", - "tokenizer_backend": "huggingface", - "num_concurrent": 2, - "timeout": 300, - "max_retries": 2, - "stream": false - } - } - }, - "versions": { - "wikitext": 2.0 - }, - "n-shot": { - "wikitext": 0 - }, - "higher_is_better": { - "wikitext": { - "word_perplexity": false, - "byte_perplexity": false, - "bits_per_byte": false - } - }, - "n-samples": { - "wikitext": { - "original": 62, - "effective": 62 - } - }, - "config": { - "model": "local-completions", - "model_args": "base_url=http://localhost:3826,model=qwen3-8b-teacher,tokenized_requests=True,tokenizer=/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218,tokenizer_backend=huggingface,num_concurrent=2,timeout=300,max_retries=2,stream=False", - "batch_size": 1, - "batch_sizes": [], - "device": null, - "use_cache": "/results/lm_cache", - "limit": null, - "downsampling_ratio": null, - "bootstrap_iters": 100000, - "gen_kwargs": { - "temperature": 1e-07, - "top_p": 0.9999999 - }, - "random_seed": 42, - "numpy_seed": 42, - "torch_seed": 42, - "fewshot_seed": 42 - }, - "git_hash": null, - "date": 1791469340.7379298, - "pretty_env_info": "name 'get_pretty_env_info' is not defined", - "transformers_version": "4.50.3", - "lm_eval_version": "No package metadata was found for lm_eval", - "upper_git_hash": null, - "tokenizer_pad_token": [ - "<|endoftext|>", - "151643" - ], - "tokenizer_eos_token": [ - "<|im_end|>", - "151645" - ], - "tokenizer_bos_token": [ - null, - "None" - ], - "eot_token_id": 151645, - "max_length": 2047, - "task_hashes": { - "wikitext": "233109c41076287b5ba5a338ab7b085728acacf36d237a7ab99f34fcda161641" - }, - "model_source": "local-completions", - "model_name": "qwen3-8b-teacher", - "model_name_sanitized": "qwen3-8b-teacher", - "system_instruction": null, - "system_instruction_sha": null, - "fewshot_as_multiturn": false, - "chat_template": null, - "chat_template_sha": null, - "start_time": 309831.550427795, - "end_time": 309851.959472321, - "total_evaluation_time_seconds": "20.409044525993522" -} \ No newline at end of file From 5616ccdc6282edf7daf5f30552bef401e883d51c Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 01:23:08 -0700 Subject: [PATCH 06/19] Simplify nel configs (tokenizer_path variable), not using hf_cache Signed-off-by: Daniel Korzekwa --- .../results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml | 2 +- .../qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml | 2 +- .../qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml | 2 +- .../wikitext_full/nel_config.yaml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml index 1ac23d2ed40..2bbb547dd80 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml @@ -34,7 +34,7 @@ deployment: HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + tokenizer_path: /checkpoint port: 18104 evaluation: tasks: diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml index 33f31c4e269..dfd42dcc412 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml @@ -35,5 +35,5 @@ deployment: HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + tokenizer_path: /checkpoint port: 18106 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml index 0615a9dba11..2d7e30220bb 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml @@ -34,7 +34,7 @@ deployment: HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + tokenizer_path: /checkpoint port: 18108 evaluation: tasks: diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml index bf394beddd6..1d4b1313206 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml @@ -35,5 +35,5 @@ deployment: HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + tokenizer_path: /checkpoint port: 18110 From ec7d2447b6908285893d08b3ff8c12243db50509 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 01:33:17 -0700 Subject: [PATCH 07/19] Fix teacher checkpoint symlinks and align evaluation tokenizer paths Signed-off-by: Daniel Korzekwa --- .../mmlu_full/nel_config.yaml | 11 +++++++---- .../wikitext_full/nel_config.yaml | 11 +++++++---- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml index 60aa86cddc7..74dedbdff8e 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml @@ -20,8 +20,12 @@ execution: mount_home: false deployment: /path/to/hf-cache: /hf-cache + # NEL mounts checkpoint_path at /checkpoint; cached files link to /blobs. + /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: /path/to/hf-cache: /hf-cache + /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218: /checkpoint + /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: image: vllm/vllm-openai:v0.28.0-cu129 checkpoint_path: ${oc.env:QWEN3_HF_CACHE}/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 @@ -30,10 +34,9 @@ deployment: data_parallel_size: 1 env_vars: HF_HOME: lit:/hf-cache - command: vllm serve /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 --served-model-name ${deployment.served_model_name} - --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization - 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto + tokenizer_path: /checkpoint port: 18100 evaluation: tasks: diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml index 3d0da2e88bb..e16fa88365b 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml @@ -21,8 +21,12 @@ execution: mount_home: false deployment: /path/to/hf-cache: /hf-cache + # NEL mounts checkpoint_path at /checkpoint; cached files link to /blobs. + /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: /path/to/hf-cache: /hf-cache + /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218: /checkpoint + /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: image: vllm/vllm-openai:v0.28.0-cu129 checkpoint_path: ${oc.env:QWEN3_HF_CACHE}/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 @@ -31,8 +35,7 @@ deployment: data_parallel_size: 1 env_vars: HF_HOME: lit:/hf-cache - command: vllm serve /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 --served-model-name ${deployment.served_model_name} - --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization - 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto + tokenizer_path: /checkpoint port: 18102 From c6938356372882daf1a2b580b45159a2d97b2f00 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 01:57:42 -0700 Subject: [PATCH 08/19] Use the same oc.env:MODEL_CHECKPOINT across different evaluated models Signed-off-by: Daniel Korzekwa --- .../results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml | 2 +- .../qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml | 2 +- .../mmlu_full/nel_config.yaml | 2 +- .../wikitext_full/nel_config.yaml | 2 +- .../qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml | 4 ++-- .../qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml | 4 ++-- 6 files changed, 8 insertions(+), 8 deletions(-) diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml index 2bbb547dd80..938e66eee13 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml @@ -26,7 +26,7 @@ execution: /path/to/checkpoint: /checkpoint deployment: image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:QWEN3_MODEL_CHECKPOINT} + checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-ptq tensor_parallel_size: 1 data_parallel_size: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml index dfd42dcc412..f766804609d 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml @@ -27,7 +27,7 @@ execution: /path/to/checkpoint: /checkpoint deployment: image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:QWEN3_MODEL_CHECKPOINT} + checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-ptq tensor_parallel_size: 1 data_parallel_size: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml index 2d7e30220bb..2a2f40f2b53 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml @@ -26,7 +26,7 @@ execution: /path/to/checkpoint: /checkpoint deployment: image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:QWEN3_MODEL_CHECKPOINT} + checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-ptq-then-qad tensor_parallel_size: 1 data_parallel_size: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml index 1d4b1313206..20b28295f6b 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml @@ -27,7 +27,7 @@ execution: /path/to/checkpoint: /checkpoint deployment: image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:QWEN3_MODEL_CHECKPOINT} + checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-ptq-then-qad tensor_parallel_size: 1 data_parallel_size: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml index 74dedbdff8e..6d36ec6d715 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml @@ -24,11 +24,11 @@ execution: /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: /path/to/hf-cache: /hf-cache - /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218: /checkpoint + /path/to/checkpoint: /checkpoint /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:QWEN3_HF_CACHE}/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-teacher tensor_parallel_size: 1 data_parallel_size: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml index e16fa88365b..b110f2dfb42 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml @@ -25,11 +25,11 @@ execution: /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: /path/to/hf-cache: /hf-cache - /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218: /checkpoint + /path/to/checkpoint: /checkpoint /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:QWEN3_HF_CACHE}/hub/models--Qwen--Qwen3-8B/snapshots/b968826d9c46dd6066d109eabc6255188de91218 + checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-teacher tensor_parallel_size: 1 data_parallel_size: 1 From 20905a6eb3b1c83c744bb0c3783e049c81b92b50 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 02:21:31 -0700 Subject: [PATCH 09/19] move evaluation section to shared mmlu yaml file Signed-off-by: Daniel Korzekwa --- .../llm_qat/evaluation/nel_configs/mmlu.yaml | 4 ++-- .../mmlu_full/nel_config.yaml | 18 ------------------ .../mmlu_full/nel_config.yaml | 18 ------------------ .../mmlu_full/nel_config.yaml | 18 ------------------ 4 files changed, 2 insertions(+), 56 deletions(-) diff --git a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml index 0651f8f377d..a85b58f3cf0 100644 --- a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml +++ b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml @@ -4,7 +4,7 @@ evaluation: nemo_evaluator_config: config: params: - parallelism: 2 + parallelism: 1 request_timeout: 300 max_retries: 2 temperature: @@ -33,4 +33,4 @@ evaluation: tokenizer: ${deployment.tokenizer_path} tokenizer_backend: huggingface tokenized_requests: true - args: --seed 42 + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml index 938e66eee13..6b816c515de 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml @@ -36,21 +36,3 @@ deployment: 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint port: 18104 -evaluation: - tasks: - - name: lm-evaluation-harness.mmlu - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - extra: - num_fewshot: 5 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" - nemo_evaluator_config: - config: - params: - parallelism: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml index 2a2f40f2b53..b59ddbfe8d3 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml @@ -36,21 +36,3 @@ deployment: 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint port: 18108 -evaluation: - tasks: - - name: lm-evaluation-harness.mmlu - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - extra: - num_fewshot: 5 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" - nemo_evaluator_config: - config: - params: - parallelism: 1 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml index 6d36ec6d715..5e82dbf70b7 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml @@ -38,21 +38,3 @@ deployment: 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint port: 18100 -evaluation: - tasks: - - name: lm-evaluation-harness.mmlu - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - extra: - num_fewshot: 5 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" - nemo_evaluator_config: - config: - params: - parallelism: 1 From f8b4b1734eaf34cc57bea1948bb6cd43ff5869ad Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 02:27:22 -0700 Subject: [PATCH 10/19] Remove reference to unused /hf_cache - replaced with /checkpoint Signed-off-by: Daniel Korzekwa --- .../results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml | 4 ---- .../qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml | 4 ---- .../mmlu_full/nel_config.yaml | 4 ---- .../wikitext_full/nel_config.yaml | 4 ---- .../qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml | 4 ---- .../qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml | 4 ---- 6 files changed, 24 deletions(-) diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml index 6b816c515de..fbe9b032448 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml @@ -19,10 +19,8 @@ execution: mounts: mount_home: false deployment: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint evaluation: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint deployment: image: vllm/vllm-openai:v0.28.0-cu129 @@ -30,8 +28,6 @@ deployment: served_model_name: qwen3-8b-ptq tensor_parallel_size: 1 data_parallel_size: 1 - env_vars: - HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml index f766804609d..cfddb1ee29d 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml @@ -20,10 +20,8 @@ execution: mounts: mount_home: false deployment: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint evaluation: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint deployment: image: vllm/vllm-openai:v0.28.0-cu129 @@ -31,8 +29,6 @@ deployment: served_model_name: qwen3-8b-ptq tensor_parallel_size: 1 data_parallel_size: 1 - env_vars: - HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml index b59ddbfe8d3..b410fb9094f 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml @@ -19,10 +19,8 @@ execution: mounts: mount_home: false deployment: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint evaluation: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint deployment: image: vllm/vllm-openai:v0.28.0-cu129 @@ -30,8 +28,6 @@ deployment: served_model_name: qwen3-8b-ptq-then-qad tensor_parallel_size: 1 data_parallel_size: 1 - env_vars: - HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml index 20b28295f6b..f2a7da6a43a 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml @@ -20,10 +20,8 @@ execution: mounts: mount_home: false deployment: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint evaluation: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint deployment: image: vllm/vllm-openai:v0.28.0-cu129 @@ -31,8 +29,6 @@ deployment: served_model_name: qwen3-8b-ptq-then-qad tensor_parallel_size: 1 data_parallel_size: 1 - env_vars: - HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml index 5e82dbf70b7..887433bd309 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml @@ -19,11 +19,9 @@ execution: mounts: mount_home: false deployment: - /path/to/hf-cache: /hf-cache # NEL mounts checkpoint_path at /checkpoint; cached files link to /blobs. /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: @@ -32,8 +30,6 @@ deployment: served_model_name: qwen3-8b-teacher tensor_parallel_size: 1 data_parallel_size: 1 - env_vars: - HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml index b110f2dfb42..5781fde3a8a 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml @@ -20,11 +20,9 @@ execution: mounts: mount_home: false deployment: - /path/to/hf-cache: /hf-cache # NEL mounts checkpoint_path at /checkpoint; cached files link to /blobs. /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: - /path/to/hf-cache: /hf-cache /path/to/checkpoint: /checkpoint /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: @@ -33,8 +31,6 @@ deployment: served_model_name: qwen3-8b-teacher tensor_parallel_size: 1 data_parallel_size: 1 - env_vars: - HF_HOME: lit:/hf-cache command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto tokenizer_path: /checkpoint From eccbd99a7046fb8e842cb03814d2de4e7bb4d7f1 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 02:49:45 -0700 Subject: [PATCH 11/19] Create a dedicated deployment nel file Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 2 +- .../nel_configs/qwen3_8b_vllm_deployment.yaml | 9 +++++++++ .../qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml | 8 +------- .../wikitext_full/nel_config.yaml | 8 +------- .../mmlu_full/nel_config.yaml | 8 +------- .../wikitext_full/nel_config.yaml | 8 +------- .../mmlu_full/nel_config.yaml | 11 +++-------- .../wikitext_full/nel_config.yaml | 11 +++-------- 8 files changed, 20 insertions(+), 45 deletions(-) create mode 100644 examples/llm_qat/evaluation/nel_configs/qwen3_8b_vllm_deployment.yaml diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index f76bbdb0df0..2d895a66e97 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -16,5 +16,5 @@ QAD improves both metrics over PTQ but remains below the BF16 teacher. The quant NeMo Evaluator Launcher (NEL) configurations and evaluation results for the [Hugging Face QAT/QAD tutorial](../README.md). -- `nel_configs/`: shared benchmark recipes. +- `nel_configs/`: shared benchmark recipes and vLLM deployment settings. - `results///`: each evaluation's config `nel_config.yaml`. diff --git a/examples/llm_qat/evaluation/nel_configs/qwen3_8b_vllm_deployment.yaml b/examples/llm_qat/evaluation/nel_configs/qwen3_8b_vllm_deployment.yaml new file mode 100644 index 00000000000..68068dfe4e1 --- /dev/null +++ b/examples/llm_qat/evaluation/nel_configs/qwen3_8b_vllm_deployment.yaml @@ -0,0 +1,9 @@ +# Shared vLLM deployment for Qwen3-8B teacher, PTQ, and PTQ → QAD. +deployment: + image: vllm/vllm-openai:v0.28.0-cu129 + checkpoint_path: ${oc.env:MODEL_CHECKPOINT} + tensor_parallel_size: 1 + data_parallel_size: 1 + command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size + 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto + tokenizer_path: /checkpoint diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml index fbe9b032448..fb6d01c5f0a 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml @@ -1,6 +1,7 @@ defaults: - execution: slurm/default - deployment: vllm + - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/mmlu@_global_ - _self_ execution: @@ -23,12 +24,5 @@ execution: evaluation: /path/to/checkpoint: /checkpoint deployment: - image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-ptq - tensor_parallel_size: 1 - data_parallel_size: 1 - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /checkpoint port: 18104 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml index cfddb1ee29d..28c111b762d 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml @@ -2,6 +2,7 @@ defaults: - execution: slurm/default - deployment: vllm + - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/wikitext@_global_ - _self_ execution: @@ -24,12 +25,5 @@ execution: evaluation: /path/to/checkpoint: /checkpoint deployment: - image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-ptq - tensor_parallel_size: 1 - data_parallel_size: 1 - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /checkpoint port: 18106 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml index b410fb9094f..4987e17ff2f 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml @@ -1,6 +1,7 @@ defaults: - execution: slurm/default - deployment: vllm + - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/mmlu@_global_ - _self_ execution: @@ -23,12 +24,5 @@ execution: evaluation: /path/to/checkpoint: /checkpoint deployment: - image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-ptq-then-qad - tensor_parallel_size: 1 - data_parallel_size: 1 - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /checkpoint port: 18108 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml index f2a7da6a43a..93ae06a59a5 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml @@ -2,6 +2,7 @@ defaults: - execution: slurm/default - deployment: vllm + - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/wikitext@_global_ - _self_ execution: @@ -24,12 +25,5 @@ execution: evaluation: /path/to/checkpoint: /checkpoint deployment: - image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-ptq-then-qad - tensor_parallel_size: 1 - data_parallel_size: 1 - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /checkpoint port: 18110 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml index 887433bd309..a1e14b09577 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml @@ -1,6 +1,7 @@ defaults: - execution: slurm/default - deployment: vllm + - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/mmlu@_global_ - _self_ execution: @@ -19,18 +20,12 @@ execution: mounts: mount_home: false deployment: - # NEL mounts checkpoint_path at /checkpoint; cached files link to /blobs. + /path/to/checkpoint: /checkpoint + # Cached checkpoint files link to /blobs. /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: /path/to/checkpoint: /checkpoint /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: - image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-teacher - tensor_parallel_size: 1 - data_parallel_size: 1 - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /checkpoint port: 18100 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml index 5781fde3a8a..b249ed6ef68 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml @@ -2,6 +2,7 @@ defaults: - execution: slurm/default - deployment: vllm + - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/wikitext@_global_ - _self_ execution: @@ -20,18 +21,12 @@ execution: mounts: mount_home: false deployment: - # NEL mounts checkpoint_path at /checkpoint; cached files link to /blobs. + /path/to/checkpoint: /checkpoint + # Cached checkpoint files link to /blobs. /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: /path/to/checkpoint: /checkpoint /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: - image: vllm/vllm-openai:v0.28.0-cu129 - checkpoint_path: ${oc.env:MODEL_CHECKPOINT} served_model_name: qwen3-8b-teacher - tensor_parallel_size: 1 - data_parallel_size: 1 - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto - tokenizer_path: /checkpoint port: 18102 From bd9bfbddde753aa7027a26304747c6d5cf1db568 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 02:52:54 -0700 Subject: [PATCH 12/19] Created a shared slurm execution nel config file Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 2 +- .../nel_configs/h100_slurm_execution.yaml | 20 +++++++++++++++++++ .../mmlu_full/nel_config.yaml | 20 +------------------ .../wikitext_full/nel_config.yaml | 20 +------------------ .../mmlu_full/nel_config.yaml | 20 +------------------ .../wikitext_full/nel_config.yaml | 20 +------------------ .../mmlu_full/nel_config.yaml | 17 +--------------- .../wikitext_full/nel_config.yaml | 17 +--------------- 8 files changed, 27 insertions(+), 109 deletions(-) create mode 100644 examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index 2d895a66e97..2fb3232bfa7 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -16,5 +16,5 @@ QAD improves both metrics over PTQ but remains below the BF16 teacher. The quant NeMo Evaluator Launcher (NEL) configurations and evaluation results for the [Hugging Face QAT/QAD tutorial](../README.md). -- `nel_configs/`: shared benchmark recipes and vLLM deployment settings. +- `nel_configs/`: shared benchmark recipes, Slurm execution settings, and vLLM deployment settings. - `results///`: each evaluation's config `nel_config.yaml`. diff --git a/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml b/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml new file mode 100644 index 00000000000..d6299ef8feb --- /dev/null +++ b/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml @@ -0,0 +1,20 @@ +# Shared single-GPU Slurm execution settings for H100 evaluation. +execution: + hostname: ${oc.env:NEL_SLURM_HOST} + username: ${oc.env:SLURM_USER} + account: ${oc.env:NEL_SLURM_ACCOUNT} + partition: ${oc.env:NEL_SLURM_PARTITION} + output_dir: ${oc.env:NEL_OUTPUT_DIR} + num_nodes: 1 + ntasks_per_node: 1 + gres: gpu:1 + walltime: 04:00:00 + sbatch_extra_flags: + exclusive: false + mem: 64G + mounts: + mount_home: false + deployment: + /path/to/checkpoint: /checkpoint + evaluation: + /path/to/checkpoint: /checkpoint diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml index fb6d01c5f0a..f3720d90056 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml @@ -1,28 +1,10 @@ defaults: - execution: slurm/default + - ../../../nel_configs/h100_slurm_execution@_global_ - deployment: vllm - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/mmlu@_global_ - _self_ -execution: - hostname: ${oc.env:NEL_SLURM_HOST} - username: ${oc.env:SLURM_USER} - account: ${oc.env:NEL_SLURM_ACCOUNT} - partition: ${oc.env:NEL_SLURM_PARTITION} - output_dir: ${oc.env:NEL_OUTPUT_DIR} - num_nodes: 1 - ntasks_per_node: 1 - gres: gpu:1 - walltime: 04:00:00 - sbatch_extra_flags: - exclusive: false - mem: 64G - mounts: - mount_home: false - deployment: - /path/to/checkpoint: /checkpoint - evaluation: - /path/to/checkpoint: /checkpoint deployment: served_model_name: qwen3-8b-ptq port: 18104 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml index 28c111b762d..bf3ba04f386 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml @@ -1,29 +1,11 @@ # Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: - execution: slurm/default + - ../../../nel_configs/h100_slurm_execution@_global_ - deployment: vllm - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/wikitext@_global_ - _self_ -execution: - hostname: ${oc.env:NEL_SLURM_HOST} - username: ${oc.env:SLURM_USER} - account: ${oc.env:NEL_SLURM_ACCOUNT} - partition: ${oc.env:NEL_SLURM_PARTITION} - output_dir: ${oc.env:NEL_OUTPUT_DIR} - num_nodes: 1 - ntasks_per_node: 1 - gres: gpu:1 - walltime: 04:00:00 - sbatch_extra_flags: - exclusive: false - mem: 64G - mounts: - mount_home: false - deployment: - /path/to/checkpoint: /checkpoint - evaluation: - /path/to/checkpoint: /checkpoint deployment: served_model_name: qwen3-8b-ptq port: 18106 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml index 4987e17ff2f..cfa1e96854b 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml @@ -1,28 +1,10 @@ defaults: - execution: slurm/default + - ../../../nel_configs/h100_slurm_execution@_global_ - deployment: vllm - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/mmlu@_global_ - _self_ -execution: - hostname: ${oc.env:NEL_SLURM_HOST} - username: ${oc.env:SLURM_USER} - account: ${oc.env:NEL_SLURM_ACCOUNT} - partition: ${oc.env:NEL_SLURM_PARTITION} - output_dir: ${oc.env:NEL_OUTPUT_DIR} - num_nodes: 1 - ntasks_per_node: 1 - gres: gpu:1 - walltime: 04:00:00 - sbatch_extra_flags: - exclusive: false - mem: 64G - mounts: - mount_home: false - deployment: - /path/to/checkpoint: /checkpoint - evaluation: - /path/to/checkpoint: /checkpoint deployment: served_model_name: qwen3-8b-ptq-then-qad port: 18108 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml index 93ae06a59a5..5f240ca05b3 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml @@ -1,29 +1,11 @@ # Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: - execution: slurm/default + - ../../../nel_configs/h100_slurm_execution@_global_ - deployment: vllm - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/wikitext@_global_ - _self_ -execution: - hostname: ${oc.env:NEL_SLURM_HOST} - username: ${oc.env:SLURM_USER} - account: ${oc.env:NEL_SLURM_ACCOUNT} - partition: ${oc.env:NEL_SLURM_PARTITION} - output_dir: ${oc.env:NEL_OUTPUT_DIR} - num_nodes: 1 - ntasks_per_node: 1 - gres: gpu:1 - walltime: 04:00:00 - sbatch_extra_flags: - exclusive: false - mem: 64G - mounts: - mount_home: false - deployment: - /path/to/checkpoint: /checkpoint - evaluation: - /path/to/checkpoint: /checkpoint deployment: served_model_name: qwen3-8b-ptq-then-qad port: 18110 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml index a1e14b09577..bff5c5d251a 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml @@ -1,30 +1,15 @@ defaults: - execution: slurm/default + - ../../../nel_configs/h100_slurm_execution@_global_ - deployment: vllm - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/mmlu@_global_ - _self_ execution: - hostname: ${oc.env:NEL_SLURM_HOST} - username: ${oc.env:SLURM_USER} - account: ${oc.env:NEL_SLURM_ACCOUNT} - partition: ${oc.env:NEL_SLURM_PARTITION} - output_dir: ${oc.env:NEL_OUTPUT_DIR} - num_nodes: 1 - ntasks_per_node: 1 - gres: gpu:1 - walltime: 04:00:00 - sbatch_extra_flags: - exclusive: false - mem: 64G mounts: - mount_home: false deployment: - /path/to/checkpoint: /checkpoint - # Cached checkpoint files link to /blobs. /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: - /path/to/checkpoint: /checkpoint /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: served_model_name: qwen3-8b-teacher diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml index b249ed6ef68..36afba92582 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml @@ -1,31 +1,16 @@ # Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: - execution: slurm/default + - ../../../nel_configs/h100_slurm_execution@_global_ - deployment: vllm - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - ../../../nel_configs/wikitext@_global_ - _self_ execution: - hostname: ${oc.env:NEL_SLURM_HOST} - username: ${oc.env:SLURM_USER} - account: ${oc.env:NEL_SLURM_ACCOUNT} - partition: ${oc.env:NEL_SLURM_PARTITION} - output_dir: ${oc.env:NEL_OUTPUT_DIR} - num_nodes: 1 - ntasks_per_node: 1 - gres: gpu:1 - walltime: 04:00:00 - sbatch_extra_flags: - exclusive: false - mem: 64G mounts: - mount_home: false deployment: - /path/to/checkpoint: /checkpoint - # Cached checkpoint files link to /blobs. /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs evaluation: - /path/to/checkpoint: /checkpoint /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs deployment: served_model_name: qwen3-8b-teacher From 3799ef62a8e7b196738f453068175bdbe1d7b4ac Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 02:59:14 -0700 Subject: [PATCH 13/19] improve nyl config clarify Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/nel_configs/mmlu.yaml | 2 +- .../nel_configs/qwen3_8b_vllm_deployment.yaml | 15 +++++++++++++-- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml index a85b58f3cf0..ddd0bc74a10 100644 --- a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml +++ b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml @@ -33,4 +33,4 @@ evaluation: tokenizer: ${deployment.tokenizer_path} tokenizer_backend: huggingface tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=1,timeout=300,max_retries=2,stream=False,max_length=32768" + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=${evaluation.nemo_evaluator_config.config.params.parallelism},timeout=${evaluation.nemo_evaluator_config.config.params.request_timeout},max_retries=${evaluation.nemo_evaluator_config.config.params.max_retries},stream=False,max_length=32768" diff --git a/examples/llm_qat/evaluation/nel_configs/qwen3_8b_vllm_deployment.yaml b/examples/llm_qat/evaluation/nel_configs/qwen3_8b_vllm_deployment.yaml index 68068dfe4e1..638ec64d6f9 100644 --- a/examples/llm_qat/evaluation/nel_configs/qwen3_8b_vllm_deployment.yaml +++ b/examples/llm_qat/evaluation/nel_configs/qwen3_8b_vllm_deployment.yaml @@ -4,6 +4,17 @@ deployment: checkpoint_path: ${oc.env:MODEL_CHECKPOINT} tensor_parallel_size: 1 data_parallel_size: 1 - command: vllm serve /checkpoint --served-model-name ${deployment.served_model_name} --host 0.0.0.0 --port ${deployment.port} --dtype bfloat16 --tensor-parallel-size - 1 --data-parallel-size 1 --max-model-len 32768 --max-num-seqs 2 --gpu-memory-utilization 0.85 --seed 42 --kv-cache-dtype auto + command: >- + vllm serve /checkpoint + --served-model-name ${deployment.served_model_name} + --host 0.0.0.0 + --port ${deployment.port} + --dtype bfloat16 + --tensor-parallel-size ${deployment.tensor_parallel_size} + --data-parallel-size ${deployment.data_parallel_size} + --max-model-len 32768 + --max-num-seqs 2 + --gpu-memory-utilization 0.85 + --seed 42 + --kv-cache-dtype auto tokenizer_path: /checkpoint From f9a15b5845260828c9ce416507feccfba0d1420c Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Tue, 13 Oct 2026 03:14:24 -0700 Subject: [PATCH 14/19] Add per subject mmlu results Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 67 ++++++++++++++++++++++++++- 1 file changed, 66 insertions(+), 1 deletion(-) diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index 2fb3232bfa7..1192e5ae4ab 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -4,7 +4,7 @@ Qwen3-8B was evaluated using NEL with vLLM on one H100 per evaluation. QAD starts from the PTQ checkpoint and uses the BF16 model as teacher. -| Checkpoint | Deployed precision | MMLU accuracy ↑ | WikiText-2 perplexity ↓ | +| Checkpoint | Deployed precision | MMLU accuracy ↑ (14,042 questions) | WikiText-2 perplexity ↓ (241,335 words) | |---|---|---:|---:| | Teacher (BF16) | BF16 | 74.94% | 13.485 | | PTQ (NVFP4) | NVFP4 weights / BF16 activations | 73.59% | 13.883 | @@ -12,6 +12,71 @@ Qwen3-8B was evaluated using NEL with vLLM on one H100 per evaluation. QAD start QAD improves both metrics over PTQ but remains below the BF16 teacher. The quantized checkpoints use Marlin's W4A16 fallback on H100; these results do not measure native NVFP4 W4A4 execution on Blackwell. KV cache is BF16 for all checkpoints. +
+MMLU accuracy by subject + +| Subject | Questions | Teacher (BF16) | PTQ (NVFP4) | PTQ → QAD (NVFP4) | +|---|---:|---:|---:|---:| +| Abstract algebra | 100 | 55.00% | 56.00% | 56.00% | +| Anatomy | 135 | 70.37% | 68.89% | 68.89% | +| Astronomy | 152 | 90.13% | 88.82% | 90.13% | +| Business ethics | 100 | 78.00% | 77.00% | 79.00% | +| Clinical knowledge | 265 | 78.49% | 80.38% | 79.62% | +| College biology | 144 | 87.50% | 86.81% | 87.50% | +| College chemistry | 100 | 56.00% | 59.00% | 56.00% | +| College computer science | 100 | 73.00% | 72.00% | 68.00% | +| College mathematics | 100 | 58.00% | 59.00% | 58.00% | +| College medicine | 173 | 79.19% | 78.03% | 78.03% | +| College physics | 102 | 64.71% | 59.80% | 63.73% | +| Computer security | 100 | 81.00% | 81.00% | 80.00% | +| Conceptual physics | 235 | 84.68% | 85.11% | 84.68% | +| Econometrics | 114 | 65.79% | 64.91% | 68.42% | +| Electrical engineering | 145 | 80.69% | 80.69% | 82.76% | +| Elementary mathematics | 378 | 78.84% | 75.40% | 76.19% | +| Formal logic | 126 | 67.46% | 66.67% | 60.32% | +| Global facts | 100 | 52.00% | 44.00% | 43.00% | +| High school biology | 310 | 90.97% | 91.29% | 91.61% | +| High school chemistry | 203 | 75.86% | 72.91% | 71.92% | +| High school computer science | 100 | 86.00% | 86.00% | 86.00% | +| High school European history | 165 | 85.45% | 85.45% | 87.27% | +| High school geography | 198 | 86.87% | 86.36% | 86.87% | +| High school government and politics | 193 | 93.26% | 92.23% | 93.26% | +| High school macroeconomics | 390 | 81.03% | 79.49% | 80.00% | +| High school mathematics | 270 | 53.70% | 50.00% | 51.85% | +| High school microeconomics | 238 | 93.70% | 91.60% | 92.02% | +| High school physics | 151 | 72.85% | 69.54% | 70.20% | +| High school psychology | 545 | 91.19% | 90.46% | 90.83% | +| High school statistics | 216 | 78.24% | 75.93% | 75.46% | +| High school US history | 204 | 86.27% | 85.29% | 83.82% | +| High school world history | 237 | 86.50% | 85.65% | 85.65% | +| Human aging | 223 | 74.44% | 75.34% | 73.99% | +| Human sexuality | 131 | 83.21% | 88.55% | 84.73% | +| International law | 121 | 73.55% | 73.55% | 73.55% | +| Jurisprudence | 108 | 81.48% | 83.33% | 79.63% | +| Logical fallacies | 163 | 82.21% | 80.37% | 82.21% | +| Machine learning | 112 | 64.29% | 64.29% | 65.18% | +| Management | 103 | 87.38% | 89.32% | 87.38% | +| Marketing | 234 | 93.59% | 92.74% | 91.88% | +| Medical genetics | 100 | 83.00% | 82.00% | 82.00% | +| Miscellaneous | 783 | 85.82% | 83.78% | 83.40% | +| Moral disputes | 346 | 76.01% | 72.25% | 73.99% | +| Moral scenarios | 895 | 55.42% | 53.85% | 53.85% | +| Nutrition | 306 | 81.05% | 81.70% | 80.07% | +| Philosophy | 311 | 77.81% | 74.28% | 76.85% | +| Prehistory | 324 | 83.95% | 82.72% | 83.02% | +| Professional accounting | 282 | 59.22% | 58.51% | 57.09% | +| Professional law | 1,534 | 51.17% | 49.02% | 50.26% | +| Professional medicine | 272 | 81.99% | 80.51% | 80.88% | +| Professional psychology | 612 | 78.59% | 76.31% | 76.14% | +| Public relations | 110 | 70.00% | 68.18% | 70.00% | +| Security studies | 245 | 79.59% | 80.00% | 80.41% | +| Sociology | 201 | 86.57% | 84.58% | 84.08% | +| US foreign policy | 100 | 84.00% | 83.00% | 85.00% | +| Virology | 166 | 56.02% | 52.41% | 53.61% | +| World religions | 171 | 87.13% | 85.38% | 84.21% | + +
+ ## File structure NeMo Evaluator Launcher (NEL) configurations and evaluation results for the [Hugging Face QAT/QAD tutorial](../README.md). From 3514389913f90ff082621c5554463717a9259b8c Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 14 Oct 2026 04:44:08 -0700 Subject: [PATCH 15/19] Add explictly arguments for wikitext evaluation task into wikitext.yaml Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/nel_configs/wikitext.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/llm_qat/evaluation/nel_configs/wikitext.yaml b/examples/llm_qat/evaluation/nel_configs/wikitext.yaml index 455a7aeffac..ba1fd7ad4f7 100644 --- a/examples/llm_qat/evaluation/nel_configs/wikitext.yaml +++ b/examples/llm_qat/evaluation/nel_configs/wikitext.yaml @@ -33,4 +33,4 @@ evaluation: tokenizer: ${deployment.tokenizer_path} tokenizer_backend: huggingface tokenized_requests: true - args: --seed 42 + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=${evaluation.nemo_evaluator_config.config.params.parallelism},timeout=${evaluation.nemo_evaluator_config.config.params.request_timeout},max_retries=${evaluation.nemo_evaluator_config.config.params.max_retries},stream=False,max_length=2048" From c55cc2686e4805dc999cb0e2c68dfcebbde38038 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 14 Oct 2026 04:59:01 -0700 Subject: [PATCH 16/19] Clean up nel config files for wikitext Signed-off-by: Daniel Korzekwa --- .../qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml | 1 - .../wikitext_full/nel_config.yaml | 1 - .../qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml | 1 - 3 files changed, 3 deletions(-) diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml index bf3ba04f386..58de46e767a 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml @@ -1,4 +1,3 @@ -# Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: - execution: slurm/default - ../../../nel_configs/h100_slurm_execution@_global_ diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml index 5f240ca05b3..de5f2abc124 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml @@ -1,4 +1,3 @@ -# Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: - execution: slurm/default - ../../../nel_configs/h100_slurm_execution@_global_ diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml index 36afba92582..d1d82e7a3af 100644 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml +++ b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml @@ -1,4 +1,3 @@ -# Full benchmark coverage; NEL 0.2.6. Replace mount placeholders; see ../../../README.md. defaults: - execution: slurm/default - ../../../nel_configs/h100_slurm_execution@_global_ From 68f124cf51322fd99534fc897cc2d9aecca4c095 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 14 Oct 2026 05:52:38 -0700 Subject: [PATCH 17/19] Simplify nel config files Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 36 +++++++++-- examples/llm_qat/evaluation/nel_config.yaml | 61 +++++++++++++++++++ .../nel_configs/h100_slurm_execution.yaml | 7 +-- .../llm_qat/evaluation/nel_configs/mmlu.yaml | 36 ----------- .../evaluation/nel_configs/wikitext.yaml | 36 ----------- .../mmlu_full/nel_config.yaml | 10 --- .../wikitext_full/nel_config.yaml | 10 --- .../mmlu_full/nel_config.yaml | 10 --- .../wikitext_full/nel_config.yaml | 10 --- .../mmlu_full/nel_config.yaml | 16 ----- .../wikitext_full/nel_config.yaml | 16 ----- 11 files changed, 96 insertions(+), 152 deletions(-) create mode 100644 examples/llm_qat/evaluation/nel_config.yaml delete mode 100644 examples/llm_qat/evaluation/nel_configs/mmlu.yaml delete mode 100644 examples/llm_qat/evaluation/nel_configs/wikitext.yaml delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml delete mode 100644 examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index 1192e5ae4ab..17ff5979225 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -77,9 +77,37 @@ QAD improves both metrics over PTQ but remains below the BF16 teacher. The quant -## File structure +## Run evaluations + +From `examples/llm_qat/evaluation/`, configure access to your Slurm cluster: + +```bash +pip install "nemo-evaluator-launcher[all]==0.2.6" + +export NEL_SLURM_HOST="" +export SLURM_USER="" +export NEL_SLURM_ACCOUNT="" +export NEL_SLURM_PARTITION="" +export NEL_OUTPUT_DIR="/shared/path/to/evaluation-results" +export MODEL_CHECKPOINT="/shared/path/to/checkpoint" + +# Run MMLU and WikiText. +nemo-evaluator-launcher run --config nel_config.yaml \ + -o deployment.served_model_name=qwen3-8b-ptq +``` -NeMo Evaluator Launcher (NEL) configurations and evaluation results for the [Hugging Face QAT/QAD tutorial](../README.md). +Use the BF16 teacher or an exported Hugging Face PTQ/QAD checkpoint. The checkpoint directory must be accessible on the cluster and contain the model files, without symlinks to unmounted locations. + +For the teacher, download a self-contained directory on the cluster; no additional `/blobs` mounts are needed: + +```bash +hf download Qwen/Qwen3-8B --local-dir /shared/path/to/qwen3-8b-teacher +export MODEL_CHECKPOINT="/shared/path/to/qwen3-8b-teacher" +``` + +Add `--dry-run` to preview, or `-t lm-evaluation-harness.mmlu` / `-t lm-evaluation-harness.wikitext` to select one benchmark. NEL 0.2.6 launches one Slurm job per benchmark; results are saved under `NEL_OUTPUT_DIR`. + +## File structure -- `nel_configs/`: shared benchmark recipes, Slurm execution settings, and vLLM deployment settings. -- `results///`: each evaluation's config `nel_config.yaml`. +- `nel_config.yaml`: entry point containing the MMLU and WikiText recipes. +- `nel_configs/`: shared Slurm execution and vLLM deployment settings. diff --git a/examples/llm_qat/evaluation/nel_config.yaml b/examples/llm_qat/evaluation/nel_config.yaml new file mode 100644 index 00000000000..41f91a559fa --- /dev/null +++ b/examples/llm_qat/evaluation/nel_config.yaml @@ -0,0 +1,61 @@ +# MMLU and WikiText recipes for nemo-evaluator-launcher==0.2.6. +defaults: + - execution: slurm/default + - nel_configs/h100_slurm_execution@_global_ + - deployment: vllm + - nel_configs/qwen3_8b_vllm_deployment@_global_ + - _self_ +deployment: + # Override when launching to identify the checkpoint, e.g., -o deployment.served_model_name=qwen3-8b-ptq + served_model_name: qwen3-8b + port: 18100 +evaluation: + nemo_evaluator_config: + target: + api_endpoint: + api_key_name: + adapter_config: + use_reasoning: false + use_request_logging: true + use_response_logging: true + max_logged_requests: 2 + max_logged_responses: 2 + tasks: + - name: lm-evaluation-harness.mmlu + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + parallelism: 1 + request_timeout: 300 + max_retries: 2 + temperature: + top_p: + max_new_tokens: + task: mmlu + extra: + num_fewshot: 5 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=${..parallelism},timeout=${..request_timeout},max_retries=${..max_retries},stream=False,max_length=32768" + - name: lm-evaluation-harness.wikitext + endpoint_type: completions + container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 + nemo_evaluator_config: + config: + params: + parallelism: 2 + request_timeout: 300 + max_retries: 2 + temperature: + top_p: + max_new_tokens: + task: wikitext + extra: + num_fewshot: 0 + tokenizer: ${deployment.tokenizer_path} + tokenizer_backend: huggingface + tokenized_requests: true + args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=${..parallelism},timeout=${..request_timeout},max_retries=${..max_retries},stream=False,max_length=2048" diff --git a/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml b/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml index d6299ef8feb..8b96b017ff3 100644 --- a/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml +++ b/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml @@ -1,4 +1,5 @@ # Shared single-GPU Slurm execution settings for H100 evaluation. +# NEL mounts deployment.checkpoint_path in the server; mount it in the evaluator too. execution: hostname: ${oc.env:NEL_SLURM_HOST} username: ${oc.env:SLURM_USER} @@ -14,7 +15,5 @@ execution: mem: 64G mounts: mount_home: false - deployment: - /path/to/checkpoint: /checkpoint - evaluation: - /path/to/checkpoint: /checkpoint + deployment: {} + evaluation: '${oc.create:''{"${oc.env:MODEL_CHECKPOINT}": "/checkpoint"}''}' diff --git a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml b/examples/llm_qat/evaluation/nel_configs/mmlu.yaml deleted file mode 100644 index ddd0bc74a10..00000000000 --- a/examples/llm_qat/evaluation/nel_configs/mmlu.yaml +++ /dev/null @@ -1,36 +0,0 @@ -# Shared benchmark recipe for nemo-evaluator-launcher==0.2.6. -# All 57 MMLU subjects, test split, 5-shot log-likelihood accuracy. No sample limit by default. -evaluation: - nemo_evaluator_config: - config: - params: - parallelism: 1 - request_timeout: 300 - max_retries: 2 - temperature: - top_p: - max_new_tokens: - task: mmlu - target: - api_endpoint: - api_key_name: - adapter_config: - use_reasoning: false - use_request_logging: true - use_response_logging: true - max_logged_requests: 2 - max_logged_responses: 2 - tasks: - - name: lm-evaluation-harness.mmlu - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - task: ${evaluation.nemo_evaluator_config.config.params.task} - extra: - num_fewshot: 5 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=${evaluation.nemo_evaluator_config.config.params.parallelism},timeout=${evaluation.nemo_evaluator_config.config.params.request_timeout},max_retries=${evaluation.nemo_evaluator_config.config.params.max_retries},stream=False,max_length=32768" diff --git a/examples/llm_qat/evaluation/nel_configs/wikitext.yaml b/examples/llm_qat/evaluation/nel_configs/wikitext.yaml deleted file mode 100644 index ba1fd7ad4f7..00000000000 --- a/examples/llm_qat/evaluation/nel_configs/wikitext.yaml +++ /dev/null @@ -1,36 +0,0 @@ -# Shared benchmark recipe for nemo-evaluator-launcher==0.2.6. -# WikiText-2 test split, zero-shot rolling log-likelihood perplexity. No sample limit by default. -evaluation: - nemo_evaluator_config: - config: - params: - parallelism: 2 - request_timeout: 300 - max_retries: 2 - temperature: - top_p: - max_new_tokens: - task: wikitext - target: - api_endpoint: - api_key_name: - adapter_config: - use_reasoning: false - use_request_logging: true - use_response_logging: true - max_logged_requests: 2 - max_logged_responses: 2 - tasks: - - name: lm-evaluation-harness.wikitext - endpoint_type: completions - container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 - nemo_evaluator_config: - config: - params: - task: ${evaluation.nemo_evaluator_config.config.params.task} - extra: - num_fewshot: 0 - tokenizer: ${deployment.tokenizer_path} - tokenizer_backend: huggingface - tokenized_requests: true - args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=${evaluation.nemo_evaluator_config.config.params.parallelism},timeout=${evaluation.nemo_evaluator_config.config.params.request_timeout},max_retries=${evaluation.nemo_evaluator_config.config.params.max_retries},stream=False,max_length=2048" diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml deleted file mode 100644 index f3720d90056..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/mmlu_full/nel_config.yaml +++ /dev/null @@ -1,10 +0,0 @@ -defaults: - - execution: slurm/default - - ../../../nel_configs/h100_slurm_execution@_global_ - - deployment: vllm - - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - - ../../../nel_configs/mmlu@_global_ - - _self_ -deployment: - served_model_name: qwen3-8b-ptq - port: 18104 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml deleted file mode 100644 index 58de46e767a..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_nvfp4/wikitext_full/nel_config.yaml +++ /dev/null @@ -1,10 +0,0 @@ -defaults: - - execution: slurm/default - - ../../../nel_configs/h100_slurm_execution@_global_ - - deployment: vllm - - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - - ../../../nel_configs/wikitext@_global_ - - _self_ -deployment: - served_model_name: qwen3-8b-ptq - port: 18106 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml deleted file mode 100644 index cfa1e96854b..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/mmlu_full/nel_config.yaml +++ /dev/null @@ -1,10 +0,0 @@ -defaults: - - execution: slurm/default - - ../../../nel_configs/h100_slurm_execution@_global_ - - deployment: vllm - - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - - ../../../nel_configs/mmlu@_global_ - - _self_ -deployment: - served_model_name: qwen3-8b-ptq-then-qad - port: 18108 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml deleted file mode 100644 index de5f2abc124..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_ptq_then_qad_nvfp4/wikitext_full/nel_config.yaml +++ /dev/null @@ -1,10 +0,0 @@ -defaults: - - execution: slurm/default - - ../../../nel_configs/h100_slurm_execution@_global_ - - deployment: vllm - - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - - ../../../nel_configs/wikitext@_global_ - - _self_ -deployment: - served_model_name: qwen3-8b-ptq-then-qad - port: 18110 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml deleted file mode 100644 index bff5c5d251a..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/mmlu_full/nel_config.yaml +++ /dev/null @@ -1,16 +0,0 @@ -defaults: - - execution: slurm/default - - ../../../nel_configs/h100_slurm_execution@_global_ - - deployment: vllm - - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - - ../../../nel_configs/mmlu@_global_ - - _self_ -execution: - mounts: - deployment: - /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs - evaluation: - /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs -deployment: - served_model_name: qwen3-8b-teacher - port: 18100 diff --git a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml b/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml deleted file mode 100644 index d1d82e7a3af..00000000000 --- a/examples/llm_qat/evaluation/results/qwen3-8b_h100_teacher_bf16/wikitext_full/nel_config.yaml +++ /dev/null @@ -1,16 +0,0 @@ -defaults: - - execution: slurm/default - - ../../../nel_configs/h100_slurm_execution@_global_ - - deployment: vllm - - ../../../nel_configs/qwen3_8b_vllm_deployment@_global_ - - ../../../nel_configs/wikitext@_global_ - - _self_ -execution: - mounts: - deployment: - /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs - evaluation: - /path/to/hf-cache/hub/models--Qwen--Qwen3-8B/blobs: /blobs -deployment: - served_model_name: qwen3-8b-teacher - port: 18102 From 64303ae53f80357aff89f9e3c98716e7c72049c2 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 14 Oct 2026 05:59:51 -0700 Subject: [PATCH 18/19] improve clarity of eval readme Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 60 +++++++++++++-------------- 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index 17ff5979225..29d1cae050a 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -12,6 +12,36 @@ Qwen3-8B was evaluated using NEL with vLLM on one H100 per evaluation. QAD start QAD improves both metrics over PTQ but remains below the BF16 teacher. The quantized checkpoints use Marlin's W4A16 fallback on H100; these results do not measure native NVFP4 W4A4 execution on Blackwell. KV cache is BF16 for all checkpoints. +## Run evaluations + +From `examples/llm_qat/evaluation/`, configure access to your Slurm cluster: + +```bash +pip install "nemo-evaluator-launcher[all]==0.2.6" + +export NEL_SLURM_HOST="" +export SLURM_USER="" +export NEL_SLURM_ACCOUNT="" +export NEL_SLURM_PARTITION="" +export NEL_OUTPUT_DIR="/shared/path/to/evaluation-results" +export MODEL_CHECKPOINT="/shared/path/to/checkpoint" + +# Run MMLU and WikiText. +nemo-evaluator-launcher run --config nel_config.yaml \ + -o deployment.served_model_name=qwen3-8b-ptq +``` + +Use the BF16 teacher or an exported Hugging Face PTQ/QAD checkpoint. The checkpoint directory must be accessible on the cluster and contain the model files, without symlinks to unmounted locations. + +For the teacher, download a self-contained directory on the cluster; no additional `/blobs` mounts are needed: + +```bash +hf download Qwen/Qwen3-8B --local-dir /shared/path/to/qwen3-8b-teacher +export MODEL_CHECKPOINT="/shared/path/to/qwen3-8b-teacher" +``` + +Add `--dry-run` to preview, or `-t lm-evaluation-harness.mmlu` / `-t lm-evaluation-harness.wikitext` to select one benchmark. NEL 0.2.6 launches one Slurm job per benchmark; results are saved under `NEL_OUTPUT_DIR`. +
MMLU accuracy by subject @@ -77,36 +107,6 @@ QAD improves both metrics over PTQ but remains below the BF16 teacher. The quant
-## Run evaluations - -From `examples/llm_qat/evaluation/`, configure access to your Slurm cluster: - -```bash -pip install "nemo-evaluator-launcher[all]==0.2.6" - -export NEL_SLURM_HOST="" -export SLURM_USER="" -export NEL_SLURM_ACCOUNT="" -export NEL_SLURM_PARTITION="" -export NEL_OUTPUT_DIR="/shared/path/to/evaluation-results" -export MODEL_CHECKPOINT="/shared/path/to/checkpoint" - -# Run MMLU and WikiText. -nemo-evaluator-launcher run --config nel_config.yaml \ - -o deployment.served_model_name=qwen3-8b-ptq -``` - -Use the BF16 teacher or an exported Hugging Face PTQ/QAD checkpoint. The checkpoint directory must be accessible on the cluster and contain the model files, without symlinks to unmounted locations. - -For the teacher, download a self-contained directory on the cluster; no additional `/blobs` mounts are needed: - -```bash -hf download Qwen/Qwen3-8B --local-dir /shared/path/to/qwen3-8b-teacher -export MODEL_CHECKPOINT="/shared/path/to/qwen3-8b-teacher" -``` - -Add `--dry-run` to preview, or `-t lm-evaluation-harness.mmlu` / `-t lm-evaluation-harness.wikitext` to select one benchmark. NEL 0.2.6 launches one Slurm job per benchmark; results are saved under `NEL_OUTPUT_DIR`. - ## File structure - `nel_config.yaml`: entry point containing the MMLU and WikiText recipes. From a3961470c23d7696479c74756714368f09074ade Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 14 Oct 2026 09:09:59 -0700 Subject: [PATCH 19/19] clarify eval running configuration Signed-off-by: Daniel Korzekwa --- examples/llm_qat/evaluation/README.md | 5 ++++- .../evaluation/nel_configs/h100_slurm_execution.yaml | 8 ++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/examples/llm_qat/evaluation/README.md b/examples/llm_qat/evaluation/README.md index 29d1cae050a..82b67d5616f 100644 --- a/examples/llm_qat/evaluation/README.md +++ b/examples/llm_qat/evaluation/README.md @@ -26,9 +26,12 @@ export NEL_SLURM_PARTITION="" export NEL_OUTPUT_DIR="/shared/path/to/evaluation-results" export MODEL_CHECKPOINT="/shared/path/to/checkpoint" +# See execution.mounts.evaluation in nel_configs/h100_slurm_execution.yaml +# for why the checkpoint mount is passed via -o. # Run MMLU and WikiText. nemo-evaluator-launcher run --config nel_config.yaml \ - -o deployment.served_model_name=qwen3-8b-ptq + -o deployment.served_model_name=qwen3-8b-ptq \ + -o "++execution.mounts.evaluation={$MODEL_CHECKPOINT: /checkpoint}" ``` Use the BF16 teacher or an exported Hugging Face PTQ/QAD checkpoint. The checkpoint directory must be accessible on the cluster and contain the model files, without symlinks to unmounted locations. diff --git a/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml b/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml index 8b96b017ff3..f4c5296d222 100644 --- a/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml +++ b/examples/llm_qat/evaluation/nel_configs/h100_slurm_execution.yaml @@ -1,5 +1,5 @@ # Shared single-GPU Slurm execution settings for H100 evaluation. -# NEL mounts deployment.checkpoint_path in the server; mount it in the evaluator too. +# NEL mounts the server checkpoint automatically; pass the evaluator mount with -o. execution: hostname: ${oc.env:NEL_SLURM_HOST} username: ${oc.env:SLURM_USER} @@ -16,4 +16,8 @@ execution: mounts: mount_home: false deployment: {} - evaluation: '${oc.create:''{"${oc.env:MODEL_CHECKPOINT}": "/checkpoint"}''}' + # NEL 0.2.6 validates this field as a dictionary before resolving oc.create, + # so the expression deriving the mount from MODEL_CHECKPOINT was rejected. + # The README therefore supplies a concrete mapping via the CLI instead: + # -o "++execution.mounts.evaluation={$MODEL_CHECKPOINT: /checkpoint}" + evaluation: {}