Repository navigation
Add Qwen3-8B HF QAD evaluation results and reusable NEL configuration #2763
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| # Evaluation | ||
|
|
||
| ## Results | ||
|
|
||
| Qwen3-8B was evaluated using NEL with vLLM on one H100 per evaluation. QAD starts from the PTQ checkpoint and uses the BF16 model as teacher. | ||
|
|
||
| | Checkpoint | Deployed precision | MMLU accuracy ↑ (14,042 questions) | WikiText-2 perplexity ↓ (241,335 words) | | ||
| |---|---|---:|---:| | ||
| | Teacher (BF16) | BF16 | 74.94% | 13.485 | | ||
| | PTQ (NVFP4) | NVFP4 weights / BF16 activations | 73.59% | 13.883 | | ||
| | PTQ → QAD (NVFP4) | NVFP4 weights / BF16 activations | 73.76% | 13.684 | | ||
|
|
||
| QAD improves both metrics over PTQ but remains below the BF16 teacher. The quantized checkpoints use Marlin's W4A16 fallback on H100; these results do not measure native NVFP4 W4A4 execution on Blackwell. KV cache is BF16 for all checkpoints. | ||
|
Comment on lines
+7
to
+13
Contributor
There was a problem hiding this comment. [SUGGESTION] The headline table reports single-run point estimates and then draws a conclusion the margins do not clearly support. The MMLU gap between PTQ (73.59%) and PTQ → QAD (73.76%) is 0.17 pp, i.e. about 24 questions out of 14,042. The binomial standard error at n=14,042 and p≈0.74 is ~0.37 pp, so the standard error on the difference is larger than the difference itself. Yet the text states flatly:
The per-subject breakdown makes this visible: QAD is worse than PTQ on Formal logic (60.32% vs 66.67%), College computer science (68.00% vs 72.00%), and High school chemistry (71.92% vs 72.91%), while better elsewhere — the pattern of noise, not a uniform shift. The WikiText perplexity improvement (13.883 → 13.684) is the stronger of the two signals and recovers ~50% of the teacher gap, which is a more defensible claim. Suggested rewording, keeping the result but matching the evidence:
The sibling tutorial reports repeat counts and ± figures per benchmark ( |
||
|
|
||
| ## Run evaluations | ||
|
|
||
| From `examples/llm_qat/evaluation/`, configure access to your Slurm cluster: | ||
|
|
||
| ```bash | ||
| pip install "nemo-evaluator-launcher[all]==0.2.6" | ||
|
|
||
| export NEL_SLURM_HOST="<cluster-login-host>" | ||
| export SLURM_USER="<cluster-username>" | ||
| export NEL_SLURM_ACCOUNT="<slurm-account>" | ||
| export NEL_SLURM_PARTITION="<slurm-partition>" | ||
| export NEL_OUTPUT_DIR="/shared/path/to/evaluation-results" | ||
| export MODEL_CHECKPOINT="/shared/path/to/checkpoint" | ||
|
|
||
| # See execution.mounts.evaluation in nel_configs/h100_slurm_execution.yaml | ||
| # for why the checkpoint mount is passed via -o. | ||
| # Run MMLU and WikiText. | ||
| nemo-evaluator-launcher run --config nel_config.yaml \ | ||
| -o deployment.served_model_name=qwen3-8b-ptq \ | ||
| -o "++execution.mounts.evaluation={$MODEL_CHECKPOINT: /checkpoint}" | ||
|
Comment on lines
+26
to
+34
Contributor
There was a problem hiding this comment. [IMPORTANT Compatibility] The committed configs are not the ones that produced the results table above, and the PR body lists the cross-check as pending ("rerun both benchmarks on cw-dfw-cs-001 ... and compare with the published results"). That matters because this README presents the table as reproducible by running the command in this block. Two concrete gaps between this block and the commands that actually generated the numbers (per the PR description):
Please either land this after the rerun confirms the numbers, or add a one-line note here stating which NEL version and config revision produced the table so a reader who gets different numbers knows where to look. Also on the -o "++execution.mounts.evaluation={$MODEL_CHECKPOINT: /checkpoint}"Hydra's override grammar does not accept an arbitrary filesystem path as a bare dict key — a key like -o "++execution.mounts.evaluation={'$MODEL_CHECKPOINT': /checkpoint}"Worth confirming during the rerun, since this is the documented primary path and a parse error here blocks every evaluation. |
||
| ``` | ||
|
|
||
| Use the BF16 teacher or an exported Hugging Face PTQ/QAD checkpoint. The checkpoint directory must be accessible on the cluster and contain the model files, without symlinks to unmounted locations. | ||
|
|
||
| For the teacher, download a self-contained directory on the cluster; no additional `/blobs` mounts are needed: | ||
|
|
||
| ```bash | ||
| hf download Qwen/Qwen3-8B --local-dir /shared/path/to/qwen3-8b-teacher | ||
| export MODEL_CHECKPOINT="/shared/path/to/qwen3-8b-teacher" | ||
| ``` | ||
|
|
||
| Add `--dry-run` to preview, or `-t lm-evaluation-harness.mmlu` / `-t lm-evaluation-harness.wikitext` to select one benchmark. NEL 0.2.6 launches one Slurm job per benchmark; results are saved under `NEL_OUTPUT_DIR`. | ||
|
|
||
| <details> | ||
| <summary>MMLU accuracy by subject</summary> | ||
|
|
||
| | Subject | Questions | Teacher (BF16) | PTQ (NVFP4) | PTQ → QAD (NVFP4) | | ||
| |---|---:|---:|---:|---:| | ||
| | Abstract algebra | 100 | 55.00% | 56.00% | 56.00% | | ||
| | Anatomy | 135 | 70.37% | 68.89% | 68.89% | | ||
| | Astronomy | 152 | 90.13% | 88.82% | 90.13% | | ||
| | Business ethics | 100 | 78.00% | 77.00% | 79.00% | | ||
| | Clinical knowledge | 265 | 78.49% | 80.38% | 79.62% | | ||
| | College biology | 144 | 87.50% | 86.81% | 87.50% | | ||
| | College chemistry | 100 | 56.00% | 59.00% | 56.00% | | ||
| | College computer science | 100 | 73.00% | 72.00% | 68.00% | | ||
| | College mathematics | 100 | 58.00% | 59.00% | 58.00% | | ||
| | College medicine | 173 | 79.19% | 78.03% | 78.03% | | ||
| | College physics | 102 | 64.71% | 59.80% | 63.73% | | ||
| | Computer security | 100 | 81.00% | 81.00% | 80.00% | | ||
| | Conceptual physics | 235 | 84.68% | 85.11% | 84.68% | | ||
| | Econometrics | 114 | 65.79% | 64.91% | 68.42% | | ||
| | Electrical engineering | 145 | 80.69% | 80.69% | 82.76% | | ||
| | Elementary mathematics | 378 | 78.84% | 75.40% | 76.19% | | ||
| | Formal logic | 126 | 67.46% | 66.67% | 60.32% | | ||
| | Global facts | 100 | 52.00% | 44.00% | 43.00% | | ||
| | High school biology | 310 | 90.97% | 91.29% | 91.61% | | ||
| | High school chemistry | 203 | 75.86% | 72.91% | 71.92% | | ||
| | High school computer science | 100 | 86.00% | 86.00% | 86.00% | | ||
| | High school European history | 165 | 85.45% | 85.45% | 87.27% | | ||
| | High school geography | 198 | 86.87% | 86.36% | 86.87% | | ||
| | High school government and politics | 193 | 93.26% | 92.23% | 93.26% | | ||
| | High school macroeconomics | 390 | 81.03% | 79.49% | 80.00% | | ||
| | High school mathematics | 270 | 53.70% | 50.00% | 51.85% | | ||
| | High school microeconomics | 238 | 93.70% | 91.60% | 92.02% | | ||
| | High school physics | 151 | 72.85% | 69.54% | 70.20% | | ||
| | High school psychology | 545 | 91.19% | 90.46% | 90.83% | | ||
| | High school statistics | 216 | 78.24% | 75.93% | 75.46% | | ||
| | High school US history | 204 | 86.27% | 85.29% | 83.82% | | ||
| | High school world history | 237 | 86.50% | 85.65% | 85.65% | | ||
| | Human aging | 223 | 74.44% | 75.34% | 73.99% | | ||
| | Human sexuality | 131 | 83.21% | 88.55% | 84.73% | | ||
| | International law | 121 | 73.55% | 73.55% | 73.55% | | ||
| | Jurisprudence | 108 | 81.48% | 83.33% | 79.63% | | ||
| | Logical fallacies | 163 | 82.21% | 80.37% | 82.21% | | ||
| | Machine learning | 112 | 64.29% | 64.29% | 65.18% | | ||
| | Management | 103 | 87.38% | 89.32% | 87.38% | | ||
| | Marketing | 234 | 93.59% | 92.74% | 91.88% | | ||
| | Medical genetics | 100 | 83.00% | 82.00% | 82.00% | | ||
| | Miscellaneous | 783 | 85.82% | 83.78% | 83.40% | | ||
| | Moral disputes | 346 | 76.01% | 72.25% | 73.99% | | ||
| | Moral scenarios | 895 | 55.42% | 53.85% | 53.85% | | ||
| | Nutrition | 306 | 81.05% | 81.70% | 80.07% | | ||
| | Philosophy | 311 | 77.81% | 74.28% | 76.85% | | ||
| | Prehistory | 324 | 83.95% | 82.72% | 83.02% | | ||
| | Professional accounting | 282 | 59.22% | 58.51% | 57.09% | | ||
| | Professional law | 1,534 | 51.17% | 49.02% | 50.26% | | ||
| | Professional medicine | 272 | 81.99% | 80.51% | 80.88% | | ||
| | Professional psychology | 612 | 78.59% | 76.31% | 76.14% | | ||
| | Public relations | 110 | 70.00% | 68.18% | 70.00% | | ||
| | Security studies | 245 | 79.59% | 80.00% | 80.41% | | ||
| | Sociology | 201 | 86.57% | 84.58% | 84.08% | | ||
| | US foreign policy | 100 | 84.00% | 83.00% | 85.00% | | ||
| | Virology | 166 | 56.02% | 52.41% | 53.61% | | ||
| | World religions | 171 | 87.13% | 85.38% | 84.21% | | ||
|
|
||
| </details> | ||
|
|
||
| ## File structure | ||
|
|
||
| - `nel_config.yaml`: entry point containing the MMLU and WikiText recipes. | ||
| - `nel_configs/`: shared Slurm execution and vLLM deployment settings. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # MMLU and WikiText recipes for nemo-evaluator-launcher==0.2.6. | ||
| defaults: | ||
| - execution: slurm/default | ||
| - nel_configs/h100_slurm_execution@_global_ | ||
| - deployment: vllm | ||
| - nel_configs/qwen3_8b_vllm_deployment@_global_ | ||
| - _self_ | ||
| deployment: | ||
| # Override when launching to identify the checkpoint, e.g., -o deployment.served_model_name=qwen3-8b-ptq | ||
| served_model_name: qwen3-8b | ||
| port: 18100 | ||
|
Contributor
There was a problem hiding this comment. [IMPORTANT Compatibility] Both tasks share a single The README says "NEL 0.2.6 launches one Slurm job per benchmark", and This is exactly the hazard the PR description works around manually ("Each evaluation uses a different port to avoid collisions between these runs if jobs share a node"), but the committed README does not carry that workaround: its one-liner runs both tasks with the shared default port and no Three ways to close it, any one is enough:
Option 1 is the smallest change and matches the existing convention. |
||
| evaluation: | ||
| nemo_evaluator_config: | ||
| target: | ||
| api_endpoint: | ||
| api_key_name: | ||
| adapter_config: | ||
| use_reasoning: false | ||
| use_request_logging: true | ||
| use_response_logging: true | ||
| max_logged_requests: 2 | ||
| max_logged_responses: 2 | ||
| tasks: | ||
| - name: lm-evaluation-harness.mmlu | ||
| endpoint_type: completions | ||
| container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 | ||
| nemo_evaluator_config: | ||
| config: | ||
| params: | ||
| parallelism: 1 | ||
| request_timeout: 300 | ||
| max_retries: 2 | ||
| temperature: | ||
|
Comment on lines
+29
to
+33
Contributor
There was a problem hiding this comment. [IMPORTANT Performance] lm-eval's WikiText uses Suggested fix — raise MMLU concurrency and the server's batch ceiling together, since # nel_config.yaml, mmlu task
parallelism: 16# nel_configs/qwen3_8b_vllm_deployment.yaml
--max-num-seqs 16Loglikelihood scoring is order-independent, so concurrency does not change the MMLU numbers. If |
||
| top_p: | ||
| max_new_tokens: | ||
| task: mmlu | ||
| extra: | ||
| num_fewshot: 5 | ||
| tokenizer: ${deployment.tokenizer_path} | ||
| tokenizer_backend: huggingface | ||
| tokenized_requests: true | ||
| args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=${..parallelism},timeout=${..request_timeout},max_retries=${..max_retries},stream=False,max_length=32768" | ||
| - name: lm-evaluation-harness.wikitext | ||
| endpoint_type: completions | ||
| container: nvcr.io/nvidia/eval-factory/lm-evaluation-harness:26.03 | ||
| nemo_evaluator_config: | ||
| config: | ||
| params: | ||
| parallelism: 2 | ||
| request_timeout: 300 | ||
| max_retries: 2 | ||
| temperature: | ||
| top_p: | ||
| max_new_tokens: | ||
| task: wikitext | ||
| extra: | ||
| num_fewshot: 0 | ||
| tokenizer: ${deployment.tokenizer_path} | ||
| tokenizer_backend: huggingface | ||
| tokenized_requests: true | ||
| args: --seed 42 --model_args "base_url=http://127.0.0.1:${deployment.port}/v1/completions,model=${deployment.served_model_name},tokenized_requests=True,tokenizer=${deployment.tokenizer_path},tokenizer_backend=huggingface,num_concurrent=${..parallelism},timeout=${..request_timeout},max_retries=${..max_retries},stream=False,max_length=2048" | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Shared single-GPU Slurm execution settings for H100 evaluation. | ||
| # NEL mounts the server checkpoint automatically; pass the evaluator mount with -o. | ||
| execution: | ||
| hostname: ${oc.env:NEL_SLURM_HOST} | ||
| username: ${oc.env:SLURM_USER} | ||
| account: ${oc.env:NEL_SLURM_ACCOUNT} | ||
| partition: ${oc.env:NEL_SLURM_PARTITION} | ||
| output_dir: ${oc.env:NEL_OUTPUT_DIR} | ||
| num_nodes: 1 | ||
| ntasks_per_node: 1 | ||
| gres: gpu:1 | ||
|
Contributor
There was a problem hiding this comment. [SUGGESTION] Same file: |
||
| walltime: 04:00:00 | ||
| sbatch_extra_flags: | ||
| exclusive: false | ||
| mem: 64G | ||
| mounts: | ||
| mount_home: false | ||
| deployment: {} | ||
| # NEL 0.2.6 validates this field as a dictionary before resolving oc.create, | ||
| # so the expression deriving the mount from MODEL_CHECKPOINT was rejected. | ||
| # The README therefore supplies a concrete mapping via the CLI instead: | ||
| # -o "++execution.mounts.evaluation={$MODEL_CHECKPOINT: /checkpoint}" | ||
| evaluation: {} | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # Shared vLLM deployment for Qwen3-8B teacher, PTQ, and PTQ → QAD. | ||
| deployment: | ||
| image: vllm/vllm-openai:v0.28.0-cu129 | ||
| checkpoint_path: ${oc.env:MODEL_CHECKPOINT} | ||
| tensor_parallel_size: 1 | ||
| data_parallel_size: 1 | ||
| command: >- | ||
| vllm serve /checkpoint | ||
| --served-model-name ${deployment.served_model_name} | ||
| --host 0.0.0.0 | ||
| --port ${deployment.port} | ||
| --dtype bfloat16 | ||
| --tensor-parallel-size ${deployment.tensor_parallel_size} | ||
| --data-parallel-size ${deployment.data_parallel_size} | ||
| --max-model-len 32768 | ||
| --max-num-seqs 2 | ||
| --gpu-memory-utilization 0.85 | ||
| --seed 42 | ||
| --kv-cache-dtype auto | ||
|
Contributor
There was a problem hiding this comment. [IMPORTANT Compatibility] With
So if the exported PTQ/QAD checkpoints carry Two options:
Please check Separately on line 14, |
||
| tokenizer_path: /checkpoint | ||
|
Contributor
There was a problem hiding this comment. [SUGGESTION] It is interpolated into both tasks' A pointer comment here would make the dependency discoverable at the point of use: # Valid only when the checkpoint is mounted at /checkpoint in the evaluation
# container; see the -o ++execution.mounts.evaluation override in ../README.md.
tokenizer_path: /checkpoint |
||
Uh oh!
There was an error while loading. Please reload this page.