LLM 4B model selection for v7.0 Mobile MLPerf #43

Activity

1. Mostelk

   assigned

   Mostelk

   ,

   freedomtan

   and

   mohitmundhragithub

   on Aug 26, 2026
2. Mostelk commented on Sep 1, 2026

   

   Mostelk

   on Sep 1, 2026

   ContributorAuthor

   More actions

   @farook-edev please share public litert-torch conversion scripts for these models
3. freedomtan commented on Sep 15, 2026

   

   freedomtan

   on Sep 15, 2026

   Contributor

   More actions

   @farook-edev added #45
4. AhmedTElthakeb commented on Sep 22, 2026

   

   AhmedTElthakeb

   on Sep 22, 2026

   Last edited by AhmedTElthakeb

   More actions

   model	TinyMMLU (few_shot_2)	IFEval-33 average	MMLU-Pro (5-shot exact_match)
   gemma-4-e4b-it (FP baseline)	63.00%	0.8121	0.6692* ± 0.0041
   gemma-4-E4B-it-qat-q4_0-unquantized	57.00%	0.7777	0.6496 ± 0.0042
   gemma-4-E4B-it-qat-w4a16-ct	60.00%	0.8200	0.6269 ± 0.0042
   gemma-4-E4B-it-qat-mobile-transformers	60.00%	0.8124	ongoing
   gemma-4-E4B-it-qat-mobile-ct	60.00%	0.8314	0.6451 ± 0.0042

   lm_eval \
     --model vllm \
     --model_args pretrained=google/gemma-4-e4b-it \
     --tasks mmlu_pro \
     --apply_chat_template \
     --fewshot_as_multiturn \
     --batch_size auto \
     --output_path ./output_lmeval \
     2>&1 | tee mmlupro_lmeval.log
   

   *(vs official report: 0.6940 !)
5. freedomtan commented on Sep 29, 2026

   

   freedomtan

   on Sep 29, 2026

   Contributor

   More actions

   @mohitmundhragithub Qwen 3.5 4B is preferred one. Gemma 4 is acceptable. But Qwen first. @Mostelk
6. freedomtan commented on Oct 6, 2026

   

   freedomtan

   on Oct 6, 2026

   Contributor

   More actions

   Check if it's OK to keep both of them as optional models, @mohitmundhragithub @Mostelk @freedomtan
7. freedomtan commented on Oct 13, 2026

   

   freedomtan

   on Oct 13, 2026

   Contributor

   More actions

   • drop llama 3B: YES
   • if it's OK to keep both (gemma 4 e4b and qwen 3.5 4b) of them as optional models: YES for all
     • @Mostelk to provide model reference URLs to models
8. Mostelk commented on Oct 28, 2026

   

   Mostelk

   on Oct 28, 2026

   ContributorAuthor

   More actions

   All models below are optional for submission:

   V7.0 LLM Models	Float Reference	Quantized Reference
   Llama 3.1 8B	https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
   Gemma 4 E4B	https://huggingface.co/google/gemma-4-E4B-it	https://huggingface.co/google/gemma-4-E4B-it-qat-mobile-transformers
   Qwen 3.5 4B	https://huggingface.co/Qwen/Qwen3.5-4B
9. freedomtan commented on Nov 3, 2026

   

   freedomtan

   on Nov 3, 2026

   Contributor

   More actions

   @farook-edev please compare with these Gemma 4 numbers.
10. freedomtan commented on Nov 10, 2026

    

    freedomtan

    on Nov 10, 2026

    Contributor

    More actions

    @Mostelk's post

    model	TinyMMLU (few_shot_2)	IFEval-33 average	MMLU-Pro (5-shot exact_match)
    gemma-4-e4b-it (FP baseline)	63.00%	0.8121	0.6692* ± 0.0041
    gemma-4-E4B-it-qat-q4_0-unquantized	57.00%	0.7777	0.6496 ± 0.0042
    gemma-4-E4B-it-qat-w4a16-ct	60.00%	0.8200	0.6269 ± 0.0042
    gemma-4-E4B-it-qat-mobile-transformers	60.00%	0.8124	ongoing
    gemma-4-E4B-it-qat-mobile-ct	60.00%	0.8314	0.6451 ± 0.0042

    @farook-edev on device with the App

    Model	TinyMMLU	IFEval
    Gemma E4B	47.0000%	65.9209%
    Qwen 3.5 4B	52.0000%	86.2216%