FLAP vs LoRA — SmolLM2-1.7B on MetaMathQA

A fine-tuning run that beat a LoRA baseline on GSM8K while giving up less general ability than LoRA did, and everything needed to check that claim.

Model	GSM8K strict	GSM8K flexible	ARC-Easy	ARC-Easy norm	HellaSwag	HellaSwag norm
SmolLM2-1.7B (untouched)	30.02	30.17	77.82	73.53	53.48	71.44
LoRA baseline	46.10	39.35	67.63	60.40	53.89	71.02
FLAP	48.45	41.32	68.43	61.41	53.98	70.46

GSM8K is 5-shot with a 256-token generation budget; ARC-Easy and HellaSwag are 0-shot. All three rows come from the same lm_eval commands on the same machine — eval/summarize.py prints those settings and refuses to present a table whose rows were scored differently.

Both fine-tunes start from HuggingFaceTB/SmolLM2-1.7B, train on the same MetaMathQA corpus and use the same adapter shape. The untouched base is scored too, because a difference between two fine-tunes means little without the point they both started from.

FLAP leads on five of the six metrics — all but HellaSwag-norm, where it is 0.56 behind. The headline is GSM8K strict: +2.35 pp.

How strong is that?

Strong enough to act on, not strong enough to call settled, and the difference matters — so it goes up front rather than in a footnote.

• +2.35 pp on GSM8K strict is 1.2σ. Each model's own standard error on that benchmark is ≈1.38 pp, so the standard error of the difference is ≈1.94 pp. A 2σ result would need roughly +3.9 pp on this eval set. This one does not clear that bar.
• What lifts it above a coin flip is that the lead is consistent: GSM8K strict +2.35, GSM8K flexible +1.97, ARC-Easy +0.80, ARC-Easy-norm +1.01, HellaSwag +0.09, HellaSwag-norm −0.56. Five of six pointing the same way says more than any single margin does alone — but it is not a substitute for the seeds and replications that would actually settle it.
• This is one run: one base model, one corpus, one seed, one LoRA baseline. Replication across seeds is planned and cheap (a few GPU-hours), and this file will be updated with the outcome whether or not it agrees.

Read +2.35 as "a consistent lead on one configuration". Anyone quoting it as "FLAP beats LoRA" is quoting it harder than the data does.

About the retention claim

An earlier version of this README said fine-tuning for maths usually costs general ability and "here it did not". That was wrong, and the base row is what shows it. Both fine-tunes cost a lot of ARC-Easy against the untouched model: 77.82 → 68.43 (FLAP) and 67.63 (LoRA). That price belongs to the corpus, and both methods pay it.

The honest claim is narrower and still worth something: FLAP pays 0.80 pp less of it while gaining 2.35 pp more GSM8K. A better exchange rate, not a free lunch.

What is in this repository

• eval/results/ — raw lm_eval output, so the numbers can be read rather than believed.
• eval/summarize.py — builds the table above straight from those JSONs. Every number here comes out of it; nothing is typed in by hand. python eval/summarize.py --check fails if this README has drifted from them.
• eval/run_evals.sh — the exact commands that produced the JSONs.
• MODEL_CARD.md — base model, data, licence, intended use, limits.
• charts/ — the same numbers, drawn.
• Model weights — on the Hugging Face Hub (large binaries belong there, not in git).

What is NOT in this repository

How FLAP works. The training method is the product and it stays closed. This repository is a claim plus the means to check it, not a recipe.

You can still verify everything that matters: the weights are real, the evaluation is a third-party harness, and the commands are here. If the numbers do not reproduce, that is a fact about the claim and it should be reported.

Reproduce the evaluation

pip install lm_eval
bash eval/run_evals.sh <path-or-hf-id-of-the-model>
python eval/summarize.py

Licence

The weights inherit the licence of HuggingFaceTB/SmolLM2-1.7B. The training data is MetaMathQA. Everything in this repository that is ours is MIT.