Skip to content

About

Does LoRA fine-tuning inject new knowledge, and does model size change the answer? A controlled experiment across 3 model families and a 55x size range.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Does LoRA fine-tuning actually inject knowledge?

A controlled experiment across three model families and a 55× size range, on facts no model could have seen before.

11 fine-tunes 135M to 7.6B MLX MIT


The question

Fine-tuning demos usually end at "the loss went down." That is not evidence a model learned anything: held-out perplexity drops after almost any fine-tune simply from memorising the prompt format, and it drops whether or not a single fact was retained.

So I built an experiment that can distinguish four different things, on data the models provably did not have:

  1. Recall — can it retrieve a trained fact when the question is phrased in a way it never saw?
  2. Generalisation — does that survive a second unseen phrasing?
  3. Composition — given A → B and B → C trained separately, can it answer A → C?
  4. Hallucination — asked about entities absent from training, does it abstain or invent?

And then ran it across three model families at three sizes each.

Headline results

Model Params Recall Generalisation Composition Train time
SmolLM2‑135M 0.135B 50% 50% 21% 40s
SmolLM2‑360M 0.362B 72% 80% 17% 56s
Qwen2.5‑0.5B 0.494B 100% 100% 17% 68s
Falcon3‑1B 1.669B 83% 86% 17% 92s
SmolLM2‑1.7B 1.711B 94% 95% 21% 124s
Qwen2.5‑3B 3.090B 98% 99% 17% 276s
Falcon3‑3B 3.228B 90% 90% 17% 164s
Falcon3‑7B 7.456B 100% 99% 29% 360s
Qwen2.5‑7B 7.620B 100% 100% 21% 428s

Teacher-forced multiple choice, 126 held-out questions per suite. Chance = 25%.

Three findings:

Recall saturates early. The climb from 50% → 100% happens almost entirely below 0.5B. Above that, every model lands between 83% and 100% — mostly inside each other's noise. If your goal is injecting facts, a 0.5B model may be all you need.

Family rivals size in the middle of the range. Qwen2.5‑0.5B (100%) beats Falcon3‑1B (83%) and SmolLM2‑1.7B (94%) while being 3–4× smaller. Parameter count is not the only axis.

Nothing composed — at any size. Both hops were trained separately and the direct link appears in zero training rows. Every model sat on the chance line, the 7Bs included. Memorisation was trivial; the one-step inference never happened. This is the result I found most interesting, and it is a negative one.

Run-to-run spread from three seeds of the same cell is ±4.8pt, so treat any gap under ~10pt as noise. Full report: results/matrix/report.md


Why the data is invented

Measuring knowledge injection on a real domain is unfalsifiable: you cannot prove a model has not seen an obscure topic, and any contamination silently corrupts the result — a model that already knew the answers looks like a successful fine-tune.

So the corpus is a completely fabricated pharmacology: the Kestrel Formulary. Every compound, drug class, antidote, marker and condition is coined. Because the domain cannot exist in any pretraining set, base models score at chance by construction — verified before every run — and any gain is provably attributable to the fine-tune.

{"messages":[{"role":"user","content":"Which drug class does Velmoxryne belong to?"},
             {"role":"assistant","content":"Velmoxryne is a thalamic dampener."}],
 "factId":"Velmoxryne.class","answer":"thalamic dampener",
 "distractors":["cortical primer","vagal stabiliser","renal shunt agent"]}

24 compounds × 5 attributes + 6 class-level facts = 132 atomic facts, each rendered through 12 question templates → 1,228 training rows. Ten templates train; two are held back so every test question is a phrasing the model has never seen.

The generator is deterministic and self-policing

No RNG and no seed — values are assigned by index arithmetic with coprime strides over closed sets, so the corpus is a pure function of the fact table and regenerates byte-identically. It also asserts its own validity, and those assertions rejected two of my earlier drafts:

  • No answer may share a 4-gram with its own entity name. This caught ten coinages where the answer was guessable from morphology alone — Ovaldrine → "lateral drine palsy" — which would have made the task partly solvable without any training at all.
  • No training row may link a compound to its antidote. My first draft made the antidote a property of the compound, which quietly made the composition suite answerable by plain recall for a third of rows, and self-contradictory for the rest. The antidote is now a property of the class, so compound → class → antidote genuinely requires two hops.
npm run data   # regenerates the corpus; assertions run every time

Measuring it honestly

The metrics are ordered by how much they can be trusted, and the code is explicit about which may be compared across models:

Metric What it is Cross-model?
Multiple choice Teacher-forced: score each candidate, lowest loss wins Yes — immune to verbosity
Free-text accuracy Generated answer graded against a closed candidate set Mostly — verbosity-sensitive
Bits per byte Loss over the answer's UTF-8 bytes Yes — same denominator for every tokenizer
Perplexity Per-token, completion-only No — tokenizer-dependent

Perplexity is deliberately demoted. In an earlier run a 135M model scored a lower base perplexity (12.99) than a 7B one (17.88) on identical text — not because it models English better, but because per-token perplexity divides by a tokenizer-specific token count. Bits-per-byte uses the same denominator for every model, so it is the one likelihood number that survives a cross-family comparison.

Grading scans a closed answer set on word boundaries rather than normalising and substring-matching. The naive approach is quietly wrong on exactly this kind of data:

Naive rule Breaks on Result
strip articles, then contains gold Hepatitis A, output Hepatitis B wrong answer marked correct
strip punctuation, then contains gold 12.5 mg → 125 mg, distractor 25 mg correct answer marked wrong

Both are regression tests. Answers grade as correct / wrong / abstain / ambiguous, so a model that declines to answer is never scored the same as one that confidently guesses wrong — which is what makes the hallucination suite meaningful.


The bug that produced a confident, wrong answer

My first sweep reported that a 135M model beat a 7B model — SmolLM2 at 99%, Qwen2.5‑7B at 21%. That is a publishable-looking finding and it was entirely an artifact.

MLX applies the LoRA scale as a raw multiplier (y + scale·z), not as alpha/rank the way PEFT does. So the effective step size is learningRate × scale. I set a textbook lr 2e-4 on top of MLX's default scale 20, giving 4e-3 — twenty times too high. The narrow model absorbed it; the wider ones diverged and never recovered:

Qwen2.5-7B train loss     iter 10        20        30        40
  lr 2e-4 × scale 20        4.12      4.96     13.54      9.90   → final val 3.53, recall 21%
  lr 1e-5 × scale 20        7.08      1.25      1.40      0.83   → best val 0.002, recall 100%

A probe confirmed the mechanism — only the product matters, not how it is split:

lr scale product val@200 early spike
2e-4 20 4e-3 4.508 13.64
2e-4 2 4e-4 0.203 3.04
1e-4 2 2e-4 0.192 3.42
1e-5 20 2e-4 0.185 3.38

The two rows at 2e-4 differ in both knobs and agree to three decimals. The failed sweep is kept in results/diverged/ rather than deleted, because "a plausible hyperparameter produced a confident and completely wrong conclusion about model size" is the most transferable thing in this repo.

A second silent failure, fixed in the harness

MLX writes numbered checkpoints during training but loads adapters.safetensors — the final iteration. Across my earlier runs, validation loss consistently bottomed out around iteration 150 and degraded through 300, meaning every run shipped an adapter measurably worse than one already sitting on disk. The runner now promotes the lowest-validation-loss checkpoint automatically; it fired on 3 of the 11 cells here (Qwen2.5‑7B: iteration 650, val 0.002).


Running it

Requires Apple Silicon (the trainer is MLX) and Node ≥ 22. The first run creates a Python venv and installs mlx-lm automatically.

npm install
npm run data     # generate the corpus (deterministic, self-checking)
npm run bench    # run every cell in bench/matrix.json
npm test         # 18 unit tests — no GPU, no network, no Python

The sweep is a standalone process, not a UI feature, because a multi-hour run has to survive closing the terminal. It checkpoints after every cell, so re-running resumes rather than restarting, and a lockfile prevents two sweeps from overlapping and poisoning the timings.

bench/
├─ gen-dataset.ts    corpus generator — deterministic, asserts its own validity
├─ matrix.json       the model matrix + shared hyperparameters
├─ probe.json        the learning-rate × scale probe
└─ run-matrix.ts     resumable sweep runner → experiment.json + report.md
src/
├─ train.ts          LoRA runner: seeded splits, prompt masking, best-checkpoint promotion
├─ eval.ts           scoring + the closed-set grader
├─ mlx_eval.py       perplexity / bits-per-byte / multiple choice / batched generation
└─ *.test.ts         regression tests, including both grader failure modes above
results/
├─ matrix/           the 11-cell sweep — report, raw JSON, full trainer log
├─ diverged/         the failed sweep, kept as evidence
└─ probe/            the learning-rate × scale probe

The training and evaluation code is extracted from local-language-machine, an offline AI coding IDE, where it powers the built-in fine-tuning panel.


What this experiment does not show

  • Size is still confounded with learning-rate optimality. One lr × scale was used across a 55× width span. It is stable everywhere, but not necessarily optimal everywhere — a per-cell sweep would be the honest next step.
  • The hallucination suite is underpowered. 20 items gives roughly a ±20pt interval. Directional only.
  • Family comparisons are ragged. SmolLM2 tops out at 1.7B, so only Qwen2.5 ↔ Falcon3 are cleanly comparable at the top tier.
  • Timings are not benchmarks. Cells ran serially on one machine, so later runs were thermally disadvantaged.
  • One dataset, one task shape. Short factual question–answer pairs. Nothing here generalises to style transfer, reasoning distillation, or long-form generation.

License

MIT — see LICENSE.

About

Does LoRA fine-tuning inject new knowledge, and does model size change the answer? A controlled experiment across 3 model families and a 55x size range.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages