[Example] NdLinear + LoRA Fine-Tuning on SmallViT (MNIST)Add files via upload

[aryanator]

This notebook demonstrates an efficient fine-tuning strategy for vision transformers by combining:

• NdLinear: A compressed linear layer that introduces tensor factorization to reduce parameter redundancy.
• LoRA (Low-Rank Adaptation): Lightweight fine-tuning via trainable low-rank matrices.

🧠 What’s included:

• A wrapper (NdLinearAdapter) to replace nn.Linear with NdLinear across ViT blocks.
• LoRA injection into NdLinear using pre-forward hooks.
• Training loop for three model variants:
  • Standard SmallViT
  • SmallViT with NdLinear
  • SmallViT with NdLinear + LoRA
• Visualization of loss, accuracy, and singular value distribution
• Final comparison of parameter counts, file size, and accuracy

📊 Results Summary

Model	Parameters	Accuracy	Model Size
Standard ViT	5.52M	95.01%	22.11 MB
NdLinear ViT	5.52M	95.81%	22.11 MB
NdLinear + LoRA	5.67M	94.86%	22.70 MB

📍 Notes:

• Dataset: MNIST
• Architecture: SmallViT from timm
• Self-contained, Colab-compatible, and easy to follow

This notebook is intended as a research-style educational example for the examples/ directory.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB