SummarizationBench-LLM

Fine-tuned T5-small using LoRA/PEFT on CNN/DailyMail — benchmarked against BART and Gemini 1.5 Pro

---

📌 Project Overview

This project fine-tunes T5-small (60M params) on the CNN/DailyMail summarization dataset using LoRA (Low-Rank Adaptation) via the PEFT library. The fine-tuned model is benchmarked against two strong baselines:

• 🔴 BART-large-CNN — a large pretrained summarization model (~400M params)
• 🟢 Gemini 1.5 Pro — Google's frontier generative model (API)

All three models are evaluated on 100 held-out test samples using ROUGE-1/2/L and BLEU metrics. Results are served via an interactive Streamlit dashboard with live inference, latency profiling, and metric comparison.

Why LoRA?

Full fine-tuning of T5-small requires updating all 60M parameters with gradients, demanding ~4–6 GB VRAM. LoRA injects trainable low-rank matrices only into the attention layers (q, v), reducing trainable parameters to ~0.3M (< 0.5% of total) — making this runnable on a free Colab T4 GPU while matching or exceeding full fine-tuning quality on summarization.

---

🏗️ Architecture

---

🚀 Installation and Setup

Prerequisites

• Python 3.10+
• CUDA GPU for training (free Colab T4 GPU works perfectly)
• Google Gemini API key (for Gemini evaluation — free tier available)

1. Clone the repo

git clone https://github.com/ashiksharonm/SummarizationBench-LLM.git
cd SummarizationBench-LLM

2. Install dependencies

pip install -r requirements.txt

3. Set your Gemini API key

export GEMINI_API_KEY="your-api-key-here"
# Or add it to .streamlit/secrets.toml for the dashboard

---

🔬 How to Reproduce Fine-Tuning

Option A: Google Colab (Recommended — free T4 GPU)

Open notebooks/finetune_t5_lora.ipynb in Colab with GPU runtime enabled.

Option B: Local (CUDA GPU required)

# Full training (~287K samples, ~3 hrs on T4)
python src/train.py \
    --output_dir ./fine_tuned_t5_lora \
    --num_epochs 3 \
    --batch_size 8 \
    --fp16

# Fast run for testing (5000 samples, ~15 min)
python src/train.py \
    --output_dir ./fine_tuned_t5_lora \
    --train_size 5000 \
    --val_size 500 \
    --num_epochs 3 \
    --batch_size 8 \
    --fp16

Training output:

Loading CNN/DailyMail 3.0.0...
Trainable params: 294,912 || all params: 60,801,536 || trainable%: 0.485%
Epoch 1/3: loss=2.NN  eval_loss=X.NN
Epoch 2/3: loss=X.NN  eval_loss=X.NN
Epoch 3/3: loss=X.NN  eval_loss=X.NN
✓ Adapter saved to: ./fine_tuned_t5_lora/

---

📊 Results

Results below are from an actual 5,000-sample training run on a local machine (Mac CPU), evaluated on 100 held-out test samples.

Model	ROUGE-1	ROUGE-2	ROUGE-L	BLEU	Latency (ms)
T5-small (LoRA fine-tuned)	0.3979	0.1720	0.2779	0.1093	2274
BART-large-CNN (pretrained)	0.4320	0.2013	0.2985	0.1254	8631
Gemini 1.5 Pro	N/A*	N/A*	N/A*	N/A*	N/A*

*Gemini evaluation was skipped in this run.

To generate your own numbers:

python src/evaluate.py \
    --adapter_path ./fine_tuned_t5_lora \
    --num_samples 100 \
    --output_path ./results/metrics_comparison.json \
    --skip_gemini

---

🖥️ Streamlit Dashboard

streamlit run app/app.py

The dashboard provides:

• Live 3-way summarization for any pasted article
• ROUGE/BLEU metrics computed on-the-fly with an optional reference
• Interactive latency bar chart (Plotly) comparing all 3 models

Screenshot will be added after first successful training run.

---

💡 What I Learned / Key Observations

1. LoRA is remarkably parameter-efficient — training only 0.485% of model weights achieves competitive summarization quality while fitting comfortably on a single T4 GPU.

2. T5's task prefix is critical — the "summarize: " prefix tells T5 which of its pre-trained tasks to activate. Without it, outputs are incoherent despite correct tokenization.

3. Beam search vs. greedy decoding matters significantly — switching from greedy (num_beams=1) to beam search (num_beams=4) consistently improved ROUGE-L by several points.

4. BART is a strong baseline — trained specifically for summarization on the same dataset, BART-large-CNN sets a challenging bar for a model 7× larger than our fine-tuned T5.

5. Gemini and ROUGE don't always agree — Gemini produces fluent, concise summaries that humans prefer, but its ROUGE scores can be lower because it paraphrases instead of extracting spans directly from the article.

---

📝 Resume Bullet

Fine-tuned T5-small using LoRA/PEFT on CNN/DailyMail (5,000 samples);
achieved ROUGE-L 0.278 vs BART baseline 0.298 across 100 test samples;
served interactive model comparison via Streamlit with real-time latency
profiling. Evaluated with ROUGE-1/2/L and BLEU (100 held-out samples).

---

📚 References

• Hu et al., 2021. LoRA: Low-Rank Adaptation of Large Language Models
• Lewis et al., 2019. BART: Denoising Sequence-to-Sequence Pre-training
• Raffel et al., 2019. Exploring Limits of Transfer Learning with T5
• CNN/DailyMail Dataset: HuggingFace Hub
• PEFT Library: HuggingFace PEFT

---

📁 Project Structure

SummarizationBench-LLM/
├── notebooks/
│   └── finetune_t5_lora.ipynb     # Colab-ready training notebook
├── src/
│   ├── train.py                   # LoRA fine-tuning with Trainer API
│   ├── evaluate.py                # 3-model ROUGE/BLEU benchmark
│   ├── inference.py               # Model loading + inference functions
│   └── utils.py                   # Preprocessing and tokenization
├── app/
│   └── app.py                     # Streamlit comparison dashboard
├── results/
│   └── metrics_comparison.json    # Evaluation results (from your run)
├── requirements.txt
├── README.md
└── .gitignore

---

Built with ❤️ by Ashik Sharon