MNIST CNN Training Stabilization: A Case Study

From Unstable Training to 99.54% Accuracy

A practical demonstration of how to identify and fix training instability in CNN models using MNIST digit classification.

---

📋 Table of Contents

• Overview
• The Problem: Training Instability
• The Solution: Three Key Optimizations
• Results Comparison
• Model Architecture
• Installation
• Usage
• Testing Your Own Handwriting
• What I Learned

---

🎯 Overview

This project documents my journey from building a CNN that suffered from training instability (massive accuracy drops during training) to a stable, optimized model achieving 99.54% test accuracy on MNIST.

Background: After completing Andrew Ng's Deep Learning courses, I wanted to practice by building a CNN from scratch. This repository shows the real challenges I faced and how I solved them.

---

⚠️ The Problem: Training Instability

Initial Model (Unstable Version)

My first CNN implementation with aggressive data augmentation showed severe training instability:

Key Issues:

• Epoch 7: Validation accuracy dropped from 99.14% to 90.25% (9% loss!)
• Validation loss: Spiked from 0.027 to 0.280 (10x increase)
• Root causes identified:
  • Aggressive data augmentation (10° rotation, 10% shifts)
  • No learning rate adaptation
  • Batch Normalization + heavy augmentation conflict

Final Performance:

• Test Accuracy: 98.78%
• Misclassified: 122/10,000 samples

---

✅ The Solution: Three Key Optimizations

1. Conservative Data Augmentation

Before:

ImageDataGenerator(
    rotation_range=10,      # Too aggressive
    width_shift_range=0.1,
    height_shift_range=0.1,
    zoom_range=0.1
)

After:

ImageDataGenerator(
    rotation_range=5,       # More conservative
    width_shift_range=0.05,
    height_shift_range=0.05,
    zoom_range=0.05
)

2. Learning Rate Scheduling

Added ReduceLROnPlateau callback to automatically reduce learning rate when validation loss plateaus:

ReduceLROnPlateau(
    monitor='val_loss',
    factor=0.5,           # Reduce LR by 50%
    patience=2,           # Wait 2 epochs before reducing
    min_lr=0.00001
)

Result: Learning rate adapted 4 times during training:

• 0.001 → 0.0005 (Epoch 4)
• 0.0005 → 0.00025 (Epoch 7)
• 0.00025 → 0.000125 (Epoch 12)

3. Early Stopping

Prevent overfitting and save the best model:

EarlyStopping(
    monitor='val_accuracy',
    patience=5,                    # Stop if no improvement for 5 epochs
    restore_best_weights=True      # Load best model weights
)

Result: Training stopped at Epoch 13 (out of 20), restoring weights from Epoch 8 (best validation accuracy).

---

📊 Results Comparison

Before vs After

Metric	Unstable (v1)	Optimized (v2)	Improvement
Test Accuracy	98.78%	99.54%	+0.76%
Misclassified	122/10,000	46/10,000	-62% errors
Training Stability	❌ Epoch 7 drop	✅ Smooth	Stable
Training Time	10 epochs	13 epochs (stopped early)	Efficient
Val Acc Drop	9% drop	No drops	Fixed

Optimized Model Training Curves

Key Improvements:

• ✅ No sudden drops in validation accuracy
• ✅ Smooth convergence throughout training
• ✅ Learning rate adapted automatically when needed
• ✅ Early stopping prevented unnecessary training

---

🏗️ Model Architecture

Total Parameters: 242,954 (949 KB)
Trainable Parameters: 242,250 (946 KB)

Layer	Type	Output Shape	Parameters
Conv2D	32 filters (3×3)	(26, 26, 32)	320
BatchNormalization	-	(26, 26, 32)	128
MaxPooling2D	(2×2)	(13, 13, 32)	0
Conv2D	64 filters (3×3)	(11, 11, 64)	18,496
BatchNormalization	-	(11, 11, 64)	256
MaxPooling2D	(2×2)	(5, 5, 64)	0
Conv2D	128 filters (3×3)	(3, 3, 128)	73,856
BatchNormalization	-	(3, 3, 128)	512
Flatten	-	(1152)	0
Dense	128 units	(128)	147,584
BatchNormalization	-	(128)	512
Dropout	0.5	(128)	0
Dense (Output)	10 units (softmax)	(10)	1,290

---

🚀 Installation

Requirements

pip install tensorflow numpy matplotlib pillow

Or use requirements.txt:

tensorflow>=2.10.0
numpy>=1.23.0
matplotlib>=3.6.0
pillow>=9.3.0

pip install -r requirements.txt

---

💻 Usage

1. Train the Model

python main.py

Output:

• training_history.png - Loss and accuracy plots
• predictions.png - Random test samples with predictions
• error_analysis.png - Misclassified examples
• best_model.keras - Best model (saved automatically)
• mnist_cnn_final.keras - Final model

2. Model Performance

Test Accuracy: 99.54%
Misclassified: 46 out of 10,000 samples (0.46%)

3. Random Test Predictions

All 5 random predictions were correct ✅

4. Error Analysis

Common misclassifications:

• 4 ↔ 6 (75% confidence errors)
• 9 ↔ 4 (68-98% confidence)
• 5 ↔ 3 (87-94% confidence)
• 8 ↔ 9 (83% confidence)

These errors show that even at 99.54% accuracy, the model struggles with visually similar digits written in unusual styles.

---

✍️ Testing Your Own Handwriting

How to Test

1. Draw a digit (0-9) in Paint or any drawing software
2. Use a BLACK pen on WHITE background
3. Save as my_digit.png in the project folder
4. Run python main.py

My Custom Test: Digit "8"

Result:

• Predicted: 8 ✅
• Confidence: 80.36%

Important Notes

⚠️ MNIST expects white digits on black background, so if your image has:

• Black digit on white background → Uncomment line 225 in main.py:

  img_array = 255 - img_array  # Invert colors

⚠️ Common issues:

• Thin lines: Draw thicker digits (model expects ~3-4px width)
• Too small/large: Keep digit centered and reasonably sized
• Unclear shapes: Make sure digit features are clear (e.g., 8's two loops should be visible)

---

📚 What I Learned

Technical Lessons

1. Data Augmentation is powerful but dangerous

   • Start conservative, increase gradually
   • Monitor validation metrics carefully

2. Learning Rate Scheduling is crucial

   • Static LR can cause instability
   • ReduceLROnPlateau adapts automatically

3. Early Stopping saves time and prevents overfitting

   • Saved 7 epochs of unnecessary training
   • Automatically restored best weights

4. Batch Normalization needs careful tuning

   • Works great with moderate augmentation
   • Can conflict with aggressive augmentation

Practical Insights

• Always visualize training curves - Spot issues early
• Start simple, add complexity gradually - Easier to debug
• Test on real data (custom images) - Reveals generalization issues
• Error analysis is valuable - Shows where model struggles

---

📁 Project Structure

mnist-cnn-optimization/
├── main.py                      # Main training script
├── requirements.txt             # Python dependencies
├── README.md                    # This file
├── my_digit.png                 # (Optional) Your custom digit
├── training_history.png         # Generated: Training curves
├── predictions.png              # Generated: Test predictions
├── error_analysis.png           # Generated: Misclassified samples
├── custom_prediction.png        # Generated: Custom digit result
├── best_model.keras             # Generated: Best model weights
└── mnist_cnn_final.keras        # Generated: Final model

---

🎓 Background

This project was created as my first hands-on CNN implementation after completing:

• Andrew Ng's Deep Learning Specialization (Coursera)
• Theoretical understanding of CNNs, backpropagation, and optimization

Goal: Translate theory into practice and learn from real challenges.

---

📝 License

MIT License - Feel free to use this code for learning and experimentation.

---