Feed-Forward Neural Network for MNIST Classification

Hello reader, this cool learning project I did which is a simple Feed-Forward Neural Network (FFNN) built from scratch using only NumPy to classify handwritten digits from the famous MNIST dataset.

It was genuinely a very fun experience demystifying the once abstracted Machine Learning Algorithms. Here, there is a direct application of the architecture of the Neural Network, forward pass algorithm which comes along with activation functions (Softmax, ReLU), backpropagation which is ofcourse based on the chosen loss function (CE), and ofcourse the lovely optimization via the Gradient Descent (used the mini-batch technique). Have fun reading this shamelessly AI-generated README.md file — signing off!

Features

• Built with NumPy: The entire neural network (forward pass, backpropagation, weight updates) is implemented using only the NumPy library.
• One Hidden Layer: A simple architecture with one hidden layer using the ReLU activation function.
• Softmax Output Layer: The output layer uses Softmax activation, making it suitable for multi-class classification.
• Cross-Entropy Loss: Implements the categorical cross-entropy loss function to measure model performance.
• Mini-Batch Gradient Descent: The model is trained using mini-batch gradient descent for efficient optimization.
• Visualizations: The script generates plots for:
  • Training loss and accuracy over epochs.
  • Examples of correct and incorrect predictions on the test set.

---

Requirements

You'll need Python 3 and the following libraries to run the script:

• numpy
• scikit-learn
• matplotlib
• pandas

You can install all the required dependencies with a single command:

pip install numpy scikit-learn matplotlib pandas

---

How to Run

1. Clone the repository or download the Python script (.py) file.

2. Install the dependencies as mentioned in the requirements section.

3. Run the script from your terminal:

   python your_script_name.py

The script will first download the MNIST dataset (this might take a minute the first time), then preprocess the data, train the network, and finally display the evaluation results and plots.

---

Code Overview

The script is structured into several key parts:

1. Activation Functions:

   • relu(): The Rectified Linear Unit function for the hidden layer.
   • relu_derivative(): The derivative of ReLU, needed for backpropagation.
   • softmax(): The Softmax function to convert output layer logits into probabilities.

2. Data Loading and Preprocessing:

   • The MNIST dataset is loaded using scikit-learn.
   • Pixel values are scaled to be between 0 and 1.
   • Labels are one-hot encoded to work with the cross-entropy loss function.
   • The data is split into training and testing sets.

3. Network Initialization:

   • The network architecture is defined (input size, hidden layer size, output size).
   • Weights and biases are initialized with small random values to break symmetry.

4. Training Loop:

   • The network iterates through a set number of epochs.
   • Forward Pass: Data is passed through the network to generate a prediction.
   • Loss Calculation: The cross-entropy loss between the prediction and the true labels is computed.
   • Backpropagation: The error is propagated backward through the network to calculate the gradients of the loss with respect to the weights and biases.
   • Weight Update: The weights and biases are updated using gradient descent to minimize the loss.

5. Evaluation and Visualization:

   • After training, the model's final accuracy is calculated on the unseen test set.
   • matplotlib is used to plot the training history and to visualize some of the model's predictions.

---

Expected Results

After running the script, you can expect the following:

1. Console Output: The training progress will be printed to the console every 10 epochs, showing the loss and accuracy. The final test accuracy will be displayed at the end.

Dataset loaded
Starting training...
Epoch 0, Loss: 0.0179, Accuracy: 0.9455
Epoch 10, Loss: 0.0029, Accuracy: 0.9922
Epoch 20, Loss: 0.0009, Accuracy: 0.9986
Epoch 30, Loss: 0.0003, Accuracy: 0.9999
Epoch 40, Loss: 0.0002, Accuracy: 1.0000
Epoch 50, Loss: 0.0001, Accuracy: 1.0000
Epoch 60, Loss: 0.0001, Accuracy: 1.0000
Epoch 70, Loss: 0.0001, Accuracy: 1.0000
Epoch 80, Loss: 0.0001, Accuracy: 1.0000
Epoch 90, Loss: 0.0001, Accuracy: 1.0000
Congratulations! Your MAGIKARP evolved into GYDRADOS!

Final Test Accuracy: 0.9746

Found 13645 correct predictions.
Found 355 incorrect predictions.

2. Training Plots:

4. Prediction Visualization: