🧠 ML From Scratch

Every Machine Learning Algorithm, Implemented From Scratch in Python.

The only repo you need to understand how ML truly works under the hood — and ace your interviews.

---

🎯 What Is This?

This is a comprehensive, interview-ready collection of 48 machine learning algorithms — all coded from scratch using only Python and NumPy. No scikit-learn. No TensorFlow. No PyTorch. Just pure math turned into working code.

Every algorithm comes with:

• ✅ Clean Python implementation — class-based, well-commented, importable
• ✅ README with theory — intuition, math, pseudocode, complexity analysis
• ✅ Runnable example — demo on a toy dataset with printed output
• ✅ Interview Q&A — real questions you'll face, with crisp answers

---

🤔 Who Is This For?

• ML/AI learners who want to go beyond calling model.fit() and actually understand what happens inside
• Engineers preparing for interviews at FAANG, startups, or ML-focused companies
• Students supplementing coursework with hands-on implementations
• Anyone who believes the best way to learn an algorithm is to build it

---

📁 Repository Structure

ml-from-scratch/
│
├── README.md                        ← you are here
├── pyproject.toml                   ← pip install -e . to install as package
├── requirements.txt
├── LICENSE
│
├── utils/                           ← shared helpers used across all algorithms
│   ├── __init__.py
│   ├── metrics.py                   ← accuracy, precision, recall, F1, MSE, R², etc.
│   ├── distances.py                 ← euclidean, manhattan, cosine, minkowski
│   ├── activations.py               ← sigmoid, relu, tanh, softmax + derivatives
│   ├── losses.py                    ← MSE, cross-entropy, hinge loss, etc.
│   └── preprocessing.py             ← standard scaler, train/test split, one-hot
│
└── algorithms/
    ├── 01_supervised_regression/
    │   ├── linear_regression_normal_equation/
    │   ├── linear_regression_gradient_descent/
    │   ├── polynomial_regression/
    │   ├── ridge_regression/
    │   ├── lasso_regression/
    │   └── elastic_net/
    │
    ├── 02_supervised_classification/
    │   ├── logistic_regression/
    │   ├── k_nearest_neighbors/
    │   ├── naive_bayes_gaussian/
    │   ├── naive_bayes_multinomial/
    │   ├── perceptron/
    │   ├── svm_linear/
    │   ├── svm_kernel/
    │   ├── decision_tree_classifier/
    │   └── decision_tree_regressor/
    │
    ├── 03_ensemble_methods/
    │   ├── bagging/
    │   ├── random_forest/
    │   ├── adaboost/
    │   ├── gradient_boosting/
    │   └── stacking/
    │
    ├── 04_clustering/
    │   ├── k_means/
    │   ├── k_means_plus_plus/
    │   ├── hierarchical_clustering/
    │   ├── dbscan/
    │   ├── mean_shift/
    │   └── gaussian_mixture_models/
    │
    ├── 05_dimensionality_reduction/
    │   ├── pca/
    │   ├── svd/
    │   ├── lda/
    │   └── tsne/
    │
    ├── 06_neural_networks/
    │   ├── single_layer_perceptron/
    │   ├── multi_layer_perceptron/
    │   ├── cnn/
    │   ├── rnn_vanilla/
    │   ├── lstm/
    │   └── autoencoder/
    │
    ├── 07_optimization/
    │   ├── batch_gradient_descent/
    │   ├── stochastic_gradient_descent/
    │   ├── mini_batch_gradient_descent/
    │   ├── momentum/
    │   ├── rmsprop/
    │   └── adam/
    │
    ├── 08_recommender_systems/
    │   ├── collaborative_filtering/
    │   └── matrix_factorization/
    │
    ├── 09_reinforcement_learning/
    │   ├── q_learning/
    │   └── sarsa/
    │
    └── 10_probabilistic/
        ├── hidden_markov_model/
        └── apriori/

Each algorithm folder contains:

<algorithm_name>/
├── README.md            ← theory, math, pseudocode, interview Q&A
├── <algorithm_name>.py  ← from-scratch implementation (class with fit/predict)
├── example.py           ← runnable demo with output

---

🖼️ Visual Gallery

Run the notebooks in notebooks/ to generate these visualizations:

Category	What You'll See
Regression	6 regression lines (Linear, Polynomial, Ridge, Lasso, Elastic Net) on the same scatter plot — compare fits visually
Classification	Decision boundaries for Logistic Regression, KNN, Naive Bayes, Decision Tree — 2×2 grid on 2D data
Ensembles	Single tree vs Random Forest vs AdaBoost decision boundaries — see how ensembles reduce overfitting
Clustering	K-Means, DBSCAN, GMM side-by-side on blob data — compare cluster assignments
Dimensionality Reduction	PCA, SVD, LDA 2D projections from 10D data — see which preserves class separation
Neural Networks	MLP learning XOR — decision boundary before/after training + loss curve
Optimizers	Momentum, RMSProp, Adam converging on a simple 1D loss surface — see trajectory differences

cd notebooks/
jupyter notebook

---

📋 Full Algorithm Index

0. Utils — Shared Building Blocks

File	What It Covers
metrics.py	Accuracy, Precision, Recall, F1, MSE, RMSE, MAE, R²
distances.py	Euclidean, Manhattan, Cosine, Minkowski
activations.py	Sigmoid, ReLU, Tanh, Softmax + all derivatives
losses.py	MSE, Binary/Categorical Cross-Entropy, Hinge
preprocessing.py	StandardScaler, MinMaxScaler, train_test_split, one-hot encoding

1. Supervised Learning — Regression

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 1 | Linear Regression (Normal Equation) | 01_supervised_regression/linear_regression_normal_equation/ | ✅ | | 2 | Linear Regression (Gradient Descent) | 01_supervised_regression/linear_regression_gradient_descent/ | ✅ | | 3 | Polynomial Regression | 01_supervised_regression/polynomial_regression/ | ✅ | | 4 | Ridge Regression (L2) | 01_supervised_regression/ridge_regression/ | ✅ | | 5 | Lasso Regression (L1) | 01_supervised_regression/lasso_regression/ | ✅ | | 6 | Elastic Net | 01_supervised_regression/elastic_net/ | ✅ |

2. Supervised Learning — Classification

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 7 | Logistic Regression | 02_supervised_classification/logistic_regression/ | ✅ | | 8 | K-Nearest Neighbors | 02_supervised_classification/k_nearest_neighbors/ | ✅ | | 9 | Naive Bayes (Gaussian) | 02_supervised_classification/naive_bayes_gaussian/ | ✅ | | 10 | Naive Bayes (Multinomial) | 02_supervised_classification/naive_bayes_multinomial/ | ✅ | | 11 | Perceptron | 02_supervised_classification/perceptron/ | ✅ | | 12 | SVM (Linear) | 02_supervised_classification/svm_linear/ | ✅ | | 13 | SVM (Kernel) | 02_supervised_classification/svm_kernel/ | ✅ | | 14 | Decision Tree (Classifier) | 02_supervised_classification/decision_tree_classifier/ | ✅ | | 15 | Decision Tree (Regressor) | 02_supervised_classification/decision_tree_regressor/ | ✅ |

3. Ensemble Methods

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 16 | Bagging | 03_ensemble_methods/bagging/ | ✅ | | 17 | Random Forest | 03_ensemble_methods/random_forest/ | ✅ | | 18 | AdaBoost | 03_ensemble_methods/adaboost/ | ✅ | | 19 | Gradient Boosting | 03_ensemble_methods/gradient_boosting/ | ✅ | | 20 | Stacking | 03_ensemble_methods/stacking/ | ✅ |

4. Clustering

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 21 | K-Means | 04_clustering/k_means/ | ✅ | | 22 | K-Means++ | 04_clustering/k_means_plus_plus/ | ✅ | | 23 | Hierarchical Clustering | 04_clustering/hierarchical_clustering/ | ✅ | | 24 | DBSCAN | 04_clustering/dbscan/ | ✅ | | 25 | Mean Shift | 04_clustering/mean_shift/ | ✅ | | 26 | Gaussian Mixture Models | 04_clustering/gaussian_mixture_models/ | ✅ |

5. Dimensionality Reduction

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 27 | PCA | 05_dimensionality_reduction/pca/ | ✅ | | 28 | SVD | 05_dimensionality_reduction/svd/ | ✅ | | 29 | LDA | 05_dimensionality_reduction/lda/ | ✅ | | 30 | t-SNE | 05_dimensionality_reduction/tsne/ | ✅ |

6. Neural Networks

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 31 | Single-Layer Perceptron | 06_neural_networks/single_layer_perceptron/ | ✅ | | 32 | Multi-Layer Perceptron | 06_neural_networks/multi_layer_perceptron/ | ✅ | | 33 | CNN | 06_neural_networks/cnn/ | ✅ | | 34 | RNN (Vanilla) | 06_neural_networks/rnn_vanilla/ | ✅ | | 35 | LSTM | 06_neural_networks/lstm/ | ✅ | | 36 | Autoencoder | 06_neural_networks/autoencoder/ | ✅ |

7. Optimization Algorithms

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 37 | Batch Gradient Descent | 07_optimization/batch_gradient_descent/ | ✅ | | 38 | Stochastic Gradient Descent | 07_optimization/stochastic_gradient_descent/ | ✅ | | 39 | Mini-Batch Gradient Descent | 07_optimization/mini_batch_gradient_descent/ | ✅ | | 40 | Momentum | 07_optimization/momentum/ | ✅ | | 41 | RMSProp | 07_optimization/rmsprop/ | ✅ | | 42 | Adam | 07_optimization/adam/ | ✅ |

8. Recommender Systems

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 43 | Collaborative Filtering | 08_recommender_systems/collaborative_filtering/ | ✅ | | 44 | Matrix Factorization | 08_recommender_systems/matrix_factorization/ | ✅ |

9. Reinforcement Learning

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 45 | Q-Learning | 09_reinforcement_learning/q_learning/ | ✅ | | 46 | SARSA | 09_reinforcement_learning/sarsa/ | ✅ |

10. Probabilistic & Other

| # | Algorithm | Folder | Status | |---|---|---|---|---| | 47 | Hidden Markov Model | 10_probabilistic/hidden_markov_model/ | ✅ | | 48 | Apriori | 10_probabilistic/apriori/ | ✅ |

---

🚀 Getting Started

1. Clone the repo

git clone https://github.com/rohanmistry231/ml-from-scratch.git
cd ml-from-scratch

2. Install dependencies

pip install -r requirements.txt

Only 3 dependencies: numpy, matplotlib, and scikit-learn (used only in examples for loading datasets and benchmarking — never in the algorithm code itself).

3. Run any algorithm

cd algorithms/01_supervised_regression/linear_regression_normal_equation/
python example.py

---

🛤️ Suggested Learning Path

If you're starting fresh, follow this order. Each builds on the last.

Phase 1 — Foundations (do first)

utils/ → Linear Regression (Normal Eq) → Linear Regression (GD) → Polynomial Regression

Phase 2 — Core Supervised Learning

Logistic Regression → KNN → Naive Bayes → Perceptron → Decision Tree

Phase 3 — Regularization

Ridge → Lasso → Elastic Net

Phase 4 — Ensembles

Bagging → Random Forest → AdaBoost → Gradient Boosting

Phase 5 — Unsupervised

K-Means → DBSCAN → PCA → GMM

Phase 6 — Neural Networks

Single Perceptron → MLP → CNN → RNN → LSTM

Phase 7 — Advanced & Interview Prep

SVM → t-SNE → Optimizers (SGD → Adam) → Recommenders → RL

---

🔑 Rules of This Repo

Rule	Why
No ML library in algorithm code	The whole point is learning what's inside the box
numpy is allowed	Vectorized math is how real implementations work
scikit-learn only in example.py	For loading data and validating our results
Every algo has a README	Code without explanation teaches nothing
Every algo has interview Q&A	Make your study immediately useful

---

🤝 Contributing

1. Fork the repo
2. Pick an algorithm marked ⬜ in the index
3. Follow the folder template exactly (README.md, implementation, example.py)
4. Ensure python example.py runs and prints output
5. Open a PR

See CONTRIBUTING.md for detailed guidelines.

---

📜 License

MIT — use it, learn from it, build on it.

---

⭐ Star This Repo

If this helped you learn or land an interview — drop a ⭐. It helps others find it.

---

"You don't truly understand an algorithm until you can implement it from scratch."