Unsupervised Anomaly Detection in Financial Transaction Data: A Mathematical and Experimental Study

Project Overview

This project investigates the application of unsupervised machine learning techniques for detecting anomalous transactions in financial datasets. Since real-world financial data often lack labelled examples of fraudulent or abnormal activities, unsupervised approaches provide an effective solution for identifying unusual behavioural patterns.

The study combines mathematical concepts and experimental analysis to evaluate different anomaly detection techniques and compare their ability to identify abnormal transaction behaviour.

---

Motivation

Financial institutions process millions of transactions every day, making manual monitoring impractical. Detecting unusual transactions at an early stage can help identify potential fraud, system misuse, or abnormal customer behaviour.

The primary objective of this project is to explore how different unsupervised learning algorithms identify anomalies in financial transaction data and to compare their behaviour using a common dataset.

---

Dataset

Source: Kaggle Financial Transactions Dataset

Dataset Characteristics

• Total Records: 50,000

• Multiple numerical and categorical features

• Examples of features:

  • TransactionAmount
  • TransactionDuration
  • LoginAttempts
  • AccountBalance
  • Customer-related attributes

---

Data Preprocessing

The following preprocessing steps were performed:

• Missing value inspection
• Date and time feature engineering
• Categorical feature encoding
• Feature scaling and normalization
• Exploratory data analysis
• Feature preparation for anomaly detection models

---

Methodology

1. Principal Component Analysis (PCA)

PCA reduces the dimensionality of the dataset while preserving maximum variance.

Anomaly Detection Principle

Transactions are reconstructed from principal components and reconstruction error is calculated:

Reconstruction Error = ||x − x̂||²

Large reconstruction error indicates anomalous behaviour.

---

2. Isolation Forest

Isolation Forest identifies anomalies through random feature partitioning.

Anomaly Detection Principle

Anomalous observations require fewer partitions and therefore have shorter isolation path lengths.

$$ s(x,n)=2^{-\frac{E(h(x))}{c(n)}} $$

---

3. Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

DBSCAN identifies anomalies as observations that do not belong to dense neighbourhood regions.

Density Criterion

$$ N_{\varepsilon}(x)={x_j : d(x,x_j)\leq \varepsilon} $$

Observations with insufficient neighbours are classified as noise points and treated as anomalies.

---

4. Gaussian Mixture Model (GMM)

GMM models transaction behaviour using multiple Gaussian probability distributions.

$$ p(x)=\sum_{k=1}^{K}\pi_k \mathcal{N}(x|\mu_k,\Sigma_k) $$

Transactions with low likelihood values are identified as anomalous observations.

---

Results Summary

Method	Anomalies Detected	Detection Principle
PCA	2500	Reconstruction Error
Isolation Forest	2500	Isolation Depth
DBSCAN	2896	Density-Based Noise Detection
Gaussian Mixture Model	2500	Probabilistic Likelihood

Key Observations

• PCA successfully identified transactions with high reconstruction error.
• Isolation Forest detected observations requiring fewer partitions.
• DBSCAN identified sparse observations outside dense behavioural clusters.
• GMM detected low-probability transactions under the learned probability distribution.
• Anomalous transactions generally exhibited higher transaction amounts and increased login activity compared to normal observations.

---

Future Work

The project will be extended by incorporating:

k-Nearest Neighbour (k-NN) Anomaly Detection

Distance-based anomaly scoring:

$$ d(x_i,x_j)=\sqrt{\sum_{k=1}^{m}(x_{ik}-x_{jk})^2} $$

Autoencoder-Based Anomaly Detection

Deep learning approach based on reconstruction loss:

$$ L=|x-\hat{x}|^2 $$

Future work will focus on comparing classical machine learning techniques with deep learning approaches for anomaly detection in financial transaction datasets.

---

Technologies Used

• Python
• Pandas
• NumPy
• Matplotlib
• Seaborn
• Scikit-Learn
• Jupyter Notebook

---

Repository Structure

├── data/
│   └── financial_transactions.csv
│
├── notebooks/
│   └── anomaly_detection.ipynb
│
├── report/
│   └── Project_Report.pdf
│
├── presentation/
│   └── Viva_Presentation.pptx
│
├── README.md
│
└── requirements.txt

---

References

1. Bishop, C. M. Pattern Recognition and Machine Learning. Springer, 2006.
2. Ester, M., Kriegel, H.-P., Sander, J., & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise.
3. Aggarwal, C. C. Outlier Analysis, 2nd Edition. Springer, 2017.
4. Cover, T. M., & Hart, P. E. (1967). Nearest Neighbor Pattern Classification.
5. Géron, A. Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow.

---

Author

Rajat Kumar

Professional Master's Program in Data Science and AI

Academic Project – Unsupervised Anomaly Detection in Financial Transaction Data