Frequency-Domain Compression of LLM Weights

An experiment testing whether the weight matrices of GPT-2 can be compressed in the frequency domain using the 2D Fast Fourier Transform (2D FFT). If weights are dominated by low-frequency components, discarding high-frequency coefficients could reduce effective parameter count and memory footprint with minimal quality loss.

Result summary: the approach does not work on raw GPT-2 weight matrices. Weight matrices (attention and MLP layers) exhibit nearly uniform spectral energy distributions — the opposite of what frequency compression requires. Model quality degrades catastrophically at every compression level tested. The positional embedding is a notable exception and behaves as expected of a learnable, smooth signal.

---

Table of Contents

• Motivation
• Setup
• Experiment Structure
• Phase 1 — Spectral Analysis
• Phase 2 — Compression & Reconstruction
• Phase 3 — Quality Benchmarking
• Findings
• Success Criteria Assessment
• Future Directions
• Reproducing the Results

---

Motivation

Information in radio signals is encoded in the frequency domain rather than as flat time-series values. Could LLM weights be treated similarly — stored and retrieved as frequency-domain representations rather than flat floating-point matrices? If weight matrices have strong low-frequency structure (most energy concentrated in few coefficients), frequency-domain compression would be both memory-efficient and information-preserving.

This experiment tests that premise on GPT-2 small as a tractable proxy.

---

Setup

• Model: GPT-2 small (gpt2, HuggingFace) — ~117M parameters
• Compression method: 2D FFT coefficient thresholding (keep top-K% by magnitude, zero the rest, reconstruct via inverse FFT)
• Evaluation: Perplexity on WikiText-2 test split (500 examples)
• Hardware: CPU
• Random seed: 42

Install

pip install torch transformers numpy scipy matplotlib datasets tqdm

Run

python main.py --phase all           # All three phases
python main.py --phase 1             # Spectral analysis only
python main.py --phase 2             # Compression validation only
python main.py --phase 3 --n 100    # Benchmarking (fast, 100 examples)

---

Experiment Structure

freq-weight-compression/
├── main.py                  # Entry point — CLI for all phases
├── phase1_spectrum.py       # Spectral analysis and visualisation
├── phase2_compress.py       # Compression, reconstruction, validation
├── phase3_benchmark.py      # Perplexity benchmarking
├── utils.py                 # Shared helpers (FFT, metrics, plotting)
├── results/
│   ├── metrics.json         # Phase 3 benchmarking results
│   ├── phase1_metrics.json  # Per-layer spectral energy data
│   ├── phase2_metrics.json  # Reconstruction error by compression level
│   └── figures/             # All generated charts (PNG)
└── frequency-weight-compression-spec.md

---

Phase 1 — Spectral Analysis

Goal: Determine whether GPT-2 weight matrices have a low-frequency structure that would make them amenable to FFT-based compression.

For each 2D weight matrix, the 2D FFT is computed, and a cumulative energy curve is built: what percentage of total spectral energy is captured by the top K% of FFT coefficients (sorted by magnitude).

Cumulative Spectral Energy by Layer

The key takeaway is in how steeply these curves rise. A steeply rising curve means energy is concentrated in few coefficients — easy to compress. A diagonal (linear) curve means energy is uniformly spread — incompressible.

Attention and MLP weights (all 12 transformer blocks) form a tight cluster of nearly identical gradual curves:

Layer type	Top 1% energy	Top 10% energy	Coefficients needed for 80%
Attention (c_attn, c_proj)	~5–6%	~33–34%	~44%
MLP (c_fc, c_proj)	~6–8%	~33–35%	~44%

Needing 44% of all coefficients to capture 80% of energy is characteristic of a near-uniform distribution — roughly what you would expect from random noise. There is no meaningful low-frequency concentration.

Positional embedding (wpe) is the striking outlier:

Layer	Top 1% energy	Top 10% energy	Coefficients needed for 80%
transformer.wpe.weight	93.9%	99.1%	<1%

This makes physical sense. Position embeddings encode smooth, periodic positional signals across a 1024-position sequence — the kind of signal that naturally lives in a low-frequency basis.

Token embedding (wte) falls between: top 10% captures only 50.9%, and the full 100% is needed to cross the 80% threshold. Vocabulary-space structure is not frequency-compressible.

Spectral Heatmaps

The heatmaps make the uniform distribution visually concrete. Both the attention weight (transformer.h.0.attn.c_attn.weight, 768×2304) and the MLP weight (transformer.h.0.mlp.c_fc.weight, 768×3072) show a flat, featureless power spectrum across the full frequency range. There is no low-frequency bright region — energy is distributed everywhere equally. This is the signature of a high-entropy, noise-like matrix in the frequency domain.

---

Phase 2 — Compression & Reconstruction

Goal: Validate the compress/reconstruct pipeline mathematically and measure reconstruction fidelity before running perplexity benchmarks.

The round-trip (compress at 100%, reconstruct) is verified to be lossless: relative Frobenius error of ~3.7×10⁻⁸ — consistent with floating-point rounding only.

Reconstruction Error vs. Compression Level

Relative Frobenius error (validated on transformer.wte.weight):

Keep Fraction	Compression Ratio	Relative Frobenius Error
1.00	1×	3.7×10⁻⁸ (lossless)
0.50	2×	33.4%
0.25	4×	54.2%
0.10	10×	69.8%
0.05	20×	76.3%
0.01	100×	82.9%

A 33% reconstruction error at just 2× compression confirms the spectral analysis: there are no expendable high-frequency components to discard cleanly. Every discarded coefficient carries significant information.

Both attention and MLP layer types show virtually identical reconstruction error curves (overlapping lines in the chart), further confirming they share the same flat spectral profile.

---

Phase 3 — Quality Benchmarking

Goal: Measure end-to-end model quality (perplexity on WikiText-2) after replacing all weight matrices with their frequency-compressed approximations.

Results

Keep Fraction	Params Stored	Compression Ratio	Perplexity	Degradation
1.00 (baseline)	124M	1×	117.42	0.00%
0.50	62M	2×	4.39×10¹⁰	+3.74×10¹⁰%
0.25	31M	4×	5.47×10¹⁴	+4.66×10¹⁴%
0.10	12M	10×	1.56×10¹⁰	+1.33×10¹⁰%
0.05	6M	20×	111,296	+94,685%
0.01	1.2M	100×	1,894,916	+1,613,711%

Perplexity vs. Compression Ratio

Degradation vs. Keep Fraction

Even at the mildest compression tested (2×, keeping 50% of coefficients), perplexity jumps from 117 to ~44 billion — a complete model collapse. There is no graceful degradation curve. The model cliff-drops to random output immediately because the per-layer reconstruction errors (~33%+) compound multiplicatively across 12 transformer blocks and ~148 weight matrices.

---

Findings

Research Question Answers

Q1 — Spectral structure: GPT-2 attention and MLP weight matrices have near-uniform spectral energy distributions. The top 10% of FFT coefficients capture only ~33–35% of energy. The approach is not viable for these layers. The positional embedding is a clear exception with >93% energy in the top 1% of coefficients, but it represents only 0.6% of total parameters.

Q2 — Compression viability: Not viable at any tested ratio. Perplexity degrades catastrophically at every compression level, including 2× (the minimum tested).

Q3 — Layer variation: Yes, there is significant variation, but not in the expected direction. Attention and MLP layers are equally non-compressible and show nearly identical spectral profiles. The positional embedding is uniquely compressible. There is no meaningful attention-vs-MLP distinction.

Q4 — Compression curve shape: Immediate cliff. There is no usable compression range before catastrophic degradation. The quality-vs-compression curve is vertical at the first compression point.

---

Success Criteria Assessment

Criterion	Target	Result	Outcome
Top 10% coefficients capture spectral energy	>85%	~34% (attention/MLP)	❌
Perplexity degradation at 4× compression	<10%	+4.66×10¹⁴%	❌
Perplexity degradation at 10× compression	<25%	+1.33×10¹⁰%	❌
Attention weights more compressible than MLP	Qualitative	No difference	❌

None of the success criteria are met. The raw weight matrix approach is not supported by these results.

---

Future Directions

Based on the spec's interpretation guide and the specific failure modes observed, the most promising next steps are:

1. FFT on weight updates (gradients/deltas) — Weight changes during training may have a much lower-frequency character than the final weights. Gradient signals and optimizer updates could be frequency-compressible even if final weights are not.

2. Learned rotation before FFT — Weights may not express low-frequency structure in the standard basis but could do so in a learned basis (similar to how LoRA works). Applying FFT after a learned orthogonal rotation could reveal latent frequency structure.

3. FFT on activations or attention patterns — Attention score matrices (output of QKᵀ/√d, before softmax) are produced over structured token sequences and may have more frequency regularity than weight matrices.

4. Hybrid approach — Compress only the positional embedding layer (which is highly compressible) via FFT while using quantisation or other methods for the remaining layers.

---

Reproducing the Results

Requirements: Python 3.10+, ~4 GB RAM for GPT-2, internet access to download the model and WikiText-2 on first run.

# Install dependencies
pip install torch transformers numpy scipy matplotlib datasets tqdm

# Run spectral analysis (fast, ~2–5 min on CPU)
python main.py --phase 1

# Run compression validation (~1–2 min)
python main.py --phase 2

# Run quality benchmarking (500 examples, ~20–40 min on CPU)
python main.py --phase 3

# Or run all phases in sequence
python main.py --phase all

All numeric results are written to results/metrics.json, results/phase1_metrics.json, and results/phase2_metrics.json. All figures are written to results/figures/.