An implementation of the KV cache VRAM formula, and then verified empirically against meta-llama/Llama-3.2-1B-Instruct running on a Colab T4.
KV Cache (bytes) = 2 × B × S × L × H_kv × D_head × bytes_per_element
B— batch size (number of requests in a batch)S— sequence length (Number of tokens in a sequence)L—num_hidden_layersH_kv—num_key_value_heads(notnum_attention_heads)D_head—head_dim=hidden_size / num_attention_heads× 2— separately storing Key and Valuebytes_per_element— 2 for FP16
Configurations pulled directly from the model's config.json on HuggingFace:
| Field | Value |
|---|---|
num_hidden_layers |
16 |
hidden_size |
2048 |
num_attention_heads |
32 |
num_key_value_heads |
8 |
head_dim (2048 / 32) |
64 |
This model uses Grouped Query Attention: 32 query heads share just 8 KV groups. Plugging
num_attention_heads (32) into the formula instead of num_key_value_heads (8)
overestimates the cache by exactly the GQA ratio — 4x, for this model.
Loaded the model in FP16 on a Colab T4, ran a single prompt, generated
greedily (do_sample=False) to 800 new tokens, then checked three
independent numbers against each other:
| Measurement | Value |
|---|---|
| Hand calculation using raw output length (S=811 — wrong) | 25.34 MB |
| Hand calculation using actual cache length (S=810 — correct) | 25.31 MB |
| Ground truth — direct inspection of the cache tensors | 25.31 MB |
Peak GPU memory during generation (torch.cuda.max_memory_allocated) |
48.10 MB |
Using the correct sequence length, the formula matches the actual cache tensors exactly — not approximately. Both numbers come from the same arithmetic, just computed two different ways: one analytically from the formula, one read directly off the live tensor shapes. Using the raw output length instead measurably overshoots by ~0.03 MB — small at this scale, but it's the same fixed one-token error regardless of sequence length, so it doesn't shrink as a percentage at longer contexts.
Total sequence length (prompt + generated) was 811 tokens, but the cache only holds 810. The last generated token is the model's final output — it is never fed back into the model for another forward pass, so its own K/V is never computed or cached. Plugging in raw output length instead of actual cache length will overshoot the formula by exactly one token, every time.
torch.cuda.max_memory_allocated() reported ~48 MB — nearly 2x the real
25.31 MB cache. The cache isn't bigger than the formula predicts; peak
memory tracking also captures two other things:
- Transient duplication during cache growth. HuggingFace's default
DynamicCachegrows viatorch.cat([old_cache, new_token_kv]), which briefly holds both the old and new tensor in memory before the old one is freed. - Per-step activation memory — attention scores, MLP intermediates, logits — from the forward pass itself, which peak-tracking doesn't separate from the cache.
The only way to get the true cache size is to read the tensors directly:
total_bytes = 0
for layer in past_key_values.layers:
total_bytes += layer.keys.nelement() * layer.keys.element_size()
total_bytes += layer.values.nelement() * layer.values.element_size()nvidia-smi and max_memory_allocated() alone will overstate KV cache cost, because they report peak memory across an operation rather than the steady-state size of the cache itself. Planning VRAM capacity off that inflated peak number will make less headroom look available than actually exists.
KV_cache_calculation.py— the calculation formulaverify_on_GPU.ipynb— Colab notebook: model load, generation, peak GPU-memory measurement, and direct cache-tensor inspection, run end-to-end with output cells intact.
- Open
verify_on_GPU.ipynbin Colab, set the runtime to a T4 GPU. - Accept the Llama 3.2 license on HuggingFace and set an access token.
- Run all the cells in the notebook.
- Compare the printed direct-measurement number against
KV_cache_calculation.py's output for the same sequence length — they should match to well within 5%.