Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

8 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 

Repository files navigation

KV Cache VRAM Calculator — Llama 3.2 1B Instruct

An implementation of the KV cache VRAM formula, and then verified empirically against meta-llama/Llama-3.2-1B-Instruct running on a Colab T4.

The Formula

KV Cache (bytes) = 2 × B × S × L × H_kv × D_head × bytes_per_element
  • B — batch size (number of requests in a batch)
  • S — sequence length (Number of tokens in a sequence)
  • L — num_hidden_layers
  • H_kv — num_key_value_heads (not num_attention_heads)
  • D_head — head_dim = hidden_size / num_attention_heads
  • × 2 — separately storing Key and Value
  • bytes_per_element — 2 for FP16

The GQA Trap

Configurations pulled directly from the model's config.json on HuggingFace:

Field Value
num_hidden_layers 16
hidden_size 2048
num_attention_heads 32
num_key_value_heads 8
head_dim (2048 / 32) 64

This model uses Grouped Query Attention: 32 query heads share just 8 KV groups. Plugging num_attention_heads (32) into the formula instead of num_key_value_heads (8) overestimates the cache by exactly the GQA ratio — 4x, for this model.

Verified Result

Loaded the model in FP16 on a Colab T4, ran a single prompt, generated greedily (do_sample=False) to 800 new tokens, then checked three independent numbers against each other:

Measurement Value
Hand calculation using raw output length (S=811 — wrong) 25.34 MB
Hand calculation using actual cache length (S=810 — correct) 25.31 MB
Ground truth — direct inspection of the cache tensors 25.31 MB
Peak GPU memory during generation (torch.cuda.max_memory_allocated) 48.10 MB

Using the correct sequence length, the formula matches the actual cache tensors exactly — not approximately. Both numbers come from the same arithmetic, just computed two different ways: one analytically from the formula, one read directly off the live tensor shapes. Using the raw output length instead measurably overshoots by ~0.03 MB — small at this scale, but it's the same fixed one-token error regardless of sequence length, so it doesn't shrink as a percentage at longer contexts.

The Off-By-One

Total sequence length (prompt + generated) was 811 tokens, but the cache only holds 810. The last generated token is the model's final output — it is never fed back into the model for another forward pass, so its own K/V is never computed or cached. Plugging in raw output length instead of actual cache length will overshoot the formula by exactly one token, every time.

Why Peak GPU Memory ≠ Cache Size

torch.cuda.max_memory_allocated() reported ~48 MB — nearly 2x the real 25.31 MB cache. The cache isn't bigger than the formula predicts; peak memory tracking also captures two other things:

  1. Transient duplication during cache growth. HuggingFace's default DynamicCache grows via torch.cat([old_cache, new_token_kv]), which briefly holds both the old and new tensor in memory before the old one is freed.
  2. Per-step activation memory — attention scores, MLP intermediates, logits — from the forward pass itself, which peak-tracking doesn't separate from the cache.

The only way to get the true cache size is to read the tensors directly:

total_bytes = 0
for layer in past_key_values.layers:
    total_bytes += layer.keys.nelement() * layer.keys.element_size()
    total_bytes += layer.values.nelement() * layer.values.element_size()

nvidia-smi and max_memory_allocated() alone will overstate KV cache cost, because they report peak memory across an operation rather than the steady-state size of the cache itself. Planning VRAM capacity off that inflated peak number will make less headroom look available than actually exists.

Files

  • KV_cache_calculation.py — the calculation formula
  • verify_on_GPU.ipynb — Colab notebook: model load, generation, peak GPU-memory measurement, and direct cache-tensor inspection, run end-to-end with output cells intact.

Reproducing

  1. Open verify_on_GPU.ipynb in Colab, set the runtime to a T4 GPU.
  2. Accept the Llama 3.2 license on HuggingFace and set an access token.
  3. Run all the cells in the notebook.
  4. Compare the printed direct-measurement number against KV_cache_calculation.py's output for the same sequence length — they should match to well within 5%.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages