Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

WAKMan — Workload Aware KV Cache Manager

Prefix caching enables Large Language Model (LLM) servers to reuse the Key-Value (KV) states and accelerate prefill computation for requests that share a prefix with earlier prompts, thus reducing latency and serving cost. With limited memory, the KV cache eviction policy becomes a decisive factor in determining the overall benefits of prefix caching. Production traffic may have single-turn, multi-turn, agentic, or any mixture of these request types. Our findings indicate that agentic steps return within seconds and favor recency, while popular prompts and long conversations return after many other requests and favor frequency. Across 20 mixes, no single policy is always best: vLLM's default LRU provides 12% higher cache hits than the frequency-oriented S3FIFO on agentic-heavy mixes but trails it by up to 22% elsewhere, and even a single workload parameter can flip the winner for a fixed mix. We present WAKMan, a workload-aware KV cache manager that picks its eviction policy from live traffic without knowing request types or reading prompts. It improves hit rate over LRU by 5 percentage points on average, with the lowest mean and worst-case optimality gap of all policies.

📄 Arxiv Preprint: WAKMan: Workload Aware KVCache Manager

Method

WAKMan overview

Every T = 15 min, the estimator replays the last window of block hashes (already computed by vLLM) through shadow LRU and S3FIFO policies, and the server switches to whichever had the higher hit rate. No request types, prompt content, or offline training are needed.

Results

Prefix-cache hit rate (%) on the 20 workload mixes (S/M/A = % single-turn / multi-turn / agentic). Bold = best policy per row; Ideal = infinite cache.

Mix S/M/ALRUS3FIFOWAKManIdeal
100/0/037.849.546.550.1
70/0/3055.360.057.470.0
50/0/5061.359.561.376.1
30/0/7064.456.164.380.3
0/0/10069.357.669.284.7
70/10/2049.855.654.066.1
20/10/7062.453.462.380.3
70/20/1043.551.749.462.3
10/20/7060.452.160.480.9
70/30/031.545.442.456.9
30/30/4050.950.451.174.3
30/40/3042.847.342.873.2
0/30/7059.151.059.181.4
50/50/023.441.438.660.4
0/50/5049.546.849.278.6
30/70/014.236.634.163.1
20/70/1022.638.336.167.6
10/70/2027.439.937.971.7
0/70/3034.340.938.375.2
0/100/08.426.015.067.1
Mean regret7.092.532.04
Max regret22.411.711.0
S3FIFO minus LRU hit rate across mixes
Hit-rate difference (S3FIFO − LRU) across all mixes.
Purple: S3FIFO higher; orange: LRU higher.
  • No static policy wins everywhere: LRU is best on 8 agentic-heavy mixes; S3FIFO on 12 single-/multi-turn-heavy mixes.
  • Each static policy has a large worst case: LRU loses up to 22.4 points, S3FIFO up to 11.7.
  • WAKMan is the most robust: lowest mean and max regret, recovering a median 84% of the LRU–S3FIFO gap. Its worst case (0/100/0) comes from starting on LRU for the first 15 min of a 1-hour trace.

Ablations

Ablations within a fixed mix

Even with the request mix fixed, a single workload parameter (turns per conversation, input length) flips the better static policy by up to ~15 points. WAKMan uses neither the mix nor these parameters, yet tracks LRU where LRU wins and moves toward S3FIFO where S3FIFO wins.

Repository layout

  • analysis/ — paper_figures.ipynb regenerates the paper's result plots from experiments/; figures are saved in analysis/figures/.
  • configs/ — the golden workload configs.
  • data/ — generated workload traces + the generation how-to.
  • experiments/ — bare-minimum run artifacts, organized by algorithm (LRU/, S3FIFO/, …).
  • runner/ — the launch folder: run perf tests via run_experiments.sh (with JSON batch specs in experiment_groups/ and base_perf_test_configs/).
  • vllm/ - custom vLLM eviction policies implementation.

Authors

  • Suhas Kumar Morisetty - Columbia University
  • Vaibhavi Lokegaonkar - University of Maryland, College Park
  • Sachin Mathew Varghese - Capital One
  • Wesley Harris - Capital One

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages