Two questions, answered with real measurements on a small open model (Qwen2.5-0.5B, 24 layers, 2 KV heads, head_dim 64):
- How compressible are real KV tensors and which scheme gives the best compression ratio for the least quality loss? (Phase 1, done, CPU only)
- Can decoding be fast enough that compressed host-to-device offloading beats sending raw bytes over PCIe? (Phase 2: measured on an RTX 4090 over three sessions: fixed-width decode beats raw PCIe 2.4-3.1x; the first rANS kernel lost 4.5-6x, an optimised batched version reaches 1.2-1.7x; one GPU model only)
Every number below is produced by a script in this repo and written to results/; the tables in this README are generated by
scripts/readme_tables.py. Limitations are at the end - please read them before quoting any number.
Compression ratio (fp16 bytes / per-sequence compressed bytes, including scales and side info) vs perplexity increase on held-out PG19 text (8 sequences; a 2048-token prefix is compressed and decompressed, the next 512 tokens are scored against the lossy cache). Dashed line: Pareto frontier. Points on the bottom edge have loss below the ~0.3% measurement noise. The full 110-config screening sweep is in figures/fig_pareto_screen.png.
- Best operating points are cross-layer PCA on pre-RoPE keys + per-component quantization, with the first 4 (attention-sink) tokens kept raw: 15.5x at +0.56% perplexity with fixed-width codes, 19.6x at +0.71% and 44.0x at +1.50% with ANS entropy coding, 101x at +6.7% (tables below). The strongest simple baseline, KIVI-style int4 (K per-channel, V per-token), is 3.1x at +0.06%; at a similar loss budget (~0.6-0.75%) KIVI int3 reaches 3.8x versus ~20x for PCA.
- Decode-friendly fixed-width coding is almost as good as entropy coding: DP-allocated fixed bit widths per PCA component (bit-unpack + matmul on the GPU) land on essentially the same frontier as ECSQ+ANS up to ~20x (15.5x/+0.56% vs 16.9x/+0.56% with ANS on top; 20.5x/+0.75% vs 19.6x/+0.71%). ANS adds ~9% ratio on top of the fixed-width codes, and pulls ahead only beyond ~30x.
- Generic entropy coders do almost nothing on quantized KV: on KIVI int4, zstd -3 gives +2.9%, a gANS-like byte-wise static ANS +2.1%, per-channel-context ANS +4.0% ratio (real bytes). The gain comes from decorrelation (PCA) + per-component bit allocation, not from the entropy coder.
- What matters (all in
docs/negative_results.md, with numbers): cross-layer vs per-head/per-layer PCA (1.2% vs 14-19% loss at 30x); pre- vs post-RoPE keys (0.56% vs 1.5%); raw sinks (without them +93% to +2400%); DP bit allocation vs rank truncation; delta coding does not help. - Retrieval check (needle-in-a-haystack, 24 cases per config): fp16, KIVI int4 and the PCA configs up to 44x retrieve every needle; KIVI int2 misses 1/24 and the 101x config misses 3/24 (table below). This model is small and the test is easy, so it is weak evidence.
| stage | variants |
|---|---|
| quantizers | int8/4/3/2 and fp8 (e4m3/e5m2); per-tensor, per-token (+channel groups), per-channel (+token groups); KIVI; fp16 outlier channels |
| transforms | PCA per head / per layer / cross-layer (pre- or post-RoPE keys; truncation); closed-loop delta coding along tokens |
| bit allocation | DP allocation of integer widths per PCA component under a bit budget (KVTC-like); entropy-coded uniform quantization with one global step (ECSQ) |
| entropy coders | zstd (levels), byte-wise static-table ANS (gANS-like), constriction ANS with per-component tables fit on calibration data, chunked independent streams |
Results (8 held-out sequences, real entropy-coded bytes verified bit-exact; * = on the Pareto frontier)
Perplexity increase is exp(delta NLL) - 1 vs the fp16 cache on the same text, +- one standard error over the 8 sequences (paired). Loss differences below ~0.3% are not resolved.
Baselines and other transforms
| scheme | ratio (per sequence) | ratio incl. amortized bases/tables (100 seqs) | bits / element | perplexity increase | attn-output rel. err |
|---|---|---|---|---|---|
uni-K8t-V8t |
1.9x | 1.9x | 8.50 | +0.47% +- 0.37 | 0.04 |
fp8e4m3-token |
1.9x | 1.9x | 8.25 | +0.41% +- 0.16 | 0.09 |
uni-K4c32-V4t32-s4 |
3.1x | 3.1x | 5.17 | +0.06% +- 0.05 | 0.08 |
uni-K4c32-V4t32-byteans-s4 |
3.2x | 3.2x | 5.06 | +0.06% +- 0.05 | 0.08 |
uni-K4c32-V4t32-zstd3-s4 |
3.2x | 3.2x | 5.03 | +0.06% +- 0.05 | 0.08 |
uni-K4c32-V4t32-ans-channel-s4 |
3.2x | 3.2x | 4.97 | +0.06% +- 0.05 | 0.08 |
uni-K4t-V4t |
3.6x | 3.6x | 4.50 | +213.90% +- 22.33 | 0.35 |
uni-K3c32-V3t32-s4 |
3.8x | 3.8x | 4.19 | +0.64% +- 0.16 | 0.17 |
dpcm-K4V4-a1-s4 |
4.0x | 4.0x | 4.02 | +0.14% +- 0.19 | 0.14 |
dpcm-K4V4-a16-s4 |
4.0x | 4.0x | 4.02 | +0.41% +- 0.10 | 0.16 |
uni-K2c32-V2t32-s4 |
5.0x | 5.0x | 3.20 | +5.73% +- 0.59 | 0.44 |
dpcm-K3V3-a16-s4 |
5.3x | 5.3x | 3.03 | +3.08% +- 0.49 | 0.32 |
dpcm-K3V3-a1-s4 |
5.3x | 5.3x | 3.03 | +3.06% +- 0.30 | 0.29 |
PCA schemes (adapt = DP-allocated fixed widths, K/V average bits per element; ecsq = global-step entropy-coded; -pre = pre-RoPE keys, otherwise post-RoPE; -s4 = 4 raw sink tokens; head/layer/cross = PCA granularity)
| scheme | ratio (per sequence) | ratio incl. amortized bases/tables (100 seqs) | bits / element | perplexity increase | attn-output rel. err |
|---|---|---|---|---|---|
pca-cross-adapt-K3V3-pre-s4 |
5.3x | 4.9x | 3.03 | -0.04% +- 0.05 | 0.08 |
pca-cross-adapt-K2V2-pre-s4 |
7.9x | 7.2x | 2.03 | +0.03% +- 0.10 | 0.13 |
pca-cross-adapt-K1.5V1.5-pre-s4 * |
10.5x | 9.5x | 1.53 | +0.14% +- 0.14 | 0.17 |
pca-cross-adapt-K1V1-pre-s4 |
15.5x | 14.0x | 1.03 | +0.56% +- 0.15 | 0.23 |
pca-head-adapt-K1V1-pre-s4 |
15.5x | 15.5x | 1.03 | +9.75% +- 1.23 | 0.53 |
pca-layer-adapt-K1V1-pre-s4 |
15.5x | 15.5x | 1.03 | +6.56% +- 0.89 | 0.45 |
pca-cross-adapt-K1V1-s4 |
15.5x | 14.0x | 1.03 | +1.51% +- 0.36 | 0.28 |
pca-cross-adapt-K1V1-pre-ans-channel-s4 * |
16.9x | 15.1x | 0.95 | +0.56% +- 0.15 | 0.23 |
pca-cross-ecsq-K1V1-pre-ans-channel-s4 * |
19.6x | 15.9x | 0.82 | +0.71% +- 0.14 | 0.24 |
pca-cross-ecsq-K1V1-pre-ans-channel |
20.4x | 16.4x | 0.79 | +92.71% +- 10.44 | 0.47 |
pca-head-ecsq-K2V2-pre-ans-channel-s4 |
20.4x | 20.3x | 0.78 | +9.31% +- 0.67 | 0.52 |
pca-cross-adapt-K0.75V0.75-pre-s4 * |
20.5x | 18.5x | 0.78 | +0.75% +- 0.22 | 0.27 |
pca-cross-ecsq-K1V1-ans-channel-s4 |
21.3x | 16.8x | 0.75 | +2.17% +- 0.32 | 0.34 |
pca-cross-r512-trunc-K4V4-pre-s4 |
22.6x | 21.4x | 0.71 | +3.65% +- 0.58 | 0.38 |
pca-layer-ecsq-K2V2-pre-ans-channel-s4 |
23.9x | 23.6x | 0.67 | +7.28% +- 0.67 | 0.48 |
pca-cross-adapt-K0.5V0.5-pre-s4 |
30.2x | 27.2x | 0.53 | +1.23% +- 0.35 | 0.32 |
pca-cross-adapt-K0.5V0.5-s4 |
30.2x | 27.1x | 0.53 | +2.81% +- 0.54 | 0.39 |
pca-layer-adapt-K0.5V0.5-pre-s4 |
30.2x | 30.0x | 0.53 | +14.55% +- 1.51 | 0.61 |
pca-head-adapt-K0.5V0.5-pre-s4 |
30.2x | 30.1x | 0.53 | +19.03% +- 1.99 | 0.69 |
pca-cross-ecsq-K1.5V1.5-pre-ans-channel-s4 * |
30.9x | 23.8x | 0.52 | +0.96% +- 0.20 | 0.30 |
pca-cross-adapt-K0.5V0.5-pre-ans-channel-s4 * |
32.8x | 29.3x | 0.49 | +1.23% +- 0.35 | 0.32 |
pca-cross-r256-trunc-K4V4-pre-s4 |
42.7x | 40.6x | 0.37 | +4.56% +- 0.84 | 0.44 |
pca-cross-ecsq-K2V2-pre-ans-channel-s4 * |
44.0x | 32.7x | 0.36 | +1.50% +- 0.24 | 0.35 |
pca-cross-ecsq-K2V2-ans-channel-s4 |
50.2x | 36.5x | 0.32 | +4.68% +- 0.48 | 0.47 |
pca-cross-adapt-K0.25V0.25-pre-s4 |
57.0x | 51.6x | 0.28 | +3.18% +- 0.62 | 0.42 |
pca-cross-ecsq-K2.5V2.5-pre-ans-channel-s4 * |
58.1x | 42.3x | 0.28 | +2.29% +- 0.43 | 0.39 |
pca-cross-ecsq-K4V4-pre-ans-channel-s4 * |
101.3x | 74.0x | 0.16 | +6.73% +- 0.87 | 0.48 |
Failure modes (loss >= 100%)
| scheme | ratio (per sequence) | ratio incl. amortized bases/tables (100 seqs) | bits / element | perplexity increase | attn-output rel. err |
|---|---|---|---|---|---|
uni-K4t-V4t |
3.6x | 3.6x | 4.50 | +213.90% +- 22.33 | 0.35 |
pca-cross-adapt-K1V1-pre |
16.0x | 14.4x | 1.00 | +2418.79% +- 394.74 | 0.56 |
The PCA basis is shared by every sequence (one per model), so it is reported as amortized; charging it to only 100 sequences of 2048 tokens costs 10-25% of the ratio at the high-compression end. The "per-sequence" ratio assumes the basis is shared across very many sequences.
| config | exact match (teacher-forced) | mean log-prob of gold digits | ctx 1024 / 2048 |
|---|---|---|---|
fp16 |
24/24 | -0.03 | 1.00 / 1.00 |
uni-K4c32-V4t32-s4 |
24/24 | -0.03 | 1.00 / 1.00 |
uni-K2c32-V2t32-s4 |
23/24 | -0.40 | 1.00 / 0.92 |
pca-cross-adapt-K1V1-pre-s4 |
24/24 | -0.02 | 1.00 / 1.00 |
pca-cross-adapt-K0.5V0.5-pre-s4 |
24/24 | -0.03 | 1.00 / 1.00 |
pca-cross-ecsq-K2V2-pre-ans-channel-s4 |
24/24 | -0.04 | 1.00 / 1.00 |
pca-cross-ecsq-K4V4-pre-ans-channel-s4 |
21/24 | -0.68 | 0.75 / 1.00 |
The ECSQ(lambda=2)+rANS payload is cut into independent streams of (tokens x components); one GPU lane decodes one stream. The cost is the 8 B state + 4 B offset per stream (scripts/chunk_cost.py, real bytes of the GPU bitstream format, decode verified bit-exact):
| tokens x components per stream | streams / 2048-token sequence | payload overhead vs ideal entropy | ratio |
|---|---|---|---|
| 1 x 1024 | 8176 | 17.2% | 43.2x |
| 2 x 2048 | 2044 | 4.1% | 48.7x |
| 4 x 2048 | 1022 | 2.1% | 49.7x |
| 8 x 1024 | 1020 | 2.1% | 49.7x |
| 16 x 512 | 889 | 1.8% | 49.8x |
| 32 x 256 | 882 | 1.8% | 49.8x |
About 1-2k streams per 2048-token sequence cost 2-4% of ratio, and the stream count scales linearly with cache length.
Value distributions, outlier channels, token autocorrelation, covariance spectra and quantized-symbol entropies, pre- vs post-RoPE, per layer/head: docs/findings.md, numbers in docs/analysis_tables.md, figures in figures/.
Verdict. On an RTX 4090 the fixed-width path (bit-unpack + dequant, PCA GEMM, RoPE/layout) is the one that clearly beats sending raw bytes: 2.4-3.1x faster delivery of a 1.5 GB raw KV cache over a 24-26 GB/s link, limited by decode (52-77 GB/s), not by PCIe. The entropy-coded rANS paths have higher ratios (19x / 43x), but the first kernel lost to raw by 4.5-6x (latency-bound: one sequence gives only 1-2k lanes). After profiling-driven fixes (lookup-table symbol search, 8-32 sequences per launch, PCA GEMM tile order) they deliver 1.16-1.74x on a gen4 link (up to 2.21x on a slow 13.4 GB/s link): positive, but below fixed-width. Gains depend directly on how slow the raw link is. One GPU model, one model, 2k-token sequences.
Delivery of 64 sequences (24 MB raw each, 1.5 GB), double-buffered, speedup vs a raw pinned copy; ratio and quality are from Phase 1 (results/final.csv):
| path | ratio | ppl increase | decode GB/s (best) | session A: 26.3 GB/s link | session B: 13.4 GB/s link | session C: 24.1 GB/s gen4 link |
|---|---|---|---|---|---|---|
| fixed-dp1 | 15.5x | +0.56% | 52-56 | 2.40x | not run | 2.45x |
| fixed-dp0.5 | 30.1x | +1.2% | 77 | 3.06x | not run | not run |
| rans-ecsq1 | 19x | +0.71% | 3.6 -> 31 | 0.17x (original kernel) | 1.29x | 1.16x |
| rans-ecsq2 | 43x | +1.50% | 5.8 -> 44 | 0.22x (original kernel) | 2.21x | 1.74x |
Session A = results/gpu_NVIDIA_GeForce_RTX_4090.json (one sequence per launch, binary-search kernel). B = gpu_batched_*.json (batched, LUT kernel, 13.4 GB/s link, old GEMM order). C = gpu_batched2.json + gpu_linkcheck_*.json (batched, best of the LUT variants, n-fastest GEMM order, link recorded as gen4 x16). rANS entries use the best group size (8-32 sequences per launch); decode GB/s shows "original -> best". Sessions are different rented hosts, so columns are not a controlled comparison; full per-stage numbers, variance (~5% on the raw copy alone) and caveats are in docs/negative_results.md #16 and the JSON files.
What was and was not verified: all four blobs decode on the GPU to within fp16 matmul noise of the CPU reference (relative error 2.3e-4 to 3.6e-4, correctness in the JSON; kernels compiled on first try, so the suspected first-compile problems did not occur). The kernel test suite passed 15/15 on the GPU in the second session (results/gpu_tests2.log; that run predates the dynamic-tail kernel and the GEMM tile order); the logs of the first and third sessions' test runs were not returned, but the benchmark asserts bit-exactness of every LUT variant against the binary-search kernel on the real blobs, and that batched decode equals single decodes. Phase-1 quality for these operating points is unchanged (ratio / perplexity increase: fixed-dp1 15.5x / +0.56%, fixed-dp0.5 30.2x / +1.2%, ecsq1 19.6x / +0.71%, ecsq2 44.0x / +1.50%; the GPU ratios of the rANS sets are 19.2x / 42.5x because they include the stream-split overhead). Ratios here exclude the PCA bases, which are assumed resident on the GPU (17-30 MB per model, amortized; see the limitations).
Two decode paths for the chosen operating region (cross-layer PCA on pre-RoPE keys, 4 raw sink tokens):
| path | bitstream | decode kernel | why |
|---|---|---|---|
fixed-width (fixed-dp1, fixed-dp0.5) |
DP-allocated width per PCA component; component-major byte-aligned bit planes | unpack_dequant_kernel: every (token, component) is an independent 1-2 byte gather + dequant |
trivially parallel; ~equal ratio to entropy coding up to ~20x (Phase 1) |
entropy-coded (rans-ecsq1, rans-ecsq2) |
static per-component rANS (64-bit state, 32-bit renormalisation, 16-bit probabilities) cut into independent (2 tokens x <=2048 components) streams, one per GPU lane | rans_decode_kernel with fused dequantization |
more ratio at >= 30x; stream split costs ~4% (table above) |
Both then run pca_recon_kernel (fp16 matmul over only the active components + mean) and rope_layout_kernel (re-applies RoPE to pre-RoPE keys and writes the [layer, head, token, dim] cache layout).
Tested: tests/test_triton.py (kernels vs numpy reference, bit-exact unpack, end-to-end synthetic decode for both formats; run in a separate venv with triton-windows, skipped in the default env) and scripts/check_blob_kernels.py (real exported blobs vs the codec's CPU reconstruction, interpreter mode).
Files: kvc/gpu/{triton_kernels,format,decode}.py, scripts/make_precomputed.py (builds data/precomputed/<tag>, each bitstream self-checked with a numpy decoder), scripts/bench_gpu.py (pinned/pageable H2D bandwidth, tuned decode throughput with per-stage split, double-buffered raw-vs-compressed delivery of 1-64 sequences, break-even model min(PCIe x ratio, decode), optional nvCOMP ANS/Zstd/LZ4 on the same symbols), scripts/plot_gpu.py (regenerates figures and the table below from results/gpu_*.json), scripts/run_gpu_box.sh (one command on the box), docs/phase2_runbook.md (copy-paste steps, time/cost estimate, troubleshooting).
NVIDIA GeForce RTX 4090 (24 GB, PCIe gen2 x16; torch 2.13.0+cu129, triton 3.7.1): measured pinned H2D peak 26.3 GB/s.
| scheme (format) | ratio | decode GB/s (decompressed) | symbol decode K / V (ms) | PCA matmul K (ms) | raw H2D -> compressed + decode, largest cache | speedup |
|---|---|---|---|---|---|---|
fixed-dp1 (fixed) |
15.5x | 51.7 | 0.06 / 0.06 | 0.18 | 61 ms -> 26 ms (1533 MB raw) | 2.40x |
fixed-dp0.5 (fixed) |
30.1x | 76.8 | 0.06 / 0.06 | 0.11 | 61 ms -> 20 ms (1533 MB raw) | 3.06x |
rans-ecsq1 (rans) |
19.2x | 3.6 | 3.16 / 2.51 | 0.24 | 62 ms -> 367 ms (1533 MB raw) | 0.17x |
rans-ecsq2 (rans) |
42.5x | 5.8 | 1.74 / 2.47 | 0.17 | 61 ms -> 274 ms (1533 MB raw) | 0.22x |
fixed-dp1: delivery time vs cache size and break-even (effective = min(PCIe x ratio, decode)):
| sequences x tokens | raw KV (MB) | compressed (MB) | raw pinned H2D (ms) | compressed H2D + decode (ms) | speedup |
|---|---|---|---|---|---|
| 1 x 2044 | 24 | 1.5 | 1.1 | 0.6 | 1.86x |
| 4 x 2044 | 96 | 6.2 | 3.9 | 1.9 | 2.09x |
| 16 x 2044 | 383 | 24.7 | 15.4 | 7.0 | 2.21x |
| 64 x 2044 | 1533 | 98.9 | 61.3 | 25.6 | 2.40x |
| PCIe GB/s | effective GB/s | vs raw |
|---|---|---|
| 8 | 51.7 | 6.46x |
| 12 | 51.7 | 4.31x |
| 25 | 51.7 | 2.07x |
| 26.3 | 51.7 | 1.97x |
| 50 | 51.7 | 1.03x |
| 100 | 51.7 | 0.52x |
| nvCOMP on the same quantized symbols | ratio vs fp16 KV | decode GB/s (fp16-equivalent) |
|---|---|---|
| ANS | 7.3 | 302.1 |
| Zstd | 8.3 | 9.3 |
| LZ4 | 3.6 | 10.7 |
fixed-dp0.5: delivery time vs cache size and break-even (effective = min(PCIe x ratio, decode)):
| sequences x tokens | raw KV (MB) | compressed (MB) | raw pinned H2D (ms) | compressed H2D + decode (ms) | speedup |
|---|---|---|---|---|---|
| 1 x 2044 | 24 | 0.8 | 1.1 | 0.4 | 2.61x |
| 4 x 2044 | 96 | 3.2 | 3.9 | 1.4 | 2.76x |
| 16 x 2044 | 383 | 12.7 | 15.4 | 5.0 | 3.07x |
| 64 x 2044 | 1533 | 50.9 | 61.4 | 20.1 | 3.06x |
| PCIe GB/s | effective GB/s | vs raw |
|---|---|---|
| 8 | 76.8 | 9.60x |
| 12 | 76.8 | 6.40x |
| 25 | 76.8 | 3.07x |
| 26.3 | 76.8 | 2.92x |
| 50 | 76.8 | 1.54x |
| 100 | 76.8 | 0.77x |
| nvCOMP on the same quantized symbols | ratio vs fp16 KV | decode GB/s (fp16-equivalent) |
|---|---|---|
| ANS | 14.5 | 232.2 |
| Zstd | 16.1 | 9.4 |
| LZ4 | 7.0 | 10.7 |
rans-ecsq1: delivery time vs cache size and break-even (effective = min(PCIe x ratio, decode)):
| sequences x tokens | raw KV (MB) | compressed (MB) | raw pinned H2D (ms) | compressed H2D + decode (ms) | speedup |
|---|---|---|---|---|---|
| 1 x 2044 | 24 | 1.2 | 1.2 | 5.9 | 0.20x |
| 4 x 2044 | 96 | 5.0 | 3.9 | 23.1 | 0.17x |
| 16 x 2044 | 383 | 20.0 | 15.9 | 92.2 | 0.17x |
| 64 x 2044 | 1533 | 79.9 | 61.9 | 367.4 | 0.17x |
| PCIe GB/s | effective GB/s | vs raw |
|---|---|---|
| 8 | 3.6 | 0.44x |
| 12 | 3.6 | 0.30x |
| 25 | 3.6 | 0.14x |
| 26.3 | 3.6 | 0.14x |
| 50 | 3.6 | 0.07x |
| 100 | 3.6 | 0.04x |
| nvCOMP on the same quantized symbols | ratio vs fp16 KV | decode GB/s (fp16-equivalent) |
|---|---|---|
| ANS | 4.4 | 94.7 |
| Zstd | 8.7 | 13.2 |
| LZ4 | 3.5 | 14.7 |
rans-ecsq2: delivery time vs cache size and break-even (effective = min(PCIe x ratio, decode)):
| sequences x tokens | raw KV (MB) | compressed (MB) | raw pinned H2D (ms) | compressed H2D + decode (ms) | speedup |
|---|---|---|---|---|---|
| 1 x 2044 | 24 | 0.6 | 1.2 | 4.4 | 0.26x |
| 4 x 2044 | 96 | 2.3 | 3.9 | 17.2 | 0.23x |
| 16 x 2044 | 383 | 9.0 | 15.3 | 68.7 | 0.22x |
| 64 x 2044 | 1533 | 36.1 | 61.3 | 274.0 | 0.22x |
| PCIe GB/s | effective GB/s | vs raw |
|---|---|---|
| 8 | 5.8 | 0.72x |
| 12 | 5.8 | 0.48x |
| 25 | 5.8 | 0.23x |
| 26.3 | 5.8 | 0.22x |
| 50 | 5.8 | 0.12x |
| 100 | 5.8 | 0.06x |
| nvCOMP on the same quantized symbols | ratio vs fp16 KV | decode GB/s (fp16-equivalent) |
|---|---|---|
| ANS | 7.8 | 260.0 |
| Zstd | 19.4 | 19.6 |
| LZ4 | 6.8 | 17.1 |
uv sync # CPU torch, pinned in uv.lock / requirements.lock
make capture # K/V (pre- and post-RoPE) + real queries for 24 PG19 sequences (~8 min on 4 cores)
make analyze && make plots # statistics + figures
make sweep # configs/sweeps/main.yaml (screen). Also: ext.yaml, final.yaml (see below)
uv run python scripts/sweep.py --sweep configs/sweeps/final.yaml --out results/final --real
uv run python scripts/summarize.py && uv run python scripts/plot.py --only pareto && uv run python scripts/readme_tables.py
make testFree-tier notebooks: notebooks/01_capture_and_analysis.ipynb, notebooks/02_pareto.ipynb (thin: they call kvc/; use configs/capture_colab.yaml).
kvc/ package (capture, rope, analysis, quant/, transforms/, entropy/, pipeline, accounting, eval/, gpu/), scripts/ one CLI per stage, configs/ YAML (models, datasets, sweeps), tests/ (roundtrip, bit-exactness, bounded error, RoPE identity, kernels), docs/ (design.md, findings.md, negative_results.md, analysis_tables.md), results/ raw JSON/CSV, figures/.
Data protocol, size accounting and what was verified about KVTC: docs/design.md. One-page summary: WRITEUP.md.
- One small model (Qwen2.5-0.5B), 2 KV heads, head_dim 64; larger models, GQA with more heads, and other families (Llama) were not run. Cross-layer PCA gains may differ.
- One text domain (PG19), 24 sequences of 4096 tokens: 12 calibration, 4 validation (analysis only), 8 test; prefix N=2048, M=512 scored tokens. Per-config standard errors are in the tables; differences below ~0.3% are not resolved. Long-context behaviour (>2k prefix) was not evaluated, so effects that scale with length (raw recent window, amortized overheads, error accumulation) are untested.
- The needle test is small (24 cases per config, context <= 2048) and the model retrieves perfectly uncompressed; it only detects gross failures.
- The "KVTC-like" allocator is implemented from the paper's description with static per-component ranges; parts of the paper (the exact DP, whether group scales are static) could not be read, see
docs/design.md. No claim is made to reproduce the paper's numbers or its decode speed. - GPU measurements: one GPU model, three rented sessions (RTX 4090; pinned H2D peaks 26.3, 13.4 and 24.1 GB/s; only the third recorded the link, gen4 x16 - the first reported gen2 x16 when queried, probably idle link down-shifting - so cross-session comparisons are not controlled). No H100 / PCIe gen5 / NVLink-C2C numbers; claims about those are model extrapolations (
min(PCIe x ratio, decode)), not measurements. The raw baseline is a pinned-memory copy of random bytes; the compressed path copies the real streams, double-buffered, and decodes into fp16[layer, head, token, dim]on the same GPU. The first rANS kernel was slow because of low occupancy per sequence (profiled); the optimised variants are still hand-written Triton, with a dense PCA GEMM taking about half of the remaining decode time, and say nothing about what an nvCOMP-style GPU rANS could do. Decode competing with model compute on the same GPU was not measured. The nvCOMP rows decode generic byte streams of the unpacked int16 symbols (no dequant / PCA), so their ratios are not comparable to ours and their throughput is only a reference for symbol decoding. The "gANS-like" numbers in Phase 1 remain a CPU model of byte-wise static ANS, not nvCOMP. - The screening sweep uses an ideal-cross-entropy estimate for ANS sizes (about 1% optimistic vs real bytes); the headline table uses real bytes.
- Amortized bases are shared across sequences; per-sequence ratios exclude them (both are reported).



