Skip to content

About

KV-cache compression for host-to-device offloading

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

KV-cache compression for host-to-device offloading: a measured study

Two questions, answered with real measurements on a small open model (Qwen2.5-0.5B, 24 layers, 2 KV heads, head_dim 64):

  1. How compressible are real KV tensors and which scheme gives the best compression ratio for the least quality loss? (Phase 1, done, CPU only)
  2. Can decoding be fast enough that compressed host-to-device offloading beats sending raw bytes over PCIe? (Phase 2: measured on an RTX 4090 over three sessions: fixed-width decode beats raw PCIe 2.4-3.1x; the first rANS kernel lost 4.5-6x, an optimised batched version reaches 1.2-1.7x; one GPU model only)

Every number below is produced by a script in this repo and written to results/; the tables in this README are generated by scripts/readme_tables.py. Limitations are at the end - please read them before quoting any number.

Phase 1 headline

ratio vs quality

Compression ratio (fp16 bytes / per-sequence compressed bytes, including scales and side info) vs perplexity increase on held-out PG19 text (8 sequences; a 2048-token prefix is compressed and decompressed, the next 512 tokens are scored against the lossy cache). Dashed line: Pareto frontier. Points on the bottom edge have loss below the ~0.3% measurement noise. The full 110-config screening sweep is in figures/fig_pareto_screen.png.

  • Best operating points are cross-layer PCA on pre-RoPE keys + per-component quantization, with the first 4 (attention-sink) tokens kept raw: 15.5x at +0.56% perplexity with fixed-width codes, 19.6x at +0.71% and 44.0x at +1.50% with ANS entropy coding, 101x at +6.7% (tables below). The strongest simple baseline, KIVI-style int4 (K per-channel, V per-token), is 3.1x at +0.06%; at a similar loss budget (~0.6-0.75%) KIVI int3 reaches 3.8x versus ~20x for PCA.
  • Decode-friendly fixed-width coding is almost as good as entropy coding: DP-allocated fixed bit widths per PCA component (bit-unpack + matmul on the GPU) land on essentially the same frontier as ECSQ+ANS up to ~20x (15.5x/+0.56% vs 16.9x/+0.56% with ANS on top; 20.5x/+0.75% vs 19.6x/+0.71%). ANS adds ~9% ratio on top of the fixed-width codes, and pulls ahead only beyond ~30x.
  • Generic entropy coders do almost nothing on quantized KV: on KIVI int4, zstd -3 gives +2.9%, a gANS-like byte-wise static ANS +2.1%, per-channel-context ANS +4.0% ratio (real bytes). The gain comes from decorrelation (PCA) + per-component bit allocation, not from the entropy coder.
  • What matters (all in docs/negative_results.md, with numbers): cross-layer vs per-head/per-layer PCA (1.2% vs 14-19% loss at 30x); pre- vs post-RoPE keys (0.56% vs 1.5%); raw sinks (without them +93% to +2400%); DP bit allocation vs rank truncation; delta coding does not help.
  • Retrieval check (needle-in-a-haystack, 24 cases per config): fp16, KIVI int4 and the PCA configs up to 44x retrieve every needle; KIVI int2 misses 1/24 and the 101x config misses 3/24 (table below). This model is small and the test is easy, so it is weak evidence.

Compression schemes evaluated (all in kvc/, each a composable stage with exact byte accounting)

stage variants
quantizers int8/4/3/2 and fp8 (e4m3/e5m2); per-tensor, per-token (+channel groups), per-channel (+token groups); KIVI; fp16 outlier channels
transforms PCA per head / per layer / cross-layer (pre- or post-RoPE keys; truncation); closed-loop delta coding along tokens
bit allocation DP allocation of integer widths per PCA component under a bit budget (KVTC-like); entropy-coded uniform quantization with one global step (ECSQ)
entropy coders zstd (levels), byte-wise static-table ANS (gANS-like), constriction ANS with per-component tables fit on calibration data, chunked independent streams

Results (8 held-out sequences, real entropy-coded bytes verified bit-exact; * = on the Pareto frontier)

Perplexity increase is exp(delta NLL) - 1 vs the fp16 cache on the same text, +- one standard error over the 8 sequences (paired). Loss differences below ~0.3% are not resolved.

Baselines and other transforms

scheme ratio (per sequence) ratio incl. amortized bases/tables (100 seqs) bits / element perplexity increase attn-output rel. err
uni-K8t-V8t 1.9x 1.9x 8.50 +0.47% +- 0.37 0.04
fp8e4m3-token 1.9x 1.9x 8.25 +0.41% +- 0.16 0.09
uni-K4c32-V4t32-s4 3.1x 3.1x 5.17 +0.06% +- 0.05 0.08
uni-K4c32-V4t32-byteans-s4 3.2x 3.2x 5.06 +0.06% +- 0.05 0.08
uni-K4c32-V4t32-zstd3-s4 3.2x 3.2x 5.03 +0.06% +- 0.05 0.08
uni-K4c32-V4t32-ans-channel-s4 3.2x 3.2x 4.97 +0.06% +- 0.05 0.08
uni-K4t-V4t 3.6x 3.6x 4.50 +213.90% +- 22.33 0.35
uni-K3c32-V3t32-s4 3.8x 3.8x 4.19 +0.64% +- 0.16 0.17
dpcm-K4V4-a1-s4 4.0x 4.0x 4.02 +0.14% +- 0.19 0.14
dpcm-K4V4-a16-s4 4.0x 4.0x 4.02 +0.41% +- 0.10 0.16
uni-K2c32-V2t32-s4 5.0x 5.0x 3.20 +5.73% +- 0.59 0.44
dpcm-K3V3-a16-s4 5.3x 5.3x 3.03 +3.08% +- 0.49 0.32
dpcm-K3V3-a1-s4 5.3x 5.3x 3.03 +3.06% +- 0.30 0.29

PCA schemes (adapt = DP-allocated fixed widths, K/V average bits per element; ecsq = global-step entropy-coded; -pre = pre-RoPE keys, otherwise post-RoPE; -s4 = 4 raw sink tokens; head/layer/cross = PCA granularity)

scheme ratio (per sequence) ratio incl. amortized bases/tables (100 seqs) bits / element perplexity increase attn-output rel. err
pca-cross-adapt-K3V3-pre-s4 5.3x 4.9x 3.03 -0.04% +- 0.05 0.08
pca-cross-adapt-K2V2-pre-s4 7.9x 7.2x 2.03 +0.03% +- 0.10 0.13
pca-cross-adapt-K1.5V1.5-pre-s4 * 10.5x 9.5x 1.53 +0.14% +- 0.14 0.17
pca-cross-adapt-K1V1-pre-s4 15.5x 14.0x 1.03 +0.56% +- 0.15 0.23
pca-head-adapt-K1V1-pre-s4 15.5x 15.5x 1.03 +9.75% +- 1.23 0.53
pca-layer-adapt-K1V1-pre-s4 15.5x 15.5x 1.03 +6.56% +- 0.89 0.45
pca-cross-adapt-K1V1-s4 15.5x 14.0x 1.03 +1.51% +- 0.36 0.28
pca-cross-adapt-K1V1-pre-ans-channel-s4 * 16.9x 15.1x 0.95 +0.56% +- 0.15 0.23
pca-cross-ecsq-K1V1-pre-ans-channel-s4 * 19.6x 15.9x 0.82 +0.71% +- 0.14 0.24
pca-cross-ecsq-K1V1-pre-ans-channel 20.4x 16.4x 0.79 +92.71% +- 10.44 0.47
pca-head-ecsq-K2V2-pre-ans-channel-s4 20.4x 20.3x 0.78 +9.31% +- 0.67 0.52
pca-cross-adapt-K0.75V0.75-pre-s4 * 20.5x 18.5x 0.78 +0.75% +- 0.22 0.27
pca-cross-ecsq-K1V1-ans-channel-s4 21.3x 16.8x 0.75 +2.17% +- 0.32 0.34
pca-cross-r512-trunc-K4V4-pre-s4 22.6x 21.4x 0.71 +3.65% +- 0.58 0.38
pca-layer-ecsq-K2V2-pre-ans-channel-s4 23.9x 23.6x 0.67 +7.28% +- 0.67 0.48
pca-cross-adapt-K0.5V0.5-pre-s4 30.2x 27.2x 0.53 +1.23% +- 0.35 0.32
pca-cross-adapt-K0.5V0.5-s4 30.2x 27.1x 0.53 +2.81% +- 0.54 0.39
pca-layer-adapt-K0.5V0.5-pre-s4 30.2x 30.0x 0.53 +14.55% +- 1.51 0.61
pca-head-adapt-K0.5V0.5-pre-s4 30.2x 30.1x 0.53 +19.03% +- 1.99 0.69
pca-cross-ecsq-K1.5V1.5-pre-ans-channel-s4 * 30.9x 23.8x 0.52 +0.96% +- 0.20 0.30
pca-cross-adapt-K0.5V0.5-pre-ans-channel-s4 * 32.8x 29.3x 0.49 +1.23% +- 0.35 0.32
pca-cross-r256-trunc-K4V4-pre-s4 42.7x 40.6x 0.37 +4.56% +- 0.84 0.44
pca-cross-ecsq-K2V2-pre-ans-channel-s4 * 44.0x 32.7x 0.36 +1.50% +- 0.24 0.35
pca-cross-ecsq-K2V2-ans-channel-s4 50.2x 36.5x 0.32 +4.68% +- 0.48 0.47
pca-cross-adapt-K0.25V0.25-pre-s4 57.0x 51.6x 0.28 +3.18% +- 0.62 0.42
pca-cross-ecsq-K2.5V2.5-pre-ans-channel-s4 * 58.1x 42.3x 0.28 +2.29% +- 0.43 0.39
pca-cross-ecsq-K4V4-pre-ans-channel-s4 * 101.3x 74.0x 0.16 +6.73% +- 0.87 0.48

Failure modes (loss >= 100%)

scheme ratio (per sequence) ratio incl. amortized bases/tables (100 seqs) bits / element perplexity increase attn-output rel. err
uni-K4t-V4t 3.6x 3.6x 4.50 +213.90% +- 22.33 0.35
pca-cross-adapt-K1V1-pre 16.0x 14.4x 1.00 +2418.79% +- 394.74 0.56

Where the bits go

bits breakdown

The PCA basis is shared by every sequence (one per model), so it is reported as amortized; charging it to only 100 sequences of 2048 tokens costs 10-25% of the ratio at the high-compression end. The "per-sequence" ratio assumes the basis is shared across very many sequences.

Needle in a haystack (top configs; scripts/needle.py)

config exact match (teacher-forced) mean log-prob of gold digits ctx 1024 / 2048
fp16 24/24 -0.03 1.00 / 1.00
uni-K4c32-V4t32-s4 24/24 -0.03 1.00 / 1.00
uni-K2c32-V2t32-s4 23/24 -0.40 1.00 / 0.92
pca-cross-adapt-K1V1-pre-s4 24/24 -0.02 1.00 / 1.00
pca-cross-adapt-K0.5V0.5-pre-s4 24/24 -0.03 1.00 / 1.00
pca-cross-ecsq-K2V2-pre-ans-channel-s4 24/24 -0.04 1.00 / 1.00
pca-cross-ecsq-K4V4-pre-ans-channel-s4 21/24 -0.68 0.75 / 1.00

Making the entropy-coded stream parallel-decodable (CPU measurement)

The ECSQ(lambda=2)+rANS payload is cut into independent streams of (tokens x components); one GPU lane decodes one stream. The cost is the 8 B state + 4 B offset per stream (scripts/chunk_cost.py, real bytes of the GPU bitstream format, decode verified bit-exact):

tokens x components per stream streams / 2048-token sequence payload overhead vs ideal entropy ratio
1 x 1024 8176 17.2% 43.2x
2 x 2048 2044 4.1% 48.7x
4 x 2048 1022 2.1% 49.7x
8 x 1024 1020 2.1% 49.7x
16 x 512 889 1.8% 49.8x
32 x 256 882 1.8% 49.8x

About 1-2k streams per 2048-token sequence cost 2-4% of ratio, and the stream count scales linearly with cache length.

Analysis of the tensors

Value distributions, outlier channels, token autocorrelation, covariance spectra and quantized-symbol entropies, pre- vs post-RoPE, per layer/head: docs/findings.md, numbers in docs/analysis_tables.md, figures in figures/.

pca

Phase 2: GPU decode (RTX 4090, three sessions)

Verdict. On an RTX 4090 the fixed-width path (bit-unpack + dequant, PCA GEMM, RoPE/layout) is the one that clearly beats sending raw bytes: 2.4-3.1x faster delivery of a 1.5 GB raw KV cache over a 24-26 GB/s link, limited by decode (52-77 GB/s), not by PCIe. The entropy-coded rANS paths have higher ratios (19x / 43x), but the first kernel lost to raw by 4.5-6x (latency-bound: one sequence gives only 1-2k lanes). After profiling-driven fixes (lookup-table symbol search, 8-32 sequences per launch, PCA GEMM tile order) they deliver 1.16-1.74x on a gen4 link (up to 2.21x on a slow 13.4 GB/s link): positive, but below fixed-width. Gains depend directly on how slow the raw link is. One GPU model, one model, 2k-token sequences.

Delivery of 64 sequences (24 MB raw each, 1.5 GB), double-buffered, speedup vs a raw pinned copy; ratio and quality are from Phase 1 (results/final.csv):

path ratio ppl increase decode GB/s (best) session A: 26.3 GB/s link session B: 13.4 GB/s link session C: 24.1 GB/s gen4 link
fixed-dp1 15.5x +0.56% 52-56 2.40x not run 2.45x
fixed-dp0.5 30.1x +1.2% 77 3.06x not run not run
rans-ecsq1 19x +0.71% 3.6 -> 31 0.17x (original kernel) 1.29x 1.16x
rans-ecsq2 43x +1.50% 5.8 -> 44 0.22x (original kernel) 2.21x 1.74x

Session A = results/gpu_NVIDIA_GeForce_RTX_4090.json (one sequence per launch, binary-search kernel). B = gpu_batched_*.json (batched, LUT kernel, 13.4 GB/s link, old GEMM order). C = gpu_batched2.json + gpu_linkcheck_*.json (batched, best of the LUT variants, n-fastest GEMM order, link recorded as gen4 x16). rANS entries use the best group size (8-32 sequences per launch); decode GB/s shows "original -> best". Sessions are different rented hosts, so columns are not a controlled comparison; full per-stage numbers, variance (~5% on the raw copy alone) and caveats are in docs/negative_results.md #16 and the JSON files.

What was and was not verified: all four blobs decode on the GPU to within fp16 matmul noise of the CPU reference (relative error 2.3e-4 to 3.6e-4, correctness in the JSON; kernels compiled on first try, so the suspected first-compile problems did not occur). The kernel test suite passed 15/15 on the GPU in the second session (results/gpu_tests2.log; that run predates the dynamic-tail kernel and the GEMM tile order); the logs of the first and third sessions' test runs were not returned, but the benchmark asserts bit-exactness of every LUT variant against the binary-search kernel on the real blobs, and that batched decode equals single decodes. Phase-1 quality for these operating points is unchanged (ratio / perplexity increase: fixed-dp1 15.5x / +0.56%, fixed-dp0.5 30.2x / +1.2%, ecsq1 19.6x / +0.71%, ecsq2 44.0x / +1.50%; the GPU ratios of the rANS sets are 19.2x / 42.5x because they include the stream-split overhead). Ratios here exclude the PCA bases, which are assumed resident on the GPU (17-30 MB per model, amortized; see the limitations).

Two decode paths for the chosen operating region (cross-layer PCA on pre-RoPE keys, 4 raw sink tokens):

path bitstream decode kernel why
fixed-width (fixed-dp1, fixed-dp0.5) DP-allocated width per PCA component; component-major byte-aligned bit planes unpack_dequant_kernel: every (token, component) is an independent 1-2 byte gather + dequant trivially parallel; ~equal ratio to entropy coding up to ~20x (Phase 1)
entropy-coded (rans-ecsq1, rans-ecsq2) static per-component rANS (64-bit state, 32-bit renormalisation, 16-bit probabilities) cut into independent (2 tokens x <=2048 components) streams, one per GPU lane rans_decode_kernel with fused dequantization more ratio at >= 30x; stream split costs ~4% (table above)

Both then run pca_recon_kernel (fp16 matmul over only the active components + mean) and rope_layout_kernel (re-applies RoPE to pre-RoPE keys and writes the [layer, head, token, dim] cache layout). Tested: tests/test_triton.py (kernels vs numpy reference, bit-exact unpack, end-to-end synthetic decode for both formats; run in a separate venv with triton-windows, skipped in the default env) and scripts/check_blob_kernels.py (real exported blobs vs the codec's CPU reconstruction, interpreter mode).

Files: kvc/gpu/{triton_kernels,format,decode}.py, scripts/make_precomputed.py (builds data/precomputed/<tag>, each bitstream self-checked with a numpy decoder), scripts/bench_gpu.py (pinned/pageable H2D bandwidth, tuned decode throughput with per-stage split, double-buffered raw-vs-compressed delivery of 1-64 sequences, break-even model min(PCIe x ratio, decode), optional nvCOMP ANS/Zstd/LZ4 on the same symbols), scripts/plot_gpu.py (regenerates figures and the table below from results/gpu_*.json), scripts/run_gpu_box.sh (one command on the box), docs/phase2_runbook.md (copy-paste steps, time/cost estimate, troubleshooting).

gpu

NVIDIA GeForce RTX 4090 (24 GB, PCIe gen2 x16; torch 2.13.0+cu129, triton 3.7.1): measured pinned H2D peak 26.3 GB/s.

scheme (format) ratio decode GB/s (decompressed) symbol decode K / V (ms) PCA matmul K (ms) raw H2D -> compressed + decode, largest cache speedup
fixed-dp1 (fixed) 15.5x 51.7 0.06 / 0.06 0.18 61 ms -> 26 ms (1533 MB raw) 2.40x
fixed-dp0.5 (fixed) 30.1x 76.8 0.06 / 0.06 0.11 61 ms -> 20 ms (1533 MB raw) 3.06x
rans-ecsq1 (rans) 19.2x 3.6 3.16 / 2.51 0.24 62 ms -> 367 ms (1533 MB raw) 0.17x
rans-ecsq2 (rans) 42.5x 5.8 1.74 / 2.47 0.17 61 ms -> 274 ms (1533 MB raw) 0.22x

fixed-dp1: delivery time vs cache size and break-even (effective = min(PCIe x ratio, decode)):

sequences x tokens raw KV (MB) compressed (MB) raw pinned H2D (ms) compressed H2D + decode (ms) speedup
1 x 2044 24 1.5 1.1 0.6 1.86x
4 x 2044 96 6.2 3.9 1.9 2.09x
16 x 2044 383 24.7 15.4 7.0 2.21x
64 x 2044 1533 98.9 61.3 25.6 2.40x
PCIe GB/s effective GB/s vs raw
8 51.7 6.46x
12 51.7 4.31x
25 51.7 2.07x
26.3 51.7 1.97x
50 51.7 1.03x
100 51.7 0.52x
nvCOMP on the same quantized symbols ratio vs fp16 KV decode GB/s (fp16-equivalent)
ANS 7.3 302.1
Zstd 8.3 9.3
LZ4 3.6 10.7

fixed-dp0.5: delivery time vs cache size and break-even (effective = min(PCIe x ratio, decode)):

sequences x tokens raw KV (MB) compressed (MB) raw pinned H2D (ms) compressed H2D + decode (ms) speedup
1 x 2044 24 0.8 1.1 0.4 2.61x
4 x 2044 96 3.2 3.9 1.4 2.76x
16 x 2044 383 12.7 15.4 5.0 3.07x
64 x 2044 1533 50.9 61.4 20.1 3.06x
PCIe GB/s effective GB/s vs raw
8 76.8 9.60x
12 76.8 6.40x
25 76.8 3.07x
26.3 76.8 2.92x
50 76.8 1.54x
100 76.8 0.77x
nvCOMP on the same quantized symbols ratio vs fp16 KV decode GB/s (fp16-equivalent)
ANS 14.5 232.2
Zstd 16.1 9.4
LZ4 7.0 10.7

rans-ecsq1: delivery time vs cache size and break-even (effective = min(PCIe x ratio, decode)):

sequences x tokens raw KV (MB) compressed (MB) raw pinned H2D (ms) compressed H2D + decode (ms) speedup
1 x 2044 24 1.2 1.2 5.9 0.20x
4 x 2044 96 5.0 3.9 23.1 0.17x
16 x 2044 383 20.0 15.9 92.2 0.17x
64 x 2044 1533 79.9 61.9 367.4 0.17x
PCIe GB/s effective GB/s vs raw
8 3.6 0.44x
12 3.6 0.30x
25 3.6 0.14x
26.3 3.6 0.14x
50 3.6 0.07x
100 3.6 0.04x
nvCOMP on the same quantized symbols ratio vs fp16 KV decode GB/s (fp16-equivalent)
ANS 4.4 94.7
Zstd 8.7 13.2
LZ4 3.5 14.7

rans-ecsq2: delivery time vs cache size and break-even (effective = min(PCIe x ratio, decode)):

sequences x tokens raw KV (MB) compressed (MB) raw pinned H2D (ms) compressed H2D + decode (ms) speedup
1 x 2044 24 0.6 1.2 4.4 0.26x
4 x 2044 96 2.3 3.9 17.2 0.23x
16 x 2044 383 9.0 15.3 68.7 0.22x
64 x 2044 1533 36.1 61.3 274.0 0.22x
PCIe GB/s effective GB/s vs raw
8 5.8 0.72x
12 5.8 0.48x
25 5.8 0.23x
26.3 5.8 0.22x
50 5.8 0.12x
100 5.8 0.06x
nvCOMP on the same quantized symbols ratio vs fp16 KV decode GB/s (fp16-equivalent)
ANS 7.8 260.0
Zstd 19.4 19.6
LZ4 6.8 17.1

Quickstart

uv sync                                   # CPU torch, pinned in uv.lock / requirements.lock
make capture                              # K/V (pre- and post-RoPE) + real queries for 24 PG19 sequences (~8 min on 4 cores)
make analyze && make plots                # statistics + figures
make sweep                                # configs/sweeps/main.yaml (screen). Also: ext.yaml, final.yaml (see below)
uv run python scripts/sweep.py --sweep configs/sweeps/final.yaml --out results/final --real
uv run python scripts/summarize.py && uv run python scripts/plot.py --only pareto && uv run python scripts/readme_tables.py
make test

Free-tier notebooks: notebooks/01_capture_and_analysis.ipynb, notebooks/02_pareto.ipynb (thin: they call kvc/; use configs/capture_colab.yaml).

Repo layout

kvc/ package (capture, rope, analysis, quant/, transforms/, entropy/, pipeline, accounting, eval/, gpu/), scripts/ one CLI per stage, configs/ YAML (models, datasets, sweeps), tests/ (roundtrip, bit-exactness, bounded error, RoPE identity, kernels), docs/ (design.md, findings.md, negative_results.md, analysis_tables.md), results/ raw JSON/CSV, figures/. Data protocol, size accounting and what was verified about KVTC: docs/design.md. One-page summary: WRITEUP.md.

Limitations (please read)

  • One small model (Qwen2.5-0.5B), 2 KV heads, head_dim 64; larger models, GQA with more heads, and other families (Llama) were not run. Cross-layer PCA gains may differ.
  • One text domain (PG19), 24 sequences of 4096 tokens: 12 calibration, 4 validation (analysis only), 8 test; prefix N=2048, M=512 scored tokens. Per-config standard errors are in the tables; differences below ~0.3% are not resolved. Long-context behaviour (>2k prefix) was not evaluated, so effects that scale with length (raw recent window, amortized overheads, error accumulation) are untested.
  • The needle test is small (24 cases per config, context <= 2048) and the model retrieves perfectly uncompressed; it only detects gross failures.
  • The "KVTC-like" allocator is implemented from the paper's description with static per-component ranges; parts of the paper (the exact DP, whether group scales are static) could not be read, see docs/design.md. No claim is made to reproduce the paper's numbers or its decode speed.
  • GPU measurements: one GPU model, three rented sessions (RTX 4090; pinned H2D peaks 26.3, 13.4 and 24.1 GB/s; only the third recorded the link, gen4 x16 - the first reported gen2 x16 when queried, probably idle link down-shifting - so cross-session comparisons are not controlled). No H100 / PCIe gen5 / NVLink-C2C numbers; claims about those are model extrapolations (min(PCIe x ratio, decode)), not measurements. The raw baseline is a pinned-memory copy of random bytes; the compressed path copies the real streams, double-buffered, and decodes into fp16 [layer, head, token, dim] on the same GPU. The first rANS kernel was slow because of low occupancy per sequence (profiled); the optimised variants are still hand-written Triton, with a dense PCA GEMM taking about half of the remaining decode time, and say nothing about what an nvCOMP-style GPU rANS could do. Decode competing with model compute on the same GPU was not measured. The nvCOMP rows decode generic byte streams of the unpacked int16 symbols (no dequant / PCA), so their ratios are not comparable to ours and their throughput is only a reference for symbol decoding. The "gANS-like" numbers in Phase 1 remain a CPU model of byte-wise static ANS, not nvCOMP.
  • The screening sweep uses an ideal-cross-entropy estimate for ANS sizes (about 1% optimistic vs real bytes); the headline table uses real bytes.
  • Amortized bases are shared across sequences; per-sequence ratios exclude them (both are reported).

About

KV-cache compression for host-to-device offloading

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages