Skip to content

PR1: per-layer KV storage (mixed-precision layer tables) - #1

Open
Astrangemaninhere wants to merge 8 commits into
masterfrom
pr1/per-layer-kv
Open

PR1: per-layer KV storage (mixed-precision layer tables)#1
Astrangemaninhere wants to merge 8 commits into
masterfrom
pr1/per-layer-kv

Conversation

@Astrangemaninhere

@Astrangemaninhere Astrangemaninhere commented Aug 31, 2026

Copy link
Copy Markdown
Owner

PR1: per-layer KV storage (mixed-precision layer tables)

Background

The KV cache previously used a single global storage format for every layer. Attention layers differ sharply in how much quantization error they tolerate, so a uniform format either wastes memory on robust layers (all-INT8) or degrades quality on sensitive ones (all-NVFP4). This PR makes the storage format a per-layer decision.

Changes

  • Per-layer dtype table: each full-attention layer selects BF16 / INT8 / FP8 independently via --kv-layer-storage or the variant default table.
  • Layer plane-base prefix sums: mixed-format layers have unequal plane counts and strides; the page geometry now tracks each layer's plane offset.
  • Scale-plane resolution: FP8 per-layer scale planes are FP16 (not the global format), and per-layer quant_group is resolved into layer views.
  • Table propagation: layer_kv_dtypes is copied into the sequence candidate so the table actually reaches the cache (previously never applied).

Files

  • src/targets/qwen3_6/impl/state/decoder_state.cpp (plane geometry, per-layer views)
  • src/targets/qwen3_6/impl/runtime/layouts_impl.h (candidate propagation)
  • src/core/paged_kv_cache.h (layer views carry dtype/group/plane base)

Dependency

Foundation for PR2 (cold pool), PR5 (YaRN), and PR6 (NVFP4 tier). Merge first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant