feat: add paged-attention monolithic decode-only AR foundation

[Graffioh]

Summary

• Add an exact fixed-block paged K/V allocator (transactional allocation, stale-handle detection, immediate release).
• Add a CUDA/HIP GGML_OP_PAGED_ATTN decode op for D=256 GQA with F16, Q4_0, and Q8_0 K/V.
• Optimize decode: one warp computes all GQA heads of a K/V group per K/V row loaded (batched q8_1 dots); 32-token score tiles with a skip-if-unchanged accumulator rescale and exp2f softmax; Q quantized once per step by a pre-pass; partition partials stored normalized in f16 (half the scratch traffic); occupancy-aware context partitioning with a stable split-softmax merge.
• Keep paging metadata (block table + sequence length) resident in the persistent target cache next to the K/V pool: decode steps upload 4 bytes per fresh 16-token block instead of the whole live table every token.
• Integrate 16-token paged autoregressive decode for dense Qwen3.5/Qwen3.6 behind --paged-attention (one CUDA or HIP device).
• Fix first-token AR bookkeeping so the first sampled token is forwarded and written at its committed K/V and RoPE position.
• For now: keep exact chunked prefill; reject incompatible snapshots, speculative decoding, DDTree, KVFlash, layer split, and finite FA windows.
• Add allocator, nine-type numerical, block-boundary, uniform-kernel, and ragged-capacity tests.

Scope

Paged K/V is the memory-management foundation for heterogeneous 4–8-user serving. This PR establishes exact single-request Qwen integration and a multi-sequence GPU primitive first, following the block-table design in ggml-org/llama.cpp#21961.

HTTP integration stays one active sequence: Qwen3.6's Gated DeltaNet conv/SSM state is backend-global and the worker is run-to-completion. Sequence-indexed recurrent state, batched decode, and a continuous-batching scheduler are follow-up work. The primary benefit is memory capacity and request-lifecycle flexibility; the benchmarks below measure attention operations, not whole-model tokens/s.

Opt in with --paged-attention. Unsupported combinations fail explicitly. Requests reaching --max-ctx finish cleanly.

Benchmark results

Qwen attention dims (D=256, Hq=24, Hkv=4). Single metric: paged ÷ contiguous throughput — above 1.0x = paged faster.

Uniform batches (identical context per sequence, Q4_0 K/V, batches 1/2/4/8)

GPU / backend	4K	32K	128K
RTX 3090 / CUDA	1.11–1.19x	1.13–1.27x	1.27–2.02x
Radeon 8060S / HIP	4.28–5.60x	4.72–5.16x	5.00–6.42x

Both rows reflect the reworked kernel. Measured on the same GPUs, the pre-rework kernel gave 0.46–0.77x on CUDA and 2.06–2.92x on HIP, so the rework is worth roughly 1.5–2.6x on CUDA and 1.3–2.3x on HIP in absolute paged step time. The HIP ratios are inflated by a weak native HIP flash_attn_ext_vec baseline — don't extrapolate them to whole-model throughput.

Ragged 8-request workload (128K + 7 × 8K, Q4_0 K/V)

Standalone synthetic multi-sequence kernel benchmark (constructs the interleaved block table directly; does not use PagedKvPool or demonstrate concurrent requests — the server integration remains single-sequence). Paging stores only live, block-rounded tokens (188,416) instead of padding every sequence to 128K (1,048,576 slots).

GPU / backend	contiguous	paged	ratio
RTX 3090 / CUDA	3.139 ms	2.234 ms	1.40x
Radeon 8060S / HIP	11.665 ms	2.389 ms	4.88x

Small-context ragged batches (e.g. 4K + 7×256) remain behind contiguous: short sequences carry dead partitions sized by the longest sequence. The stream-K/persistent-kernel rework staged with continuous batching addresses this.

K/V capacity (projection over Qwen3.6's 16 full-attention layers from the one measured layer; excludes weights, activations, recurrent state):

layout	one-layer K/V	16-layer K/V
padded per-sequence contiguous	1152 MiB	18.000 GiB
paged	207 MiB	3.234 GiB

→ 5.57x less K/V (82.03% saving), now alongside faster attention on CUDA rather than at its expense.

Roadmap

1. This PR: decode-only ggml_paged_attn, block manager, CUDA/HIP integration, benchmarks.
2. Scheduler groundwork: sequence-indexed DeltaNet/conv state, iteration-level scheduling, dynamic decode batches.
3. Paged-aware chunk prefill: arbitrary block-table writes, multi-token queries over preceding paged K/V, causal attention within the chunk.
4. Continuous admission and interleaving: token budgets, decode prioritization, admission control, cancellation, prefill/decode interleaving; plus the stream-K decode kernel for ragged batches.
5. Variable-length batched prefill.
6. Prefix caching/CoW: shared prefix blocks with reference counting and copy-on-write.

Validation

• All nine F16/Q4_0/Q8_0 K/V combinations pass the numerical oracle on CUDA in partitioned and direct modes, including the narrower 3/1-heads-per-warp fallbacks and forced single-partition (max abs ≤ 1.2e-4 vs the 5e-4 bound; the increase over the pre-rework ~5e-6 is f16-partial rounding, below KV quantization error). Bench float64-oracle error ~5–8e-6, same magnitude as the contiguous kernel.
• Generation-level A/B on Qwen3.6-27B (RTX 3090, greedy, 8 prompts × 4 configs): paged vs dense diverges on 3/8 long open-ended prompts — the same rate and prompts as the two controls (dense vs DFLASH_QWEN35_NO_KVPAD=1 dense, and paged vs GGML_CUDA_PAGED_ATTN_FORCE_PARTITIONS=1 paged), so divergence is greedy argmax FP chaos, not a kernel defect. All checkable answers correct; paged output run-to-run deterministic; decode tok/s at parity.
• Persistent-metadata path exercised end-to-end: prefill → many block boundaries → pool reset across requests, including a 10K-token prompt.
• Correctness covers empty sequences, page boundaries, shuffled blocks, long-context partitioning, and corrupt block-table bounds guards.
• HIP gfx1151 (Radeon 8060S): all nine K/V combinations pass in partitioned and direct modes with identical error to CUDA, paged_kv_pool passes, and the full 4K/32K/128K plus ragged benchmark matrix completed. Note the native HIP contiguous path's own oracle error reaches 3.3e-3–4.5e-3 at long context (near the benchmark's 5e-3 gate) while paged stays at ~5e-6, so paged is also the numerically better path on this backend.
• git diff --check passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Vkkox5Yo66K2Rn1oPuVoX7

[cubic-dev-ai]

All reported issues were addressed across 28 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

the first emitted output token from prefill writes KV row only during the subsequent forward (that's right)

BUT here on the first decode step, we increment committed by 1, even though for cache we are still considering the first emitted token from prefill, leading to a gap of 1

was there a reason for this?

(in case this needs to be fixed also in other model's backend)

[Graffioh]

we want this as user options or we fix as 16? (vllm default)

[Graffioh]

for now kernel impl and optimizied for kv head dim 256 but must be generic somehow