Skip to content

kv: replace eligible dense causal masks with compact prefixes - #7

Open
GenerelSchwerz wants to merge 23 commits into
beellama/mainfrom
exp/compact-causal-mask-pre-pr4
Open

kv: replace eligible dense causal masks with compact prefixes#7
GenerelSchwerz wants to merge 23 commits into
beellama/mainfrom
exp/compact-causal-mask-pre-pr4

Conversation

@GenerelSchwerz

@GenerelSchwerz GenerelSchwerz commented Aug 21, 2026

Copy link
Copy Markdown
Owner

Status

Official isolated head: ea066a68ad6eb6db701251d5ae23dfea3948ee26
Measured runtime head: 222eec2e7c91296b5f9335389dcff0cfc8a08129
Exact base: 4b86269fdf001de44dd96e9c9ae26a9e25091cca (beellama/main)

This PR is ready for review after a normal merge of current beellama/main.
It remains unmerged. The evidence-only commit after the runtime head changes no
binary source.

Scope

Eligible single-stream contiguous standard-KV graphs pass the existing I64
write-index tensor directly to FlashAttention as an exclusive causal-prefix
descriptor. Unsupported layouts fail closed to the existing dense mask.

  • no public flag, user CMake option, model/architecture special case, context
    threshold, or prompt/decode policy;
  • no native-Q8, KVarN implementation, workspace, allocator-policy, VMM-policy,
    host-staging, speculation, or development-branch feature;
  • CPU and CUDA descriptor consumers, with CUDA capability registration;
  • compile-time <..., false> dense and <..., true> compact vector
    specializations retained across the existing cache-format matrix;
  • explicit block-owned shared-memory lifetime for the compact MMA first bound;
    and
  • one optional graph value owns representation presence and retained K span,
    removing an invalid boolean/count state without hot-path work.

The main merge is c542f137c; the isolated lifetime repair is 1a3e6fcf3.
Focused compiled-capability coverage replaces compact-specific source-text
assertions. The retained two-turn runner now takes model and artifact paths as
arguments rather than embedding a machine model path.

Correctness and quality

  • Fresh Release CUDA/FA SM120a builds, default quant matrix, KVarN off, maximum
    12 build jobs under /tmp/beellama-cuda-build.lock.
  • Compiled CUDA dense-versus-compact oracle: 6/6, including D=256 F16 MMA
    with no GQA and GQA 6.
  • Focused graph, allocator, vector-generation, route, fallback, parser, and
    replay CTests: 10/10.
  • Main/candidate/main PPL with -b 512 -ub 256 --chunks 4: identical chunks
    1.9315, 2.1279, 2.2498, 2.1674 and final
    2.1674 +/- 0.03849; measured increase zero.

Every GPU lifecycle used the complete
flock /tmp/beellama-single-gpu.lock -c 'COMMAND' form, fresh processes, and
native progress. The one 128K placement setup failure is explicitly invalid
and contributes no result.

Matched runtime and resources

Qwen3.8 27B, q8_0 K/V, batch/ubatch 512/256, RTX 5070 Ti, CUDA 13.3, driver
610.57.04:

gate compact effect
4K GPU-KV prefill -0.0736% time-adjusted; neutral
4K GPU-KV decode +0.0482%
30K GPU-KV prefill / decode +0.7765% / +0.4059%
128K CPU-pinned-KV prefill / decode +1.2745% / +0.4923%

The user-requested closing A4 anchor makes 4K prefill chronological
A B A B A B A. Baseline falls monotonically from 1810.6711 to 1798.3230
tok/s. A linear elapsed-time/source model estimates -1.3282 tok/s (-0.0736%)
with 1.7246 tok/s standard error; local interpolated effects are -0.1665%,
-0.0330%, and -0.2066%. This is neutral within observed time/process variation,
not a speedup and not a detected regression. The 128K decode series retains all
four processes/source; its +0.4923% aggregate has wide process variance and is
not promoted to a strict speedup claim.

Independent 100 ms resource sampling measured process-VRAM deltas of
-2 / 0 / -58 MiB at 4K/30K/128K prefill. At 128K, total compute backing
falls 123.5 MiB, while pinned KV context, resident KV, ordinary-host buffers,
and CUDA VMM high-water are unchanged. Allocation-class telemetry and the
process sampler are separate from timing.

Evidence

The durable design, commands, identities, limitations, and results are in
docs/compact-causal-mask-vram-investigation.md. The 135-file local artifact
manifest has SHA-256
d4246230238f39cf643a2a2a1037835cc4a20dd490844abc1dc976d1180206a1.

The final main-relative audit is 27 files and contains no PR 4, workspace,
allocator-policy, VMM-policy, server, or development-only source. This PR must
not be merged by the readiness maintainer; normal review owns landing.

@github-actions github-actions Bot added documentation Improvements or additions to documentation testing CUDA ggml labels Aug 21, 2026
@GenerelSchwerz
GenerelSchwerz force-pushed the exp/compact-causal-mask-pre-pr4 branch from a71c9f6 to d4183ad Compare August 21, 2026 05:45
Give each eligible compact causal descriptor consumer its own graph view so scheduler copies die with their attention split instead of fragmenting the CUDA compute arena. Include the two-turn reproduction runner and the allocation, exactness, PPL, and 4K/30K/34K evidence.
@GenerelSchwerz
GenerelSchwerz force-pushed the exp/compact-causal-mask-pre-pr4 branch from d4183ad to 565233f Compare August 21, 2026 06:50
Preserve the exact-source 30K prefill and 128K decode gates, production profiler findings, and the configuration-matched 240K load-to-idle allocation check. Document the measured-versus-inferred causal conclusion and the bounded symbolic/tile-scheduling redesign plan while PR 7 remains draft.
Reuse the proven consecutive KV write-index contract directly as the compact causal descriptor. Load one bound per query tile or warp, derive later bounds by query offset, and keep fully visible CUDA attention tiles on the unmasked inner path.

Cover partial vector, tile, and MMA query tiles bit-exactly and record the rejected designs, direct screens, clean-process performance, memory, PPL, profiler, and KVarN compatibility evidence.
@GenerelSchwerz
GenerelSchwerz changed the base branch from beellama-kv-cpu-offload to beellama/main August 22, 2026 22:38
Move the compact prefix bound out of aliased dynamic shared-memory padding and into block-owned storage. Broadcast it per warp and cover D256 F16 MMA layouts with and without GQA on the isolated branch.
Store causal-prefix presence and its retained KV span in one optional value. This removes the invalid bool/count combinations while leaving graph construction, reuse checks, and attention kernels unchanged.
Require the compiled CUDA backend capability whenever the compact descriptor oracle is selected, and remove compact-only source-text assertions from the shared KVarN route-policy test.
Replace the attempt transcript with the durable design, accepted evidence, reproducible gates, source boundaries, and limitations. Require the lifecycle runner model path explicitly instead of embedding a host-specific location.
@GenerelSchwerz
GenerelSchwerz marked this pull request as ready for review August 23, 2026 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CUDA documentation Improvements or additions to documentation ggml server testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants