perf(runtime): keep the Linear Attention state in L2 across the decode graph

[MichaelDementii]

Rebased onto 487f8977. Where the numbers below come from

Base. This package was written against ad0f3d38 and now targets upstream 487f8977
(perf(sparse_moe): one CTA per token in small-T S2, and a warp merge in place of eight dependent rounds), 83 commits further on. ad0f3d38 and the earlier heads still named
throughout this report are the commits the work was done and measured on. They are left
standing rather than rewritten, so that every number below stays attached to the tree it was
actually taken from.

Provenance of every number. Every timing, ratio, percentage, bandwidth, TFLOP/s figure,
register and shared-memory count and ctest line in this report was measured on ad0f3d38, or
on an earlier head named at the point of use. Nothing here has been re-measured on
487f8977. Two changes to the instrument itself bear on that, and are stated once here
instead of being repeated at every table:

• The engine context cache changed sides. On 487f8977 ninfer_bench switches it off
  itself - engine_options.context_cache.enabled = false,
  bench/targets/qwen3_6_27b/ninfer_bench.cpp:157, introduced upstream by 385b30ce. On
  ad0f3d38 that line does not exist at all, so the bench inherited the engine default
  ContextCacheOptions::enabled = true (include/ninfer/types.h:130 there, :132 here) and
  every end-to-end run behind this report was taken with the cache constructed. Two bounds
  on how far that reaches, both stated because the difference has not been measured. From
  above: a live context cache can serve a prefix and shorten a prefill, which for prefill and
  TTFT would be a first-order difference rather than a rounding one. From below: the bench also
  sets options.execution.allow_prefix_reuse = false on every request
  (bench/targets/qwen3_6_27b/ninfer_bench.cpp:65) - byte-identical on both bases - so no
  request in these runs was eligible for a prefix hit at all, and what the old configuration
  carried was a constructed-but-unused cache (host state slots and pinned host KV) rather than
  served prefixes. Which bound is nearer the truth is an open question here; until it is
  measured, read the prefill and TTFT figures as taken in a configuration the current bench no
  longer builds.
• The bench flag was renamed. --mtp-draft-tokens no longer exists anywhere in the tree.
  The current spelling is --spec mtp --draft-tokens N
  (bench/targets/qwen3_6_27b/ninfer_bench_support.cpp:356-359), the same pair the product CLI
  takes (apps/cli/options.cpp:141-145). Every reproduction line below has been rewritten to the
  current spelling; the runs behind the numbers used the old one. One trap in that rename is
  worth naming: --spec mtp --draft-tokens 0 is rejected - validate_speculative_cli_options
  requires [1,5] for MTP (src/product/speculative_options.h:41) - so the zero-draft arm,
  written --mtp-draft-tokens 0 throughout the old text and called mtp0 below, is spelled on
  the current bench by passing neither flag, which is already the default
  (SpeculativeBackend::None).

Registered tests. tests/CMakeLists.txt moved as well. The suite that registered 104
targets on ad0f3d38 registers 114 on 487f8977 (+10 targets), and one more of them is
artifact-gated: 20 named targets now carry an explicit SKIP_RETURN_CODE 77 against
19 on ad0f3d38, on top of the blanket rule inside ninfer_add_op_test that marks
every op test skippable. The addition is ninfer_qwen3_8_27b_dflash2_real_test, gated on
weights like the rest, so one further test skips on a host that carries none. Every ctest count printed below is consequently a
statement about ad0f3d38 or an earlier head, not about the current base; the counts are left
in place rather than restated, with the base each belongs to named beside it.

No ctest run in this report has been repeated on 487f8977, and no round "114 of 114" is
claimed for it. Two facts forbid that claim. ninfer_attn_input_proj_test is red on the
bare base - an upstream defect, filed as issue #196, not something this branch introduces.
And ctest and a direct run of the same test binary have been observed to disagree on this
host, so a green ctest is not by itself evidence of anything. The only formulation this
package will stand behind on the new base is the exact one: the arm's result matched the
base, and the single failure reproduces on bare 487f8977.

Line references re-checked against 487f8977. Every file:line citation in this report was
opened on the new base. The ones that moved are corrected in place; the number each one used to
carry is kept here so nothing is replaced silently.

citation	printed here before	on 487f8977	what stands at the old number now
include/ninfer/types.h, ProposalHead::Full	:80	:82	:80 is the comment // Startup-fixed K: MTP 1..5; DFlash and DFlash2 1..15 (query width K+1).

What this does

The captured decode graph now carries a CUDA access policy window over the Linear Attention
continuation state pool, with hitProp = cudaAccessPropertyPersisting, so that state stays
resident in L2 between decode rounds.

Every decode round reads the whole pool and writes it back at a fixed address.
recurrent_fold_kernel streams all of it in a single launch. That makes it the one decode
consumer with both a fixed footprint and enough reuse to be worth an L2 reservation — and, as
the profile below shows, the one that was already leaning on L2 opportunistically.

Two details are load-bearing:

• The window is installed before cudaStreamBeginCapture and removed from the stream right
  after the capture ends. Capture bakes the stream window into every kernel node and does not
  re-read the attribute at replay, so this is the only placement that reaches graph kernels while
  still leaving eager phases under the default access policy window. The device-wide reserve is
  a separate setting and is deliberately not restored: it stays in force outside the graph
  too, and that is exactly where the prefill cost reported below comes from. Verified by walking
  the finished graph on an instrumented build: 520 of 520 kernel nodes carry a non-empty
  window with hitProp = Persisting, and zero do on stock.
• When the pool is larger than persistingL2CacheMaxSize, hitRatio is lowered to the fraction
  that physically fits instead of being left at 1. Leaving it at 1 makes the lines evict each
  other. Read off this device: l2CacheSize 96.00 MiB, persistingL2CacheMaxSize 60.00 MiB
  (62 914 560 B), accessPolicyMaxWindowSize 128.00 MiB, default persisting limit 18.00 MiB.
  The pool is 122.81 MiB, so the window is the whole pool (it fits under the 128 MiB window
  cap), the reserve clamps to 60.00 MiB, and the ratio is 0.4885 — which is exactly the
  configuration the numbers below were taken at.
• Nothing is installed at all — no window and no reserve — when the pool is wider than
  accessPolicyMaxWindowSize. That is a different limit from the reserve limit, and the
  difference matters: on this target the window is 122.81 MiB against a 60.00 MiB reserve, so a
  guard written against the reserve limit would switch off the configuration that was measured
  as the best one. Above the window limit a window stops being a window over the pool and
  becomes a window over its prefix, while the device-wide reserve is still taken in full and
  still paid for in prefill. The threshold is read from cudaDeviceProp, not compiled in. See
  the second target below for what that costs when it is not guarded.

Numbers

Qwen3.6-35B-A3B, RTX 5090, -pg 2048,384 -r 6 --warmup 2 --prefill-chunk 8192 --spec mtp --draft-tokens 3. Eight alternating passes, pass 0 dropped as warm-up, arm order rotated
each pass, delta taken as pairs inside a pass, median over the remaining seven. Two different
cards — two 5090s in the same host, so the GPUs are independent but the CPU is shared. Three
ladders in all: one on card A, two on card B.

	card A	card B
decode throughput	+0.789 % (sem 0.043 %, n=7)	+0.539 % (sem 0.072 %, n=7)
prefill throughput	−0.709 % (sem 0.054 %)	not measured
total wall time, pp2048+tg384	+0.581 % (sem 0.035 %)	not measured

Instrument noise floor in these runs: base standard deviation across passes 0.164 % on card A
and 0.161 % on card B. A null-control arm — the mechanism compiled in and armed, but installing
no window and no reserve — reads −0.082 % (sem 0.059) on card A and +0.009 % (sem 0.077)
on card B, i.e. zero on both to within the noise floor.

Every ladder here is one connected run: a single process, its own tag, its own output file. That
is checked rather than assumed — the harness can emit exactly one arm order, so each file is
replayed against it cell for cell, and all three match 48 of 48 with no gaps, no repeats and
monotone timestamps.

A second clean ladder was then taken on card B, same binary, same conditions, half an hour
later, for one purpose: to size the run-to-run spread. It reads +0.660 % (sem 0.145, n=7),
null control +0.015 %, base standard deviation 0.141 %. Two clean ladders on the same card, the
same day, therefore differ from each other by 0.121 pp.

Two further clean ladders on card B (raw/abab_h3.txt, raw/abab_h6.txt), taken later to check
the guard described below, put the same arm at +0.752 % and +0.674 % and the guarded arm —
the one this PR submits — at +0.778 % and +0.777 %, on base spreads of 0.053 % and
0.066 %. That widens the same-card spread of a single arm from 0.121 pp to 0.213 pp and leaves
the reading below unchanged: a range, not a point.

Those two ladders are the only measurement of the guarded arm, so their cells are the whole
evidence for the submitted number. They are in this package: 32 cells each, arms
off / x@nr / n60w30@nr / g60w30@nr, replayable with
scripts/analyze2.py raw/abab_h3.txt and the same on h6.

Read the absolute figure as a range, not a point. The sem inside one ladder understates the
real uncertainty, and the ladders show by how much. A sibling arm of identical geometry appears in
six single-run ladders across the two cards and comes out at
+0.386 / +0.288 / +0.257 / +0.425 / +0.187 / +0.363 % — a spread of 0.238 pp against within-run
sems of 0.013–0.099. Those six, in that order, are raw/abab_r3.txt, abab_r6.txt,
abab_r7.txt, abab_a8.txt, abab_rb1.txt, abab_rb2.txt, arm n60w30. (Two of those six ran
while the card had a foreign tenant, which is what their wider base standard deviation reflects;
all six are single connected runs, checked as above. A seventh ladder,
raw/abab_a9_disqualified_control.txt, put the arm at +0.012 %, but that one is disqualified by
its own internal control and is quoted here only so the discard is visible; it is shipped under
that name so the discard cannot be mistaken for a clean ladder.) The headline arm reads
+0.789 % on card A and +0.539 % / +0.660 % on card B. So the honest reading of this change is
+0.5 to +0.8 % of decode throughput, not "+0.789 % exactly".

The absolute level does not replicate across cards to the 0.15 pp we treat as the limit for
calling a cross-card replication successful: +0.789 % against +0.539 % is 0.250 pp apart. The
second card B ladder, +0.660 %, sits 0.129 pp from card A and would pass that limit — and that is
the point rather than a rescue. The two card B ladders are 0.121 pp apart from each other, so the
limit is smaller than the spread a single ladder carries, and no single pair of ladders can settle
the question either way. We are not claiming the absolute figure replicates across cards.

One contrast inside these ladders does reproduce tightly: the same window with the prefill cache
reset put back, which accounts for 45 %, 67 % and 63 % of the gain on the three ladders. It reads
−0.354 % (sem 0.059) on card A against −0.363 % (sem 0.088) and −0.414 % (sem 0.146) on card B —
0.009 pp and 0.060 pp apart, a spread of 0.060 pp across the three ladders against the headline's
0.250 pp. Of the fifteen arm-to-arm quantities the three ladders have in common, that contrast is
the tightest and the headline delta is the second widest. Both rest on exactly these three
ladders, so that is a like-for-like comparison; the six-ladder cross-check quoted above exists
only for the sibling arm, since the other three ladders carry no arm this contrast could be taken
against.

That is a fact about those two arms, not a general property of pairing, and it is not offered
instead of the headline. Every figure quoted here is already a pair taken inside one pass, the
headline included, and their cross-ladder spreads run from 0.060 pp to 0.489 pp — the widest of the
fifteen contains no baseline arm at all, while baseline against the null control is third tightest
at 0.097 pp. Nor does the contrast measure what this PR does: both of its arms carry the change and
master carries neither. What the change is worth against master is the +0.5 to +0.8 % above, and
that is the figure whose absolute level does not replicate across cards.

How the arms were switched. All arms in a ladder come from one binary, with the arm
selected by an environment variable, so no build-to-build difference can leak into a delta. The
code submitted here has that arm compiled in unconditionally. The submitted build was compiled
and tested (below), but its throughput was not measured separately from the arm-switching
build; the two differ only in whether the same call is behind an if.

Prefill pays, and the trade depends on the mix. The reservation is also in force while a
prefill chunk streams through L2, and the profile names the kernel that pays for it. At this
prompt and generation length the decode gain more than covers it; a prefill-dominated workload
would not be a good trade. That is a deliberate choice, not an oversight, and the numbers for
both halves are above.

The second target, and why the guard is in the code

capture_graph is shared, so this change reaches every target of this runtime. It was measured
on a second one, Qwen3.8-27B NVFP4, and there it was a pure loss — which is what the guard
above exists to stop.

That target's state pool is 307 888 128 B, 2.29× the 134 217 728 B window limit. The window
therefore covered 43.6 % of the pool while the reserve was taken and paid for in full. Nothing
reported this: cudaDeviceSetLimit and cudaStreamSetAttribute both returned cudaSuccess,
the requested limit was granted exactly, and the truncation happened in our own code before the
call. Measured cost of that silence, same protocol, two clean ladders per column, four in all,
one binary, arms switched by environment variable:

	without the guard	with the guard
decode throughput	−0.039 % / −0.041 %	−0.013 % / −0.008 %
prefill throughput	−0.550 % / −0.521 %	−0.001 % / +0.005 %
total wall time	−0.089 % / −0.108 %	−0.013 % / −0.018 %
base spread of the ladder	0.027 % / 0.062 %	same ladders

Half a percent of prefill for nothing in decode, and after the guard nothing either way: every
figure in the right column is inside the base spread of its own ladder. Walking the captured
graph says the same thing without reference to timing — 0 of 811 kernel nodes carry a
window with the guard, against 811 of 811 without it, and the persisting limit stays at the
18 874 368 B driver default instead of being raised to 60 MiB. The mechanism is off, not
weakened.

The reason it gains nothing there is structural rather than a matter of degree: the window
covers 43.6 % of the pool, of which half is the second state slot that nothing reads, so about
20 % of the live state ends up resident against 49 % on the target above. And there is little
for it to give back — a kernel census on the nearest neighbour of that target (Qwen3.6-27B
NVFP4: same target code, different weights) puts recurrent_fold_kernel at 0.9 % of decode
kernel time and weight GEMMs at 89.8 %, already running at 89.7 % of the measured read ceiling.
Arithmetic straight off the submitted target's own benchmark row agrees on the second half of
that: 20 826 889 216 B of weights per round in 17.98 ms is 1158.5 GB/s, 68.6 % of the measured
read ceiling, and that is a lower bound.

On Qwen3.6-35B-A3B the guard does not fire and changes nothing. That is the control
against a false trip, and it is checked two ways. By audit: the instrumented build prints
identical geometry with and without the guard — 520 of 520 kernel nodes windowed, window
128 778 240 B, reserve 62 914 560 B, hitRatio 0.4885 — on every cell of both ladders. And by
ladder: taken as pairs inside a pass, the guarded arm against the unguarded one reads
+0.030 pp and −0.072 pp of decode, 0.080 pp and 0.052 pp of prefill, and 0.002 pp and
0.016 pp of total wall time, on ladders whose base spread is 0.053 % and 0.066 %. The sign of
the decode difference flips between the two ladders, which is how a quantity whose true value
is zero behaves.

Three of seven ladders taken for this were discarded by their own pre-registered noise
control (base spread 1.436 %, 1.562 %, 0.217 % against a 0.20 % limit written down before the
runs); the card witness was clean on all of them and no foreign process was on the card, so the
excursions are unexplained. They are quoted nowhere above. For the record they agree in sign
with the four that qualified.

Because nothing in the driver reports the truncation, the guard says so once per process on
stderr when it fires, rather than switching the mechanism off silently — silent truncation is
the failure being fixed here, and a silent guard would reproduce it. Checked on the submitted
binary itself, not only on the instrumented one: the 35B artifact prints nothing, and a 27B
NVFP4 artifact prints exactly one line naming both sizes.

Where the time moves

nsys profile -t cuda --cuda-graph-trace=node, no capture range. ncu counters are closed on
this host, so there is no direct L2 hit counter here. Instead the mechanism rests on two things.
First a structural-zero control. In one ladder (base standard deviation 0.132 %), a window of
identical geometry — same reserve, same size, same hit ratio — pointed at a buffer nothing
reads measures −1.082 % (sem 0.065) against stock, while the real window in the same ladder
measures +0.257 % (sem 0.053). Same cost, opposite sign: the gain is not an artefact of
taking a reserve or of installing a window, it comes from pinning this range. (Both arms of
that ladder also carried an extra cache-reset call that the submitted code does not have, which
is why +0.257 % sits below the headline; the two carry it equally, so the contrast is unaffected.)
Second, the roofline below.

kernel	launches	median, stock → patched
gated_delta_net::recurrent_fold_kernel	231	54 944 → 41 216 ns (−25.0 %)
gated_delta_net::recurrent_record_kernel	6 990	5 344 → 4 737 ns (−11.4 %)
gated_delta_net::recurrent_bf16_direct_kernel	30	2 656 → 2 096 ns (−21.1 %)
sparse_moe_prefill_reduce_kernel (prefill)	80	38 193 → 48 288.5 ns (+26.4 %)
sparse_moe_d4_nine_warp_kernel	37	10 912 → 11 808 ns (+8.2 %)
rmsnorm_warp_bf16x2_kernel (this instantiation)	60	19 904 → 21 328 ns (+7.2 %)
sparse_moe_d3_path_tiled_kernel (this instantiation)	233	49 089 → 51 105 ns (+4.1 %)
every other kernel		no median moves by more than 3.1 %

The table is the full picture, not a top-N: both runs contain the same 84 distinct kernels, none
appears or disappears, and every one of the 84 was compared. The largest regression,
sparse_moe_prefill_reduce_kernel, is a prefill kernel, which is consistent with prefill
throughput being the half that pays — though that attribution is by kernel name and launch
count, not by a per-phase timeline. Summed over the whole pp2048+tg384 run, total kernel time
still drops: 1434.0 → 1430.3 ms.

The reservation is not free, and it is instructive who pays for it. With the reserve taken but
nothing pinned, recurrent_fold_kernel goes 54 944 → 73 921 ns (+34.5 %) and accounts for
115 % of the entire slowdown, while the high-launch-count MoE streaming kernels
(sparse_moe_d3_path_tiled, 9 320 launches; sparse_moe_d4_token, 8 621) do not move at all —
+0.00 % median on both. The kernel that pays for the reservation is the same kernel the
reservation is for: it was already served in part by L2, the carve-out takes that away, and the
window gives it back with interest.

Roofline, against ceilings measured on this card

The ceilings below are direct measurements of this card, each from two independent runs agreeing
to within 0.2 %: DRAM copy 1503.4 GB/s, DRAM pure read 1689.4 GB/s, L2 aggregate read
7132–7461 GB/s. That is more honest than quoting a constant compiled into a benchmark, but it is
our own measurement, not a vendor figure.

Per launch, recurrent_fold_kernel reads and rewrites the live slot of all 30 layers:
30 × (49 152 + 2 097 152) = 64 389 120 B each way, 128 778 240 B of traffic.

	launch time	read+write rate	vs DRAM copy ceiling	read-only rate	vs DRAM read ceiling
stock	54 944 ns	2343.8 GB/s	1.56×	1171.9 GB/s	69.4 %
reserve taken, nothing pinned	73 921 ns	1742.1 GB/s	1.16×	871.0 GB/s	51.6 %
patched	41 216 ns	3124.5 GB/s	2.08×	1562.2 GB/s	92.5 %

The kernel exceeds the measured DRAM copy ceiling already on stock and by 2.08× when pinned.
That traffic cannot be coming from DRAM, which is arithmetic rather than interpretation: with no
hit counter available, exceeding one medium's ceiling is what establishes that another medium is
carrying the difference. Against the aggregate L2 read ceiling the patched kernel takes 21.9 %.

recurrent_record_kernel reads one layer (2 097 152 B) per launch: 392.4 GB/s = 23.2 % of the
read ceiling on stock, 442.7 GB/s = 26.2 % patched. That one is latency-bound, not
bandwidth-bound, which is why its gain is half the size.

Correctness

Output is bit-identical. Greedy decoding through the CLI (where prefix reuse is off in code),
--spec mtp --draft-tokens 3 --prefill-chunk 8192 --max-new 192, three prompts, seven arms
including the exact configuration submitted here (reserve 60 MiB, window over the whole pool,
missProp = Normal). The reflexive comparison — stock against stock — was taken first, because
a gate that does not agree with itself proves nothing in either direction.

21 of 21 comparisons IDENTICAL; exactly 3 distinct digests, one per prompt

That was taken on the revision before the guard. It was re-run with the guard in place, on both
targets, arms off (twice, reflexive first), the armed-but-inert control, the unguarded window
and the guarded window:

Qwen3.6-35B-A3B   12 of 12 IDENTICAL; 3 distinct digests   (guard does not fire)
Qwen3.8-27B       12 of 12 IDENTICAL; 3 distinct digests   (guard fires)

Independently, spec_acceptance_rate and spec_rounds agree to the last digit across every
clean measurement cell of every ladder — the patch does not change a single accepted draft.

ctest: 104/104 pass on both arms from the same build directory — base 464.9 s, branch 460.5 s.
Both arms are on ad0f3d38, the base this branch sits on, whose tests/CMakeLists.txt registers
exactly those 104 targets; 487f8977 registers 114 and neither arm was re-run there.

Worth being explicit about one thing: six of those 104 are opt-in real-artifact tests that skip
unless their weights variable is set, and they are the only ones in the suite that reach the code
this change touches. So they were run explicitly against real artifacts:
ninfer_qwen3_6_35b_a3b_real_test (36.9 s), ninfer_qwen3_6_35b_a3b_dflash_real_test (62.4 s)
and ninfer_qwen3_6_27b_prefix_real_test (103.2 s) all pass. The first two build the engine with
use_cuda_graph = true, so the captured decode graph — and with it the window installed in
capture_graph — is actually exercised; the third runs on a target whose pool is 307 888 128 B,
so it exercises the guarded path instead. Without those, a green ctest here would have meant
very little.

clang-format (23.1.0, repository config) reports zero findings on all three touched C++ files,
and zero on the pre-existing one at the base commit as well — checked from a separate worktree at
the base rather than through a pipe, because clang-format does not find the repository config
when reading stdin — so no formatting of existing code is mixed into this change.

The change touches cache policy only; it does not alter arithmetic, kernels, register or shared
memory usage, or any launch geometry.

What was measured and rejected on the way here

Recording these so the choice of parameters is not mistaken for a guess:

• Reserve size is not a tuning knob with an interior optimum. Sweeping the reserve from 6 to
  60 MiB at a fixed window gives a monotone curve — −0.418 %, −0.185 %, −0.040 %, +0.093 %,
  +0.270 %, +0.386 % (n=6 per point) — so the best reserve is the card's maximum, and the card's
  maximum is the 60.00 MiB read out of persistingL2CacheMaxSize above. Measured in a run whose
  base standard deviation was 0.071 %.
• Narrowing the window hurts. A window over 14 layers at hitRatio 1.0 is 0.297 % (sem
  0.103 %) worse than the whole pool at hitRatio 0.4885.
• There are no hot layers to prefer. Pinning the lowest 14 layers and the highest 14 layers
  differ by −0.011 % (sem 0.081 %).
• The card's non-zero default reserve is nothing to reclaim. Setting the limit explicitly to
  0 and to its 18 MiB default come out at +0.019 % — but the band on that one is sem 0.140 %,
  wide enough to hide an effect the size of the ones above, so read it as "no evidence of a
  difference", not as a demonstrated zero.

Two caveats on that list, both worth stating rather than leaving to be found. The last three
items come from ladders taken while the card had a second tenant (base standard deviation
0.243 % and 0.132 %, against 0.164 %, 0.161 % and 0.141 % for the three clean ladders the
headline numbers come from), which is why they are quoted only as pairs taken inside a pass,
so that the raised base level does not enter the delta. (That keeps the pass baseline out of each
number; it is not a claim that pairing makes a quantity reproduce across ladders — see the spread
of the fifteen above.) And every arm on that list — the reserve sweep included — still carried an extra
cudaCtxResetPersistingL2Cache call on the prefill path that this change does not have. It sits
in both halves of each pair equally, so the comparisons hold; it does mean the absolute values
there are not directly comparable to the headline.

Known limitations

The reserve stays in force during prefill, which is where the −0.709 % comes from. Dropping the
limit before a prefill chunk and restoring it afterwards may recover part of that; the cost of
cudaDeviceSetLimit in a hot path has not been measured, so it is not in this change.

About half the reserve is spent on bytes nothing reads. In this configuration the state pool
holds two slots per layer, laid out [conv s0][conv s1][rec s0][rec s1], so live and dead bytes
alternate with a 2 MiB stride inside every layer — and an access policy window, being a single
contiguous range with no stride, cannot tell them apart. Laying the pool out by slot rather than
by layer would roughly double the share of the reserve that holds live bytes; whether that
converts into gain has not been measured, and in any case it is a change to the state allocator
rather than to cache policy, so it is not part of this PR.

The guard is a size test, not a benefit test. It switches the mechanism off where it provably
cannot work — a window that cannot span the pool — and it says nothing about targets whose pool
does fit but whose decode has no reuse to recover. Two targets have been measured, one on each
side of the limit; a third has not. If a fitting target ever turns out not to benefit, the answer
would be a per-target opt-out rather than a wider size test, and there is no evidence today that
one is needed.

Qwen3.8-27B was measured on one card only. For a result that is a zero, and whose main evidence
is a graph audit showing no window installed at all, a second card would move the digits and not
the conclusion.

Speculation counters: how much of this number is round count, and how much is round speed

Added 2026-09-06 after a dedicated audit of every decode headline I have prepared for this
project. That audit was run with my own harness outside this repository and is not a file
of this package; it is named only to say where the section came from, and nothing below
leans on it — its own logs can be supplied on request. The section answers, from this
package's own raw, the one question a throughput figure in tokens per second cannot answer
by itself.

ninfer_bench reports decode as generated tokens over elapsed time. A change that alters the
numbers the model produces can alter draft acceptance, and with it the number of speculative
rounds spent on the same fixed output length. Such a change raises tokens per second without any
kernel running faster. The two effects separate exactly, because the bench prints the round count
itself:

decode_time(base) / decode_time(arm)  =  [ rounds(base) / rounds(arm) ]  *  [ t(base) / t(arm) ]

where t = decode_seconds_mean / spec_rounds is the time of one round. spec_rounds is an exact
integer printed by the engine and decode_seconds_mean is the measured time, so the split is an
identity rather than a model.

This package: the round count is identical across every arm, so the whole figure is round speed.
Raw, all of it in this package - twelve ladders, 522 cells, each cell a full CSV row:

file	cells	arms	model	spec_rounds / spec_acceptance_rate in every cell	what it carries
raw/abab_a8.txt	48	6	35B	678 / 0.7994100295	headline card A, +0.789 %; null control −0.082 %
raw/abab_r3.txt	49	7	35B	678 / 0.7994100295	window-length sweep; sibling arm +0.386 %
raw/abab_r6.txt	56	7	35B	678 / 0.7994100295	sibling arm +0.288 %
raw/abab_r7.txt	48	6	35B	678 / 0.7994100295	sibling arm +0.257 %
raw/abab_r8.txt	65	6	35B	678 / 0.7994100295	+0.597 %; the one file here that is not a single connected run - two start headers, 30 cells written after done; split and re-analysed in raw/reanalyze_r8_split.txt
raw/abab_rb1.txt	48	6	35B	678 / 0.7994100295	headline card B, +0.539 %
raw/abab_rb2.txt	48	6	35B	678 / 0.7994100295	second card B ladder, +0.660 %
raw/abab_h3.txt	32	4	35B	678 / 0.7994100295	guarded arm +0.778 %
raw/abab_h6.txt	32	4	35B	678 / 0.7994100295	guarded arm +0.777 %
raw/abab_a9_disqualified_control.txt	32	4	35B	678 / 0.7994100295	discarded ladder, +0.012 %, kept so the discard is visible
raw/abab_h2.txt	32	4	27B	576 / 1	second target, guard check
raw/abab_h7.txt	32	4	27B	576 / 1	second target, guard check

In every one of the 458 cells of the ten 35B ladders the engine prints the same three
counters: spec_rounds 678, spec_fallback_steps 0, spec_acceptance_rate
0.7994100295. The two 27B ladders are a different target and print 576 / 0 / 1 in all
64 of their cells - also constant across arms, which is the property that matters here.

Each of the twelve files is replayed cell for cell against the one arm order the harness can
emit; the witness for all twelve is raw/verify_all_ladders_2026-09-06.txt. Eleven come back
"one connected run, no gaps, no repeats, monotone timestamps". abab_r8.txt does not, and is
reported as such rather than quietly dropped: it carries two start headers and 30 cells
appended after its done line, and the split is worked through in raw/reanalyze_r8_split.txt.
No headline number in this PR rests on it.

The measurement runs at --spec mtp --draft-tokens 3, which is where a round-count substitution could
hide. With the round count equal in all 458 35B cells, +0.789% on stand A and +0.539% on stand
B are the time of one round and nothing else, and so is the submitted +0.778% / +0.777%.

This is what a bit-exact change has to do, and it is stated here because the identity gate alone
would not settle it: under greedy verification the emitted text is the target model's greedy
sequence whatever the acceptance rate, so equal output does not by itself imply an equal number of
rounds. The counters do.

Which proposal head this was taken at. Every cell cited above ran at the product default
ProposalHead::Full (include/ninfer/types.h:82). The audit put the same question to
--lm-head-draft, the configuration docs/performance.md publishes in, on Qwen3.6-35B-A3B at
-pg 2048,384 --prefill-chunk 8192 --spec mtp --draft-tokens 3: a bit-exact arm and the base agree there
as well - 690 rounds and acceptance 0.7819767442 in both - so the split is the same under
either head. A non-bit-exact arm measured beside them in the same ladder does move the count
(714 rounds, acceptance 0.7422969188), which is what shows the instrument would have caught a
substitution had there been one.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T08:10:45.802642Z	0deee4d	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 59d1bccb49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Restore the device-wide L2 reservation on teardown

cudaDeviceSetLimit changes context-wide mutable state, but no Program or Engine owner restores the previous limit. After destroying a graph-enabled Engine whose state fits the window, creating another Engine with CUDA graphs disabled—or one whose state exceeds accessPolicyMaxWindowSize—retains the enlarged reservation without installing any access-policy window, reducing the L2 available to that subsequent workload. Tie the limit to Program lifetime and restore it during teardown.

AGENTS.md reference: AGENTS.md:L51-L53

Useful? React with 👍 / 👎.

[MichaelDementii]

The Codex finding is correct, and this is the fix: 0deee4d8.

cudaDeviceSetLimit(cudaLimitPersistingL2CacheSize, ...) was context state that nothing restored. unpin takes the window off the stream and never touches the limit, so an Engine destroyed after capture left the set-aside standing for whatever ran next — including a target where pin_range bails at the accessPolicyMaxWindowSize guard and installs no window at all. The comment in this file already said the reserve "is taken in full and paid for everywhere, prefill included", so the cost was ours to own.

What the commit does. l2p::Reservation is a counted handle in DeviceContext. The limit is per device rather than per process, so requests are reconciled per device ordinal: the driver limit is held at the largest live request for that device, and returns to the limit in force before the first request once the last is released.

I put the owner in DeviceContext rather than in Program as suggested. Program is narrower than the graphs need — DecodeGraphFamily lives in Program storage, and reaching it from capture_graph means threading a reference through ExecutionCore for nothing. DeviceContext is held by reference by both Program (program.h:622) and EngineCore (engine_core.h:2028) and is a member of Engine::Impl declared before core, so it outlives every graph captured under the window.

Behaviour while the reserve is live is unchanged. The 35B state pool is 128 778 240 B, the reserve clamps to 62 914 560 B, and that is above this driver's 18 874 368 B default under both the old code and the new. Only teardown differs. Verified end to end by interposing the driver calls on ninfer_bench: the reserve holds across all 93 graph launches, and the limit returns to 18 874 368 at ~DeviceContext, before CUDA unloads. With --no-cuda-graph, nothing is touched at all.

Three defects were found and fixed while writing this, each by a review that did not trust the previous one.

• The handle's fields were outside the registry lock. Two threads on one handle publish two requests and withdraw one, leaving the device at a raised limit with no window — the exact state this change exists to prevent. Reproduced with 2×20000 request/release pairs, then fixed.
• A failed cudaSetDevice was swallowed and the limit written anyway, to whichever device happened to be current. Now reported instead.
• The bookkeeping was surrendered before the restoring write landed, and a request that recorded no baseline could still take one back on failure. A request above persistingL2CacheMaxSize returns cudaErrorInvalidValue on healthy hardware, so that path is reachable without any device fault; it leaked 12 MiB of L2 for the life of the process. Measured on an injection harness, then closed.

The test's discrimination is measured, not assumed. tests/test_l2_reservation.cpp covers nesting, equal-sized twins, 32 cycles without baseline drift, request(0) as a non-release, moves, replacement on a live handle, a refused request, and concurrent use of one handle. Six mutations of the source were built and run against it. The first version of the test killed two of six: two of its assertions could not fail, because the size they asked for was below this driver's default limit — including the one that claimed to verify that the limit comes back down. Cases were added until the survivors died. Two mutants survive by construction: both need a limit write to fail, which one healthy card cannot stage, and they are recorded as uncovered rather than as passed.

Checks. ctest 114 of 115; clang-format clean on all six files; the new test does not flake over 25 consecutive runs. The single ctest failure is ninfer_attn_input_proj_test, which fails the same way on untouched master — built from master and run three times, same set of T, all under graph phase=1 — and is the subject of #196. It reaches neither pin_range nor capture_graph.

Not covered. Multi-GPU: the per-device reconciliation and the bind-failure path are read off the code, not measured, because this stand has one card. On a two-card host the state that distinguishes the third fix above is reachable without any injection, since cudaSetDevice to another device can fail on healthy hardware — so that fix is not theoretical, merely unverifiable here. Kernel-node enumeration was not repeated on this base; that the window is installed before capture with the same geometry is verified, that it is baked into 520 of 520 nodes is carried from the earlier session.

🤖 Generated with Claude Code

[MichaelDementii]

The branch was replaced after the finding above; the review on this PR still points at the previous head. Current head is 0deee4d8.

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 0deee4d8d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".