perf(text): skip NFC normalisation when the text is already pure ASCII

[MichaelDementii]

Rebased onto a16b6442. Where the numbers below come from

Base. This package was written against ad0f3d38 and now targets upstream a16b6442
(docs: organize performance reports and update 35b results). Its parent is 487f8977
(perf(sparse_moe): one CTA per token in small-T S2, and a warp merge in place of eight dependent rounds), which is the commit the rebase and the probe re-measurement below were
carried out on; master moved one commit further while this was being prepared. That one commit
touches thirteen files, all of them documentation - README.md, docs/, eval/README.md,
model-cards/, tools/bench/README.md - and no source, header, test or bench file, so every
code citation and every number in this report reads the same on 487f8977 and on a16b6442.
ad0f3d38 and the earlier heads still named throughout this report are the commits the original
work was done and measured on. They are left standing rather than rewritten, so that every
number below stays attached to the tree it was actually taken from.

Provenance of every number. Except where a table says otherwise, every timing, ratio,
percentage, bandwidth, TFLOP/s figure, register and shared-memory count and ctest line in this
report was measured on ad0f3d38, or on an earlier head named at the point of use.
One thing has been re-measured on 487f8977: the normalize_nfc probe. It came back at
x46 to x57 against the x62 to x71 this report claimed; the mechanism, the sign and the
order of magnitude all held, the multiplier did not. Both are printed side by side in
"The probe re-measured on the new base". Nothing else here has been re-measured on 487f8977

• not the Tokenizer::encode column of the isolated bench, not the end-to-end rows, not the
  server TTFT rows, not the identity corpus and not any ctest run. Two changes to the instrument
  itself bear on that, and are stated once here instead of being repeated at every table:

• The engine context cache changed sides. On 487f8977 ninfer_bench switches it off
  itself - engine_options.context_cache.enabled = false,
  bench/targets/qwen3_6_27b/ninfer_bench.cpp:157, introduced upstream by 385b30ce. On
  ad0f3d38 that line does not exist at all, so the bench inherited the engine default
  ContextCacheOptions::enabled = true (include/ninfer/types.h:130 there, :132 here) and
  every end-to-end run behind this report was taken with the cache constructed. Two bounds
  on how far that reaches, both stated because the difference has not been measured. From
  above: a live context cache can serve a prefix and shorten a prefill, which for prefill and
  TTFT would be a first-order difference rather than a rounding one. From below: the bench also
  sets options.execution.allow_prefix_reuse = false on every request
  (bench/targets/qwen3_6_27b/ninfer_bench.cpp:65) - byte-identical on both bases - so no
  request in these runs was eligible for a prefix hit at all, and what the old configuration
  carried was a constructed-but-unused cache (host state slots and pinned host KV) rather than
  served prefixes. Which bound is nearer the truth is an open question here; until it is
  measured, read the prefill and TTFT figures as taken in a configuration the current bench no
  longer builds.
• This paragraph is about ninfer_bench, and it does not cover the server rows of this
  report. The two binaries are configured differently and the distinction decides whether the
  TTFT claim below means anything, so it is spelled out rather than left to be inferred.
  bench/targets/qwen3_6_27b/ninfer_bench.cpp:65 hard-codes allow_prefix_reuse = false and can
  never take a prefix hit. apps/ninfer-serve does the opposite: src/serve/serve_options.h:50
  is bool allow_prefix_reuse = true; - byte-identical on ad0f3d38 and 487f8977 - and it
  is turned off only by an explicit --no-prefix-reuse
  (src/serve/serve_options.cpp:286), which the server run below does not pass. From there
  src/serve/generation_service.cpp:291 gives the request CacheParticipation::ReadWrite. The
  server rows are therefore taken with prefix reuse on, at the shipped default, and the
  request log confirms hits actually happened rather than merely being permitted:
  computed_prefill_tokens=0, prefix_cache_hit_tokens=8475 on requests 2..40.
• The bench flag was renamed. --mtp-draft-tokens no longer exists anywhere in the tree.
  The current spelling is --spec mtp --draft-tokens N
  (bench/targets/qwen3_6_27b/ninfer_bench_support.cpp:356-359), the same pair the product CLI
  takes (apps/cli/options.cpp:141-145). Every reproduction line below has been rewritten to the
  current spelling; the runs behind the numbers used the old one. One trap in that rename is
  worth naming: --spec mtp --draft-tokens 0 is rejected - validate_speculative_cli_options
  requires [1,5] for MTP (src/product/speculative_options.h:41) - so the zero-draft arm,
  written --mtp-draft-tokens 0 throughout the old text and called mtp0 below, is spelled on
  the current bench by passing neither flag, which is already the default
  (SpeculativeBackend::None).

Registered tests. tests/CMakeLists.txt moved as well. The suite that registered 104
targets on ad0f3d38 registers 114 on 487f8977 (+10 targets), and one more of them is
artifact-gated: 20 named targets now carry an explicit SKIP_RETURN_CODE 77 against
19 on ad0f3d38, on top of the blanket rule inside ninfer_add_op_test that marks
every op test skippable. The addition is ninfer_qwen3_8_27b_dflash2_real_test, gated on
weights like the rest, so one further test skips on a host that carries none. Every ctest count printed below is consequently a
statement about ad0f3d38 or an earlier head, not about the current base; the counts are left
in place rather than restated, with the base each belongs to named beside it.

No ctest run in this report has been repeated on 487f8977, and no round "114 of 114" is
claimed for it. Two facts forbid that claim. ninfer_attn_input_proj_test is red on the
bare base - an upstream defect, filed as issue #196, not something this branch introduces.
And ctest and a direct run of the same test binary have been observed to disagree on this
host, so a green ctest is not by itself evidence of anything. The only formulation this
package will stand behind on the new base is the exact one: the arm's result matched the
base, and the single failure reproduces on bare 487f8977.

Line references re-checked against 487f8977. Every file:line citation in this report was
opened on the new base. The ones that moved are corrected in place; the number each one used to
carry is kept here so nothing is replaced silently.

citation	printed here before	on 487f8977	what stands at the old number now
src/targets/qwen3_6/impl/state/round_state.cpp, the DFlash allocation gate	:176	:174	:176 is inside the block, not its condition: DFlashDecodeStateLayout& decode = layout.dflash_decode.emplace();
include/ninfer/types.h, the backend default	:78	:79	:78 is the opening line struct SpeculativeOptions {
src/serve/serve_options.h, SpeculativeOptions speculative;	:46	:46	unchanged - re-checked, still correct

The gate this report quotes no longer exists, and the real one is wider. The text below
quoted if (layout.spec.enable_dflash). The field enable_dflash is gone from the tree
entirely - git grep enable_dflash origin/master finds it only inside the frozen code samples
under eval/corpora/. round_state.cpp:174 now reads
if (is_masked_draft_backend(layout.spec.backend)), and that predicate is
backend == SpeculativeBackend::DFlash || backend == SpeculativeBackend::DFlash2
(src/targets/qwen3_6/export/ninfer/targets/qwen3_6/startup_features.h:7-9). So the gate admits
both --spec dflash and --spec dflash2, not --spec dflash alone, and the quotation and
the "reachable only through --spec dflash" wording have been corrected wherever they appeared.
The conclusion the paragraph draws is unaffected: the default is still
SpeculativeBackend::None, and no measurement in this report passes either flag.

Base: origin/master @ ad0f3d38, checked 2026-09-05 with
git ls-remote https://github.com/Neroued/ninfer.git refs/heads/master rather than
through a clone's origin. This branch was written against a140e7ae and has been rebased
straight onto ad0f3d38 across every intervening head: no conflict, and
patch-id --stable of the change is unchanged. Head 076bd9bd -> ac6a69e3, 1 commit(s)
above the base.

The two commits added since b8786751 are 863aa8a5 (fix(dflash): allow vision prompts) and ad0f3d38 (chore: add project funding information). Nothing here is
re-measured because of them, and that is settled by a line of source rather than by the
commit subject: everything the DFlash fix adds is allocated under
if (is_masked_draft_backend(layout.spec.backend)) (src/targets/qwen3_6/impl/state/round_state.cpp:174),
behind a speculative backend whose default is SpeculativeBackend::None in every entry
point (include/ninfer/types.h:79, src/serve/serve_options.h:46), reachable only
through an explicit --spec dflash or --spec dflash2 that no measurement here passes.

Note that master and dev no longer agree: dev is one commit behind, at 863aa8a5.
Details: the rebase note that travels with this package - a working file of mine, not a
file of this repository, supplied on request.

Scope

One host function, ninfer::text::normalize_nfc, one file, src/text/unicode.cpp, +12 lines,
-0. No signature, no header, no public contract, no CLI flag, no workspace, no graph profile, no
artifact format, no device code. Token ids are bit-identical; the change is a fast path, not a
different answer.

Level of the claim: host operator, with the caller measured on both sides of it. The operator
number is large and easy; the end-to-end number is the honest part, because it depends entirely on
whether there is any prefill left to hide behind. Both regimes are measured and both are below.

Environment

Two hosts. Nothing in this report mixes them, and every table says which one it comes from.

Measurement host. AMD EPYC 7B12, 64 cores / 256 threads, 503 GiB, one RTX 5090; Ubuntu 24.04,
g++ 13.3.0, Release, -DCMAKE_CUDA_ARCHITECTURES=120a, CUDA 13.1.115. Base origin/master
a140e7ae; four arms built from one tree in turn, git am --3way clean, tree not dirty, zero debug
markers.

Test host. AMD EPYC 9V74, 160 threads, 188 GiB, one RTX 5090, driver 580.95.05, CUDA 13.1.115,
g++ 13.3.0, Ubuntu 24.04.4, cmake 3.28.3, ninja 1.11.1. This is where ctest was run, because the
suite links one static executable per test and needs the disk for it.

Vocabulary is the product one, not a stand-in: the tokenizer was read out of a
qwen3_6_35b_a3b.ninfer artifact - 248 044 tokens, 247 587 merges, 26 added tokens. The operator
rows do not involve a vocabulary at all; the encode rows do, and this is the real denominator.

Method for every ratio below. One process per (arm, fixture, round); arms interleaved inside a
round; 3 warm-ups then 15 timed repeats; the cell is the median over 7 rounds of the minimum of
those 15; two fully independent passes, both printed. Server rows are 40 requests per arm per
pass over 16 sessions with --request-log-jsonl at full precision.

The noise floor is measured, not asserted. The same sweep contains 54 cells whose true value
is exactly 1.000, because in those pairs the arm does not touch src/text/unicode.cpp and
md5sum shows unicode.o byte-identical between the two binaries. Those 54 cells run
0.902 to 1.025, median 0.997, and the low end reproduces in both passes on the same fixture, so
it is binary layout rather than the scheduler. Within-arm dispersion of round minima on the base arm,
72 cells: median 1.40%, p90 3.14%, max 8.82%. Working rule: anything inside 0.90..1.03 on this
bench is not an effect. Every number claimed as an effect below is far outside it - 0.014 is seven
times further from 1.000 than the edge of the band.

Observation

normalize_nfc is 0.79 ms of a 5.86 ms Tokenizer::encode on a 33 KB pure-ASCII chat prompt, and
2.87 ms on a 135 KB one. It is called on every prompt, on every request, before anything reaches the
GPU. On an artifact where a hit prefix cache brings prefill down to 1.6 ms, that host work is
79 to 81% of time-to-first-token.

Root cause

src/text/unicode.cpp:18 hands every prompt to utf8proc_map with
UTF8PROC_STABLE | UTF8PROC_COMPOSE, which decodes the text to codepoints, maps each one,
recomposes, and writes the result into a fresh malloc'd buffer that the function then copies into a
std::string and frees.

Every byte below 0x80 is a starter with canonical combining class zero that takes part in no
canonical composition. Pure ASCII is therefore already in NFC, and the entire pass is the identity on
it - decode, map, recompose, allocate, copy, free, to reproduce the input.

Change

One linear pass over the bytes; if none is >= 0x80, return a copy and skip utf8proc entirely.

 std::string normalize_nfc(std::string_view text) {
+    // Every byte below 0x80 is a starter with combining class zero and takes part in no canonical
+    // composition, so pure ASCII is already in NFC and utf8proc has nothing to do with it. The
+    // scan is one pass the compiler vectorises against utf8proc's decode, map and recompose.
+    bool ascii_only = true;
+    for (const char byte : text) {
+        if (static_cast<unsigned char>(byte) >= 0x80u) {
+            ascii_only = false;
+            break;
+        }
+    }
+    if (ascii_only) { return std::string(text); }
+
     utf8proc_uint8_t* mapped = nullptr;

The test is on raw bytes, not on decoded codepoints, and that is what makes it safe. Every byte
of a multi-byte UTF-8 sequence is >= 0x80 - lead bytes 0xC2..0xF4, continuation bytes
0x80..0xBF - so any non-ASCII input, well-formed or not, leaves the loop on its first such byte and
takes the unchanged path. The classification can only be conservative: it cannot admit a non-ASCII
string to the fast path. The rejection contract is unchanged for the same reason - invalid UTF-8
necessarily contains such a byte, still reaches utf8proc_map, and still throws the same exception.

Nothing is added to any object's lifetime: no getenv, no thread_local, no module-global state.
The fast path replaces the malloc/free pair inside utf8proc_map with the std::string copy the
function already performed on its way out.

Correctness evidence

The premise is checked exhaustively against the tree's own vendored utf8proc, not argued.
normalize_nfc(s) == s for every ASCII string of length 1, 2 and 3 - all 2 113 664 of them,
including 0x00 and every control character - plus 2 000 000 random ones of length 4..32. Total
4 113 664 strings: 0 mismatches, 0 exceptions raised by the unpatched function. Two consequences
worth naming: STABLE | COMPOSE carries no NLF2LF, so CR/LF and the C0 controls are fixed points;
and utf8proc_map passes an embedded NUL through rather than rejecting it, so a NUL in a prompt
behaves the same on both paths.

Output is byte-identical. 18 fixtures - English prose, code, Russian, Chinese, emoji and ZWJ
joins, malformed boundaries, control runs - at seven max_tokens values in both added-token parsing
modes: 252 digest cells, byte-identical on all four arms. An exhaustive max_tokens sweep of
0..3000 on 8 fixtures, 24 008 encodes per arm, gives an identical rolling digest. The probes are
not vacuous: precomposed and decomposed spellings of the same text converge to the same normalised
digest, so composition really happens on the slow path.

ctest, run on both arms on the test host.

arm	hash	result
base	a140e7ae	104 of 104 passed, 0 failed (6 skipped: real-weight tests, no weights in env)
this branch	076bd9bd	104 of 104 passed, 0 failed (same 6 skipped)

Re-run with the real artifacts in the environment, so the six weight-gated tests actually execute:

arm	result
base a140e7ae	4 passed, 1 skipped (27b_load_plan), 1 FAILED: ninfer_qwen3_6_27b_prefix_real_test - normalized first response did not restore its response checkpoint: path=5 reused=267
this branch 076bd9bd	4 passed, 1 skipped, 1 FAILED: the same test, the same message, the same numbers

That failure is upstream's, not this branch's. It is on a140e7ae itself with an unmodified
tree, it reproduces identically on both arms down to path=5 reused=267, and it has been reproduced
on three separate machines. It is not touched, worked around, or hidden here; it is reported as a
separate defect.

A passing suite proves nothing until it is shown to look at the changed code, so that was
measured too. A throwaway probe branch (not part of this PR) nulls the added fast path -
if (ascii_only) { return std::string(); } - and the suite goes red:

21/104 Test #21: ninfer_qwen3_6_frontend_test ...***Failed
  priority BPE changed rank or leftmost merge semantics
  boundary-aware tokenizer changed crossing-token or result-order semantics
  bounded tokenizer output is not the exact prefix of unbounded tokenization
  text frontend did not render/tokenize chat
  text frontend did not preserve prefix/thinking identity
  ... 12 assertions in all

So the suite does observe this code path - through 12 distinct assertions - and the 104/104 above is
a real result rather than an absence of coverage. It is worth being precise about how thin that
coverage is: exactly one of the 104 tests dies under the probe. Changing unicode.cpp relinks 23
of the test executables, but only ninfer_qwen3_6_frontend_test actually checks what
normalize_nfc returns.

clang-format, repo .clang-format, version 23.1.0, counted as "lines the formatter would
change in src/text/unicode.cpp": 0 before the change, 0 after. The file is clean, as is the one
it replaces.

Performance

Operator and caller, isolated

Ratios to base, pass 1 / pass 2. The non-ASCII bytes column is the reason each row lands where it
does.

fixture	bytes	non-ASCII bytes	normalize_nfc	Tokenizer::encode
asciien_chat_32k - ASCII chat	33 824	0	0.014 / 0.014	0.858 / 0.858
asciien_plain_32k - ASCII, single turn	32 768	0	0.014 / 0.014	0.863 / 0.861
chat_code_32k - chat with code	33 824	0	0.014 / 0.014	0.836 / 0.834
plain_code_32k - code, single turn	32 768	0	0.014 / 0.014	0.842 / 0.844
chat_en_8k - short ASCII chat	8 516	0	0.015 / 0.014	0.855 / 0.855
chat_unique_32k - ASCII, rare tokens	33 824	0	0.016 / 0.016	0.778 / 0.773
unique_ascii_32k - ASCII, rarest tokens	32 769	0	0.016 / 0.016	0.775 / 0.773
uniqrand_chat_32k	33 824	0	0.015 / 0.015	0.860 / 0.854
uniqrand_plain_32k	32 769	0	0.015 / 0.016	0.866 / 0.865
chat_en_32k - repo documentation	33 784	9 156	1.019 / 1.025	0.923 / 0.924
chat_en_128k - same, long	134 986	28 311	0.998 / 0.996	0.932 / 0.929
plain_en_32k	32 768	> 0, not counted	1.003 / 1.004	0.974 / 0.971
plain_en_128k	131 072	> 0, not counted	0.994 / 0.994	0.992 / 0.991
chat_mixed_32k - languages + emoji	33 787	21 829	0.999 / 0.992	0.995 / 0.994
chat_zh_32k - Chinese	33 792	32 736	0.977 / 0.978	0.997 / 0.997
chat_ru_32k - Russian	33 776	29 984	1.011 / 1.012	1.000 / 0.995
nonascii_only_32k - not one ASCII byte	32 767	32 767	0.950 / 0.947	0.990 / 0.994
ascii_then_high_32k - worst case	32 767	2	1.011 / 0.984	1.002 / 0.999

The two plain_en_* rows carry non-ASCII bytes - their operator ratio says so - but the exact count
was not recorded for them; every other row's count is exact.

Nine pure-ASCII fixtures land at 0.014..0.016, that is x62 to x71 - on ad0f3d38, which is the
base this table was taken on. Re-measured on 487f8977 the multiplier is x46 to x57; the
subsection below prints the re-run beside it. In absolute terms on
asciien_chat_32k, normalize_nfc goes 0.7948 ms -> 0.0111 ms; on chat_en_8k,
0.1990 -> 0.0029 ms. Host Tokenizer::encode on those same fixtures is 0.773 to 0.866, i.e.
-13 to -23%.

The probe re-measured on the new base

Two arms built from the same 487f8977 tree - base carrying src/text/unicode.cpp from the bare
base, the arm carrying this branch's version - plus a copy of base as a null control. Six rounds,
arms alternated between rounds, round 1 discarded, median over the remaining five of the minimum of
15 repeats, a fresh process per cell. tokenizer.cpp and the vendored utf8proc are the same blob
in both arms (md5 equal).

fixture	bytes	base, ms	arm, ms	ratio	multiplier	null
prose_ascii_256k	262 144	2.7921	0.0489	0.0175	x57.1	1.0085
prose_ascii_64k	65 536	0.6655	0.0123	0.0186	x53.8	1.0158
code_ascii_64k	65 536	0.6555	0.0122	0.0187	x53.5	1.0063
conv_ascii_64k	65 536	0.6448	0.0122	0.0189	x52.9	1.0011
conv_plain_64k	65 536	0.6274	0.0122	0.0194	x51.5	1.0107
repeat2_ascii_64k	65 536	0.5855	0.0122	0.0208	x48.1	0.9682
unique_ascii_64k	65 536	0.5702	0.0122	0.0212	x47.2	1.0161
longword_unique_64k	65 536	0.5641	0.0122	0.0216	x46.3	1.0305

x46 to x57, against the x62 to x71 claimed above. The non-ASCII fixtures behave as before and
stay inside the noise band - russian_64k 0.9750, nonascii_only_64k 0.9739, cjk_only_64k
1.0136, mixed_64k 0.9770, and the worst case ascii_then_high_64k 0.9481, against a null control
of 0.9986 to 1.0240 on those same fixtures. Tokenizer::encode re-reads 0.776 to 0.870 on
ASCII, i.e. -13 to -22%, which is the same statement as the -13 to -23% above.

Why the multiplier moved, as far as it can be established - and it cannot be established. The
input is not the variable: src/text/unicode.cpp is the same blob on both bases,
193b92ad1e3267a27421b6dcaf482b36ad9bf426 on ad0f3d38 and on 487f8977 alike, and
third_party/utf8proc is unchanged as well, so the two runs compiled byte-identical sources. What
did change between them is the compiler and the host together - g++ 13.3.0 on the machine the
first table was taken on, g++ 15.2.0 on the machine the re-run was taken on, on different silicon.
The whole ratio is set by how fast the base arm scans, so a faster base arm shrinks the multiplier
without anything happening to this change at all, and both a newer compiler and a different CPU
would do that. That is a plausible account and not a measurement: the two were not separated,
because g++ 13 is not installed on the machine that has the newer one, so no A/B over the compiler
alone was run. Neither this report nor its author knows which of the two moved the number.

What this does and does not do to the claim. The mechanism is unchanged - a pure-ASCII string is
returned untouched instead of being decoded, mapped and recomposed - and it reproduced on two hosts,
two compilers and two fixture sets, with the null control flat in both. What is not stable across
hosts is the size of the multiplier, so this report quotes x46 to x57 and treats the old x62 to
x71 as a figure of one particular host. Anyone reproducing on a third machine should expect a third
number in that neighbourhood; the load-bearing statement is Tokenizer::encode at -13 to -22%, which
did reproduce.

The nine rows that carry non-ASCII bytes are in the same table and the same runs, and the operator
does not move on any of them: 0.947 to 1.025, every cell inside the measured 0.902..1.025 band.
Their encode column is a different matter and is the next subsection: two of them do move, because
encode normalises per segment rather than per prompt.

The worst case is measured rather than assumed. ascii_then_high_32k is the most expensive shape
a prompt can construct: ASCII for its whole length with a single high byte at the very end, so the
scan walks all 32 KB and buys nothing. It reads 1.011 / 0.984 - inside the band, i.e. below this
bench's resolution. An earlier and cruder measurement of the same shape put it at 1.8%; on the
product vocabulary with the noise floor measured, it does not resolve at all.

The fast path fires more often than a whole-prompt ASCII test suggests

This is the finding that changes how the applicability should be read. Look at chat_en_32k: it is
27% non-ASCII by byte (9 156 of 33 784), so normalize_nfc over the whole document is 1.019 -
nothing. But encode on that same text is 0.923.

The reason is that encode calls normalisation once per stretch of ordinary text between added
tokens (src/targets/qwen3_6/impl/frontend/tokenizer.cpp:652 - the only tokenizer.cpp in the
tree, and not under src/text/, where unicode.cpp lives), not once per prompt. Most turns of a
conversation are pure ASCII even when one of them carries Cyrillic, so most of those calls take
the fast path. A "is the whole
prompt ASCII" test understates where this helps.

The effect has no length threshold

Sweeping the distance between control tokens from 128 bytes to 256 KB, encode holds at
0.843 to 0.859 across the whole range - a flat band, ten points, both passes. There is no
crossover to argue about and no constant to justify.

End to end, both regimes

Cold prompt, prefill on the card: no measurable effect, and that is the honest answer. 48 CLI
runs, all 48 points recorded quiet:

prompt	tokens	prepare, ms (base)	prefill, ms	prepare as % of total
asciien_chat_32k	8 475	8.52	767	1.10%
chat_code_32k	8 147	6.85	742	0.91%
chat_en_32k	7 952	9.27	728	1.26%
chat_en_128k	32 516	32.9	3 600	0.91%

Host preparation is 0.9 to 1.3% of the run, and at three runs per cell the prepare column does
not resolve a 2 ms effect: the arm that must be faster reads slower on three of the four fixtures
(+2.4, +1.1, +2.0 ms) and faster on the fourth (-1.1 ms). Total time is flat - ratios 1.0000, 0.9987,
1.0000, 1.0000. The correct statement for cold prompts is "no measurable change", not a small
gain.

Hit prefix cache: the whole difference is visible. The instrument here is apps/ninfer-serve,
not ninfer_bench, and the distinction is what makes the rows mean anything, so it is stated
before the numbers. ninfer-serve ships with prefix reuse on - src/serve/serve_options.h:50,
bool allow_prefix_reuse = true;, byte-identical on ad0f3d38 and 487f8977 - and turns it off
only for an explicit --no-prefix-reuse (src/serve/serve_options.cpp:286). The command below
passes no such flag, so both arms ran at the shipped default with the cache live; that is the
configuration these four cells describe, and it is the configuration a deployed server has.
ninfer_bench, by contrast, hard-codes allow_prefix_reuse = false at
bench/targets/qwen3_6_27b/ninfer_bench.cpp:65 on both bases, which is why no ninfer_bench
number anywhere in this report is a prefix-hit number and none is offered as one.

That the cache was permitted is a code fact; that it was hit is a logged fact, and the log is
the evidence: ninfer-serve --request-log-jsonl, 40 requests per arm, two passes, 16 sessions.
Request 1 prefills 8 475 tokens (762.6 ms); requests 2..40 record
computed_prefill_tokens=0 and prefix_cache_hit_tokens=8475 for this fixture (the second
fixture in the same log reports 7952) - a complete hit, every token of the
prompt served from the cache - and prefill collapses to 1.6 ms.

prompt	arm	tokenize, ms	TTFT, ms	tokenize as % of TTFT	TTFT vs base
asciien_chat_32k	base	14.18 / 14.36	17.5 / 18.0	81% / 80%	-
	this branch	11.94 / 11.88	15.5 / 15.3	77% / 78%	0.884 / 0.852
chat_en_32k	base	13.59 / 13.52	17.2 / 17.1	79% / 79%	-
	this branch	13.07 / 12.15	16.3 / 15.4	80% / 79%	0.951 / 0.901

Tokenisation is 79 to 81% of TTFT there, and the change takes 4.9 to 14.8% off TTFT - four
cells, 0.852, 0.884, 0.901, 0.951. The two instruments agree on the mechanism: the isolated bench
puts encode at 0.858 on this fixture, the server's own tokenize record at 0.827..0.842.

Scope of that claim, stated exactly. It is a statement about ninfer-serve at its default
allow_prefix_reuse = true, on requests the log shows were served entirely from the prefix cache.
It is not a statement about ninfer_bench at any setting, and it does not follow from any
ninfer_bench row in this report. These server rows were taken on ad0f3d38 and have not been
re-measured on 487f8977; what has been checked on both bases is only that the option they depend
on reads the same, which it does.

So the honest headline is conditional. This is worth nothing on a cold long prompt, where prefill
dominates by two orders of magnitude, and it is worth 5 to 15% of TTFT on cached-prefix traffic -
agents and chat UIs that resend a growing conversation, which is exactly the traffic the prefix cache
exists for. Both numbers are above; neither is presented without the other.

Tradeoffs

• The only cost the change can carry is the scan itself, and it is bounded above by the worst
  shape a prompt can take. Measured, that shape is inside the noise band on this bench: 1.011 /
  0.984 on ascii_then_high_32k, and 0.947..1.025 across all nine non-ASCII fixtures.
• Nothing else moves. Workspace, resident memory, transfers and every device execution path are
  untouched; the numerical behaviour of the tokenizer is unchanged bit for bit.
• Memory traffic is strictly lower on the fast path, not merely relabelled: one malloc/free
  pair and one intermediate buffer disappear per call.
• Where it does not apply, it does nothing - and unlike a threshold constant, there is no value
  to fit and no artifact to fit it on. The predicate is a property of the input, not of the model.

Reproduction

# build, both arms from one tree
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_CUDA_ARCHITECTURES=120a -DBUILD_TESTING=ON \
      -DNINFER_BUILD_APPS=ON -DNINFER_BUILD_BENCHMARKS=ON
ninja -C build

# suite, no weights in the environment: 104/104 on both arms
ctest --test-dir build --output-on-failure

# the six weight-gated tests: 4 pass, 1 skips, prefix_real fails identically on both arms
NINFER_QWEN3_6_27B_WEIGHTS=... NINFER_QWEN3_6_35B_A3B_WEIGHTS=... \
ctest --test-dir build --output-on-failure \
  -R "27b_prefix_real|27b_score_real|27b_load_plan|35b_a3b_real|35b_a3b_dflash_real|35b_a3b_dflash_load_plan"

# the coverage probe: replace the fast-path return with `return std::string();`
# and the suite goes 103/104 - ninfer_qwen3_6_frontend_test, 12 assertions

# server rows: prefix cache hit, 40 requests per arm
build/apps/ninfer-serve qwen3_6_35b_a3b.ninfer --request-log-jsonl <file>
# the artifact path is positional (src/serve/serve_options.cpp:139); the server has no --weights flag

The operator and encode rows come from a standalone harness that compiles the tree's own
src/text/unicode.cpp, src/targets/qwen3_6/impl/frontend/tokenizer.cpp and vendored utf8proc and
calls uni::normalize_nfc and fi::Tokenizer::encode directly, with the tokenizer built from the
product artifact's own vocabulary. The harness is research scaffolding and is not part of this PR.

Checks not run, and the resulting limitations

• The share of pure-ASCII text in real production traffic was not measured. Every number above is
  against fixtures whose composition is stated explicitly in the table; carrying them over to a
  traffic mix without measuring that mix would not be sound. The per-segment finding says the
  applicable share is larger than a whole-prompt test suggests, but it does not say how large.

• The cold-prompt regime is a null result, not a small win. The CLI prepare column at three
  runs per cell cannot resolve 2 ms; it is reported as flat rather than fitted.

• The performance numbers were taken on the measurement host, ctest on the test host. No
  performance figure in this report was taken on the test host and none is compared across the two.

• One microarchitecture per figure, and two toolchains across editions. The rows in this
  report are g++ 13.3.0 at -O3 on Zen 2 and Zen 4; the re-measurement quoted above ran under
  g++ 15.2.0 on a different host, and the two moved together, so this report does not separate
  them. The scan is a
  plain byte loop, so the shape should carry, but the absolute scan rate is these machines'.

• The server rows are one artifact and one prompt shape per fixture, 40 requests per arm per
  pass. The within-arm spread of the server's own tokenize record is large (41 to 81% between min
  and max), which is why medians over 40 requests and two passes are reported rather than single
  runs.

• ninfer_qwen3_6_27b_prefix_real_test was red on a140e7ae before this branch existed, and is
  not red any more. On a140e7ae it failed identically on both arms and was reported rather than
  fixed. Upstream b8786751 (fix(runtime): correct aliased state ownership) is the maintainer's
  fix for exactly that failure: re-measured on one host with the same artifacts, the test fails on
  bare a140e7ae in 49.31 s with the same message and counters (path=5 reused=267) and
  passes on the newer base in 126.77 s, together with the other five weight-gated tests, 6 of 6.

  Retracted, 2026-09-05, and the correction runs the other way. The line above used to read:
  "ninfer_qwen3_6_27b_prefix_real_test is red on a140e7ae before this branch exists. It is
  reported here, identically on both arms, and left alone." That was true on a140e7ae and is
  false on the base this branch now targets. On today's base it is a live check, and a failure
  of it on this arm would be this arm's, not a known upstream red to be reported around. The
  run that governs is this arm's own on the current base, below.

• ctest on the base this branch targets, taken 2026-09-05 from one build directory, both arms.
  Base ad0f3d38 (tree af48897f): 104 of 104 passed, 0 failed, six weight-gated tests skipped.
  This branch ac6a69e3 (tree ced69cfe): 104 of 104 passed, 0 failed, the same six skipped.
  Same CMake configuration for both, -DBUILD_TESTING=ON, CUDA 13.1; the arm was produced by
  checking the branch out in the same worktree and rebuilding incrementally, so the two runs differ
  in nothing but the commit. Both runs were taken under the host's GPU lock, one arm at a time.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-07T09:06:20.103551Z	52cace9	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.