[jishnuvenugopal]

Thank you, @ashhart, for the GPU round and the head-depth rule in round_depth.zig that this refines, and for the
window-cost table the Metal backend already measures at load (copy_lanes.Costs): it is what lets the depth be priced
instead of fixed.

What this changes

round_depth.Depth (zig/src/families/nemotron/round_depth.zig) started a GPU round at all 8 levels and dropped to a
fixed 4 when the levels past 4 brought under a tenth of the tokens. That treats a verify window of up to 5 rows as
costing about one row, which the backend's window-cost table contradicts on a chip whose row projections run without
tensor units (zig/kernels/metal/core/row_projection.metal): on an M4 each row adds 7 to 9 ms to a 20 ms window.
Depth.init now takes the table and the head step time from gpu_round.run and prices each depth s as
costs.at(s + 1) + s * head_ms. It starts at the depth with the most expected tokens per ms under a 0.75 per-level
survival prior, or at min(4, most) when the table is empty. Every 8 rounds Depth.next compares the realized tokens
per ms of each depth up to the current one, and goes one level deeper only when the current depth's full windows would
pay for the extra row. A fixed depth is unchanged. round_depth.zig joins nemotron.zig's test block for its two new
tests.

This is the depth of one stream's GPU round on Metal. It does not touch the depth rule of shared rounds, where several
streams verify together, which #1059 is about (thanks, @plotarmordev).

Verified outcome

• Why: on an M4, drafted decoding of one request ran between 5% faster and 9% slower than plain decoding: on
  essay-style text it was 9% slower (42.8 against 47.2 tok/s). A round that drafted 4 or 8 levels deep paid for verify
  rows whose tokens were mostly thrown away.
• What was done: the round now chooses how deep to draft from the chip's own measured cost of each window size, and
  goes deeper only when the deeper rows keep paying for themselves.
• Result: on this M4, drafted decoding of one request is 18% to 27% faster than before with tensorfold run (51.1
  to 60.1 tok/s on code, 42.8 to 54.3 on essay text) and 16% to 26% faster under serve, 26% to 33% faster on
  high-acceptance copy prompts (78.6 to 99.0 tok/s copying a paragraph, 81.5 to 108.3 copying an 8K list), and now 15%
  to 139% above plain decoding on every prompt measured. Every reply is token for token the same as before. Four
  requests at once, prompt speed and memory are unchanged. Not measured: any chip with tensor units (M5), where this
  changes the depth a round starts at.

Receipt

• Environment: branch perf/nemotron-round-depth-costs on main at d1018b3 (1.1.1), Zig 0.17.0, Apple M4 (32 GB),
  macOS 27.0.1, TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit at d9d758fb. Load averages 1.4 to 2.9 (no
  other workload), recorded per run.

• Exactness: tensorfold run, 256 tokens, temperature 0, four prompts (fibonacci-raw, gpu-chat-no-think,
  sql-chat-no-think, essay-chat-no-think). Each prompt's plain run (--no-drafts), fixed depths 1 to 4, and two runs of
  main and two of this branch at --gpu-round 8: all 8 drafted runs per prompt equal the plain run's 256 tokens (32 of
  32: 54195f74313a, dc6df8c9622e, bdba18f79184, 498e92e027a3). serve: tools/bench_openai.py token_sha
  c91869ce1808 and 5ef050084ac3 on all four launches, tools/bench_concurrent.py --alone --serial 0 unequal, 0 failed on all four, and bench/d4_copy.py text_sha 8d5581a06df6, fd621d8d4fc4, dcd1880a0a4f (server token_sha
  5414bb3db7e3, 15045d7e74d1, eba0eb1969af) equal across all 16 drafted and plain runs per prompt.

• Decode before and after, tensorfold run --max-tokens 256 --warmup, one window-cost table timed by the first drafted
  run and kept by every later run of both builds (1 to 5 rows: 19.74, 27.22, 35.96, 43.36, 52.50 ms; head step 1.505
  ms), tok/s (the depth the rule chose, in rounds):

  Prompt	Plain	Fixed 1, 2, 3, 4	Main, run 1	Main, run 2	This branch, run 1	This branch, run 2
  fibonacci-raw	48.4	59.1, 54.6, 64.2, 53.3	51.1 (4: 46, 8: 24)	51.0	60.1 (1: 68, 2: 27)	60.1
  gpu-chat-no-think	49.0	59.9, 57.9, 54.8, 49.6	48.7 (4: 104, 8: 9)	48.7	58.5 (1: 113, 2: 39)	58.4
  sql-chat-no-think	49.1	60.4, 59.8, 55.5, 51.0	49.3 (4: 101, 8: 9)	49.2	60.1 (1: 102, 2: 42)	60.1
  essay-chat-no-think	47.2	55.7, 53.3, 48.7, 44.0	42.8 (4: 116, 8: 13)	42.7	54.5 (1: 118, 2: 40)	54.1

  The rule picks a depth from the table and the prompt's own acceptance; on fibonacci-raw a fixed depth 3 is faster
  still (64.2) and a fixed depth 1 is 0.3 to 1.4 tok/s ahead on the other three, so a better prior is possible, but no
  single fixed depth is best on all four prompts, and a fixed 1 would be wrong on a chip whose table is flat.

• Decode before and after, serve --parallel 4 --context 34816 --prompt-cache-gib 0, four alternated launches (this
  branch, main, this branch, main) on one shared window-cost table timed by main, measured with tools/bench_openai.py --tokens 256 --reps 2 --temperatures 0, tools/bench_concurrent.py --levels 1,4 --tokens 128 --reps 2 --temperatures 0 --alone --serial, and bench/d4_copy.py (four more alternated launches on the same table; 256 tokens, 172 for the
  paragraph, temperature 0, median of two reps), tok/s:

  Workload	Main (launches 2, 4)	This branch (launches 1, 3)
  bench_openai.py fibonacci-raw	49.51, 49.56 (TTFT 0.128, 0.127 s)	57.78, 57.59 (TTFT 0.129, 0.128 s)
  bench_openai.py gpu-chat-no-think	46.68, 46.39 (TTFT 0.326, 0.331 s)	56.33, 56.25 (TTFT 0.325, 0.325 s)
  bench_concurrent.py code, 1 stream (128 tok)	47.8, 47.8	58.5, 58.3
  bench_concurrent.py chat, 1 stream (128 tok)	45.5, 45.6	57.6, 56.8
  bench_concurrent.py code, 4 streams (aggregate / steady)	88.8, 89.4 (95.1, 95.6)	89.8, 90.1 (96.2, 96.5)
  bench_concurrent.py chat, 4 streams (aggregate / steady)	78.3, 74.1 (91.9, 87.2)	75.3, 77.5 (87.7, 92.6)
  d4_copy.py copy-paragraph, drafted (plain: 46.3 to 47.3)	76.60, 80.53 (21 to 22 rounds)	98.26, 99.72 (15 rounds)
  d4_copy.py copy-8k-list, drafted (plain: 45.0 to 45.9)	81.61, 81.45 (25 rounds)	107.63, 108.90 (17 rounds)
  d4_copy.py chat-control, drafted (plain: 46.8 to 47.3)	48.52, 48.35 (106 rounds)	57.88, 57.71 (147 to 148 rounds)

  Four streams at once is unchanged within noise: that path does not use the GPU round's depth rule.

• Prompt speed before and after, tools/bench_cache.py --contexts 2048,8192,32768 --max-tokens 1 --require-cold at
  serve --parallel 4 --context 34816 --prompt-cache-gib 0 on one shared window-cost table (cache_state: "cold" and
  identical token_sha 7d7179c146d0, 28c7d2ef7f39, ffa6059b954a on every run), median TTFT (and tok/s). Taken on
  d46d17a (1.1.0 and docs), this branch's previous base; the prompt pass does not read the round depth, and
  d1018b3 changes only imports in the Nemotron Metal path:

  Build	2,048 (5 cold runs)	8,192 (5 cold runs)	32,768 (3 cold runs)
  Main	4.566 s (448.6 tok/s)	19.302 s (424.4 tok/s)	92.033 s (356.0 tok/s)
  This branch	4.618 s (443.5 tok/s)	19.371 s (422.9 tok/s)	90.683 s (361.3 tok/s)

• Peak physical footprint (vmmap -summary, after the benchmark suites): 23.0 GiB current and 23.0 GiB peak on all
  four decode launches of both builds; 22.3 to 22.4 GiB current and 22.7 GiB peak on the prefill launches. 0 leak lines
  in every server log.

• Tests run, uncached, on this branch and on main at d1018b3: zig fmt --check build.zig zig, git diff --check origin/main, zig build -Dcpu=apple_m1, zig build test -Dcpu=apple_m1 (4930/4951 passed, 21 skipped; main
  4927/4948, 21 skipped), zig build test-kernel-contract (65/65), zig build test-golden (306/319 and 26/26 equal, 0
  unexpected differences, as on main), zig build fuzz-cortheon (11/11), flashnext_batch_serve_test.py (8 OK),
  cross-builds -Dtarget=x86_64-linux-gnu -Dcpu=x86_64_v3 and -Dtarget=aarch64-linux-gnu, zig build native -Dcpu=apple_m1 -Dversion=1.0.0, and lean_new.py (443 problems, 444 on the base, 0 new). The 3 more tests are the 2
  new ones and 1 existing one the macOS test binary now reaches through round_depth.zig (copy rows that keep landing price a long window; ones that miss price none), all passing. New, in round_depth.zig: a steep per-row window table starts at 1 level and steps up only when windows saturate (expected 1, found 4 when Depth.init starts at
  main's @min(4, cap), the only failure) and a flat tile window table starts at 4 levels, steps up on full windows, and drops back when deep levels miss (an invented flat table, see below). They run in the macOS test binary; the
  ubuntu and Windows jobs do not build the Nemotron Metal family.

• Not run, and why: no M5 or other tensor-unit chip here. On such a chip the rows of a window share one tile, so the
  table is flat to 16 rows and this rule will start deeper than 1; main starts at 8 and drops to 4. Whether the start
  the table picks matches or beats main there is unmeasured, and the flat table in the test is invented. Please run
  tensorfold run --gpu-round 8 on an M5, before and after, before this lands. M1 to M3 and CUDA: not run (CUDA is
  not touched).

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  (Above: drafted equals plain 32 of 32 on the CLI; bench_openai.py, bench_concurrent.py --alone --serial, and
  d4_copy.py all match across main and this branch.)
• No precision traded for speed. (No arithmetic changes: the depth only decides how many levels are drafted, and
  every reply's tokens are equal.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's
  SUMMARY line is in the receipt. (n/a: no kernel is added or changed.)
• Prompt processing is no slower than the last release. (Above: bench_cache.py --require-cold at 2K, 8K and 32K.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (M4 only; M5 is
  the open question above.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (Above.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.