[jishnuvenugopal]

Thank you, @ashhart, for cache_fit.budgetBeside (zig/src/native/cache_fit.zig), which Gemma 4 and the 27B already use for the memory their
streams take on first use, and for 1.1.1's change that gives the 27B's DeltaNet rollback scratch only to the slots that need it, the same idea this
applies to Nemotron's Mamba states. Thank you, @akol1, for the /memory endpoint (zig/src/server/routes.zig) that reports the process footprint and
its peak beside the prompt cache: it made the gap between the footprint at startup and after the first request directly visible.

What this changes

nemotron.backend.Metal.init (zig/src/families/nemotron/backend.zig) sized state.Pool for rows + o.streams + 1 Mamba slots, 67 at two streams
(3.10 GiB), although a stream holds at most two (its state and its pending forward) and a lone GPU round a third. It now takes
state.poolSlots(o.streams), 2 * streams + 3, and o.streams + 2 replay ids instead of o.streams + 4 (only a stream's cache takes one); a
test fills a pool to that worst case and shows the next slot is the last. With one spare
left, queueFn and verifyFn now return a fresh slot with errdefer when the round fails after taking it (before, the slot stayed out of the pool
for the life of the process while the lane host failed the round and served on), as snapshot.restore already does. Pool.init and
Scratch.init (state.zig) zero-fill their buffers, so their pages are in the process footprint before cache_fit reads it whether the
window-cost table was timed or read from disk. native/metal.zig then calls cache_fit.budgetBeside with the KV and window buffers each lane
allocates on its first prompt (Cache.bytes at the full window, checked against Cache.init by a test), as Gemma 4 and the 27B do. So a server
without --context (the 262,144-token window) now reserves 1.75 GiB a lane for those buffers: on a 32 GB Mac it keeps no prompt cache, and an
explicit --prompt-cache-gib stops at startup with the cap message (--prompt-cache-over-cap still overrides it).

Verified outcome

• Why: on a 32 GB Mac the server planned its prompt cache from the memory in use at startup, before its streams had used any. With a saved
  timing table it planned 1.7 GiB, then the first request added 4.0 GiB and the server passed the 22.4 GiB (70%) limit by 1.25 GiB. On a launch
  that had to time the table it saw 22.2 GiB already in use and kept no prompt cache at all.
• What was done: the pool of Mamba states is sized to the streams that can exist and gets back any state a failed round took, its memory is
  counted before the cache is planned, and the memory each stream will take on first use is counted beside it.
• Result: on this M4 the server now stays under the limit at its peak (20.9 GiB, 2.8 GiB less than before) with either kind of launch, and
  the launch that times the table keeps a 0.7 GiB prompt cache instead of none. A saved-table launch plans 0.9 GiB instead of 1.7, because 1.7
  was never available, and a server left at the model's full context window plans none, because two full-length streams would need that memory.
  Replies are token for token the same as before, and startup, decode speed and prompt speed are unchanged.

Receipt

• Environment: branch fix/nemotron-prompt-cache-budget on main at d1018b3 (1.1.1), Zig 0.17.0, Apple M4 (32 GB, limit 22.4 GiB), macOS 27.0.1,
  TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit at d9d758fb. serve --parallel 2 --context 16384, default prompt-cache budget, each
  launch with its own HOME. The table and exactness runs below were taken on d46d17a (1.1.0) with load averages 1.0 to 2.3, then repeated once
  per row on d1018b3 (load 1.6 to 2.6): the same budgets, the same peaks to two decimals (23.66, 22.89, 20.87, 20.78 GiB) and the same token hashes.

• Memory and prompt cache, bench/p2_memory.py (/memory footprint, prompt-cache log), four launches per build: saved table (kept) and an empty
  HOME that times a table (fresh), alternated, each pair repeated and identical to two decimals. GiB:

  Build, launch	Budget	Ready	After 1 request	8 prompt states filled	Peak	Prompt states kept
  Main, kept	1.7	18.73	22.75	23.61	23.65	9 (789 of 1,710 MiB)
  Main, fresh	0.0	22.21 to 22.22	22.75 to 22.76	22.83	22.88 to 22.89	none
  This branch, kept	0.9	19.28	19.97	20.82	20.87	9 (789 of 919 MiB)
  This branch, fresh	0.7	19.44	19.97	20.74	20.78	8 (701 of 717 to 723 MiB), 1 evicted

  Loaded in 11.1 to 11.5 s (kept) and 39.9 to 40.8 s (fresh) on both builds.

• Without --context (262,144-token window, --parallel 2, saved table, d1018b3): main plans 1.7 GiB (18.7 GiB in use once loaded); this branch
  plans 0.0 GiB (20.2 GiB in use once loaded and 3.5 GiB its streams take on first use). With --prompt-cache-gib 1, main starts with 1.0 GiB and
  this branch exits with the cap message. The 0.9 GiB more in use once loaded is the two attention scratch sets (the backend's and the MTP head's)
  whose split-K partials are sized by the window, now zero-filled: about 0.5 GiB each at this window and 0.03 GiB at 16,384.

• Exactness: bench_openai.py --tokens 256 fibonacci-raw c91869ce1808 and gpu-chat-no-think 5ef050084ac3 at --parallel 1, 2 and 4, the same
  hashes as main. bench_concurrent.py --levels 1,N --alone --serial at temperatures 0 and 1.0: 0 unequal, 0 failed at all three. Resumed
  prompts, bench/p2_restore.py (a 5,974-token prompt: cold, repeat, N at once, N-1 arriving while another stream decodes) on this branch and on main at
  --parallel 1, 2 and 4: 5,968 tokens cached on every resume, every reply equal between the builds including a 160-token reply decoding beside
  the resumes, 0 errors and no slot or replay-id error in any server log. tf-train-check on the 30B checkpoint (streams = 2, 7 Mamba slots vs 67 on
  main): loss 12.34732 (with gradients 12.34732) and every gradient norm and finite-difference ratio across all 11 checked layers (51 down to 0)
  equal to main (ok: worst ratio off 1 by 0.102).

• Decode speed before and after, bench_openai.py --tokens 256 --reps 2 --temperatures 0 at serve --parallel 2 --context 16384, four alternated
  launches (this branch, main, this branch, main) on one shared window-cost table: fibonacci-raw 48.92, 47.58 tok/s (TTFT 0.123, 0.126 s) on this
  branch vs 48.77, 46.48 tok/s (TTFT 0.123, 0.130 s) on main; gpu-chat-no-think 48.25, 46.29 tok/s (TTFT 0.315, 0.327 s) on this branch vs
  47.77, 47.56 tok/s (TTFT 0.320, 0.322 s) on main. Prompt speed: n/a, the prompt pass is unchanged.

• Peak footprint is in the table above. 0 leak lines in every server log.

• Tests run, uncached, on this branch and on main at d1018b3: zig fmt --check build.zig zig, git diff --check origin/main, zig build -Dcpu=apple_m1,
  zig build test -Dcpu=apple_m1 (4934/4955 passed, 21 skipped; main 4927/4948, 21 skipped), zig build test-kernel-contract (65/65), zig build test-golden (306/319 and 26/26 equal, 0 unexpected differences, as on main), zig build fuzz-cortheon (11/11), flashnext_batch_serve_test.py (8 OK),
  cross-builds -Dtarget=x86_64-linux-gnu -Dcpu=x86_64_v3 and -Dtarget=aarch64-linux-gnu, zig build native -Dcpu=apple_m1 -Dversion=1.0.0, and
  lean_new.py (444 problems, 444 on the base, 0 new). The 7 more tests are the 3 new ones and 4 existing ones the macOS test binary now reaches
  through backend.zig (split-K partitions follow MLX's rule, tree tables: depths and ancestors by depth, a tree's stepped lanes by depth, and each lane's ranks from its parent's step, a state's bytes: fixed Mamba states, then a key and value row per KV head and attention layer a token),
  all passing. New, each failing before its change: a pool of poolSlots holds every stream's state and pending forward, a lone round's third, and one spare (error.StateSlotsFull when poolSlots is 2 * streams + 2); Cache.bytes matches the Metal buffers Cache.init allocates (expected 2816, found 4198144 without the KV and MTP terms); a verify refused after a window took its Mamba slot gives the slot back to the pool
  (backend_slots_test.zig, a backend with no weights; expected 6, found 5 without the errdefers, and its grammar case alone fails the same way
  when the slot is taken before the grammar bits are staged). They need a Metal device: they run in the macOS test binary and skip without one, and
  the ubuntu and Windows jobs do not build the Nemotron Metal family.

• Not run, and why: M1 to M3, M5 and CUDA. The change is in the Nemotron Metal backend and the Metal host. Other memory sizes: the fix is arithmetic on
  what is wired, but the numbers above are one 32 GB machine. queueFn's errdefer has no test: the only call after its slot that can fail is the
  command buffer's submit, which needs the loaded model.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Above.)
• No precision traded for speed. (No kernel or arithmetic changes.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt.
  (n/a: no kernel is added or changed.)
• Prompt processing is no slower than the last release. (n/a: the prompt pass is unchanged.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (M4 32 GB only.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (Above.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.