Full-FT offload is host-bound: measure resident device AdamW and update the placement model #34

Problem

After resident host buffers (#26), 8B full FT on one RNGD runs at 6.2–6.4 s/step; two cards are slower (9.26 s/step, #30) because host copies and host AdamW (24 cores), not the PCIe links, set the step time. The paper's §5.3 optimizer-placement model still uses link bandwidth as the cost.

Evidence

experiments/2026-10-21-two-card-split.md: summed D2H 4.0 → 5.2 s and AdamW 7.1 → 9.7 s with two cards; model predicted 5.6 s, measured 9.26 s.

Scope

1. Precompile the resident AdamW graphs on a dedicated CPU node (npu-lora compile --full --resident-layers N; did not finish in 2.5 h on a shared node).
2. Measure 1 and 2 cards with about 8 resident layers per card.
3. Replace the link term in the placement model with host copy + host AdamW throughput.

Done when

Step time with resident layers is measured on RNGD and the model predicts it within 15 %.

Owner

gspark-etri