Repository navigation
perf: reduce CPU staging overhead in portable ATOM KV import #205
Description
Problem
Portable ATOM KV import performs a full positions/heads contiguous transpose and reverses it before encoding, causing redundant host copies. BF16→DLFloat lookup also dominates CPU staging for larger exports.
Evidence
Qwen3-4B512-token export CPU-only alternating tests: preserving the permuted view reduces about127ms to111ms; vectorized bit arithmetic matches all65536BF16 bit patterns and reduces full-array encoding about96ms to85ms on the measured host. These are not hardware/import speedup claims yet.
Scope
Remove redundant layout copies and test exact vectorized encoding in the portable ATOM backend. Retain geometry/ownership/cancellation checks. Exhaustively compare the existing encoding contract, verify real ATOM slot readback and decode logits, and measure alternating warm hardware imports before adopting.
Done when
Exact CPU conversion/layout tests and actual-device parity pass; hardware timings support the improvement or the candidate is rejected. No asynchronous DMA/overlap claim.
Activity
gspark-etri commentedon Oct 24, 2026
Native serving integration request (follow-up to #205/#206)
Problem: the native
AtomKVBlocks.import_layer_spanpath used by the current PR266 B4 serving experiments still calls its separate lookup-basedbf16_to_dlfloat, while PR206 has an exact arithmetic implementation in the portable adapter. Please consider sharing the validated encoder between these paths. This is an owner-facing request under CONTRIBUTING §8; no production engine patch was made from the research repository.New evidence: Qwen3-0.6B, B4/TP1/k5, context9216, width8192, c16, RNGD→ATOM, caches off, rebel-compiler0.11.0/optimum-rbln0.11.0.post1. First component-instrumented whole arm passed132 HTTP checks. Across66 8K requests, host import took62.986s, lookup conversion43.902s and SDK-write calls2.079s. First stream arm also passed132 checks with identical references/device binding:8K import56.337s, conversion45.315s, SDK writes2.368s. These are host aggregates, not isolated DMA times. Extra instrumentation changed some throughput observations; this does not prove conversion caused the earlier streaming regression or that a candidate improves HTTP serving.
Candidate validation: rechecked existing PR206 code against the current native lookup and an independent scalar oracle for all65,536 BF16 patterns. Read-only negative-stride, transposed,5-D and empty arrays preserve exact output, shape/dtype, input contents and independent result storage. Native source SHA256
d98a51d75a36cd8c379a6220adb92528b3607cf80145b85a032270494a429580; PR206 adapter source SHA256bf40c4cfe2463c2d73e88c130117a9006f629f3d2aa9fe1eb6900ac14d257383(PR206 head f4ebe93). This is conversion correctness, not a new serving speedup claim.Scope: reuse the existing exact converter through one shared implementation, preserving geometry validation, ownership/cancellation, signed-zero and saturation behavior. Avoid adding another native library before testing this existing candidate. Keep both whole and per-layer streaming paths covered.
Done when: engine owner supplies a reviewable integration; exhaustive/strided tests pass; same-device baseline/candidate serving comparisons retain exact tokens for Qwen3-0.6B,Qwen3-4B,KONI-8B,Gemma3-1B at2K/8K and32/128outputs; report both latency/throughput and regressions. Keep existing native model-numerics and semantic-quality failures separate. Do not infer DMA overlap.
Owner: Optimus Prime maintainer for engine integration; research harness can provide the matched validation and artifacts. Related #270, #272, #201. Component ABBA remains in progress; final results will follow.
Generated with Codex; maintainer review required.
gspark-etri commentedon Oct 24, 2026
Follow-up measurement completed. The component-instrumented whole/stream/stream/whole run passed528 HTTP responses with identical references on one ATOM device and cleanup0. Lookup conversion accounts for roughly63–85% of measured host-import intervals. Both modes issue the same SDK write count; extra Python span calls do not imply extra DMA calls. Residual host work varies, and instrumentation affects throughput, so this does not establish a single cause for streaming regressions.
The existing PR206 arithmetic encoder was then compared standalone on the same host CPU (Python3.10.12,NumPy2.2.6,96CPU affinity, OMP/MKL threads4), without loading or patching the engine. Alternating order, seven repetitions with the first excluded:
Shape is[layers,2,8,input,128], with a fixed repeated synthetic BF16 sample. All65,536 BF16 patterns match an independent oracle; all28 paired full-array comparisons match exactly. Read-only inputs and independent output storage are checked. Six timed samples per variant/shape; no exclusive-host or HTTP speedup claim. Source hashes are those in the preceding request. Raw benchmark is registered with research-hub; experiment2026-10-25-atom-encoder205.md and bench_dlfloat270.py retain setup/commands.
This supports reusing the existing encoder in the native path as the next candidate. Actual four-model serving and fault-isolation acceptance still requires owner integration and matched measurements; no default change or merge has been made.
Generated with Codex.