[gspark-etri]

Background

With B1 ATOM remote-only PD, AtomBatcher._pd_step submits a prompt to the prefill producer only once the single decode lane is free. Prefill of the next request and decode of the current one are serialized, so the producer sits idle during decode (#253).

Changes

From #255 (branch perf/253-atom-prefetch, commit 3e16c33), code and tests only. The diff was limited to these paths and applied cleanly onto current optimus-rm:

• opt_prime/opt_prime/accel/platforms/atom/batching.py: AtomBatcher(..., remote_prefetch=False).
  • When enabled, _prefetch_next() submits the next queued prompt (send_from=0, source slot 0) while the current request decodes.
  • At most one prefetch at a time. The source slot stays reserved until the owner imports the KV. After a cancel or producer error it stays reserved until the producer future completes.
  • idle stays false while a prefetch is held, so an orphaned producer is still drained.
  • Admission does not re-submit an already prefetched request.
  • Raises ValueError unless lanes == 1, a prefill_worker is set, and local_prefill=False.
• opt_prime/examples/pd/serve_batched.py: --pd-prefetch-next. build() rejects it before model loading unless --backend atom --slots 1 --pd-remote-only and a PD prefill worker are set.
• opt_prime/tests/test_atom_batching.py: test_b1_prefetch_overlaps_decode_and_holds_source_until_import, parametrized over {success, cancel, error} x {cache off, on}.
• opt_prime/tests/test_serving_prefill_cache.py: test_next_prefill_rejects_unsupported_route_before_load (4 unsupported routes).
• New in this PR: a docstring line and a help-text line that explain the prefix-cache behavior (see Compatibility and limits).

Not taken: examples/pd/http_matrix/* (the pipeline script, evidence253/, experiment record, STATUS.md, .research/) and the four exp commits.

Verification

• cd opt_prime && TORCH_DEVICE_BACKEND_AUTOLOAD=0 PYTHONPATH=$PWD python3 -m pytest -q tests/test_atom_batching.py tests/test_serving_prefill_cache.py: 35 passed.
• Fail-before: the new tests (-k "prefetch or next_prefill") run against the optimus-rm versions of batching.py / serve_batched.py give 10 failed. With this change, all pass.
• Full suite: CI.
• Performance, from the source branch Prefill one queued request during single-lane ATOM decode #255 (not re-measured here): RNGD prefill to ATOM decode, warm engine, prefetch off vs on, 3 alternating repeats, medians. Throughput is +9% to +53% at c4 and neutral at c1.

Compatibility and limits

• Off by default; nothing changes unless remote_prefetch=True / --pd-prefetch-next is set.
• Only single-lane ATOM remote-only PD with a prefill worker is supported. Every other configuration is rejected (constructor and CLI; the CLI rejects it before model loading).
• With a prefix cache, the prefetch is submitted before cache admission, so it re-sends the full prompt and overwrites the restored prefix. The result is correct, but it forfeits the cache's transfer savings.
• No transport or protocol change and no extra slab slot.

Related Issue

Refs #253
Supersedes #255

🤖 Generated with Claude Code