Skip to content

Prefill one queued request during single-lane ATOM decode #255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gspark-etri wants to merge 5 commits into from
Closed

Prefill one queued request during single-lane ATOM decode #255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3e16c33
feat: prefill one queued request during B1 ATOM decode (#253)
gspark-etri Oct 23, 2026
a95e08f
exp: verify KONI prefetch gains and live fault isolation (#253)
gspark-etri Oct 23, 2026
d8e74ff
exp: validate Qwen3-4B prefetch performance and fault isolation (#253)
gspark-etri Oct 23, 2026
4bee440
exp: validate Qwen3-0.6B ATOM prefetch and lifecycle (#253)
gspark-etri Oct 23, 2026
ffc5e6e
exp: complete four-model ATOM prefetch validation (#253)
gspark-etri Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions opt_prime/examples/pd/http_matrix/.research/events.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,7 @@
{"id": "4c7935b757264687b1310fbe1ec6399a", "ts": "2026-10-23T21:26:38+09:00", "task": "Tokenizer EOS in batched serving (aicomp #229)", "result": "Production factory union of generation-config/tokenizer EOS fixes real KONI HTTP turn end: baseline length, fixed stop, ignore_eos length, streaming stop, reuse stop. Baseline factory3fail/3pass ->22related tests pass. Preserved failed initial SSE checker log.", "next": "Publish scoped serving fix; verify NPU HTTP and retain long RNGD semantic validation.", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"model": "KONI-8B", "backend": "gpu", "max_tokens": "8"}, "artifacts": [{"path": "analysis/serving-matrix-201/eos229-koni-http-final.json", "size": 5067, "sha256": "d5be9a4a52a6485f"}], "experiments": ["experiments/2026-10-23-serving-eos229.md"], "labels": ["bug"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "e4993823527144b5a14e6441b1cd5c01", "ts": "2026-10-23T21:29:56+09:00", "task": "Tokenizer EOS in batched serving (aicomp #229)", "result": "Published production EOS fix and real KONI HTTP evidence in draft PR230 (e5fa087). Final factory/streaming/sampling/batched suite22passed; hardware checker5/5 cases passed; changed files sensitive-clean,4unchanged baseline findings in this branch.", "next": "Verify same HTTP factory on NPU; continue full four-model long semantic and heterogeneous PD validation.", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"backend": "gpu"}, "artifacts": [{"path": "analysis/serving-matrix-201/eos229-koni-http-final.json", "size": 5067, "sha256": "d5be9a4a52a6485f"}], "experiments": ["experiments/2026-10-23-serving-eos229.md"], "labels": ["bug"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "328b8349e9974a059ddc7e1a8a485e2c", "ts": "2026-10-23T21:32:23+09:00", "task": "Tokenizer EOS in batched serving (aicomp #229)", "result": "Real ATOM KONI HTTP also passes5/5 baseline/fixed/ignore/stream/reuse checks using same production factory: config-only length vs union stop. GPU+ATOM now10/10 expected outcomes; no SDK/artifact rebuild.", "next": "Update PR230 with NPU evidence; finish KONI medium-context RNGD handoff and broader PD work.", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"backend": "atom", "kv_blocks": "1"}, "artifacts": [{"path": "analysis/serving-matrix-201/eos229-koni-atom-http.json", "size": 5199, "sha256": "4735255db379ff00"}], "experiments": ["experiments/2026-10-23-serving-eos229.md"], "labels": ["bug"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "336aabdd36624e148ea41f974e84af8e", "ts": "2026-10-24T03:34:33+09:00", "task": "ATOM next-request prefetch (aicomp #253)", "result": "KONI complete: 48 batches exact, 8 fixed-output and 4 EOS groups passed; cancel/reuse and producer-exit preservation/503 gates passed. c4 median gains23.48-40.87%; c1 neutral. Qwen3-4B started.", "next": "Finish Qwen3-4B, Qwen3-0.6B and Gemma in current queue; publish KONI evidence to draft PR255", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"model": "KONI-8B", "producer_width": "8192", "prefetch": "true"}, "artifacts": [{"path": "analysis/serving-matrix-201/atom253-koni-summary.json", "size": 5612, "sha256": "b66019a0a29d1e3a"}], "experiments": ["experiments/2026-10-24-atom-prefetch253.md"], "labels": ["benchmark", "rngd"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "5ae1dbf4a057408187ad603c57f1cd7b", "ts": "2026-10-24T04:06:11+09:00", "task": "ATOM next-request prefetch (aicomp #253)", "result": "Qwen3-4B complete:48batches exact,8fixed/4EOS groups pass; cancel/reuse and producer-failure isolation/503 gates pass. c4 median gains21.20-41.00%; c1 near-neutral. Qwen0.6 started. Separate GPU-route numerical issue256 stays open.", "next": "Complete Qwen0.6/Gemma prefetch queue; publish Qwen4 results to PR255; run prepared forced-prefix diagnostic after local GPU matrix", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"model": "Qwen3-4B", "producer_width": "8192"}, "artifacts": [{"path": "analysis/serving-matrix-201/atom253-qwen4b-summary.json", "size": 5590, "sha256": "275d53eef400a031"}], "experiments": ["experiments/2026-10-24-atom-prefetch253.md"], "labels": ["benchmark", "rngd"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "28246e6d99fb4667a780a2e4750917e2", "ts": "2026-10-24T04:18:23+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "Qwen3-0.6B ATOM prefetch48 batches plus parity/EOS/lifecycle passed; 8K/output128/c4 35.016->48.231 tok/s (+37.74%). KONI GPU drift localized to pre-EOS indices64/98 at output128; output32 passes. Gemma active on both paths.", "next": "Complete Gemma matrix and GPU lifecycle; diagnose Qwen4/KONI pre-EOS drift", "state": "active", "done": false, "blocker": "", "metrics": {"tok_s": "48.23077851745031"}, "params": {"model": "Qwen3-0.6B", "input_tokens": "8192", "output_tokens": "128", "concurrency": "4"}, "artifacts": [{"path": "analysis/serving-matrix-201/atom253-qwen06b-summary.json", "size": 5583, "sha256": "f420a2447bd438af"}], "experiments": ["experiments/2026-10-24-atom-prefetch253.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "082ee581602746b68a55087a6bef1646", "ts": "2026-10-24T04:39:26+09:00", "task": "ATOM next-request prefetch (aicomp #253)", "result": "Gemma48 batches and output/EOS/lifecycle passed; 8K/output128/c4 40.000->59.148tok/s (+47.87%). All4models now192timed batches passed on/off parity. Owned prefetched Pod cleanup requested; GPU corrected Gemma still running.", "next": "Publish fourth-model evidence in PR255; finish separate GPU correctness and lifecycle coverage", "state": "active", "done": false, "blocker": "", "metrics": {"tok_s": "59.147871"}, "params": {"model": "Gemma3-1B", "input_tokens": "8192", "output_tokens": "128", "concurrency": "4"}, "artifacts": [{"path": "analysis/serving-matrix-201/atom253-gemma-summary.json", "size": 5586, "sha256": "148cb39f84cb39e8"}], "experiments": ["experiments/2026-10-24-atom-prefetch253.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
12 changes: 12 additions & 0 deletions opt_prime/examples/pd/http_matrix/STATUS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,15 @@ A real terminated subprocess producer previously left /ready200 and new requests
## Batched producer timing (#249)

Candidate exports airm_pd_producer_seconds_total{stage="prefill"|"transfer"} from existing accumulated batcher durations. Repeated scrapes do not add observations; missing timing keys are omitted. Tiny real-model admission test covers fixed worker-reported durations and legacy omissions. These counters are worker-reported admitted-handoff sums, not request latency or proof of zero cost for workers that do not measure stages. Full CPU suite:467passed,17skipped,135subtests,1existing warning. See pd-stage-metrics249 experiment.

## B1 next-request prefill (#253), 2026-10-24

Opt-in remote_prefetch / --pd-prefetch-next implemented for single-lane ATOM remote-only PD. Keep one source result reserved through owner import; cancelled writers drain before reuse. Prefix/host cache admission remains late, so early prefill transfers the full prompt.35focused tests and full511passed34skipped253subtests; real four-model hardware pipeline/performance validation pending. No speedup claim. See experiments/2026-10-24-atom-prefetch253.md.

KONI hardware prefetch validation complete:48timed batches exact,8fixed-output/4EOS groups pass, cancellation/reuse and producer-exit isolation/readiness503 pass. Same-warm-instance off/on3repeat c4 gains23.48–40.87%; c1 near-neutral. Host producer/decode overlap only in enabled mode. Remaining three models pending; no default promotion. See evidence253/atom253-koni-summary.json and experiment.

Qwen3-4B also passes48timed batches,8fixed/4EOS groups and live cancellation/producer-exit isolation. c4 median gains21.20–41.00%; c1 near-neutral. Qwen0.6/Gemma remain pending. This does not resolve the separate GPU decoder pre-EOS drift in#256.

Qwen3-0.6B: 48 prefetch off/on batches and fixed-output/EOS/lifecycle checks passed. At 8K input/output128/c4, 35.016 -> 48.231 tok/s (+37.74%); c1 roughly neutral. Gemma hardware validation remains pending.

Gemma completed48batches and output/EOS/lifecycle checks. Allfour models192batches pass prefetch on/off parity. Gemma8K/output128/c4 +47.87%; separate GPU and native-reference numerical issues remain.
Loading