Skip to content

exp: validate live RNGD GPU cancellation and producer replacement #260

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gspark-etri wants to merge 3 commits into from
Closed

exp: validate live RNGD GPU cancellation and producer replacement #260

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7b28974
exp: verify live RNGD GPU cancellation and producer recovery (#201)
gspark-etri Oct 23, 2026
f2de635
exp: complete all-four-model GPU producer recovery evidence (#201)
gspark-etri Oct 23, 2026
f9ce6e8
test: verify active GPU decode survives producer death on four models…
gspark-etri Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions opt_prime/examples/pd/http_matrix/.research/events.jsonl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,6 @@
{"id": "328b8349e9974a059ddc7e1a8a485e2c", "ts": "2026-10-23T21:32:23+09:00", "task": "Tokenizer EOS in batched serving (aicomp #229)", "result": "Real ATOM KONI HTTP also passes5/5 baseline/fixed/ignore/stream/reuse checks using same production factory: config-only length vs union stop. GPU+ATOM now10/10 expected outcomes; no SDK/artifact rebuild.", "next": "Update PR230 with NPU evidence; finish KONI medium-context RNGD handoff and broader PD work.", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {"backend": "atom", "kv_blocks": "1"}, "artifacts": [{"path": "analysis/serving-matrix-201/eos229-koni-atom-http.json", "size": 5199, "sha256": "4735255db379ff00"}], "experiments": ["experiments/2026-10-23-serving-eos229.md"], "labels": ["bug"], "review": true, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "1020452aea8445f080108c1ed7f6d643", "ts": "2026-10-24T04:35:37+09:00", "task": "Dense hybrid GPU KV import (aicomp #258)", "result": "Dense hybrid receiver fix:31focused and505full CPU tests passed,34skipped,253subtests,151.69s. Preparing isolated PR and fresh Gemma live rerun; hardware and numerical acceptance not yet claimed.", "next": "Run corrected Gemma RNGD/GPU matrix preserving original failure; complete lifecycle and numerical checks", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-24-dense-hybrid258.md"], "labels": ["bug"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "551af4e77bca4e67bf0512304142ac48", "ts": "2026-10-24T04:41:11+09:00", "task": "Dense hybrid GPU KV import (aicomp #258)", "result": "Corrected live Gemma completed24batches;20/20token checks and60/60timed texts exact to nativeGPU,84remote/0local/0errors. Shape crash fixed on hardware. Qwen0.6 GPU lifecycle passed; Qwen4 next. Owned ATOM Pod confirmed absent.", "next": "Publish hardware evidence PR259; finish remaining GPU fault coverage and numerical boundary work", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/local258-gemma-summary.json", "size": 1288, "sha256": "3afbcc7e61bb3239"}], "experiments": ["experiments/2026-10-24-dense-hybrid258.md"], "labels": ["bug"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "c41ea1b2753146b091f85134832aa680", "ts": "2026-10-24T04:44:49+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "Prepared independent live GPU lifecycle validation branch on32f18ff with executed harness, native-call observer and completed Qwen0.6/Qwen4 raw evidence; KONI remains live. CLI help verified; production runtime unchanged.", "next": "Publish draft evidence PR and complete KONI/Gemma lifecycle runs", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-24-live-gpu-fault201.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "2919d0420ffc49ceacc9aa8982bf4859", "ts": "2026-10-24T04:48:11+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "All4 live GPU lifecycle runs completed passed=true, including KONI/Gemma: cancellation/reuse, producer death503/readiness, slab unlink and replacement exact32-token controls. Started separate512-token concurrent-decode isolation suite after prior queue terminal and device-idle checks.", "next": "Publish remaining lifecycle evidence; verify currently decoding requests survive producer failure across4models", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/local201-koni-faults.json", "size": 1003, "sha256": "5ef1011fb2c2a4c6"}, {"path": "analysis/serving-matrix-201/local201-gemma-faults.json", "size": 1016, "sha256": "e707fb1f52adbb1a"}], "experiments": ["experiments/2026-10-24-live-gpu-fault201.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
{"id": "eb53d939aca44b85ab4b6fd602754651", "ts": "2026-10-24T04:58:02+09:00", "task": "Four-model serving matrix (aicomp #201)", "result": "All4 GPU concurrent-decode isolation tests passed512exact tokens after producer killed at token2, plus cancellation/503/slab-unlink/replacement. KONI paddedGPU control completed: changes output but does NOT reproduce short RNGD output; padding-only explanation rejected.", "next": "Publish all4isolation evidence; capture short KONI KV for identical-prefix numerical analysis", "state": "active", "done": false, "blocker": "", "metrics": {}, "params": {}, "artifacts": [{"path": "analysis/serving-matrix-201/koni-short-padding-diagnostic.json", "size": 5903, "sha256": "f67039817a78849a"}], "experiments": ["experiments/2026-10-24-live-gpu-fault201.md"], "labels": ["benchmark"], "review": false, "todos": [], "note": "", "agent": "codex", "host": "xeon-max", "branch": "main", "commit": "351d39d-dirty"}
6 changes: 6 additions & 0 deletions opt_prime/examples/pd/http_matrix/STATUS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,9 @@ Candidate exports airm_pd_producer_seconds_total{stage="prefill"|"transfer"} fro
#258 dense hybrid KV: actual Gemma failure reproduced; mapping/group validation corrected.31focused and505full CPU tests pass; live rerun pending.

#258 corrected live Gemma32ce5d5:24batches,20/20token checks,60/60timed texts exact;84remote/0local/0errors. Shape crash fixed in recorded hardware matrix; overall numerical/quality boundaries remain.

Live GPU faults#201: Qwen0.6/Qwen4 cancellation, producer-exit503/readiness, SHM unlink and replacement exact-output checks pass. KONI/Gemma pending; long-output numerical failures remain.

KONI/Gemma GPU lifecycle completed; allfour pass cancellation/reuse, producer death/readiness503, slab unlink and producer replacement. Concurrent-decode isolation is being tested separately and not claimed here.

Allfour live GPU isolation runs pass: active512-token decode survives other-request producer death with exact output; all baseline lifecycle gates retained.
Loading