Repository navigation
Integrate resident runtimes with standalone prepared serving and startup warmup - #33
Draft
gspark-etri wants to merge 19 commits into
Draft
gspark-etri wants to merge 19 commits into
gspark-etri wants to merge 19 commits into
Conversation
This was referenced Oct 30, 2026
Ninety main samples across eighteen n5 conditions; original raw-linked medians/IQR, native parity/device checks and source/input SHA provenance. Original measured source commits and publication history are permanently retained by annotated tag ccgrid-20261030-raw-v1 -> cec176d. Native B2 remains open in #39; optional ATOM #37 is deferred.
Add a persistent ADR-0010 B2 owner for lossless BF16 projection and LM-head rows. It stages target units, journals actual serving allocations, verifies D2H readback before publishing identity, and retains verified rollback. Failed restoration unloads; failed release quarantines and blocks replacement. Unsupported host, scaled, header and coarse ranges request reload. Native SDK 2026.4 conformance passed three declared cases, eleven independent fresh-token gates and 2,805 original compiled calls/device checks with zero compilation. A silent write corruption was rejected and actual base rows/tokens restored. This is n=1 conformance, with no repeated-latency claim. The original coarse projection failure is retained: its changed chunk also covered equal host/scaled bytes. No planner or kit assertion was weakened. [Native report and raw evidence](https://github.com/ai-computing/rngd-pd/blob/binding-rngd-20261030-raw-v1/experiments/REPORT-RNGD-BINDING-20261030.md) include both attempts, authentic Onggi plans, SHA-verified fixtures, exact harnesses and UID cleanup receipts. Run `python experiments/verify_rngd_binding_evidence.py`; CI verifies committed raw bytes and rejects source changes relative to the measured code. Annotated tag binding-rngd-20261030-raw-v1 retains measured a2ebc32 ancestry. API/LOC audit: vendor owner435 physical lines, shared transport199; full explicit RNGD scope3,343. New row-copy/readback uses public Torch APIs; inherited graph/cache code still uses private `_inner` serialization, `_C` configuration and version-sensitive native IR/cache records, enumerated in docs/rngd-binding.md. Validation: full CPU635 passed/9 skipped/4 subtests; final Linux CI41067732620 is green (642 passed/2 skipped/4 subtests and evidence verifier), logs read. Independent code/evidence review has no remaining Important finding; clean Git archive verification and source-tamper rejection pass. Neutral kit onggi PR81 is stacked on PR80/ADR PR76; its author branch is unchanged. Refs #39.
…idence (#41) Add a public ADR-0010 ATOM B1 owner: describe, resident identity, pure reload-only plans, compiled-session reuse and whole-request ownership. Replacement closes the old runtime before creating the next. Experimental #28 attachment and its private weight-free ABI remain separate from the default public AtomSession path. Native B1 passed two cases and five additional reload/generation repeats: all seven binding loads matched an independent fresh session, 120 native runtime calls, zero compilation, unchanged 15.176 GB graph SHA. The successful worker was unprivileged and exposed only its one allocated card. Three earlier device-discovery/mapping failures are preserved. Native success is physical/logical0 only; nonzero mapping has CPU format tests. n=5: full owner load including source SHA median20.048155s/IQR0.014296; post-load first token0.561485s/IQR0.000626. Parse10.491921s and runtime-create0.353489s are SDK-call wall times, not measured weight-upload/DMA latency. No cold-cache or cross-vendor speedup claim. ATOM B2 is not implemented. [Raw report, all attempts and derived statistics](https://github.com/ai-computing/rngd-pd/blob/exp/37-atom-binding/experiments/REPORT-ATOM-BINDING-20261030.md). Run `python experiments/verify_atom_binding_evidence.py`; CI checks packaged hashes/current measured-source compatibility and recomputes inclusive quartiles. Vendor owner193 physical lines +shared transport199; explicit full public scope2,193; optional private experimental path3,638, with file lists/API audit in docs/atom-binding.md. Validation: full CPU629 passed/9 skipped/4 subtests before integration; evidence verifier, clean Git archive/source-tamper checks and independent code/evidence review pass. Final combined integration CI pending. Raw native adapter sourcefbfdb8b and kit71c37d2 are identified separately; compile preparationf5fa133 was off-card. Refs #37, #28.
#42) Complete the two remaining native compliance gates from #39 on the unchanged RNGD B2 owner: apply waits across a whole native request; a real partial rollback produces D2H-verified mixed bytes, then failure unloads the model and refuses inference. Five fresh-token gates, 1275 checked native calls, zero compilation. This is n=1 compliance evidence, without a timing claim. Includes exact raw JSON/archive, authentic input/reference links, measured harness SHA, UID cleanup and experiments/REPORT-RNGD-FAULTS-20261030.md. Adds the third stdlib evidence verifier to CI. Clean Git archive checks and independent review pass; combined CI pending. No runtime, LOC, main-branch or NUMBERS.md changes. Follows merged #40 and #41; refs #39.
…44) The RNGD B2 adapter uses public Torch row operations, but no-compile load still depends on inherited SDK imports and native cache reconstruction. Add a function-level inventory pinned to be468b6, separating explicit private names, native/undocumented contracts, legacy branches and repository-owned state. The inventory records 15 dependency entries, no-compile B2 reachability, four official public alternatives and six concrete vendor clarification requests. FXB/LLM deployment is documented as a higher-level alternative; it is not claimed as a drop-in for this engine's cache ABI or resident row transactions. This is a source/docs audit with no licensing or IP legal conclusion. Validation: inspected every Furiosa import across the 18-file scope; checked 61 relative links, source line bounds and source identity against the baseline; recomputed all five LOC scopes and all report file counts/hashes. Two adapter raw hashes differ only due to CRLF conversion versus baseline LF blobs; documented and preserved the existing LOC report. git diff --check passed. Documentation only; no hardware or remote nodes used. Refs #43. Hold merge for root review; this PR makes no runtime changes.
The current transactional B2 path previously had conformance evidence but no matched repeated timing. This adds five alternating B2/reload pairs on the existing fixed B4/S256 projection-update fixture, plus a separate excluded smoke pair; runtime source is unchanged at 4482fae. Plan-to-first-native-token median [IQR] is 0.448449 [0.198801] s for B2 versus 13.783574 [0.335147] s for same-target reload. Undo, staging, synchronization and readback stay in the B2 clock; closing the previous owner stays in reload. Logical serving writes are 1,949,696 B with 469,762,048 B whole-parent readback per B2 apply, not total DMA. All ten target continuations match fresh-target references/current paired reload, preserve B2 allocation/program identity, and pass 5,100 original native-output checks with no compilation. The exact 47-file package includes raw trials/events, input/config/source identities, full post-input hash receipts, Kubernetes allocation/status/UID cleanup and source EOL reconciliation. Current final-v2 retains the earlier derived summaries, including the initial failed analysis and stale 46-file package receipt; native measurements are unchanged. The offline verifier and CI reproduce the final summary. Validation: 20 focused tests; full CI at 7fd4310 passed; independent source/raw/timing review found only the corrected stale summary link. This is a prepared local-file fixed-shape update, not graph switching, live CSI, service drain under load, or cold-cache startup. One RNGD was used only inside bounded cdx- Jobs; owned Jobs/Pods were UID-cleaned. Closes #45
The earlier delivery/restart benchmark mixed file delivery with runtime residency. This experiment keeps the same RNGD owner, shared weights and base programs while comparing four delivery paths from a base-only graph cache. It also checks ownership with the owner's actual CSI-published checkpoint and tests S512 capacity against an independent fresh target. Five alternating main cohorts (20 transitions; 10,200 native calls) give validated demand-to-first-token medians [inclusive IQR] of: | Delivery path | Seconds | |---|---:| | Graph-only local copy | 0.402495 [0.008617] | | Graph-only CSI fallback | 0.416117 [0.002011] | | Whole bundle, CSI reconstruction | 67.576389 [0.307710] | | Whole bundle, CSI rebase | 9.881452 [0.289921] | Whole-bundle delivery drops from 57.891s to 0.315s, with complete target SHA validation inside the endpoint. The graph-only local-copy baseline is already fast; CSI is slightly slower here. Default graph reuse admission refuses 22.3% same-offset content, so that result is fallback, not successful rebase. Donor preparation costs are recorded separately and cannot be ignored for a one-off deployment. The package preserves 262 exact cohort files and 141 operational records, including source/binary pins, configs, failed CPU preparations, Job/Pod UIDs, actual CSI RPC/log/lease/mount/inode evidence and cleanup receipts. All eight native Jobs used one card with deadlines/TTL and were UID-cleaned. Separate n=1 controls establish active-checkpoint refusal and P257/O8 fresh-target parity (9,231 capacity native calls); these are not latency distributions. Pre-merge review found that failed owner close could release CSI leases. The reusable harness now retains mounts/leases and partial evidence until process exit on that branch. A regression executes its real cleanup block with actual temporary lease files. The measured source pins and accepted raw observations remain unchanged; the native core has no diff. Validation: independent native/evidence review and focused cleanup re-review have no remaining Important findings. Exact-head Linux CI41387459660 at91c7de5 was read: 727 passed, 2 skipped, 4 subtests, plus all binding and native-package verifiers and exact n5 summary reproduction. Local focused tests:21 passed/1 skipped. Full local Windows collection is unavailable because httpx is absent; Linux CI runs the full suite. The annotated resident-delivery-20261102-raw-v1 tag preserves measured-source ancestry across squash merge. Scope: colocated POSIX CSI, shared uncontrolled OS cache, explicit zero-inflight caller, reconstructed native first-token endpoint. This is not a stock production FXB comparison, HTTP/Pod readiness result, network/cold-cache result or loaded-service drain study. Model payloads/vendor binaries are absent from Git; offline verification checks recorded hash witnesses. CPU same-cache/chunking and loaded-service studies remain separate work. Closes #47.
…52) A verified switch plan previously required callers to compose binding operations themselves. Both owners now expose transition(plan, target): validate the ready resident base, describe and plan, perform an admitted RNGD B2 update or an explicit lifecycle reload, and return the resulting identity under one inference-excluding owner lock. The persistent CLI exposes transition --target without changing the five conformance operations. Failed admitted updates propagate instead of silently reloading. ATOM also reuses the previous public parsed compiled model when a fresh full graph SHA matches. It closes the old device runtime before creating its replacement, reparses changed graph bytes, and releases the host object on explicit close. This reduces repeated host parsing; it does not retain device weights or implement ATOM B2. Validation: baseline73 passed/1 Unix-socket skip; new transition tests failed before implementation then passed; host-parse reuse2 regressions failed before the change then passed. Connected swap suite441 passed/1 Windows Unix-socket skip. Tests use real owners with injected CPU vendor boundaries; new native timing and output parity remain unmeasured. Existing native evidence and vendor SDK/driver files are unchanged. Three bounded read-only review passes found no Important/Critical findings. Exact-head c199ec6 CI41415336238 logs were read:753 passed,2 skipped,4 subtests, and all native evidence verifiers passed. Historical source checks now read their pinned Git blobs so runtime evolution does not relabel old native results. All raw/archive/fixture checks remain; old data and golden summaries are unchanged. Local whole-suite collection lacked httpx; the subsequent scoped Windows run also exposed two installed entry-point registration failures. The complete Linux CI is the integration gate and passes. Closes #51.
…aller (#54) Measure the common ATOM transition caller on real hardware: for an unchanged compiled graph and metadata revision, reusing the public parsed host object reduces trigger-to-first-token median from **20.8113s to10.2899s** across five alternating pairs (2.0225x ratio of medians,50.556% reduction). Both arms repeat the full graph SHA, close the old runtime and create a new native runtime; this is B1 reload with host parsing reuse, not changed-shape/weight or device-resident partial loading. The measured source is immutable fec2d37 atop runtime57e610e. The separate smoke and main evidence includes315 main native calls across21 generations, same-target fresh output equality, zero compilation, parse/create counts1/1 versus0/1 and pre/post input hashes. Both bounded unprivileged one-card Jobs were exported and UID-cleaned; final live cluster inventory has no accelerator/cdx Pods or Jobs. [Report](experiments/REPORT-ATOM-REUSE-20261102.md), [all observations/median/IQR](experiments/atom-reuse-summary-20261102.json), [60-file sealed raw package](experiments/data/atom-reuse-20261102). The SDK-free verifier checks exact exports,82 pinned Git blobs, native clocks/counters, independent statistics and one-card Pod/release evidence. Independent review findings were reproduced RED→GREEN and fixed;72 focused CPU tests pass. CI re-verifies this package and historical evidence. Earlier native raw and NUMBERS are unchanged. Closes #53.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integrate reviewed ATOM #28 into RNGD #32 while preserving SDK 2026.4 cache loading and optional host staging. The standalone cluster worker now uses the genuine shared whole-request KV implementation without requiring vLLM or injecting test modules. The optional vLLM adapter retains its existing contract.
Optional
warmup: trueexecutes every configured prepared variant before readiness and discards private warmup KV. Startup failure/cancellation closes the engine. The review fix preserves the original startup exception with cleanup failure as its cause, marks RNGD failed before synchronization, and retains buffers while a failed fence may leave native work live. Default warmup remains false.Real native proof connects Onggi init preparation → fresh standalone prepared worker → HTTP decode on one RNGD. 12 requests /96 tokens equal fresh references, including 12 supplied first tokens and 84 native-produced suffix tokens. Native calls progress 17 before readiness →136 →255 →374. Upload1 /graph load1 /compile0, with all 113 shared projection/LM buffers and compiled identities retained. Checkpoint409/context400 admission checks pass.
Experiment and limits · immutable public evidence for both attempts · Onggi integration #72.
Validation:
The first prefix-only KV harness failed before native continuation and is preserved separately. The successful retry transfers the actual full S256 native arrays and checks bitwise BF16 roundtrips. Publication changes only docs and the incorrect prefix-transfer comment; all src Python ASTs match the frozen native source.
This proof delivers graph/config files with an existing external checkpoint. No standalone-warmup latency or new speedup claim. Live prefill/gateway, multiple native B/S choices, ATOM HTTP startup, installed-vLLM execution, UCX recovery and production rollout remain separate gates. ATOM constant updates remain available through their Python API/validation harness, not a prepared HTTP factory.
Stacked on #32; no main merge, base retarget, old PR closure or deployment.