Repository navigation
E11: concurrent multi-node variant delivery, store-only versus peers #82
Description
Requested by the CCGrid paper collaborator (2026-10-31 10:14 KST).
Measure the same file-only variant switch on as many live Kubernetes nodes as feasible (minimum3), without any NPU resource allocation. Compare store-only against the existing Onggi peer / CSI peer path, with at least5 independent simultaneous-node trials per condition. Record actual store payload bytes, per-node completion latency and full-cohort completion latency, median/IQR, NIC/link/network constraints, source/input SHAs, node/cache conditions, synchronization skew, failures and cleanup. No inference or vendor-format parsing.
Plan: inspect dell_0 live kubeconfig and existing peer implementation; isolate new exp branch and bounded cdx CPU-only jobs; freeze a real artifact transition and cache/admission conditions; smoke both paths before n5. Explicitly distinguish peer seeding/setup costs, origin bytes, node peer bytes and physical network counters. Preserve every attempt and raw JSON plus deterministic aggregation. GitHub flow: issue -> exp branch -> PR -> read green CI -> merge. Shared EXPERIMENTS.md E11 and Log updated at milestones; NUMBERS.md owned by Claude and not edited.
Activity
gspark-etri commentedon Oct 31, 2026
E11 is running in draft PR83. Eligible workers are cpu-1,cpu-2,etri-001 (N=3); other workers are unreachable, cordoned or under DiskPressure, and no node settings were changed. Uses the unchanged main05c9760 CLI and real complete B4/S256/g2 -> B4/S512/g2 manifests from E6. CPU-only Jobs request no accelerators and mount no accelerator devices. Origin is colocated on etri-001; its active physical NIC reports1GbE. Full base bootstrap and all post-pull full-file hashes are outside the timed barrier.
Frozen measured harness7870cef872e201a6a4dca87ed1bb998da9c3109f passes7 stdlib regressions and independent review after fixing interrupted-response accounting and adding exact target/cache-placement gates. Exact-head CI41155969692 is green and logs read, including Go race/vet/build, both real S3 server conformance/CLI paths and the7 harness tests. Four smoke cohorts precede20main cohorts: empty or one-seed caches x store or peer x n5. Node0's seed cache is identical in BOTH arms; seeding is separately reported. Empty peers use a fixed pre-pull snapshot, so no dynamic dissemination claim.
All setup attempts are preserved: origin01 failed UID mismatch before serving; attempt02 bootstrap was stopped before any measured cohort after review improvements; attempt03 is the frozen measurement setup. No smoke/main retries or latency-based exclusions. Main result summaries remain pending. NUMBERS.md untouched.
gspark-etri commentedon Oct 31, 2026
Completed and merged as PR83, main
d56fe3e9b3d5ea7906d76601604653c7a9d748a4; merged tree equals independently reviewed/tested final head. Final CI41161011264 all three checks green, actual logs read.Results: three simultaneous CPU-only/1GbE nodes, four cache/arm conditions × five cohorts; 20 main +4 preserved smokes, zero retry/exclusion/failure. All nine complete target files match per worker. With one seed in both arms, origin pack395,822→0 B, but all successful S3 body70,089,116→69,693,294 B (0.564740% reduction), due unchanged metadata. Cohort median store→peer: empty3.102→3.240s; seeded3.296→3.014s. Report includes per-node times/IQR and limits; no general latency speedup claim.
Report, summary, 128-file raw/tag. Original26,928 request rows independently reconstruct counters; exact final-head clean Git archive reproduces summary; corruptions rejected. Model payloads absent from Git; worker-recorded full hashes compared with frozen inventory offline.
All owned CPU Jobs/Pods succeeded then UID-cleaned; namespace absent, cdx Jobs/Pods empty. No NPU/GPU used. Shared board E11 DONE and final Log updated. NUMBERS.md untouched.