Repository navigation
[prototyping] Make DiskGraph explicit I/O production-ready - #169
danielmawhirter wants to merge 1 commit into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional.
Full details: Public Resource Access
Full details: Change Map Design Review
Full details: Pr Size
|
Change mapCurrent head The current rebase preserves upstream oversized-row/null-feature behavior across fused, mmap and explicit batched feature paths, and reuses upstream seed-column classification. Generated manifests/notices retain their canonical content, including upstream license indentation. Actual optimized native BUILD_COMMIT is the exact head above, SHA256 A separate bounded actual-head parallel fork probe produced four passing child-created sampler controls and four 15-second inherited external-sampler timeouts; all eight children were reaped. This is an unresolved production blocker, with no baseline attribution or fix claimed. Keep this PR draft until a focused follow-up rejects inherited external samplers before native access or proves safe worker-local construction, with parallel-path regression coverage. Exact-head CI passed Linux x86_64 Rust 1.99.1 workspace (2,194 passed / 28 ignored) and ten strict native cases, Clippy, Linux wheel, generic native Python (931 passed / 20 opt-in skips / one deselected), CPU task checks (eight tests / 24 task-backend executions) and 15 strict Python tests with zero skips, and macOS arm64 Rust 1.99.1 wheel build/install plus 11 mmap smoke tests. Both Engine package shards passed; the linked run records other integrations. Frozen published-candidate performance measurements remain scoped to candidate
|
96a6538 to
2643ad7
Compare
2643ad7 to
5fd6218
Compare
5fd6218 to
c1a04a3
Compare
c1a04a3 to
a45b767
Compare
a45b767 to
4a340a6
Compare
4a340a6 to
acaa797
Compare
Scope and revision
Port DiskGraph #424, pinned at
f84d30cf3b9a31884c46b3b71c19d233c80fbcc3, with optional Buffered/Direct I/O, bounded caching, prefetch planning, Core/Python integration and readiness checks. Preserve mmap defaults and current seeded neighborhoods by omitting the Floyd sampler replacement. Retain the verified cancellation, process ownership, automatic fusion and profiler fixes.One SSH-signed commit,
acaa797a277982c2ac4737a89a9c577eeafdde06, sits directly on fetchedupstream/maind8aaf2b6e818027b7a03f43bd96f8c3efd6263b0; the port remains 80 paths. The rebase preserves main's oversized-row/null-feature behavior in fused, mmap and explicit batched reads, and reuses main's canonical seed-column classification. This PR has no dependency on #196, #197 or #198.Actual validation
An optimized extension and ingest binary were built from this exact commit, using local AArch64 Linux Rust 1.96.1 and Python 3.14.7. Native SHA256:
4437578a151f976fef71c0fb6e8cfc31b7a7894307fd0986c02ef118790af2e8. Loaded paths, SHA256,BUILD_COMMIT, source paths and dependency order were checked. The unchanged task suite ran with that same wheel unpacked inside an owned no-pip Python prefix; shared dependencies remained unchanged.Earlier local attempts retained setup failures separately: cached build-tool import path, canonical build directory, full-workspace libpython lookup and the task fixture's installed-prefix guard. The successful runs corrected their runtime setup without changing tests or bypassing guards. All owned services terminated; a host scan found no remaining owned processes.
Exact-head CI also passed: Linux x86_64 Rust 1.99.1 workspace (2,194 passed, 28 ignored), all ten strict native cases, Clippy and the Linux wheel; eight CPU task checks / 24 task-backend executions followed by 15 strict Python checks with zero skips; generic native Python 931 passed, 20 opt-in skips, one deselected; and macOS arm64 Rust 1.99.1 optimized wheel build/install plus 11 mmap ingestion/sampling smoke tests. Both Engine package shards passed. The CI run records the other integration jobs.
Confirmed production blocker
On this exact native build, all four parent-created external
SamplerEnginecases—mmap/Direct with cold/warmed parents—hang after entering parallel history execution throughwith_history, exceeding the 15-second deadline. All four matching child-created controls complete with 128 task roots, two native threads, 129 history offsets and 256 history rows. All eight children were positively reaped.The external sampler API and unguarded ownership predate this rebase; no baseline native build established when the issue began. Keep this PR draft and block production qualification until a focused follow-up rejects inherited external samplers before native access or establishes safe worker-local construction, with a regression that forces the parallel path. This port includes no fix for that separate boundary.
Performance scope
The original end-to-end report and physical/application IOPS report each retain their five-repetition 90-trial scope. Their performance numbers qualify their recorded candidate.
Queue-depth, CPU/off-CPU and drive profiling retain frozen published candidate
0fe02c1b, native build75ca0e2fand native SHA256a5949f96723df2684c6cdbd5884bcc25c72129c8a629bfffa49bc36f9552140f. Fresh captures have completed; offline replay and the final report are being finalized separately from this port. The published investigation checkpoint records the recovered inputs and pinned build. Those measurements do not qualify theacaa797abinary. Keep mmap as default and budget caches per process. GPU, production-worker/storage-policy and Windows readiness remain unqualified.Historical investigation · Maintained change map.