Skip to content

[prototyping] Make DiskGraph explicit I/O production-ready - #169

Draft
danielmawhirter wants to merge 1 commit into
mainfrom
dm-diskgraph-424-readiness
Draft

danielmawhirter wants to merge 1 commit into
mainfrom
dm-diskgraph-424-readiness

Conversation

@danielmawhirter

@danielmawhirter danielmawhirter commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Scope and revision

Port DiskGraph #424, pinned at f84d30cf3b9a31884c46b3b71c19d233c80fbcc3, with optional Buffered/Direct I/O, bounded caching, prefetch planning, Core/Python integration and readiness checks. Preserve mmap defaults and current seeded neighborhoods by omitting the Floyd sampler replacement. Retain the verified cancellation, process ownership, automatic fusion and profiler fixes.

One SSH-signed commit, acaa797a277982c2ac4737a89a9c577eeafdde06, sits directly on fetched upstream/main d8aaf2b6e818027b7a03f43bd96f8c3efd6263b0; the port remains 80 paths. The rebase preserves main's oversized-row/null-feature behavior in fused, mmap and explicit batched reads, and reuses main's canonical seed-column classification. This PR has no dependency on #196, #197 or #198.

Actual validation

An optimized extension and ingest binary were built from this exact commit, using local AArch64 Linux Rust 1.96.1 and Python 3.14.7. Native SHA256: 4437578a151f976fef71c0fb6e8cfc31b7a7894307fd0986c02ef118790af2e8. Loaded paths, SHA256, BUILD_COMMIT, source paths and dependency order were checked. The unchanged task suite ran with that same wheel unpacked inside an owned no-pip Python prefix; shared dependencies remained unchanged.

  • Full locked Rust workspace: 2,191 passed, 28 explicitly ignored, zero failures; ten strict native I/O/cache/error/fork cases also passed.
  • Empty-node-data Rust regression: one actual pass for each of mmap, Buffered and Direct, asserting the reader's selected backend.
  • Native/runtime Python: 60 passed, zero skips.
  • Upstream oversized-row primary-key/null-feature/catalog regression: one pass per backend, zero skips.
  • All eight task checks passed, each training, predicting and reloading on all three backends in fresh processes.

Earlier local attempts retained setup failures separately: cached build-tool import path, canonical build directory, full-workspace libpython lookup and the task fixture's installed-prefix guard. The successful runs corrected their runtime setup without changing tests or bypassing guards. All owned services terminated; a host scan found no remaining owned processes.

Exact-head CI also passed: Linux x86_64 Rust 1.99.1 workspace (2,194 passed, 28 ignored), all ten strict native cases, Clippy and the Linux wheel; eight CPU task checks / 24 task-backend executions followed by 15 strict Python checks with zero skips; generic native Python 931 passed, 20 opt-in skips, one deselected; and macOS arm64 Rust 1.99.1 optimized wheel build/install plus 11 mmap ingestion/sampling smoke tests. Both Engine package shards passed. The CI run records the other integration jobs.

Confirmed production blocker

On this exact native build, all four parent-created external SamplerEngine cases—mmap/Direct with cold/warmed parents—hang after entering parallel history execution through with_history, exceeding the 15-second deadline. All four matching child-created controls complete with 128 task roots, two native threads, 129 history offsets and 256 history rows. All eight children were positively reaped.

The external sampler API and unguarded ownership predate this rebase; no baseline native build established when the issue began. Keep this PR draft and block production qualification until a focused follow-up rejects inherited external samplers before native access or establishes safe worker-local construction, with a regression that forces the parallel path. This port includes no fix for that separate boundary.

Performance scope

The original end-to-end report and physical/application IOPS report each retain their five-repetition 90-trial scope. Their performance numbers qualify their recorded candidate.

Queue-depth, CPU/off-CPU and drive profiling retain frozen published candidate 0fe02c1b, native build 75ca0e2f and native SHA256 a5949f96723df2684c6cdbd5884bcc25c72129c8a629bfffa49bc36f9552140f. Fresh captures have completed; offline replay and the final report are being finalized separately from this port. The published investigation checkpoint records the recovered inputs and pinned build. Those measurements do not qualify the acaa797a binary. Keep mmap as default and budget caches per process. GPU, production-worker/storage-policy and Windows readiness remain unqualified.

Historical investigation · Maintained change map.

@copy-pr-bot

copy-pr-bot Bot commented Nov 3, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 3, 2026 18:18 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

Failed Failed checks: 1 error · 2 warnings

Check name Status Explanation Resolution
Public Resource Access Error The PR adds external references that anonymous users cannot access. The diff from base d8aaf2b to acaa797 adds them. Anonymous curl checks confirm the failures. GitHub itself responds with 200, so t… For each inaccessible reference, replace it with a public resource or put the required information in this repository. S3 data and reproduction paths (diskgraph-localssd-mmap-vs-direct-2026-08-18.md lines 6 and 91; diskgraph-pinterest-i8g-i…
Change Map Design Review Warning Summary: The change map (issuecomment-6611981159, author danielmawhirter, updated 2026-11-19) now matches the current PR. It covers the blocker and the stop condition. One substantive problem remains.… 1. Rewrite the runtime_batch_iterator.rs row. State that the guard covers RuntimeIndex and iterator access and drop, and sampler creation from an inherited index (lines 262-304, 721-729). State that SamplerEngine has no owner check (lib.rs …
Pr Size Warning PR #169 (NVIDIA/kumo-engine, base main, head dm-diskgraph-424-readiness) changes 80 files, which exceeds the limit of 40. Evidence from the local repository: - The head commit acaa797… The PR changes 80 files. This exceeds the limit of 40. Split PR #169 into smaller, reviewable units of 40 files or fewer. Group the work by area: - Optional Buffered/Direct I/O. - Bounded caching and prefetch planning. - Python/Core option …
Full details: Public Resource Access

Explanation

The PR adds external references that anonymous users cannot access. The diff from base d8aaf2b to acaa797 adds them. Anonymous curl checks confirm the failures. GitHub itself responds with 200, so the network test is valid. Inaccessible S3 references (HTTP 403 AccessDenied; the bucket listings also return AccessDenied): - packages/diskgraph/docs/reports/diskgraph-localssd-mmap-vs-direct-2026-08-18.md, line 6: s3://kumo-unit-test/diskgraph-datagenerator/ (data location). Not an example; the report names it as the source of the alt_detection data. - Same file, line 91: --source s3://kumo-unit-test/diskgraph-datagenerator/alt_detection_10pct/diskgraph_config.json (reproduction command). - packages/diskgraph/docs/reports/diskgraph-pinterest-i8g-io-benchmark-2026-08-09.md, lines 54, 63, 64 and 185: s3://kumo-omnisage-synth-926922431314/configs/, .../snapshots/idx_pruned_ts/ and .../snapshots/idx_unpruned_ts/ (generator configs, index snapshots, and a reproduction command). Inaccessible GitHub provenance links (HTTP 404 anonymously; the repository root also returns 404): - packages/diskgraph/docs/reports/pr-424-production-readiness-2026-10-30.md, line 232: https://github.com/kumo-ai/kumo-diskgraph/pull/424. - packages/diskgraph/docs/cached-direct-io-sampler.html, line 1910: the same URL. Accessible references checked: github.com/tokio-rs/io-uring (200) and the kumo-engine PR page (200). The S3 paths are needed to reproduce the benchmark runs that the added reports describe, so the S3 failure is material. The 404 links serve as provenance only, so they are a secondary issue. The standing failure is not refuted. Its file and line citations match the current tree.

Resolution

For each inaccessible reference, replace it with a public resource or put the required information in this repository. S3 data and reproduction paths (diskgraph-localssd-mmap-vs-direct-2026-08-18.md lines 6 and 91; diskgraph-pinterest-i8g-io-benchmark-2026-08-09.md lines 54, 63, 64 and 185): - Publish the dataset and index snapshots at a public location and update the paths, or - Commit the generator configs (for example alt_detection/diskgraph_config.json and the pinterest_omnisage_*.json files) and the reproduction steps to this repository. State how to regenerate the data locally, including the generation seed and version. kumo-diskgraph PR #424 links (pr-424-production-readiness-2026-10-30.md line 232; cached-direct-io-sampler.html line 1910): - Replace each link with upstream commit f84d30cf3b9a31884c46b3b71c19d233c80fbcc3, which the report already records. Add a public source link only if one exists. - Or state the provenance in text without a link. After the change, run anonymous access checks on every added URL and S3 path. Each must return an accessible response, or the data must be in this repository.

Full details: Change Map Design Review

Explanation

Summary: The change map (issuecomment-6611981159, author danielmawhirter, updated 2026-11-19) now matches the current PR. It covers the blocker and the stop condition. One substantive problem remains. Its guard and test rows claim coverage that the code does not provide for the external SamplerEngine path, which the map itself calls the blocker. Evidence 1. Revision: resolved. The map states head acaa797 on base d8aaf2b. The provider reports the same head and base, and the PR is open and draft. The stale-revision finding from the prior run (2643ad7 and 4a340a6) no longer applies. No 158 or 101 test figure appears in the current map or description. 2. Blocker and stop condition: present. The map states the parent-created external sampler hang. It says to keep the PR draft until a follow-up rejects inherited external samplers or proves worker-local construction, with parallel-path regression coverage. 3. Scope: the PR changes 80 files, with 18,513 insertions and 354 deletions. That is above 1,000 lines, so file-level scope is allowed. The identical rationale for output.rs and state.rs is a refinement only. 4. Guard row, runtime_batch_iterator.rs: misleading. The row says it rejects "inherited worker handles before touching parent Rayon/channel state." - The ProcessOwned guard (lines 262-304) covers the RuntimeIndex and iterator. - sampler() (lines 721-729) calls self.inner.get()?, so the index PID check applies at creation. - SamplerEngine (lib.rs lines 501-506) has no owner field. - with_history() (lines 981-1008) checks only the iterator state through get_mut(). It then clones sampler.reader and sampler.engine into HistoryIteration (lines 1002-1008) with no ownership check. - The blocker runs through that path, because history.engine.thread_pool is used at line 477. The prior run said sampler() had no PID check. That is incorrect. The gap is at with_history() and the SamplerEngine type. 5. Test row, page_io_readiness.rs: overstated. The row says "inherited unsafe access returns typed errors." The index-crate tests check inherited OnceLock, cache-lock, and cache-hit behavior (page_io_readiness_tests.rs lines 104 and 125; tests/page_io_readiness.rs line 229). I did not find typed-error assertions for these cases. The pybind readiness file says the inherited SamplerEngine boundary is "separately unqualified" (test_native_io_readiness.py lines 207-208). The row should not imply coverage of that path. 6. lightning_data.py row: accurate. The diff adds a cached RuntimeIndex handle, a PID check, and a rebuild on PID change. It also clears the PID in state. Uncertain design questions (not the basis for FAIL) - About 17 benchmark config JSONs, about 20 historical reports and proposals, the datagenerator crate, and six benchmark scripts are justified as "Carry" or "Retain." The map does not tie them to the named release requirement (the explicit I/O port). The description links the same reports at other commits. The author should justify them or move them to a separate PR. - The .agents/skills rows cite a "requested upstream development workflow." That workflow is not part of the stated release requirement. The author should justify it or move it. Prior-run items: item 1 (revision), item 2 (blocker), and item 4 (stop condition) are addressed. Item 3 (guard and test row wording) is not addressed. The benchmark-file question remains open.

Resolution

  1. Rewrite the runtime_batch_iterator.rs row. State that the guard covers RuntimeIndex and iterator access and drop, and sampler creation from an inherited index (lines 262-304, 721-729). State that SamplerEngine has no owner check (lib.rs 501-506). State that with_history() attaches an external sampler without one (runtime_batch_iterator.rs 981-1008). Do not claim rejection of inherited worker handles before parent Rayon or channel use for this path. 2. Rewrite the page_io_readiness.rs row. Name the cases actually covered (OnceLock, cache lock, cache hit) and drop "typed errors" unless the tests assert them. Add that the external SamplerEngine attachment is not covered. Make the test_native_io_readiness.py row match its own comment (lines 207-208). 3. Keep the blocker row. Add the cited code ranges: sampler() 721-729 and with_history() 981-1008. 4. Keep the stop condition. Keep the PR draft until the sampler boundary is resolved, with a forced parallel-path regression. 5. Justify the benchmark configs, historical reports, datagenerator crate, benchmark scripts, and .agents/skills rows against a named requirement, or move them to a separate PR. 6. Optional: give output.rs and state.rs separate rationales.
Full details: Pr Size

Explanation

PR #169 (NVIDIA/kumo-engine, base main, head dm-diskgraph-424-readiness) changes 80 files, which exceeds the limit of 40. Evidence from the local repository: - The head commit acaa797 has one parent, d8aaf2b. That is the upstream/main revision the PR description names as its base. - git diff --name-only -M d8aaf2b6 acaa797a lists 80 paths. Rename detection is off in --no-renames mode and the count is still 80. - The status breakdown is 47 added (A) and 33 modified (M). - Against the local origin/main (b53781b), the merge-base is b53781b and the count is larger, which the previous run recorded as 140. The local origin/main is stale, so this count is not the reference. Every base that matches the PR's stated base gives more than 40 files. The head changed from the earlier commit named in the prior failure, but the file count did not drop. The prior failure therefore still stands. The check counts files only and ignores line counts.

Resolution

The PR changes 80 files. This exceeds the limit of 40. Split PR #169 into smaller, reviewable units of 40 files or fewer. Group the work by area: - Optional Buffered/Direct I/O. - Bounded caching and prefetch planning. - Python/Core option propagation. - Native packaging and platform checks. - Readiness tests. Keep the regenerated artifact and license inventories with the code they describe. Rebase each unit on the current main before review.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@danielmawhirter

danielmawhirter commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Change map

Current head acaa797a277982c2ac4737a89a9c577eeafdde06 is one SSH-signed commit directly on fetched main d8aaf2b6e818027b7a03f43bd96f8c3efd6263b0, covering the same 80 paths. The requested release requirement is to port #424's optional explicit I/O while preserving existing seeded neighborhoods and mmap defaults. Omit the Floyd sampler replacement; retain the reproduced cancellation, process-ownership and automatic-fusion fixes. #196, #197 and #198 are not dependencies.

The current rebase preserves upstream oversized-row/null-feature behavior across fused, mmap and explicit batched feature paths, and reuses upstream seed-column classification. Generated manifests/notices retain their canonical content, including upstream license indentation.

Actual optimized native BUILD_COMMIT is the exact head above, SHA256 4437578a151f976fef71c0fb6e8cfc31b7a7894307fd0986c02ef118790af2e8. Local AArch64 Linux Rust 1.96.1 qualification: 2,191 workspace tests passed / 28 explicitly ignored, ten strict native cases, three backend-specific empty-node-data passes, 60 native/runtime Python passes, three backend-specific oversized-row passes, and all eight task checks across three backends; Python suites have zero skips. Task checks use the unchanged fixture guard with the wheel physically inside an owned no-pip interpreter prefix. Earlier setup failures remain separate evidence.

A separate bounded actual-head parallel fork probe produced four passing child-created sampler controls and four 15-second inherited external-sampler timeouts; all eight children were reaped. This is an unresolved production blocker, with no baseline attribution or fix claimed. Keep this PR draft until a focused follow-up rejects inherited external samplers before native access or proves safe worker-local construction, with parallel-path regression coverage.

Exact-head CI passed Linux x86_64 Rust 1.99.1 workspace (2,194 passed / 28 ignored) and ten strict native cases, Clippy, Linux wheel, generic native Python (931 passed / 20 opt-in skips / one deselected), CPU task checks (eight tests / 24 task-backend executions) and 15 strict Python tests with zero skips, and macOS arm64 Rust 1.99.1 wheel build/install plus 11 mmap smoke tests. Both Engine package shards passed; the linked run records other integrations. Frozen published-candidate performance measurements remain scoped to candidate 0fe02c1b / native 75ca0e2f, separately from this new native build. Keep mmap as default and budget caches per process; production worker/storage-policy, GPU and Windows qualification remain open.

File What changed Why
.agents/skills/build-ui/SKILL.md Use upstream main as the default base, dm-ui branches and explicit same-name upstream pushes. Honor the requested upstream development workflow and remove active origin/master advice.
.agents/skills/build-ui/references/decision-tables.md Check the selected base on upstream, with main as the default. Honor the requested upstream development workflow and remove active origin/master advice.
.github/actions/engine-gpu-smoke/action.yml Validate and export the chosen I/O backend, use a 1 GiB direct cache, and retain the backend in the real GPU task summary. Run the finished candidate wheel against explicit storage backends without changing runner or credential policy.
.github/workflows/kumo-engine-pr.yml Add a whitelisted manual backend choice with default mmap and pass it to the GPU task action. Make direct or buffered eight-task GPU validation selectable while preserving mmap, repository trust checks and main’s reusable build_only workflow.
.github/workflows/testing.yml Require strict native Linux checks, macOS installed-wheel checks, all 24 task/backend runs and all native Python/Core boundary cases with both opt-in flags and a no-skip guard; use main’s current action pins and Rust toolchain action. Exercise the verified candidate wheel across task and ownership/parity/seeded boundaries; deny a green result from skipped native cases.
3rdparty/notices/inventory.json Regenerate resolved Linux/Windows artifact dependencies and the 404-component notice inventory for io-uring and unified pprof 0.14.1. Account for shipped dependencies under the existing policy without retaining the affected profiler version.
THIRD-PARTY.txt Regenerate resolved Linux/Windows artifact dependencies and the 404-component notice inventory for io-uring and unified pprof 0.14.1. Account for shipped dependencies under the existing policy without retaining the affected profiler version.
packages/diskgraph/Cargo.lock Resolve io-uring 0.7.13 and benchmark dependencies alongside main’s Arrow/Parquet 60.1.0, PyO3 0.29.3 and epserde 0.13.2; unify the profiler on pprof 0.14.1. Preserve current dependency controls and avoid reintroducing the affected profiler version.
packages/diskgraph/README.md Document backend configuration, per-process native runtime and cache budgets, generator entry points and benchmark usage. Make operational settings and benchmark limits explicit before enabling native I/O.
packages/diskgraph/diskgraph-datagenerator/README.md Describe the proposed backend, generator or benchmark entry points. Connect the imported tools with their usage and evidence.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_1pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_25pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_50pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_5pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/alt_detection_full_scale.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_100pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_10pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_1pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_25pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_50pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/ecommerce_features_5pct.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/pinterest_omnisage_pruned_organic.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/pinterest_omnisage_pruned_organic_graph.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/configs/pinterest_omnisage_pruned_organic_size_matched.json Carry compressed identity, ecommerce and organic Pinterest workload configuration. Retain reproducible workloads used by the source benchmarks.
packages/diskgraph/diskgraph-datagenerator/src/config.rs Add source generator distributions, feature data and compression support. Generate the complete proposed benchmark workloads.
packages/diskgraph/diskgraph-datagenerator/src/generator.rs Add source generator distributions, feature data and compression support. Generate the complete proposed benchmark workloads.
packages/diskgraph/diskgraph-engine/Cargo.toml Add benchmark dependencies and use Unix-only pprof 0.14.1, matching the patched version already used by ingest. Retain Linux/macOS profiling while removing the affected report-builder dependency and preserving Windows compilation.
packages/diskgraph/diskgraph-engine/src/bin/dg-engine-bench.rs Add the end-to-end benchmark, retaining Unix profiling through patched pprof and rejecting unsupported Windows profiling requests. Measure equivalent workloads while preserving the experimental Windows build boundary.
packages/diskgraph/diskgraph-engine/src/engine.rs Integrate explicit planning, preserve seeded sampling, select automatic fusion from shard capabilities, and normalize empty node data in fused/mmap/explicit feature paths using main's canonical seed-column classification. Preserve main's null-feature semantics and mmap behavior while explicit backends use supported standalone feature reads, including zero-hop requests.
packages/diskgraph/diskgraph-engine/src/output.rs Integrate explicit page planning and reusable sampling state; preserve current seeded selection and engine features. Expose the new backend through the existing relation, projection and per-seed commit paths.
packages/diskgraph/diskgraph-engine/src/state.rs Integrate explicit page planning and reusable sampling state; preserve current seeded selection and engine features. Expose the new backend through the existing relation, projection and per-seed commit paths.
packages/diskgraph/diskgraph-engine/src/tests_unit.rs Preserve upstream empty-node-data regressions and prefetch parity tests; assert the reader's configured backend. Execute the same null-feature regression in fresh mmap, Buffered and Direct processes without changing the expected values.
packages/diskgraph/diskgraph-index/Cargo.toml Add io-uring 0.7.13 only for Linux targets. Preserve supported macOS and native Windows mmap builds without Linux-only dependencies.
packages/diskgraph/diskgraph-index/src/bin/dg-page-io-bench.rs Carry the page-I/O benchmark, centralize platform-specific opens, and checksum literal requested bytes in every backend. Measure equivalent work and detect byte mismatches.
packages/diskgraph/diskgraph-index/src/lib.rs Add cached explicit I/O, bounded record planning and optional mmap prefetch. Implement the proposed sampler storage path while retaining computed offset positions.
packages/diskgraph/diskgraph-index/src/page_io.rs Add explicit I/O/cache planning, consume failed reservations once and create process-owned runtime/cache state after fork. Validate backend input and reject explicit I/O on non-Linux platforms. Preserve byte correctness, prevent replacement-slot cancellation and avoid inherited locks/channels or inaccessible cache mappings.
packages/diskgraph/diskgraph-index/src/page_io_readiness_tests.rs Test exact-once cancellation and error recovery; fork while configuration or cache locks are held. Verify deterministic ownership and child initialization rather than relying on timing.
packages/diskgraph/diskgraph-index/src/prefetch.rs Add bounded prefetch, retaining Unix page advice and a non-Unix no-op. Preserve the proposed Linux optimization without introducing Unix syscall requirements on Windows.
packages/diskgraph/diskgraph-index/src/reader.rs Add explicit range/page reads and prefetch while using the shared platform-aware direct-file helper. Retain current offset semantics and reject unsupported platform operations clearly.
packages/diskgraph/diskgraph-index/tests/page_bench_checksum.rs Compare mmap, buffered and direct checksums over patterned bytes including repeated page requests. Require all benchmark backends to consume identical data.
packages/diskgraph/diskgraph-index/tests/page_io_readiness.rs Check native bytes, concurrent cancellation, I/O-error recovery, fork timing, populated cache, and inherited pending/prepared handles. Prove fresh child reads work and inherited unsafe access returns typed errors.
packages/diskgraph/diskgraph-ingest/src/lib.rs Carry the encode-budget override and buffered LZ4 output while retaining current ingestion behavior. Preserve proposal tuning and performance changes without reverting newer budget and recovery handling.
packages/diskgraph/diskgraph-pybind/python/tests/test_backend_portability.py Exercise installed-wheel default mmap ingestion/sampling, platform backend rejection, and strict backend text/invalid-UTF8 parsing. Prove supported macOS behavior and matching explicit configuration errors.
packages/diskgraph/diskgraph-pybind/python/tests/test_native_io_readiness.py Verify parent/train-table parity, projections, seeded replay, backend fusion, inherited native owners, and warmed real core DataLoader workers. Exercise actual Python/runtime/core boundaries and preserve default mmap behavior.
packages/diskgraph/diskgraph-pybind/src/runtime_batch_iterator.rs Guard native runtime-index and iterator access/drop with process ownership. Reject inherited worker handles before touching parent Rayon/channel state while preserving same-process values.
packages/diskgraph/docs/cached-direct-io-sampler.html Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/proposals/buffered-io-uring-sampler.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/proposals/direct-io-cache-sampler.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/README.md Index the imported benchmark reports and current readiness review. Keep historical performance evidence distinct from current release findings.
packages/diskgraph/docs/reports/diskgraph-all-edge-3hop-100pct-mmap-vs-direct-2026-08-07.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-all-edge-3hop-scale-matrix-2026-08-08.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-direct-cache-optimized-2026-08-07.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-ecommerce-feature-io-benchmark-2026-08-08.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-i8g-ecommerce-feature-io-benchmark-2026-08-10.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-i8g-identity-scale-matrix-2026-08-10.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-localssd-ecommerce-feature-mmap-vs-direct-2026-08-19.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-localssd-mmap-vs-direct-2026-08-18.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-pinterest-hotspot-real-pkey-2026-08-21.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-pinterest-i8g-io-benchmark-2026-08-09.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-pinterest-organic-capped-bfs-2026-08-21.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-real-pkey-e2e-mmap-vs-direct-2026-08-20.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/diskgraph-sampler-io-benchmark-2026-08-06.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/docs/reports/pr-424-production-readiness-2026-10-30.md Record observed fixes, the profiler advisory correction and inherited index-rebuild requirement; add the November 9 core-only rebase scope and distinguish historical CPU/GPU/platform qualification. Keep historical evidence distinct from the latest main integration and require fresh artifact qualification without making speedup claims.
packages/diskgraph/docs/sampling-mmap-baseline-2026-08-06.md Retain the proposal design, methodology and historical benchmark report. Carry the complete source proposal and its original evidence without claiming a fresh performance measurement.
packages/diskgraph/scripts/bench_fio_iops.sh Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction. Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/bench_fio_raw_matrix.sh Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction. Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/bench_pinterest_organic_pkey.sh Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction. Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/bench_pinterest_pruned_pkey.sh Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction. Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/bench_sampling_iops.sh Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction. Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/scripts/check_native_io.sh Run seven public native lifecycle cases, two lock-held fork cases and equal-byte benchmark separately with fixed cache/ring settings. Make native I/O capability and every named case mandatory in Linux CI without green skips.
packages/diskgraph/scripts/evict_file_cache.py Add source workload, fio and sampler benchmark helpers with file-scoped cache eviction. Reproduce the proposed measurements and retain safe, focused cache controls.
packages/diskgraph/third_party/artifact-manifest.json Regenerate resolved Linux/Windows artifact dependencies and the 404-component notice inventory for io-uring and unified pprof 0.14.1. Account for shipped dependencies under the existing policy without retaining the affected profiler version.
packages/kumo-core/src/kumo_core/diskgraph/lightning_data.py Type cached native handles and rebuild them when worker PID changes; clear ownership on serialization. Support warmed-parent DataLoader fork without inherited native runtime reuse; preserve main’s feeder-error handling and graph_schema API.
packages/kumo-core/src/kumo_core/diskgraph/stores.py Validate backend settings and select standalone automatic training mode for explicit I/O while preserving mmap sizing and worker policy. Prevent small-graph training from forcing unsupported fused execution; preserve main’s removal of legacy store APIs and make the minimal required import/spacing lint repairs.
packages/kumo-core/tests/test_diskgraph_auto_config.py Cover backend choices, exact invalid strings, edge-size thresholds and platform/start-method policies. Preserve current mmap defaults and verify explicit backend automatic policy across 118 cases.
packages/kumo-engine/test/test_native_io_tasks.py Run 24 real CPU fits with canonical entity/time inputs, finite complete predictions and own-model repeat/reload checks; compare backends using one saved model and snapshot. Cover every concrete task type through the actual Core default policy and verify full-key backend parity at absolute tolerance 1e-6.
packages/kumo-engine/test/wheel_gpu_smoke.py Record the selected edge backend and direct-cache configuration in the retained eight-task GPU summary. Associate real fit and prediction evidence with the backend requested by the trusted workflow.
test/scripts/test_engine_pr_workflow.py Verify allowed manual backend choices, the mmap default, direct-cache environment and retained summary wiring. Protect explicit backend validation while preserving existing workflow security and publication behavior.
test/train/datamodule/test_diskgraph_lightning_data.py Initialize explicit runtime handle/PID cache fields in the synthetic dataset fixture. Match the typed production dataset state used by existing DataModule regressions.
tools/migrations/diskgraph/check_native_runtime.py Add a bounded Kumo iterable, DataLoader and runtime-adapter diagnostic for each backend. Verify the actual training integration and reproduce worker failure after parent I/O.

@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 3, 2026 18:41 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 3, 2026 19:55 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 4, 2026 17:17 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter changed the base branch from main to dm-bigquery-test-eligibility November 4, 2026 17:18
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 4, 2026 17:57 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 4, 2026 18:26 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter changed the base branch from dm-bigquery-test-eligibility to dm-snapshot-checksums-portable November 4, 2026 18:43
@danielmawhirter danielmawhirter changed the title Make DiskGraph explicit I/O production-ready [prototyping] Make DiskGraph explicit I/O production-ready Nov 6, 2026
@danielmawhirter
danielmawhirter changed the base branch from dm-snapshot-checksums-portable to main November 9, 2026 19:54
@danielmawhirter
danielmawhirter force-pushed the dm-diskgraph-424-readiness branch from 96a6538 to 2643ad7 Compare November 9, 2026 19:59
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 9, 2026 19:59 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter force-pushed the dm-diskgraph-424-readiness branch from 2643ad7 to 5fd6218 Compare November 10, 2026 00:55
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 10, 2026 00:55 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter force-pushed the dm-diskgraph-424-readiness branch from 5fd6218 to c1a04a3 Compare November 10, 2026 05:38
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 10, 2026 05:38 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter force-pushed the dm-diskgraph-424-readiness branch from c1a04a3 to a45b767 Compare November 18, 2026 20:26
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 18, 2026 20:26 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter force-pushed the dm-diskgraph-424-readiness branch from a45b767 to 4a340a6 Compare November 18, 2026 20:42
@danielmawhirter
danielmawhirter temporarily deployed to gha-kumo-workflow-tests November 18, 2026 20:42 — with GitHub Actions Inactive
@danielmawhirter
danielmawhirter force-pushed the dm-diskgraph-424-readiness branch from 4a340a6 to acaa797 Compare November 19, 2026 02:23
@danielmawhirter
danielmawhirter deployed to gha-kumo-workflow-tests November 19, 2026 02:23 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
gha-kumo-workflow-tests — acaa797a Deployed Nov 19, 2026 by danielmawhirter via import #2670
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant