Skip to content

fix(diskgraph): run bench_runtime_batch_iterator with resolution-mode seeds - #330

Merged
nittin-murthi merged 2 commits into
mainfrom
fix/bench-runtime-batch-iterator-seed-mode
Nov 16, 2026
Merged

nittin-murthi merged 2 commits into
mainfrom
fix/bench-runtime-batch-iterator-seed-mode

Conversation

@nittin-murthi

@nittin-murthi nittin-murthi commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

diskgraph-pybind/python/tests/bench_runtime_batch_iterator.py fails on main before timing anything:

diskgraph.DiskgraphSystemError: Ingest failed: table train_seeds fkey ENTITY1 references missing table users

The seed table still used the removed FK-edge seed mode (external_parent fkey). This builds the seed fixture the same way as test_runtime_batch_iterator_e2e.py: an entity_node_id column, fkeys: [], and entity_node_type="users" on the iterator. The timed loop is unchanged.

Found while benchmarking #133.

Change map

File-by-file rationale

Testing Done

Against a uv pip install of packages/diskgraph at main (32afaed8), Linux:

  • Original script: fails with the error above.
  • Fixed script, default and --n-users 200000 --orders-per-user 8 --n-seeds 200000 --batch-size 512 --n-iter 300: completes for both fused and non-fused dispatch.
  • Seed mapping check: with shuffle=False, all 2,500 task.seed_node_ids equal ENTITY1 (i % n_users) for fused and non-fused iteration.

Checklist

  • No breaking changes (benchmark script only).
  • No examples or documentation updates needed.
  • No unit tests changed; this is a standalone benchmark script.

🤖 Generated with Claude Code

… seeds

The benchmark still ingested its seed table with the removed FK-edge seed
mode, so ingest failed with "fkey ENTITY1 references missing table users"
before any timing ran. Build the seed fixture the way
test_runtime_batch_iterator_e2e.py does: an entity_node_id column, no fkeys,
and entity_node_type="users" on the iterator. The timed loop is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@nittin-murthi
nittin-murthi temporarily deployed to gha-kumo-workflow-tests November 13, 2026 17:51 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Explain Change →

Important

Review skipped

Review was skipped as selected files did not have any reviewable changes.

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 5a1e519 and e85e97f.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

No actionable comments were generated in the recent review.

Walkthrough

The benchmark fixture now writes each seed’s user ID to an entity_node_id column and declares that column as u_int64. The train-seed schema no longer defines the ENTITY1 foreign-key mapping. Both the warm-up and measured iterator calls now specify entity_node_type="users".

Merge Risk: Minimal · previous review 5a1e5

This updates a benchmark fixture so it runs with resolution-mode seeds. It does not affect production code paths. The risk of merging is minimal.

Security architecture risk: Minimal · up to 5a1e5

The change updates synthetic benchmark seeds and supplies their node type to the existing iterator. It does not introduce or worsen a material security, ownership, or failure-containment risk.

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

Failed Failed checks: 3 inconclusive

Check name Status Explanation Resolution
Change Map Design Review Inconclusive Pull request inputs changed after this check was assessed; its verdict is no longer current. Re-run the check against the current pull request inputs.
Pr Size Inconclusive Pull request inputs changed after this check was assessed; its verdict is no longer current. Re-run the check against the current pull request inputs.
Public Resource Access Inconclusive Pull request inputs changed after this check was assessed; its verdict is no longer current. Re-run the check against the current pull request inputs.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between eff6c2a and 5a1e519.

Files selected for processing: 1

  • packages/diskgraph/diskgraph-pybind/python/tests/bench_runtime_batch_iterator.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@nittin-murthi

nittin-murthi commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Change map

File Function / scope What changed Why
packages/diskgraph/diskgraph-pybind/python/tests/bench_runtime_batch_iterator.py _build_fixture(), _run_one() Seed parquet gains entity_node_id (= ENTITY1, the single-shard users node id); seed ingest config drops the external_parent fkey (fkeys: []); both iterator constructions pass entity_node_type="users". FK-edge seed mode was removed, so seed ingest failed ("references missing table users") and the iterator requires entity_node_type. Mirrors the fixture in test_runtime_batch_iterator_e2e.py.

Failure: the script raised DiskgraphSystemError during fixture ingest on main (32afaed8), so no benchmark ran. Smallest fix: fixture and iterator kwargs only; timed loop and reported metrics are unchanged. Verification: the script completes at default and large sizes, and seed_node_ids match ENTITY1 for all 2,500 seeds (fused and non-fused). Stopping condition: the benchmark runs on main; it is the only file under diskgraph-pybind/python that still uses external_parent.

Merge revalidation: merged current main (08577147e832) in e85e97f90281, without conflicts. The PR diff against main remains the same single benchmark file (7 additions, 11 deletions). With the matching native DiskGraph build, the 10,000-user / 80,000-order / 10,000-seed fixture completed all 79 batches in each mode (--batch-size 128 --n-iter 100000). Diff whitespace checks passed.

@nittin-murthi

Copy link
Copy Markdown
Contributor Author

Additional validation and CPU measurements against current main (08577147e832585cc6e6094188ecb81f7bf904ae), using this PR's script at 5a1e5197565ce1a0ca8c73ddef4f67cefd3a0acb.

The unmodified main script still fails during seed ingestion with train_seeds fkey ENTITY1 references missing table users. The PR applies cleanly and runs successfully. These numbers compare fused versus non-fused dispatch after the repair, not a performance improvement introduced by this PR.

Users / seeds Orders Batch size Batches Non-fused median (range), ms Fused median (range), ms Fused speedup
10,000 80,000 128 79 40.6 (39.4–41.4) 28.0 (27.7–28.9) 1.45×
200,000 1,600,000 128 1,563 788.3 (767.6–791.3) 542.5 (538.8–561.2) 1.45×
200,000 1,600,000 512 391 668.2 (659.2–672.7) 500.9 (493.7–518.6) 1.33×

Method:

  • Apple Silicon CPU, macOS 26.7.1 arm64, 18 logical CPUs; Python 3.12.14, DiskGraph 0.2.0 built from 08577147e832, PyArrow 24.0.0.
  • Seven runs per mode/configuration (42 timed runs), each in a fresh subprocess, alternating which mode runs first. Reused the same on-disk fixture per dataset size; no OS cache dropping.
  • Used the PR's unchanged _run_one() with n_iter=100000 to traverse all seeds. Its warm-up and measured iterator constructions are unchanged: one iterator worker, automatic native thread selection, shuffle=False, rnd_seed=None, two layers with fanout 8.
  • Timings cover iteration and the existing edge-count loop; exclude ingestion, index opening, and warm-up. Fixture ingestion took approximately 0.51 s / 1.54 s for the small / large dataset.
  • Separate validation passes checked every seed ID against i % n_users: 10,000 / 200,000 seeds delivered. Both modes returned the same total edge counts (160,000 / 3,200,000), batch counts, and mean edges per batch. This is not a complete feature-by-feature equality check.
  • Median whole-process peak RSS was approximately 60 MiB for the small fixture and 101–102 MiB for the large fixture, with little difference between modes. These are fresh-subprocess measurements, not the original script's cumulative two-mode RSS.

For a single-pass reproduction with the matching native build, run the PR script from packages/diskgraph/diskgraph-pybind:

python python/tests/bench_runtime_batch_iterator.py --n-users 10000 --orders-per-user 8 --n-seeds 10000 --batch-size 128 --n-iter 100000
python python/tests/bench_runtime_batch_iterator.py --n-users 200000 --orders-per-user 8 --n-seeds 200000 --batch-size 128 --n-iter 100000
python python/tests/bench_runtime_batch_iterator.py --n-users 200000 --orders-per-user 8 --n-seeds 200000 --batch-size 512 --n-iter 100000

Fused dispatch had 1.33–1.45× higher throughput on this narrow synthetic fixture. The fixture has few feature columns; these local warmed-cache results do not establish the best dispatch mode for wide features, other graph shapes, cold storage, GPU training, or production workloads. No benchmark source changes were needed beyond this PR.

Individual timed runs (milliseconds, chronological within each mode)
Seeds Batch size Mode Seven runs, ms
10,000 128 Non-fused 39.379, 41.404, 40.934, 40.627, 40.648, 40.825, 40.546
10,000 128 Fused 27.956, 28.006, 28.049, 28.914, 28.165, 27.698, 28.478
200,000 128 Non-fused 791.281, 767.632, 788.118, 791.262, 788.254, 791.232, 786.748
200,000 128 Fused 541.553, 550.679, 547.699, 541.939, 561.177, 542.534, 538.785
200,000 512 Non-fused 659.195, 670.161, 672.692, 668.158, 664.621, 662.151, 669.535
200,000 512 Fused 518.612, 493.705, 500.919, 500.347, 501.975, 500.059, 503.070

@nittin-murthi
nittin-murthi marked this pull request as ready for review November 14, 2026 23:46
@nittin-murthi
nittin-murthi temporarily deployed to gha-kumo-workflow-tests November 14, 2026 23:48 — with GitHub Actions Inactive
@nittin-murthi
nittin-murthi enabled auto-merge (squash) November 16, 2026 21:54
@nittin-murthi
nittin-murthi merged commit 15f4c24 into main Nov 16, 2026
114 checks passed
@nittin-murthi
nittin-murthi deleted the fix/bench-runtime-batch-iterator-seed-mode branch November 16, 2026 22:04

This branch was previously deployed

1 inactive deployment
gha-kumo-workflow-tests — e85e97f9 Deployed Nov 14, 2026 by nittin-murthi via import #2393
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants