Repository navigation
Improve RNTuple IMT scheduling #23813
Description
Explain what you would like to see improved and how.
This PR #23327 introduced an improved design of the RNTuple IMT scheduling. After discussion among the team, there seems to be consensus on a different design that could improve the scheduling even further. A starting point is #23327 (review) . This should be implemented and benchmarked against the design implemented by #23327 .
ROOT version
6.42 and onwards
Installation method
Any
Operating system
Any
Additional context
No response
Activity
SamdaniSayam commentedon Oct 24, 2026
Hi @vepadulano ,
I revisited PR #23821 in light of your review. Here's the new direction I'm considering for #23813 , I'd like to confirm it's acceptable before writing any code:
Base: cleanly stacked on 20bebc0 (#23327 head), branch fix/23813-rntuple-imt-staging already created.
Minimal plan (touch only RNTupleDS.{hxx,cxx} + parallel_scheduling.md, no public-API/CMake changes):
Keep #23327's splittable-range tbb::parallel_for task layer untouched.
Add a background staging thread (ExecStagingMT) under #ifdef R__USE_IMT that stages page sources ahead of the next claimed file in a window bounded by nSlots (constructor already pre-opens file 0).
File-level tasks take the staged page source instead of creating it inline.
When all files are claimed (fNextFileToClaim >= nFiles), switch to tail-scheduling mode: remaining clusters are enqueued as individual tasks, each cloning the parent page source after acquiring a slot, with a single-cluster entry range (bounds concurrent open page sources at 2 * nSlots).
Leave #23327's IMT helpers (RLoopManager, core/imt, RLoopManager.hxx) exactly as-is.
Add the uncontroversial tests: SingleFileTailScheduling, GlobalEntryRangeMT, and the 8/16-slot fd-limit cases in datasource_ntuple.cxx.
Benchmarks & testing (reported here, not committed):
ctest -R datasource-ntuple + processor: pass.
Broad tree/dataframe regression subset: 12/12 pass.
Realistic CMS dimuon analysis on http://root.cern/files/tutorials/ntpl004_dimuon_v1.root (8-file chain = 320M entries): bit-stable results across nSlots = 0,2,4,8.
fd-limit check on 400 distinct files at nSlots = 8/16.
Is this the right direction? Specifically: keep #23327's splittable-range layer and only add staging + tail scheduling? And keep benchmarks/latency shims out of the PR?
I'll wait for a thumbs-up before pushing anything.
vepadulano commentedon Oct 24, 2026
This issue is closed. Please refrain from polluting the issue tracker.