Repository navigation
[df] Improve RNTuple IMT scheduling with sliding-window staging and tail splitting - #23821
Closed
SamdaniSayam wants to merge 2 commits into
Closed
SamdaniSayam wants to merge 2 commits into
SamdaniSayam wants to merge 2 commits into
Conversation
The MT scheduling of RNTupleDS is changed. The data source adopts now a similar pattern as TTreeDS, implementing a ProcessMT(ROOT::Detail::RDF::RLoopManager &lm) method. This allows the data source to manage the creation and scheduling of multithreaded tasks, inside of which the loop manager is invoked to apply the computation graph to the range of entries in the task. A brief summary of the scheduling approach follows: 1. One task is launched in parallel per input file 2. At the beginning of the file-based task, a page source is created and the list of clusters is computed a. If the user has not specified a global entry range, it is the actual list of clusters of the page source. This happens in ScheduleTasksForFile. b. If the user has specified a global entry range, before launching the file-based tasks the data source had computed a full list of global entry offsets across all input files. The list of clusters of the page source is then adjusted according to the position of the global entry range w.r.t. the global offsets of this file. This happens in ScheduleTasksForFileWithGlobalOffset. 3. The task goes on within ScheduleTaskForPageSource, which invokes TBB's parallel_for with two arguments, one respecting the Range and the other respecting the Body requirements. See the docs respectively at https://oneapi-spec.uxlfoundation.org/specifications/oneapi/latest/elements/onetbb/source/named_requirements/algorithms/range and https://oneapi-spec.uxlfoundation.org/specifications/oneapi/latest/elements/onetbb/source/named_requirements/algorithms/par_for_body. a. The Range argument is of type RPageSourceSplittableRange, a struct representing a range of entries (N consecutive clusters) tied to a specific page source to be processed. b. The first time parallel_for is invoked, an RPageSourceSplittableRange instance is passed to it which includes the whole list of clusters for that page source. c. TBB immediately calls the Range object's SplitWork method, which will split the cluster range in roughly two equal parts, the first part is re-assigned to the same object, the second part is used to create a new RPageSourceSplittableRange. d. TBB continues to split the task recursively, until a list of all non-splittable tasks is available. e. Only then, the TBB parallel_for will schedule the various calls to the ParallelForBody function, each taking one of the cluster ranges generated before. f. Within ParallelForBody, the computation graph is executed by reading data from the page source, restricting it to the assigned cluster range, by calling RLoopManager::RNTupleThreadTask. This approach delegates to the scheduling engine (TBB in this case) the responsibility to optimise the task processing and split tasks that may be running for too long (e.g. some files may contain many more entries than others). The strategy is also documented in a markdown file for posterity.
…ail splitting Follow-up to PR root-project#23327 (Issue root-project#23813). Replaces TBB splittable ranges with file-level task distribution, background sliding-window staging, and dynamic tail-cluster scheduling: - One file-level task per input file dispatched into a tbb::task_group. - Sequential cluster iteration per file preserves read-ahead and cluster prefetching across cluster boundaries. - Staging thread prefetches page sources ahead of the next file to be claimed with a window bounded by fNSlots. - Transition into tail-cluster scheduling once all files have been claimed; remaining clusters are scheduled individually across idle workers. - Strictly bounds concurrent open page sources to at most 2 * nSlots. - Added tests for single-file tail scheduling, global entry ranges, and file descriptor bounds with high slot counts (8/16 slots, 512 files). - Documented new architecture in tree/dataframe/doc/parallel_scheduling.md. - Added reproducible benchmark suite (benchmark_rntuple_imt.C and benchmark_remote_simulation.py). Resolves root-project#23813.
SamdaniSayam
requested review from
bellenot,
dpiparo,
hageboeck and
vepadulano
as code owners
October 23, 2026 16:52
vepadulano
requested changes
Oct 23, 2026
vepadulano
left a comment
vepadulano
left a comment
Member
There was a problem hiding this comment.
Dear @SamdaniSayam ,
I have to close this PR because it violates the guidelines in CONTRIBUTING.md, shows a complete lack of understanding of the problem and the code organisation of the project, and limits testing to a synthetic benchmark, whereas this would need an actual testing campaign with realistic analysis use cases.
Thanks for taking interest in the ROOT project. If that is truly the case, I invite you to take a look at the list of issues with the good first issue label, and to ask with a comment if help is needed on that issue.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to PR #23327, addressing Issue #23813 (Improve RNTuple IMT scheduling).
Note
This PR is stacked on top of PR #23327 (
df-rntuple-imt-work-stealing). Once #23327 merges intomaster, this PR will cleanly contain only the scheduling redesign.This PR replaces TBB's recursive range-splitting abstraction (
tbb::parallel_forwith splittable ranges) with an explicit task distribution model that restores background file staging and sequential cluster prefetching:tbb::task_group. Workers set the page source entry range to the entire file (or the trimmed file-range when a global entry range is requested), enabling sequential read-ahead across cluster boundaries.fNSlotsfiles.fNextFileToClaim >= nFiles), any file-level task detecting tail mode schedules its remaining unread clusters individually into the task group. Idle threads steal single-cluster tasks, eliminating thread straggling at the tail.RSlotStack, ensuring that at mostfNSlotsfiles are open in staging and at mostfNSlotsare open across active worker slots.Testing & Validation
tree/dataframe/doc/parallel_scheduling.mddescribing the new scheduler architecture.SingleFileTailScheduling,GlobalEntryRangeMT, and high slot count (8/16 slots, 512 files) tests totree/dataframe/test/datasource_ntuple.cxx. All 24 tests ingtest-tree-dataframe-datasource-ntuplepass cleanly.tree/dataframe/test/benchmark_rntuple_imt.Cand simulated remote latency testing intree/dataframe/test/benchmark_remote_simulation.py.Closes #23813.