Repository navigation
[DistRDF] Support return_filenames and merge options in Snapshot (fixes #17136) - #23815
muntasir-396 wants to merge 1 commit into
Conversation
root-project#17136) - Allow passing return_filenames=True to distributed Snapshot to return the list of output file paths instead of a distributed RDataFrame. - Allow passing merge=True to distributed Snapshot to merge partial task output files into the target file via ROOT.TFileMerger. - Preserve full backward compatibility when neither option is set. - Add unit tests for Snapshot operation option handling and execution.
muntasir-396
changed the title
Snapshot in distributed RDataFrame: 1. return_filenames: bool = False: When True, returns the list of snapshot file paths rather than a distributed RDataFrame. 2. merge: bool = False: When True, merges partial task snapshot files into the user-specified target file using ROOT.TFileMerger and cleans up temporary partial files. Default behavior remains completely unchanged and backward-compatible. Supersedes the stalled draft PR #18422 with a clean, per-call implementation following maintainer feedback. ### Checklist - [x] Follows ROOT coding guidelines - [x] Verified backward compatibility with existing DistRDF workflows - [x] Added unit tests for argument parsing and snapshot execution
vepadulano
left a comment
vepadulano
left a comment
There was a problem hiding this comment.
Dear @muntasir-396 ,
Thanks for taking interest again in the project. As you can see from the issue you have linked, this PR is not enough. The changes per se are fairly straightforward, but before bringing it to production this would require thorough testing with realistic analysis use case, at least with LHC open data and on distributed computing infrastructures. I estimate a proper benchmarking would require 1 month of full time work (excluding coding). At the moment I cannot commit to supervising that, but perhaps this could change in the future somewhat. In the meanwhile, you can think if you are interested in doing that as well.
|
Dear @vepadulano, thank you for the clear feedback and context! I completely understand the need for realistic LHC distributed benchmarking and that supervision isn't available right now. We can keep this on hold for the future. Thanks again |
Summary
This PR resolves #17136 by adding configurable behaviors to
Snapshotin distributed RDataFrame:return_filenames: bool = False: WhenTrue, returns the list of snapshot file paths rather than a distributedRDataFrame.merge: bool = False: WhenTrue, merges partial task snapshot files into the user-specified target file usingROOT.TFileMergerand cleans up temporary partial files.Default behavior remains completely unchanged and backward-compatible.
Supersedes the stalled draft PR #18422 with a clean, per-call implementation following maintainer feedback.
Checklist