Repository navigation
[io] Fix data loss in nested directories during batched TFileMerger - #23791
Open
JainAyush-01 wants to merge 3 commits into
Open
JainAyush-01 wants to merge 3 commits into
JainAyush-01 wants to merge 3 commits into
Conversation
When TFileMerger merges files with SetMaxOpenedFiles (e.g. hadd -n),
merging is performed incrementally across batches with kIncremental.
In TFileMerger::MergeOne, the cleanup condition previously checked:
if (ownobj && (!(type & kIncremental) || dirobj->GetFile() != target))
For nested subdirectories at depth >= 2, `target` is the parent
TDirectoryFile, whereas `dirobj->GetFile()` returns the root TFile
(`fOutputFile`). Consequently, `dirobj->GetFile() != target` always
evaluated to true for nested directories, prematurely deleting the
in-memory directory before fOutputFile->Write() could flush it,
causing silent data loss of objects merged in subsequent batches.
Comparing `dirobj->GetFile() != fOutputFile` properly ensures that
directories belonging to the output file are not prematurely deleted
during incremental merges.
Fixes root-project#23666. Added regression test NestedDirMaxOpenedFiles in
TFileMergerTests.cxx.
* Allocate TFile and TTree on the heap (to avoid corner-case issues with DeleteChangesMemory) * Clarify intent of the test: it's not to use `kIncremental` but to exercise the `-n` option in hadd, using SetMaxOpenedFiles in TFileMerger * Also check correct TTree entry values
vepadulano
force-pushed
the
fix/23666-hadd-nested-dirs
branch
from
October 23, 2026 16:35
629d49d to
d901d78
Compare
enirolf
force-pushed
the
fix/23666-hadd-nested-dirs
branch
from
October 23, 2026 16:44
3438441 to
65ae899
Compare
Test Results 23 files 23 suites 3d 18h 51m 33s ⏱️ For more details on these failures, see this check. Results for commit 65ae899. |
Author
|
Thanks @vepadulano and @enirolf for improving the test and adding the comprehensive roottest coverage! I noticed in the CI run for commit 65ae899 that the Alma10 build failed in |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This Pull request:
Fixes silent data loss when merging ROOT files containing nested subdirectories (depth >= 2) with chunking / batching enabled (
TFileMerger::SetMaxOpenedFilesorhadd -n).Changes or fixes:
Root Cause:
When
TFileMergermerges files incrementally in batches (type & kIncremental),TFileMerger::MergeOnecleans up in-memory directory objects at lines 830–834:For top-level subdirectories (depth 1),
targetis theTFile*itself, sodirobj->GetFile() == targetevaluates to true anddirobjis preserved across batches until the finalfOutputFile->Write()flushes all accumulated objects.However, for nested subdirectories at depth$\ge 2$ (e.g.,
nested/dir):targetis the parentTDirectoryFile*(nested)dirobj->GetFile()returns the rootTFile*(fOutputFile)Because a
TDirectoryFile*does not equal aTFile*,dirobj->GetFile() != targetalways evaluated totrue. This prematurely deleted the in-memory subdirectorydirobjbefore subsequent batches could write and flush into it, causing silent loss of objects merged in later batches.Fix:
Change
dirobj->GetFile() != targettodirobj->GetFile() != fOutputFile. This correctly checks whetherdirobjbelongs to the output file being accumulated across incremental batches, ensuring nested directories and their contents remain intact in memory until written.Testing:
TEST(TFileMerger, NestedDirMaxOpenedFiles)inio/io/test/TFileMergerTests.cxx.-n 2,-n 3).gtest-io-io-TFileMergertest suite viactest.Checklist:
This PR fixes #23666