You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++][Python] Option to read a memory-mapped IPC stream without MADV_WILLNEED #55656
MemoryMappedFile::ReadAt calls MemoryAdviseWillNeed on every range it returns (cpp/src/arrow/io/file.cc). With the IPC stream reader that's one call per message header and one per batch body, so reading a memory-mapped file with read_all() asks the kernel to prefetch the whole file, even when the caller only wants the zero-copy table and will touch a small part of it.
This is the case in Hugging Face datasets: load_from_disk and the dataset cache open each Arrow file with pa.ipc.open_stream(pa.memory_map(path)).read_all(), and the rows are only read later, often a few at a time. On a cold page cache the whole dataset is read from disk at load time (huggingface/datasets#2547). On a local xfs disk, with a 596 MiB dataset (2 files of 1000 batches), 100% of the files were read when opening it (1.23 s). Reading the same stream from a Python mmap advised MADV_RANDOM read 1.4% (0.13 s), but datasets can't use that as a fix because it also turns off readahead for the later reads.
strace -f -e trace=madvise python repro.py shows 2001 MADV_WILLNEED calls for this 1000-batch file.
This was discussed in #18920 / #11588 for random access in an IPC file. There, @pitrou suggested an option on the IPC reader to disable the advisory hints, and @westonpace suggested an auto_willneed flag on MemoryMappedFile::Open. The issue was closed as stale. Would one of these be acceptable now, with the default unchanged (prefetch on)? Either one would let datasets open memory-mapped files without reading them, and keep the normal readahead for later reads.
Activity