Skip to content

[C++][Python] Option to read a memory-mapped IPC stream without MADV_WILLNEED #55656

Open

Description

@behroozazarkhalili

Describe the enhancement requested

MemoryMappedFile::ReadAt calls MemoryAdviseWillNeed on every range it returns (cpp/src/arrow/io/file.cc). With the IPC stream reader that's one call per message header and one per batch body, so reading a memory-mapped file with read_all() asks the kernel to prefetch the whole file, even when the caller only wants the zero-copy table and will touch a small part of it.

This is the case in Hugging Face datasets: load_from_disk and the dataset cache open each Arrow file with pa.ipc.open_stream(pa.memory_map(path)).read_all(), and the rows are only read later, often a few at a time. On a cold page cache the whole dataset is read from disk at load time (huggingface/datasets#2547). On a local xfs disk, with a 596 MiB dataset (2 files of 1000 batches), 100% of the files were read when opening it (1.23 s). Reading the same stream from a Python mmap advised MADV_RANDOM read 1.4% (0.13 s), but datasets can't use that as a fix because it also turns off readahead for the later reads.

Reproducer (pyarrow 25.0.1, Linux):

import os
import pyarrow as pa

path = "data.arrows"
batch = pa.record_batch({"x": pa.array(range(1000), pa.int64()), "s": pa.array(["a" * 32] * 1000)})
with pa.OSFile(path, "wb") as sink, pa.ipc.new_stream(sink, batch.schema) as writer:
    for _ in range(1000):
        writer.write_batch(batch)

with pa.memory_map(path) as source:
    table = pa.ipc.open_stream(source).read_all()

strace -f -e trace=madvise python repro.py shows 2001 MADV_WILLNEED calls for this 1000-batch file.

This was discussed in #18920 / #11588 for random access in an IPC file. There, @pitrou suggested an option on the IPC reader to disable the advisory hints, and @westonpace suggested an auto_willneed flag on MemoryMappedFile::Open. The issue was closed as stale. Would one of these be acceptable now, with the default unchanged (prefetch on)? Either one would let datasets open memory-mapped files without reading them, and keep the normal readahead for later reads.

Component(s)

C++, Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions