llama : add unbuffered positional read on Windows

[nibor1896]

Overview

Closes #26541.

(not quite the same as PR #26014, but the full path to resolution:
26014 does not contain "OVERLAPPED" and no "handle / thread")

Problem:
On Windows, llama.cpp cannot do "read from byte X" - only "read from
wherever you currently are". When several threads want to read from the
same file, they get in each other's way: they keep moving each other's
position.

The code implied it could read unbuffered: it takes the flag, then does
nothing with it, answers "yes" when asked, and writes "read unbuffered"
to the log.

What I built together with Claude:
Exactly what was already promised. llama.cpp can now jump to any point
(byte X) and read unbuffered, and every thread gets its own handle, so
they no longer interfere.

The other six pieces are not features, they are dependencies this needs.

Shared handle at depth 8: 1.01x
One handle per thread: 2.22x

Additional information

tests/test-llama-file.cpp is new and covers the read path, which no
test touched before. 18 checks, all passing on this branch against
current master.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Built with Claude Code (AI). Text written by me (Robin - Human).

[nibor1896]

Adding the measurement behind this, since the PR body doesn't carry it.

Why the hard "true" from "has_direct_io()" matters in practice: external code trusts
it. #25294's Windows branch calls it, logs "MoE expert streaming uses O_DIRECT (page
cache bypassed)", and then reads buffered through one shared handle serialised behind
a "static std::mutex".

Measured through "llama_file" itself, Windows 11 on NVMe, 12.75 MiB requests:

arm	throughput	ratio
1 thread, private handle	5,009.2 MB/s	1.000x
8 threads, one handle per worker	10,533.4 MB/s	2.103x
8 threads, shared handle	4,963.3 MB/s	0.991x

A shared "llama_file" does not parallelise: eight threads on one handle land at the
speed of one. The multi-open therefore has to sit below, inside "llama_file", for any
worker pool above it to scale at all.

4 files, +895/-14, test-llama-file 18/18 builds in 20sec.

[stevenhoving]

Adding the measurement behind this, since the PR body doesn't carry it.

Why the hard "true" from "has_direct_io()" matters in practice: external code trusts it. #25294's Windows branch calls it, logs "MoE expert streaming uses O_DIRECT (page cache bypassed)", and then reads buffered through one shared handle serialised behind a "static std::mutex".

Measured through "llama_file" itself, Windows 11 on NVMe, 12.75 MiB requests:

arm throughput ratio
1 thread, private handle 5,009.2 MB/s 1.000x
8 threads, one handle per worker 10,533.4 MB/s 2.103x
8 threads, shared handle 4,963.3 MB/s 0.991x
A shared "llama_file" does not parallelise: eight threads on one handle land at the speed of one. The multi-open therefore has to sit below, inside "llama_file", for any worker pool above it to scale at all.

4 files, +895/-14, test-llama-file 18/18 builds in 20sec.

Did you look into leveraging iocp and worker threads? For me it seems that you are implementing that kind of mechanism yourself. And I don't think that is a good idea to create it yourself.

[nibor1896]

@stevenhoving

The PR doesn't implement an IOCP mechanism.

I think the confusion is the word "pool". it's a pool of HANDLES not of threads.

The PR creates no threads and has no completion machinery. The only std::thread in the diff is in the test. The implementation opens up to 18 private handles once, single-threaded, in init_direct; a caller that reads concurrently passes its own worker_id to pick one. The threads themselves belong to the caller, i.e. the worker pool that already sits above llama_file.

Everything stays synchronous. The handles are opened without FILE_FLAG_OVERLAPPED; the OVERLAPPED structure is used purely to carry the offset, so ReadFile blocks until completion. That's the classic pread emulation, matching what the POSIX side already does with pread on one fd. There is no GetOverlappedResult, no wait, no pending path anywhere.

Why per-thread handles instead of one shared handle: Windows serialises synchronous I/O on the file object, not on the position. Measured both ways on the same setup, a shared handle reaches 1.01x single-thread throughput at depth 8 via OVERLAPPED offset and 0.98x via SetFilePointerEx, while a private handle per thread reaches 2.22x. So the multi-open is the concurrency mechanism; the read mechanism makes no difference.

On IOCP specifically: it's the right tool for completion-based async I/O (FILE_FLAG_OVERLAPPED, few threads, many in-flight ops), but llama_file's contract is blocking and every caller brings its own threads. Wiring IOCP underneath would mean either changing that contract or blocking per request anyway. That adds machinery with, for this access pattern, the same throughput ceiling: the 2.22x is NVMe queue scaling, which N concurrent blocking positional reads already deliver. Happy to benchmark an IOCP variant side by side if you'd find that useful.