fix(scanner): pre-size the to_the_end read buffer from a source size …

[rockdatasrl001]

What this does

The to_the_end scanner reads an entire source into a single message with
io.ReadAll, which is given no size hint. For the file input the size is
already known — the input calls Stat() on every file (for the mod time) — but
it was never used.

This change threads an optional SizeHint through scanner.SourceDetails so the
file input can hand the scanner the size it already has, and replaces
io.ReadAll in the to_the_end scanner with a capacity-hinted read
(readAllHinted) that allocates the read buffer once, up front.

The result is a large reduction in memory allocated per read, and far fewer
allocations, when reading a whole file.

Why — the allocation mechanism (Go 1.26)

This repo targets Go 1.26.4 (now 1.26.6 on main). io.ReadAll was rewritten in
Go 1.26 and its own comment describes the strategy:

Build slices of exponentially growing size, then copy into a perfectly-sized
slice at the end.

It accumulates a list of exponentially growing chunks and then copies them into a
single right-sized slice. The returned slice is exact, but at peak the
intermediate chunks and the final copy are live simultaneously, so it
transiently holds roughly twice the content size in memory. Pre-sizing the buffer
to the known size avoids the intermediate chunks and the final copy entirely.

Note this is not documented behaviour — neither the io.ReadAll nor the append
godoc specifies an allocation strategy — so this PR is framed around the
measured allocation reduction, not around Go internals guarantees.

Measurements

Benchmark added in scanner_to_the_end_internal_test.go: a 64 MiB source fed
through a reader that returns realistic short reads (not one giant
bytes.Reader), io.ReadAll vs readAllHinted.

Raw go test -bench output (Go 1.26.6, -benchmem -count=6):

goos: linux
goarch: amd64
pkg: github.com/redpanda-data/benthos/v4/internal/impl/pure
cpu: AMD Ryzen 5 7640U w/ Radeon 760M Graphics
BenchmarkToTheEndReadAll-12          	      20	  52518838 ns/op	1277.81 MB/s	165347692 B/op	      37 allocs/op
BenchmarkToTheEndReadAll-12          	      21	  51125961 ns/op	1312.62 MB/s	165347169 B/op	      36 allocs/op
BenchmarkToTheEndReadAll-12          	      19	  54705585 ns/op	1226.73 MB/s	165347123 B/op	      36 allocs/op
BenchmarkToTheEndReadAll-12          	      24	  48705848 ns/op	1377.84 MB/s	165347139 B/op	      36 allocs/op
BenchmarkToTheEndReadAll-12          	      26	  48951980 ns/op	1370.91 MB/s	165347156 B/op	      36 allocs/op
BenchmarkToTheEndReadAll-12          	      20	  50688896 ns/op	1323.94 MB/s	165347443 B/op	      37 allocs/op
BenchmarkToTheEndReadAllHinted-12    	      43	  27296848 ns/op	2458.48 MB/s	67117074 B/op	       2 allocs/op
BenchmarkToTheEndReadAllHinted-12    	      42	  27096146 ns/op	2476.69 MB/s	67117072 B/op	       2 allocs/op
BenchmarkToTheEndReadAllHinted-12    	      44	  26971602 ns/op	2488.13 MB/s	67117077 B/op	       2 allocs/op
BenchmarkToTheEndReadAllHinted-12    	      43	  27192572 ns/op	2467.91 MB/s	67117074 B/op	       2 allocs/op
BenchmarkToTheEndReadAllHinted-12    	      43	  27140304 ns/op	2472.66 MB/s	67117074 B/op	       2 allocs/op
BenchmarkToTheEndReadAllHinted-12    	      44	  26966928 ns/op	2488.56 MB/s	67117077 B/op	       2 allocs/op

benchstat (n=6), io.ReadAll (base) vs readAllHinted:

            │  io.ReadAll  │        readAllHinted        │
            │     B/op     │     B/op      vs base       │
ToTheEnd-12   157.69Mi ± 0%   64.01Mi ± 0%  -59.41% (p=0.002)

            │  io.ReadAll  │        readAllHinted        │
            │  allocs/op   │  allocs/op    vs base       │
ToTheEnd-12    36.000 ± 3%    2.000 ± 0%   -94.44% (p=0.002)

So for a 64 MiB file the read goes from ~158 MiB allocated across 36 allocations
to ~64 MiB (the file size) in 2. The B/op figures are deterministic (± 0%),
which is why they, rather than timing, are the basis of the claim.

This benchmark measures allocation volume of the read itself, which is the
mechanism being changed. It is an in-process proxy for the peak-RSS behaviour, not
a full-pipeline RSS measurement.

The hint is an optimisation only — never a correctness input

A writer can append to or truncate a file between Stat and the read, so the hint
can be wrong in either direction. readAllHinted is written so the bytes returned
are identical to io.ReadAll for every hint value — correct, too small, too
large, zero, or negative:

• If the source is larger than the hint, the buffer grows exactly as io.ReadAll
  would, paying a copy only for the excess.
• If it is smaller, the read simply ends early.
• The capacity is hint+1, not hint: the read loop only calls Read while
  cap > len, so a buffer sized to exactly the content length would hit
  len == cap on the final iteration and grow once more, reintroducing the
  reallocation this avoids.

Tests covering this (scanner_to_the_end_internal_test.go,
scanner_to_the_end_test.go):

• Differential against io.ReadAll over a matrix of content sizes
  {0,1,511,512,513,4096,1<<20} × hints {0,1,size/2,size-1,size,size+1,size*2,-5},
  through a short-read reader (a bytes.Reader alone would never exercise the
  refill path).
• Exact-hint-no-realloc: asserts the returned buffer's capacity is size+1,
  which is the test that catches a regression of the +1 subtlety.
• End-to-end through the public scanner config path with hints that are
  absent, exact, too small, too large, and negative — content is unchanged in all
  cases.

Latent defect fixed in passing: codec/scanner.go dropped fields

To make the hint reach the scanner from the file input, a bug had to be fixed in
public/service/codec/scanner.go. That layer sits between the input and the
scanner and reconstructs the ScannerSourceDetails, copying only Name:

sDetails := service.NewScannerSourceDetails()
sDetails.SetName(details.Name())

Any field added to SourceDetails is therefore silently dropped for every input
routed through this layer, with no compile error and no test failure — which is
exactly what made site 3 easy to miss. It now propagates SizeHint too (and
nil-guards details, which was an unchecked deref).

Open question for maintainers: rather than copying fields one by one, would you
prefer this layer pass details through wholesale so future fields can't be
dropped again? That is a smaller, more robust change; I kept the explicit copy here
to match the existing style, but I'm happy to switch if you'd prefer.

Changed files

• internal/component/scanner/interface.go — add SizeHint int64 to SourceDetails.
• public/service/scanner.go — add SetSizeHint/SizeHint accessors.
• public/service/codec/scanner.go — propagate SizeHint; nil-guard details.
• internal/impl/io/input_file.go — move Stat() above scanner creation; set the
  hint from fInfo.Size(), guarded by IsRegular() (a FIFO, device or directory
  has no meaningful size to pre-allocate against).
• internal/impl/pure/scanner_to_the_end.go — capture the hint (nil-safe); replace
  io.ReadAll with readAllHinted.
• Tests + benchmark added; CHANGELOG updated.

Checks

make fmt, make lint and make test all pass (Go 1.26.6).

[CLAassistant]

All committers have signed the CLA.

[josephwoodward]

io.ReadAll has a default buffer of 512 bytes, which means anything without a hint larger than that value would change the current the runtime profile.

Perhaps a better solution would be to use that same 512 bytes as a default size, and only use the hint of larger to better align existing behaviour? ie:

defaultReadAllCap := 512 // matching io.ReadAlls

...
if hint < defaultReadAllCap {
    hint = defaultReadAllCap
}
buf := make([]byte, 0, hint+1)

[josephwoodward]

Also, given setting the hint is now exposed as an exported function on the package, we should also guard against hint overflows to ensure it never exceeds the max size, so the whole thing (including the above) looks like this:

func readAllHinted(r io.Reader, hint int64) ([]byte, error) {
      if hint < defaultReadAllCap {
              hint = defaultReadAllCap
      }
      if hint > math.MaxInt64-1 {
              hint = math.MaxInt64 - 1
      }
      buf := make([]byte, 0, hint+1)
      ...

It'd be good to get some tests around the behaviour too.

[rockdatasrl001]

Thank you, Joseph — both points are well made and have been addressed in 4d7c4ed.

On the 512-byte floor. I concur, and I would add that the consequence is somewhat broader than small files alone. The to_the_end scanner is also exercised by the socket, stdin and http_client inputs, none of which supply a size hint. Absent a floor, such no-hint reads would grow from a single-byte buffer and, for payloads below 512 bytes, incur more allocations than the io.ReadAll they replace — a regression on the very paths the change was not intended to touch. Flooring the hint at 512 (the initial capacity io.ReadAll itself uses) preserves the existing allocation profile on those paths, whilst larger hints continue to pre-size exactly.

On guarding against overflow. Agreed. As SetSizeHint is now exported, the hint must be treated as untrusted input, and the +1 capacity could otherwise overflow to a negative value. I have accordingly clamped it below math.MaxInt64. To keep the behaviour verifiable, I have extracted the floor and clamp into a small hintedCap function and covered it directly with TestHintedCap; the overflow branch is not reachable through an actual read, as the allocation would fail long beforehand. I have also added TestReadAllHintedFloor to confirm that no-hint callers retain the 512-byte buffer.

The benchmark figures are unchanged, and make fmt, make lint and make test all pass. Grateful for the review.

[squiidz]

1. Unclamped upfront allocation can panic the pipeline — scanner_to_the_end.go:93
   readAllHinted does make([]byte, 0, hint) with no ceiling. A sparse file (truncate -s 8E f) passes the IsRegular() guard and Stats at 2^63-1, so make panics makeslice: cap out of range inside NextBatch — reproduced by actually running it. Same panic for any file ≥ ~2GiB on 32-bit builds, and the exported SetSizeHint accepts arbitrary values from plugins. io.ReadAll could never crash this way since it allocated proportionally to bytes read. Fix: clamp the pre-allocation to a sane ceiling and let growth handle the rest.

2. Every non-file source regresses — scanner_to_the_end.go:92
   Sockets, stdin, and http_client pass no hint, and readAllHinted(r, 0) uses the old append-grow loop — but go.mod requires Go 1.26, whose io.ReadAll uses a chunk list with one final right-sized copy. Measured on a 64MiB stream: 393.5MB total allocations vs 165.3MB (2.4x), plus 17.5% permanently retained over-capacity. The code comment claiming parity with io.ReadAll's allocation profile is factually wrong for the required Go version. One-line fix: if hint <= 0 { return io.ReadAll(r) }.

3. Oversized hints pin memory for the message's lifetime — scanner_to_the_end.go:104
   The buffer is never right-sized on return. If a file Stats at 2GB but is truncated to 4KB before reading (copytruncate rotation), the message carries a 2GB backing array through the whole pipeline until GC. Deterministic whenever hint > content — the PR's own test passes hint = size*100. A glob over rotated files reproduces exactly the memory blow-up the PR claims to fix. Fix: copy to an exact-size slice when cap meaningfully exceeds len.

4. Nil-guard asymmetry in the public codec API — public/service/codec/scanner.go:84
   The PR nil-guards details in codecRPublic.Create (line 122) but the sibling codecRInternal.Create still calls details.Name() unconditionally. Which impl runs depends on whether the end user's YAML uses scanner: or the deprecated codec: field — so Create(rdr, ack, nil) works or panics based on config the caller can't see. Mirror the guard at line 84.