Repository navigation
perf(manifest): reuse deflate buffers while reading entries - #2644
iceberg-kitten wants to merge 4 commits into
Conversation
Reuse inflater state and buffers within each manifest reader. Preserve the block size limit and release a large buffer after a much smaller block. Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
2b8ee88 to
c277ddf
Compare
badalprasadsingh
left a comment
There was a problem hiding this comment.
approved @iceberg-kitten
laskoviymishka
left a comment
There was a problem hiding this comment.
Thanks for keeping the size limit and error text identical to the stock deflate codec. As I read it, reusing the buffer is safe at avro v1.9.0: the decoder copies strings and bytes out of the block unless AliasInput is set, and ocf.Reader strips it. With this codec reading two ~70 KiB blocks, both entries decode correctly from one shared backing array. Two small nits inline.
| // one manifest. The OCF reader makes its own copy of any bytes kept by a decoded | ||
| // entry, so the output buffer can be overwritten on the next block. |
There was a problem hiding this comment.
Could the comment name the decoder as the component that copies? slab.string/slab.bytes copy out of the block unless AliasInput is set (deser.go), and ocf.Reader drops that option (ocf.go). That pair is what the buffer reuse depends on.
| // one manifest. The OCF reader makes its own copy of any bytes kept by a decoded | |
| // entry, so the output buffer can be overwritten on the next block. | |
| // one manifest. The avro decoder copies strings and bytes out of the block unless | |
| // avro.AliasInput is set, and ocf.Reader drops that option, so the output buffer | |
| // can be overwritten on the next block. |
| func TestManifestReaderKeepsDeflateEntriesFromEarlierBlocks(t *testing.T) { | ||
| schema := NewSchema(1, NestedField{ID: 1, Name: "id", Type: PrimitiveTypes.Int64, Required: true}) | ||
| snapshotID := int64(1) | ||
| // Each path exceeds the writer's 64 KiB block threshold. |
There was a problem hiding this comment.
Should this assert that the manifest really has two blocks? The split depends on the writer's 64 KiB default (defaultBlockBytes). If that default or our writer options change, both entries can land in one block, and the test would still pass while covering only one block. Counting DecompressBounded calls would rule that out.
Summary
Reuse one inflater and output buffer across deflate blocks in each Avro manifest reader. If the buffer exceeds 1 MiB and a later block fills at most a quarter of its capacity, copy the smaller block so the larger allocation can be released. The codec applies the existing decompressed block-size limit on both full and projected reads.
Benchmark
Values are
main(e7a4d96) → this PR. The read benchmark uses 10,000-entry Go-written manifests with statistics for 10 or 100 fields. LocalPlanFilesreads eight in-memory manifests and verifies that it returns 20,000 tasks.PlanFiles, 20,000 files12 fresh processes per revision on one pinned core (Intel i9-12900HK, Linux amd64, Go 1.26.3,
GOMAXPROCS=1,GOGC=100). Revision order rotated between rounds. Values are medians from benchstat, with 95% confidence intervals for timing. Timing changes usep < 0.005.Run once per revision in each round and save each revision’s output separately.
Earlier end-to-end planning measurements
The initial buffer-reuse patch was tested by planning 100,000 files from 50 manifests with one worker on a local Hadoop warehouse. The figures are median allocated bytes per operation across three fresh processes per build and fixture, with four warm samples per process. Wall-clock timings varied on the shared host.
mainValidation
make testandmake lintpassed on this branch.