Skip to content

perf(manifest): reuse deflate buffers while reading entries - #2644

Open
iceberg-kitten wants to merge 4 commits into
apache:mainfrom
iceberg-kitten:perf/reuse-manifest-deflate-buffers
Open

iceberg-kitten wants to merge 4 commits into
apache:mainfrom
iceberg-kitten:perf/reuse-manifest-deflate-buffers

Conversation

@iceberg-kitten

@iceberg-kitten iceberg-kitten commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Reuse one inflater and output buffer across deflate blocks in each Avro manifest reader. If the buffer exceeds 1 MiB and a later block fills at most a quarter of its capacity, copy the smaller block so the larger allocation can be released. The codec applies the existing decompressed block-size limit on both full and projected reads.

Benchmark

Values are main (e7a4d96) → this PR. The read benchmark uses 10,000-entry Go-written manifests with statistics for 10 or 100 fields. Local PlanFiles reads eight in-memory manifests and verifies that it returns 20,000 tasks.

Workload Time/op Time change Allocated MiB/op Allocs/op
10 fields, full 139.53 ms ± 1% → 130.77 ms ± 1% −6.28% 31.13 → 21.49 (−30.96%) 444,013 → 442,849
10 fields, projected 48.79 ms ± 1% → 40.46 ms ± 1% −17.08% 14.72 → 5.08 (−65.49%) 64,552.5 → 63,361
10 fields, projected with pruning stats 149.12 ms ± 1% → 139.01 ms ± 2% −6.78% 29.32 → 19.69 (−32.86%) 217,108 → 215,945
100 fields, full 1.034 s ± 1% → 979.99 ms ± 1% −5.24% 247.14 → 158.35 (−35.93%) 2,257,021 → 2,247,264.5
100 fields, projected 299.74 ms ± 1% → 244.65 ms ± 1% −18.38% 94.90 → 6.11 (−93.57%) 77,256.5 → 67,485
100 fields, projected with pruning stats 1.065 s ± 0% → 1.011 s ± 1% −5.08% 229.72 → 140.93 (−38.65%) 243,319.5 → 233,551.5
Local PlanFiles, 20,000 files 58.20 ms ± 5% → 56.88 ms ± 2% Inconclusive 22.53 → 19.79 (−12.19%) 191,835 → 191,328

12 fresh processes per revision on one pinned core (Intel i9-12900HK, Linux amd64, Go 1.26.3, GOMAXPROCS=1, GOGC=100). Revision order rotated between rounds. Values are medians from benchstat, with 95% confidence intervals for timing. Timing changes use p < 0.005.

Run once per revision in each round and save each revision’s output separately.

taskset -c 19 env GOMAXPROCS=1 GOGC=100 go test . -run '^$' \
  -bench '^BenchmarkManifestEntryProjection$/^stats_fields=(10|100)$/^(full|scan|scan_with_stats)$' \
  -benchmem -benchtime=300ms
taskset -c 19 env GOMAXPROCS=1 GOGC=100 go test ./table -run '^$' \
  -bench '^BenchmarkPlanFilesTaskFinalization$/^plain$' -benchmem -benchtime=300ms
benchstat -alpha 0.005 before.txt after.txt
Earlier end-to-end planning measurements

The initial buffer-reuse patch was tested by planning 100,000 files from 50 manifests with one worker on a local Hadoop warehouse. The figures are median allocated bytes per operation across three fresh processes per build and fixture, with four warm samples per process. Wall-clock timings varied on the shared host.

Manifest writer main Initial patch Reduction
Java 262.4 MiB 170.7 MiB 35%
Go 254.7 MiB 168.2 MiB 34%

Validation

  • make test and make lint passed on this branch.
  • Tests cover paths and bounds across blocks in full and projected reads, truncated data, the decompressed size limit, and release of an oversized buffer.

Reuse inflater state and buffers within each manifest reader. Preserve the block size limit and release a large buffer after a much smaller block.

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
@iceberg-kitten
iceberg-kitten force-pushed the perf/reuse-manifest-deflate-buffers branch from 2b8ee88 to c277ddf Compare November 18, 2026 23:49

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved @iceberg-kitten

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for keeping the size limit and error text identical to the stock deflate codec. As I read it, reusing the buffer is safe at avro v1.9.0: the decoder copies strings and bytes out of the block unless AliasInput is set, and ocf.Reader strips it. With this codec reading two ~70 KiB blocks, both entries decode correctly from one shared backing array. Two small nits inline.

Comment thread manifest_deflate.go Outdated
Comment on lines +30 to +31
// one manifest. The OCF reader makes its own copy of any bytes kept by a decoded
// entry, so the output buffer can be overwritten on the next block.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the comment name the decoder as the component that copies? slab.string/slab.bytes copy out of the block unless AliasInput is set (deser.go), and ocf.Reader drops that option (ocf.go). That pair is what the buffer reuse depends on.

Suggested change
// one manifest. The OCF reader makes its own copy of any bytes kept by a decoded
// entry, so the output buffer can be overwritten on the next block.
// one manifest. The avro decoder copies strings and bytes out of the block unless
// avro.AliasInput is set, and ocf.Reader drops that option, so the output buffer
// can be overwritten on the next block.

Comment thread manifest_deflate_test.go
func TestManifestReaderKeepsDeflateEntriesFromEarlierBlocks(t *testing.T) {
schema := NewSchema(1, NestedField{ID: 1, Name: "id", Type: PrimitiveTypes.Int64, Required: true})
snapshotID := int64(1)
// Each path exceeds the writer's 64 KiB block threshold.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this assert that the manifest really has two blocks? The split depends on the writer's 64 KiB default (defaultBlockBytes). If that default or our writer options change, both entries can land in one block, and the test would still pass while covering only one block. Counting DecompressBounded calls would rule that out.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants