Skip to content

perf(table): decode v2 history logs directly - #2643

Open
iceberg-kitten wants to merge 5 commits into
apache:mainfrom
iceberg-kitten:perf/metadata-decode-v2-history-logs
Open

iceberg-kitten wants to merge 5 commits into
apache:mainfrom
iceberg-kitten:perf/metadata-decode-v2-history-logs

Conversation

@iceberg-kitten

@iceberg-kitten iceberg-kitten commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Decode ordinary snapshot-log and metadata-log entries directly during eager v2 reads. Unusual records can still use encoding/json individually. If a log fails to decode, let the standard metadata parser report the error. Each decoded metadata-log entry owns its filename.

Builds on #2641 and #2642. History-log-only diff.

Benchmark

In-memory v2 metadata parsing across four table shapes. Values are #2642 (f1f5572) → this PR. The 64-snapshot case has populated summaries.

Schemas / specs / snapshots Time/op Time change Bytes/op Allocs/op
1 / 1 / 10 217.7 µs ± 1% → 220.0 µs ± 2% Inconclusive 34,921 → 34,921 582 → 582
1 / 1 / 64 476.7 µs ± 1% → 297.3 µs ± 1% −37.63% 124,539 → 101,242 1,319 → 859
32 / 32 / 1,000 5.15 ms ± 3% → 2.46 ms ± 1% −52.23% 1,220,423 → 880,891 16,574 → 9,564
256 / 256 / 10,000 47.17 ms ± 5% → 20.48 ms ± 1% −56.57% 12,264,343.5 → 8,873,725 155,043 → 85,031.5

12 fresh processes per revision on one pinned core (Intel i9-12900HK, Linux amd64, Go 1.26.3, GOMAXPROCS=1, GOGC=100). Revision order rotated between rounds. Values are medians from benchstat, with 95% confidence intervals for timing. Timing changes use p < 0.005.

The 10-snapshot fixture is below the 4 KiB cutoff for eager decoding.

Run once per revision in each round and save each revision’s output separately.

taskset -c 19 env GOMAXPROCS=1 GOGC=100 go test ./table -run '^$' \
  -bench '^BenchmarkParseMetadataBytes$' -benchmem -benchtime=300ms
benchstat -alpha 0.005 before.txt after.txt
Earlier warm file-load measurements

The initial stack was also measured on a warm 66,321-byte v2 metadata file. Median load time fell from 16.748 ms on main to 13.951 ms with #2641, 4.624 ms through #2642, and 1.989 ms through #2643. These were three matched rounds at roughly 0.9 GHz with Go 1.26.10 and GOGC=100.

A separate single-round comparison at roughly 2.9 GHz measured 0.185 ms for Java 1.12.0, 0.339 ms for the initial Go stack, 0.420 ms for Rust (641f59049e0a), and 3.287 ms for Go main (e7a4d96). Each result is the median of four timing windows.

Validation

  • make test passed on this branch. Lint passed on the metadata stack.
  • Tests cover unusual entries, escaped filenames, duplicate fields, null and empty logs, and the document nesting limit.

Call the version-specific decoder after preflight and preserve type error offsets with and without partition spec normalization.

Signed-off-by: Minh Vu <356647812+iceberg-kitten@users.noreply.github.com>
Share wire decoding and validation with the standard path. Decode common snapshots directly, fall back per entry, and allocate history capacity on demand.

Signed-off-by: Minh Vu <356647812+iceberg-kitten@users.noreply.github.com>
Decode common log records directly and fall back for unusual records. Preserve null and empty logs, and defer errors to the standard metadata parser.

Signed-off-by: Minh Vu <356647812+iceberg-kitten@users.noreply.github.com>
@iceberg-kitten
iceberg-kitten force-pushed the perf/metadata-decode-v2-history-logs branch from ff714a3 to 4e9c8b4 Compare November 18, 2026 23:49
Minh Vu added 2 commits November 19, 2026 14:30

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant