Skip to content

perf(table): decode v2 snapshot history directly - #2642

Open
iceberg-kitten wants to merge 3 commits into
apache:mainfrom
iceberg-kitten:perf/metadata-decode-v2-snapshots
Open

iceberg-kitten wants to merge 3 commits into
apache:mainfrom
iceberg-kitten:perf/metadata-decode-v2-snapshots

Conversation

@iceberg-kitten

@iceberg-kitten iceberg-kitten commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Decode ordinary snapshot fields and summaries directly during eager v2 metadata reads. Both paths share the v2 wire decoder and metadata validation. Unusual snapshots fall back to encoding/json individually. The fast path respects the full document’s nesting limit, copies strings retained by each snapshot, and allocates only what empty and single-entry histories need.

Builds on #2641. Snapshot-only diff.

Benchmark

In-memory v2 metadata parsing across four table shapes. Values are #2641 (301247d) → this PR. The 64-snapshot case has populated summaries.

Schemas / specs / snapshots Time/op Time change Bytes/op Allocs/op
1 / 1 / 10 218.2 µs ± 2% → 217.7 µs ± 1% Inconclusive 34,921 → 34,921 582 → 582
1 / 1 / 64 2.15 ms ± 1% → 476.7 µs ± 1% −77.78% 294,007 → 124,539 5,107 → 1,319
32 / 32 / 1,000 14.49 ms ± 1% → 5.15 ms ± 3% −64.44% 2,262,881 → 1,220,423 33,578 → 16,574
256 / 256 / 10,000 135.21 ms ± 1% → 47.17 ms ± 5% −65.11% 22,661,866.5 → 12,264,343.5 325,049 → 155,043

12 fresh processes per revision on one pinned core (Intel i9-12900HK, Linux amd64, Go 1.26.3, GOMAXPROCS=1, GOGC=100). Revision order rotated between rounds. Values are medians from benchstat, with 95% confidence intervals for timing. Timing changes use p < 0.005.

The 10-snapshot fixture is below the 4 KiB cutoff for eager decoding.

Run once per revision in each round and save each revision’s output separately.

taskset -c 19 env GOMAXPROCS=1 GOGC=100 go test ./table -run '^$' \
  -bench '^BenchmarkParseMetadataBytes$' -benchmem -benchtime=300ms
benchstat -alpha 0.005 before.txt after.txt

Validation

  • make test passed on this branch. Lint passed on the metadata stack.
  • Tests cover malformed JSON, duplicate fields, escaped values, the nesting limit, and ownership of decoded strings.

Call the version-specific decoder after preflight and preserve type error offsets with and without partition spec normalization.

Signed-off-by: Minh Vu <356647812+iceberg-kitten@users.noreply.github.com>
Share wire decoding and validation with the standard path. Decode common snapshots directly, fall back per entry, and allocate history capacity on demand.

Signed-off-by: Minh Vu <356647812+iceberg-kitten@users.noreply.github.com>
@iceberg-kitten
iceberg-kitten force-pushed the perf/metadata-decode-v2-snapshots branch from dbfd60e to f1f5572 Compare November 18, 2026 23:49
…e-v2-snapshots

# Conflicts:
#	table/metadata_preflight_test.go

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant