Skip to content

perf(table): skip redundant metadata JSON scan - #2641

Merged
laskoviymishka merged 2 commits into
apache:mainfrom
iceberg-kitten:perf/metadata-skip-redundant-json-scan
Nov 19, 2026
Merged

laskoviymishka merged 2 commits into
apache:mainfrom
iceberg-kitten:perf/metadata-skip-redundant-json-scan

Conversation

@iceberg-kitten

@iceberg-kitten iceberg-kitten commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Metadata preflight already validates JSON syntax. After preflight and partition-field ID normalization, trim outer JSON whitespace and call the version-specific UnmarshalJSON directly. This skips the second syntax scan while preserving type-error offsets.

Benchmark

In-memory v2 metadata parsing across four table shapes. Values are main (e7a4d96) → this PR. The 64-snapshot case has populated summaries.

Schemas / specs / snapshots Time/op Time change Bytes/op Allocs/op
1 / 1 / 10 252.3 µs ± 1% → 218.2 µs ± 2% −13.51% 35,192 → 34,921 587 → 582
1 / 1 / 64 2.44 ms ± 1% → 2.15 ms ± 1% −12.20% 294,280 → 294,007 5,112 → 5,107
32 / 32 / 1,000 16.99 ms ± 1% → 14.49 ms ± 1% −14.74% 2,263,159.5 → 2,262,881 33,583 → 33,578
256 / 256 / 10,000 156.26 ms ± 2% → 135.21 ms ± 1% −13.47% 22,662,112 → 22,661,866.5 325,054 → 325,049

12 fresh processes per revision on one pinned core (Intel i9-12900HK, Linux amd64, Go 1.26.3, GOMAXPROCS=1, GOGC=100). Revision order rotated between rounds. Values are medians from benchstat, with 95% confidence intervals for timing. Timing changes use p < 0.005.

Use this PR’s table/metadata_preflight_bench_test.go in the main checkout to run the same fixtures.

Run once per revision in each round and save each revision’s output separately.

taskset -c 19 env GOMAXPROCS=1 GOGC=100 go test ./table -run '^$' \
  -bench '^BenchmarkParseMetadataBytes$' -benchmem -benchtime=300ms
benchstat -alpha 0.005 before.txt after.txt

Validation

  • make test passed on this branch. Lint passed on the metadata stack.
  • Tests compare type errors with encoding/json for v1, v2, and v3, including outer whitespace and partition-field ID normalization.

Call the version-specific decoder after preflight and preserve type error offsets with and without partition spec normalization.

Signed-off-by: Minh Vu <356647812+iceberg-kitten@users.noreply.github.com>
@iceberg-kitten
iceberg-kitten force-pushed the perf/metadata-skip-redundant-json-scan branch from 6e33daf to 301247d Compare November 18, 2026 23:49

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pinning the UnmarshalTypeError against encoding/json in the new test. Stdlib's object() hands UnmarshalJSON the same whitespace-trimmed span trimJSONSpan produces (decode.go), and a stdlib-only repro of that shape returns deep-equal errors with and without GOEXPERIMENT=jsonv2. Two small test nits inline.

Comment thread table/metadata_preflight_test.go Outdated
number int
target json.Unmarshaler
}{
{1, &metadataV1{}},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the inner spec table uses keyed literals ({name: "whitespace"}), so {number: 1, target: &metadataV1{}} here would read the same way. Optional.

Comment thread table/metadata_preflight_test.go Outdated
{name: "normalized", fields: `,"partition-specs":[{"spec-id":0,"fields":[{"source-id":1,"name":"id","transform":"identity"}]}]`},
} {
t.Run(spec.name, func(t *testing.T) {
input := []byte(fmt.Sprintf(" \t\r\n"+`{"format-version":%d,"last-updated-ms":1,"last-column-id":"not-an-int"%s}`+"\n",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: fmt.Appendf(nil, ...) skips the string-to-bytes conversion, and the package already uses it in tests (updates_test.go, inspect_partitions_bench_test.go). Lint doesn't require it either way.

@laskoviymishka
laskoviymishka merged commit 503c6d0 into apache:main Nov 19, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants