Repository navigation
perf(table): skip redundant metadata JSON scan - #2641
laskoviymishka merged 2 commits into
Conversation
Call the version-specific decoder after preflight and preserve type error offsets with and without partition spec normalization. Signed-off-by: Minh Vu <356647812+iceberg-kitten@users.noreply.github.com>
6e33daf to
301247d
Compare
laskoviymishka
left a comment
There was a problem hiding this comment.
Thanks for pinning the UnmarshalTypeError against encoding/json in the new test. Stdlib's object() hands UnmarshalJSON the same whitespace-trimmed span trimJSONSpan produces (decode.go), and a stdlib-only repro of that shape returns deep-equal errors with and without GOEXPERIMENT=jsonv2. Two small test nits inline.
| number int | ||
| target json.Unmarshaler | ||
| }{ | ||
| {1, &metadataV1{}}, |
There was a problem hiding this comment.
nit: the inner spec table uses keyed literals ({name: "whitespace"}), so {number: 1, target: &metadataV1{}} here would read the same way. Optional.
| {name: "normalized", fields: `,"partition-specs":[{"spec-id":0,"fields":[{"source-id":1,"name":"id","transform":"identity"}]}]`}, | ||
| } { | ||
| t.Run(spec.name, func(t *testing.T) { | ||
| input := []byte(fmt.Sprintf(" \t\r\n"+`{"format-version":%d,"last-updated-ms":1,"last-column-id":"not-an-int"%s}`+"\n", |
There was a problem hiding this comment.
nit: fmt.Appendf(nil, ...) skips the string-to-bytes conversion, and the package already uses it in tests (updates_test.go, inspect_partitions_bench_test.go). Lint doesn't require it either way.
Summary
Metadata preflight already validates JSON syntax. After preflight and partition-field ID normalization, trim outer JSON whitespace and call the version-specific
UnmarshalJSONdirectly. This skips the second syntax scan while preserving type-error offsets.Benchmark
In-memory v2 metadata parsing across four table shapes. Values are
main(e7a4d96) → this PR. The 64-snapshot case has populated summaries.12 fresh processes per revision on one pinned core (Intel i9-12900HK, Linux amd64, Go 1.26.3,
GOMAXPROCS=1,GOGC=100). Revision order rotated between rounds. Values are medians from benchstat, with 95% confidence intervals for timing. Timing changes usep < 0.005.Use this PR’s
table/metadata_preflight_bench_test.goin themaincheckout to run the same fixtures.Run once per revision in each round and save each revision’s output separately.
Validation
make testpassed on this branch. Lint passed on the metadata stack.encoding/jsonfor v1, v2, and v3, including outer whitespace and partition-field ID normalization.