You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Core, Spark: Add JMH benchmarks for Variants #15628
Report from the initial PR, which doesn't (yet) fork. It can be viewed in jmh viewer.
Nested variants have 1/8 of their toplevel fields nested, each of which has 5 children with existing (string) keys and new strings, so there's more data to write
There's a lot of sample variance; running a few thousand iterations on a physical machine otherwise completely idle would be better here.
You can see the cost of shredding in the build() benchmark.
This is what claude says of the numbers
build (construct ShreddedObject + sizeInBytes):
- Shallow and Nested are very close at all 10k params — within noise. At 100%: 2252 vs 2190us. Construction cost is dominated by the number of put() calls and sizeInBytes() computation,
not value complexity.
- Nested 10k scales cleanly: 353 → 1107 → 1707 → 2190us. Nearly linear with shredding %.
- At 0% shredded, Nested (353us) is cheaper than Shallow (567us). No put() calls in either case, so the difference is sizeInBytes() on the unshreddedObject. Nested objects may compute
size differently than flat primitives.
- 1k results are still noisy at low shredding (9-48% error) but stable at high shredding (2-9%).
serialize (pre-built object, writeTo only):
- Tightest results of all three benchmarks — most entries <2% error.
- Nested overhead is consistent: ~1.8x at 10k/100% (7045 vs 3896us). At 10k/0% they're nearly identical (2356 vs 2299us) since no nested values are in the shredded portion.
- Linear scaling with shredding % in both modes. Nested 10k: 2356 → 4392 → 5854 → 7045us. Shallow 10k: 2299 → 3161 → 3595 → 3896us. The slope is steeper for Nested — each additional
shredded field costs more bytes to write.
deserialize (parse buffer + iterate all fields):
- Flat across shredding % — confirms deserialization cost depends on total data volume, not how it was shredded.
- Nested 10k: ~4500us. Shallow 10k: ~3760us. ~1.2x overhead for Nested. Less than the 1.8x serialization overhead because deserialization just scans offsets without recursing into nested
values.
- Nested 1k: remarkably stable at ~390us across all shredding values (1-2% error). This is the cleanest data in the entire run.
- Shallow 1k/0%: 421us with 16% error — the first param combination still absorbs some residual warmup even with 100 warmup iterations.
Summary: Serialization cost scales with both shredding % and nesting depth. Deserialization is flat across shredding and only modestly affected by nesting (~20% overhead). Build cost is
dominated by field count and shredding %, with nesting having negligible impact.
For anyone looking at this, there's no significant cost for variants without shedding when creating/loading them. It takes write when shedding, but these benchmarks don't measure cost of queries over data, where the shedded variants should show significant speedup compared to just reading all the data.
This sounds expected to me. An unshredded vs shredded variant should be pretty similar size. Probably similar enough. The Variance you see may be dependent on JIT optimizations. I would expect nested writers taking more and more nested function calls which may take more runs before the JIT cleans them up.
There I've noticed that the granularity of ARM CPUs is nowhere as good as x86 rdtsc opcodes (ignoring core/socket variations there), so the very small operations need to be repeated multiple times within a single benchmark method. I'll apply that here too. Essentially: microbenchmarks need to take milliseconds to minimise clock granularity. jmh does at least pin the benchmark to a core so cross-socket and cross-core issues (older intel Si) don't surface. And by only working on the same struct the data should all be in $L1 so data cache issues shouldn't be an issue, hence nor is NUMA. Modern multisocket servers are the new datacentre.
Assesses cost of write complexity and makes sure there is now problem lurking right at the bottom.
iceberg spark module: table-level benchmark
I'm creating two tables with same content then doing queries over them. I need to produce an uneven spread of numbers so that iceberg can use the shedded metadata to filter files; without that filtering shedded is coming of slower right now, which is presumably happening underneath. Spark parquet vectorization currently doesn't work with projection on shedded variants, so I'm turning vectorization off on both tables for consistency.
@RussellSpitzer and here's the results of a the spark reader benchmark I've added to the PR.
The longer lines are all the shedded ones, even when filtering or selecting on a shedded column.
...add a deep nesting to VariantSerializationBenchmark and the cost of building deep trees is pretty bad, just from the #of hash tables created (one per SerializedObject .initOffsetsAndLengths(). I'm looking @ whether they could be replaced (this isn't a Hash Join after all), but that's potentially optimising for deep over wide
Results from the benchmark of queries through spark. Shedded parquet variants may compress well, but there's currently a serious penalty of using them compared to unshredded variants.
For operations which examine the contents of variants at all, avro wins.
Qualifier: these numbers are from the dev setup: no fork and on an arm-based MBP. Real benchmarks should be run in separate JVMs and on an x86 server VM with compute capacity and the rdtsp opcode for nanosecond resolution. I think I'm ready to try that next
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
Activity
[-]Add JMH benchmarks for Variants[/-][+]Core: Add JMH benchmarks for Variants[/+]Add JMH benchmarks for Variants
Add JMH benchmarks for Variants
steveloughran commentedon Mar 13, 2026
Report from the initial PR, which doesn't (yet) fork. It can be viewed in jmh viewer.
You can see the cost of shredding in the build() benchmark.
This is what claude says of the numbers
results.json
Core: Add JMH benchmarks for Variants
steveloughran commentedon Mar 19, 2026
For anyone looking at this, there's no significant cost for variants without shedding when creating/loading them. It takes write when shedding, but these benchmarks don't measure cost of queries over data, where the shedded variants should show significant speedup compared to just reading all the data.
RussellSpitzer commentedon Mar 19, 2026
This sounds expected to me. An unshredded vs shredded variant should be pretty similar size. Probably similar enough. The Variance you see may be dependent on JIT optimizations. I would expect nested writers taking more and more nested function calls which may take more runs before the JIT cleans them up.
steveloughran commentedon Mar 20, 2026
@RussellSpitzer Nesting is creating more hashmaps, which allocate map space.
I've been doing the parquet version of this test suite too, using this file as the foundation.
apache/parquet-java#3452
There I've noticed that the granularity of ARM CPUs is nowhere as good as x86
rdtscopcodes (ignoring core/socket variations there), so the very small operations need to be repeated multiple times within a single benchmark method. I'll apply that here too. Essentially: microbenchmarks need to take milliseconds to minimise clock granularity. jmh does at least pin the benchmark to a core so cross-socket and cross-core issues (older intel Si) don't surface. And by only working on the same struct the data should all be in $L1 so data cache issues shouldn't be an issue, hence nor is NUMA. Modern multisocket servers are the new datacentre.what I'm doing now are:
parquet-benchmark module row write with/without shedding.
Assesses cost of write complexity and makes sure there is now problem lurking right at the bottom.
iceberg spark module: table-level benchmark
I'm creating two tables with same content then doing queries over them. I need to produce an uneven spread of numbers so that iceberg can use the shedded metadata to filter files; without that filtering shedded is coming of slower right now, which is presumably happening underneath. Spark parquet vectorization currently doesn't work with projection on shedded variants, so I'm turning vectorization off on both tables for consistency.
steveloughran commentedon Mar 20, 2026
@RussellSpitzer and here's the results of a the spark reader benchmark I've added to the PR.
The longer lines are all the shedded ones, even when filtering or selecting on a shedded column.
human-readable-output.txt
I can think of some more tests there
Suggestions welcome
steveloughran commentedon Mar 20, 2026
Full test results.json
View in https://jmh.morethan.io/
steveloughran commentedon Mar 23, 2026
...add a deep nesting to
VariantSerializationBenchmarkand the cost of building deep trees is pretty bad, just from the #of hash tables created (one perSerializedObject .initOffsetsAndLengths(). I'm looking @ whether they could be replaced (this isn't a Hash Join after all), but that's potentially optimising for deep over widesteveloughran commentedon Mar 24, 2026
Results from the benchmark of queries through spark. Shedded parquet variants may compress well, but there's currently a serious penalty of using them compared to unshredded variants.
For operations which examine the contents of variants at all, avro wins.
steveloughran commentedon Mar 24, 2026
#15510 is probably the key feature for restoring performance; without it #14297 is going to slow things down
[-]Core: Add JMH benchmarks for Variants[/-][+]Core, Spark: Add JMH benchmarks for Variants[/+]steveloughran commentedon Mar 24, 2026
Qualifier: these numbers are from the dev setup: no fork and on an arm-based MBP. Real benchmarks should be run in separate JVMs and on an x86 server VM with compute capacity and the
rdtspopcode for nanosecond resolution. I think I'm ready to try that nextCore: Add JMH benchmarks for Variants
Core: Add JMH benchmarks for Variants
Core, Spark: Add JMH benchmarks for Variants
Core, Spark: Add JMH benchmarks for Variants
Core, Spark: Add JMH benchmarks for Variants
github-actions commentedon Sep 21, 2026
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
steveloughran commentedon Sep 27, 2026
not stale