Skip to content

Core, Spark: Add JMH benchmarks for Variants #15628

Open

Description

@steveloughran

Feature Request / Improvement

There's no JMH benchmarks for variants, so no way to see if behaviour scales well.

Query engine

None

Willingness to contribute

  • I can contribute this improvement/feature independently
  • I would be willing to contribute this improvement/feature with guidance from the Iceberg community
  • I cannot contribute this improvement/feature at this time

Activity

  1. changed the title [-]Add JMH benchmarks for Variants[/-] [+]Core: Add JMH benchmarks for Variants[/+] on Mar 13, 2026
  2. added 2 commits that reference this issue on Mar 13, 2026
    1c61cda
    7c4f806
  3. steveloughran commented on Mar 13, 2026

    @steveloughran
    ContributorAuthor

    Report from the initial PR, which doesn't (yet) fork. It can be viewed in jmh viewer.

    • Nested variants have 1/8 of their toplevel fields nested, each of which has 5 children with existing (string) keys and new strings, so there's more data to write
    • There's a lot of sample variance; running a few thousand iterations on a physical machine otherwise completely idle would be better here.

    You can see the cost of shredding in the build() benchmark.

    This is what claude says of the numbers

      build (construct ShreddedObject + sizeInBytes):                                                                                                                                           
      - Shallow and Nested are very close at all 10k params — within noise. At 100%: 2252 vs 2190us. Construction cost is dominated by the number of put() calls and sizeInBytes() computation,
      not value complexity.                                                                                                                                                                     
      - Nested 10k scales cleanly: 353 → 1107 → 1707 → 2190us. Nearly linear with shredding %.                                                                                                
      - At 0% shredded, Nested (353us) is cheaper than Shallow (567us). No put() calls in either case, so the difference is sizeInBytes() on the unshreddedObject. Nested objects may compute   
      size differently than flat primitives.
      - 1k results are still noisy at low shredding (9-48% error) but stable at high shredding (2-9%).
    
      serialize (pre-built object, writeTo only):
      - Tightest results of all three benchmarks — most entries <2% error.
      - Nested overhead is consistent: ~1.8x at 10k/100% (7045 vs 3896us). At 10k/0% they're nearly identical (2356 vs 2299us) since no nested values are in the shredded portion.
      - Linear scaling with shredding % in both modes. Nested 10k: 2356 → 4392 → 5854 → 7045us. Shallow 10k: 2299 → 3161 → 3595 → 3896us. The slope is steeper for Nested — each additional
      shredded field costs more bytes to write.
    
      deserialize (parse buffer + iterate all fields):
      - Flat across shredding % — confirms deserialization cost depends on total data volume, not how it was shredded.
      - Nested 10k: ~4500us. Shallow 10k: ~3760us. ~1.2x overhead for Nested. Less than the 1.8x serialization overhead because deserialization just scans offsets without recursing into nested
       values.
      - Nested 1k: remarkably stable at ~390us across all shredding values (1-2% error). This is the cleanest data in the entire run.
      - Shallow 1k/0%: 421us with 16% error — the first param combination still absorbs some residual warmup even with 100 warmup iterations.
    
      Summary: Serialization cost scales with both shredding % and nesting depth. Deserialization is flat across shredding and only modestly affected by nesting (~20% overhead). Build cost is
      dominated by field count and shredding %, with nesting having negligible impact.
    

    results.json

  4. added a commit that references this issue on Mar 13, 2026
    2be00b9
  5. steveloughran commented on Mar 19, 2026

    @steveloughran
    ContributorAuthor

    For anyone looking at this, there's no significant cost for variants without shedding when creating/loading them. It takes write when shedding, but these benchmarks don't measure cost of queries over data, where the shedded variants should show significant speedup compared to just reading all the data.

  6. RussellSpitzer commented on Mar 19, 2026

    @RussellSpitzer
    Member

    This sounds expected to me. An unshredded vs shredded variant should be pretty similar size. Probably similar enough. The Variance you see may be dependent on JIT optimizations. I would expect nested writers taking more and more nested function calls which may take more runs before the JIT cleans them up.

  7. steveloughran commented on Mar 20, 2026

    @steveloughran
    ContributorAuthor

    @RussellSpitzer Nesting is creating more hashmaps, which allocate map space.

    I've been doing the parquet version of this test suite too, using this file as the foundation.

    apache/parquet-java#3452

    There I've noticed that the granularity of ARM CPUs is nowhere as good as x86 rdtsc opcodes (ignoring core/socket variations there), so the very small operations need to be repeated multiple times within a single benchmark method. I'll apply that here too. Essentially: microbenchmarks need to take milliseconds to minimise clock granularity. jmh does at least pin the benchmark to a core so cross-socket and cross-core issues (older intel Si) don't surface. And by only working on the same struct the data should all be in $L1 so data cache issues shouldn't be an issue, hence nor is NUMA. Modern multisocket servers are the new datacentre.

    what I'm doing now are:

    parquet-benchmark module row write with/without shedding.

    Assesses cost of write complexity and makes sure there is now problem lurking right at the bottom.

    iceberg spark module: table-level benchmark

    I'm creating two tables with same content then doing queries over them. I need to produce an uneven spread of numbers so that iceberg can use the shedded metadata to filter files; without that filtering shedded is coming of slower right now, which is presumably happening underneath. Spark parquet vectorization currently doesn't work with projection on shedded variants, so I'm turning vectorization off on both tables for consistency.

  8. steveloughran commented on Mar 20, 2026

    @steveloughran
    ContributorAuthor

    @RussellSpitzer and here's the results of a the spark reader benchmark I've added to the PR.
    The longer lines are all the shedded ones, even when filtering or selecting on a shedded column.

    human-readable-output.txt

    I can think of some more tests there

    • cost of adding rows
    • extending the persisted structure (I'd parameterize for a simple vs complex variant)
    • and optionally partition by the category:int field, both direct and via the variant.

    Suggestions welcome

  9. steveloughran commented on Mar 20, 2026

    @steveloughran
    ContributorAuthor
  10. steveloughran commented on Mar 23, 2026

    @steveloughran
    ContributorAuthor

    ...add a deep nesting to VariantSerializationBenchmark and the cost of building deep trees is pretty bad, just from the #of hash tables created (one per SerializedObject .initOffsetsAndLengths(). I'm looking @ whether they could be replaced (this isn't a Hash Join after all), but that's potentially optimising for deep over wide

  11. steveloughran commented on Mar 24, 2026

    @steveloughran
    ContributorAuthor

    Results from the benchmark of queries through spark. Shedded parquet variants may compress well, but there's currently a serious penalty of using them compared to unshredded variants.
    For operations which examine the contents of variants at all, avro wins.

    Image
  12. steveloughran commented on Mar 24, 2026

    @steveloughran
    ContributorAuthor

    #15510 is probably the key feature for restoring performance; without it #14297 is going to slow things down

  13. changed the title [-]Core: Add JMH benchmarks for Variants[/-] [+]Core, Spark: Add JMH benchmarks for Variants[/+] on Mar 24, 2026
  14. steveloughran commented on Mar 24, 2026

    @steveloughran
    ContributorAuthor

    Qualifier: these numbers are from the dev setup: no fork and on an arm-based MBP. Real benchmarks should be run in separate JVMs and on an x86 server VM with compute capacity and the rdtsp opcode for nanosecond resolution. I think I'm ready to try that next

  15. added a commit that references this issue on Apr 10, 2026
    64c68b5
  16. added 4 commits that reference this issue on May 18, 2026
    4e01bb3
    2690e5c
    34f347c
    aff833a
  17. github-actions commented on Sep 21, 2026

    @github-actions

    This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

  18. steveloughran commented on Sep 27, 2026

    @steveloughran
    ContributorAuthor

    not stale

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    improvementPR that improves existing functionality

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions