You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Core, Spark: Performant queries over (shredded) Variant data #16172
Rowgroup and file skipping based on shredded field stats.
Benchmarks to evaluate this
Iceberg query performance relies on spark to pass down variant_get() calls to the rowgroup filter, so the changes are interrelated. This stuff will have to target spark 4.2 only
#14297 Spark: Support writing shredded variant in Iceberg-Spark #15510 Parquet Rowgroup skipping for variant predicate #15384 Api: Support variant extract and fix manifest bounds byte order #15385 Spark: Support variant_get predicate pushdown for file skipping #15628 Core, Spark: Add JMH benchmarks for Variants
skip files on iceberg stats, if possible.
Spark
54598 Enable Parquet rowgroup skipping for variant filters
54394
Support variant_get predicate for DSv2 filter pushdown
changed the title [-]Core, Spark: Performant queries over Variant data[/-][+]Core, Spark: Performant queries over (shredded) Variant data[/+]on Apr 30, 2026
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
Activity
[-]Core, Spark: Performant queries over Variant data[/-][+]Core, Spark: Performant queries over (shredded) Variant data[/+]qlong commentedon May 20, 2026
Discovered a gap while doing performanc test. #16448 Witout it, variant is fully reconstructed from shreddred columns
github-actions commentedon Nov 17, 2026
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.