Skip to content

Spark 4.2, Vortex: faster scans of Vortex tables - #48

Open
robert3005 wants to merge 28 commits into
mainfrom
rk/vortex-perf
Open

robert3005 wants to merge 28 commits into
mainfrom
rk/vortex-perf

Conversation

@robert3005

@robert3005 robert3005 commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Summary

Performance changes to the Spark 4.2 integration and the Vortex format.

Spark 4.2

  • e4630f4 Prune data columns with runtime filters. Scans offer their data columns, not only the partition source columns, to Spark's runtime filters. A filter drops the files whose column metrics rule it out, and it reaches the file readers with the residual. The session extensions install NestedDynamicPruning, which drops pruning filters that hold pruning filters of their own.
  • 3e45b8c Offer data columns to runtime filtering only for Vortex. A scan offers data columns only when the table writes Vortex files and every file of the scan is a Vortex file.
  • 6062c7e Add a setting for the formats that offer data columns. spark.sql.iceberg.runtime-filtering.data-columns.formats lists the formats. The default is vortex.
  • cd1735d Describe runtime filters once and shorten large IN lists in plans. The scan builds the full description when it applies a runtime filter, and plans show at most 10 values of each IN.
  • a498d62 Size adaptive splits by the bytes of the scanned columns. A scan of a few columns of a wide table no longer gets as few splits as a scan of the full rows. spark.sql.iceberg.read.column-aware-split-size.enabled turns this off.
  • b4db8b2 Fit adaptive splits to one wave of tasks. The share per core gets a third more room, because cuts fall on chunk boundaries. Adds spark.sql.iceberg.read.min-file-split-size (default 64 MB).
  • e53bb56 Find stats columns by field ID in the scan schema. After a column rename, planning failed with a NullPointerException.

Vortex

  • 54fdee5 Align byte-range splits to chunk boundaries, so that no two tasks decode the same chunk.
  • fbb6587 Push string matches that Iceberg cannot state (CONTAINS, ENDS_WITH) into Vortex scans. Spark still evaluates the predicates after the scan.
  • cc0c385 Write split offsets every 16 MB.
  • cad0a4e Plan row splits from the projected columns only.
  • 57abc8b Convert large IN sets in one native call.
  • 72b8624 Read each range with one request. Streams open with openFile, the known file length and the random read policy.
  • 994e129 Use the writer's size estimate for the length of open files. length() returns VortexWriter.estimatedSize(), plus the rows that are not in the writer yet, scaled by VortexWriter.compressionRatio().

Backports

The Spark-side changes also go to Spark 4.1, 4.0 and 3.5, with one commit per version. 3e45b8c, 6062c7e and cd1735d are in Spark 4.2 only.

Change Spark 4.1 Spark 4.0 Spark 3.5
e4630f4 Prune data columns with runtime filters cf8f62c 0285e9e 98bafb5
fbb6587 Push string matches into Vortex scans (Spark part) c8b13bb 45e5b05 b070883
a498d62 Size adaptive splits by the bytes of the scanned columns 954f144 bd6bb20 467a888
b4db8b2 Fit adaptive splits to one wave of tasks dd06cd6 ac27725 b35e14a
e53bb56 Find stats columns by field ID in the scan schema 01f6226 in 0285e9e, bd6bb20 in 98bafb5, 467a888
  • Spark 4.1 has no partition predicates, so its scans keep their constructors and only get the data column flag.
  • Spark 4.0 and 3.5 have no SparkRuntimeFilterableScan or BaseSparkScanBuilder. The runtime filter logic goes into SparkBatchQueryScan, and the string match conversion goes into SparkScanBuilder.pushPredicates.
  • Spark 3.5 has no variant stats columns, so its scan builder asks only for the stats that these changes use.

Dependency

cad0a4e, 57abc8b and 994e129 use Vortex JNI methods that are not in a released version yet (vortex-data/vortex#11096 and vortex-data/vortex#11357). Merge this PR after a Vortex release that holds both, and bump the Vortex version.

Scans offer their data columns, not only the partition source columns, to
Spark's runtime filters such as dynamic pruning. A join on any key can then
prune an Iceberg table, which helps unpartitioned tables most.

- The Iceberg session extensions install NestedDynamicPruning. The rule
  drops each pruning filter whose filtering plan holds pruning filters of
  its own, which otherwise grow the plan exponentially through
  self-joined common table expressions. Scans offer data columns only
  when the session runs the rule.
- A runtime filter drops the files whose column metrics rule it out. The
  scan asks for the metrics of the columns it offers.
- The runtime filter on data columns reaches the file readers with the
  residual, so they skip row groups and chunks. An IN with more than
  spark.sql.iceberg.runtime-filtering.max-values values reaches them as
  its range.
- Scans of row-level operations do not offer data columns. Spark keys the
  groups of rows they rewrite by every filter attribute, and a null in a
  data column matches no key.
- spark.sql.iceberg.runtime-filtering.data-columns.enabled turns the
  feature off.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Map each split boundary to the nearest natural split of the file, so
scans of neighbouring splits do not both decode the chunk that holds
the boundary.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…ex scans

Spark predicates such as CONTAINS and ENDS_WITH have no Iceberg
expression, so Iceberg reads every column of every row and Spark filters
afterwards. The scan builder now converts them into LIKE filters for the
Vortex reader, which applies them in the scan and reads the other
columns only for the rows that match. Spark still evaluates the
predicates after the scan.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Adaptive split sizing aims for one split per shuffle partition by file
bytes. A scan of a few columns of a wide table then makes many tasks
that each read little data, and every task adds partial aggregation and
shuffle output. The adaptive size now counts the column sizes that the
manifests record for the columns the scan reads, and converts the result
back to file bytes, with at least one split per core.

spark.sql.iceberg.read.column-aware-split-size.enabled turns this off.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Adaptive split sizing now makes splits of at least
spark.sql.iceberg.read.min-file-split-size (64 MB by default), unless that
leaves cores without work. The share per core also gets a third more room,
because bin packing cannot fill each split. Before this change, two 16 MiB
pieces were slightly larger than the even share per core, so a scan of
store_sales ran 60 tasks in two waves on 32 cores. It now runs 30 tasks.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Finer split offsets let adaptive split sizing give each core a task for
tables of about 1 GB. On TPC-DS SF10 with Spark 4.2 this changes the total
from 104.8 s to 96.6 s (q22 15.4 s to 4.4 s, q72 7.4 s to 4.9 s).

Signed-off-by: Robert Kruszewski <github@robertk.io>
Each task found the row splits of its file from the layouts of all the
columns. Use only the projected columns, or the first column when the
projection is empty.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Integer, long and string sets go to native code as one array, rather than
one Java literal expression per value.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Scans offer their data columns to
Spark's runtime filters, drop files by the column metrics, and give the
filter to the file readers. The Iceberg session extensions install
NestedDynamicPruning, which stops pruning subqueries from nesting into
each other.

Spark 4.1 has no partition predicates, so the scans keep their Spark 4.1
constructors and only add the data column flag.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…into Vortex scans

Backport of the Spark 4.2 change. A CONTAINS or ENDS_WITH predicate on a
string column, which has no Iceberg expression, reaches the Vortex reader
as a filter that skips rows early. Spark still evaluates the predicate
after the scan.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…columns

Backport of the Spark 4.2 change. Adaptive split sizing counts only the
bytes of the columns that a scan reads, from the column sizes in the
file metrics. A scan of a few columns of a wide table then gets enough
splits for the cluster.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. The share of bytes per core gets a
third more room, because cuts fall on split offsets and packing cannot
fill each split. Without it, a piece slightly larger than the share makes
the scan take two waves of tasks. spark.sql.iceberg.read.min-file-split-size
sets the smallest split.

Signed-off-by: Robert Kruszewski <github@robertk.io>
A batch scan that pins its snapshot finds the names of its stats columns
in the schema of that snapshot. Column-aware split sizing asked for the
stats of the projected columns by their names in the table schema. After
a column rename, the snapshot schema does not hold the new name, and
planning failed with a NullPointerException. Spark then planned the query
without runtime filters.

The scan builder now gives the schema that the scan reads with to
includeStats, and finds each stats column there by its field ID. The
stats of a renamed column stay available.

TestRuntimeFiltering.testRenamedSourceColumnTable covers this.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 fix. A batch scan that pins its snapshot finds
the names of its stats columns in the schema of that snapshot. After a
column rename, planning failed with a NullPointerException, and Spark
planned the query without runtime filters. The scan builder now finds
each stats column by its field ID in the schema that the scan reads with.

TestRuntimeFiltering.testRenamedSourceColumnTable covers this.

Signed-off-by: Robert Kruszewski <github@robertk.io>
spotlessApply moves the VortexLikeFilters import into order.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Scans offer their data columns to
Spark's runtime filters, drop files by the column metrics, and give the
filter to the file readers. The Iceberg session extensions install
NestedDynamicPruning, which stops pruning subqueries from nesting into
each other.

Spark 4.0 has no SparkRuntimeFilterableScan, so the runtime filter logic
goes into SparkBatchQueryScan, which implements runtime filtering there.
The scan builder finds stats columns by field ID in the schema that the
scan reads with, as in the Spark 4.2 fix for renamed columns.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…into Vortex scans

Backport of the Spark 4.2 change. The scan builder turns CONTAINS and
ENDS_WITH predicates on string columns, which Iceberg expressions cannot
state, into filters for the Vortex reader. Spark still evaluates these
predicates after the scan.

Spark 4.0 has no BaseSparkScanBuilder, so the conversion goes into
SparkScanBuilder.pushPredicates.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…columns

Backport of the Spark 4.2 change. Adaptive split sizing counts only the
bytes of the columns that a scan reads, from the column sizes in the
file metrics. A scan of a few columns of a wide table then gets enough
splits for the cluster.

The scan builder finds the stats columns by field ID in the schema that
the scan reads with, as in the Spark 4.2 fix for renamed columns.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Adaptive split sizing makes splits of
at least spark.sql.iceberg.read.min-file-split-size, unless that leaves
cores without work. The share per core also gets a third more room,
because bin packing cannot fill each split.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Scans offer their data columns to
Spark's runtime filters, drop files by the column metrics, and give the
filter to the file readers. The Iceberg session extensions install
NestedDynamicPruning, which stops pruning subqueries from nesting into
each other.

As in Spark 4.0, the runtime filter logic goes into SparkBatchQueryScan.
Spark 3.5 has no variant stats columns, so the scan builder asks only
for the stats of the data filter columns.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…into Vortex scans

Backport of the Spark 4.2 change. The scan builder turns CONTAINS and
ENDS_WITH predicates on string columns, which Iceberg expressions cannot
state, into filters for the Vortex reader. Spark still evaluates these
predicates after the scan.

As in Spark 4.0, the conversion goes into SparkScanBuilder.pushPredicates.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…columns

Backport of the Spark 4.2 change. Adaptive split sizing counts only the
bytes of the columns that a scan reads, from the column sizes in the
file metrics. A scan of a few columns of a wide table then gets enough
splits for the cluster.

The scan builder finds the stats columns by field ID in the schema that
the scan reads with, as in the Spark 4.2 fix for renamed columns.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Adaptive split sizing makes splits of
at least spark.sql.iceberg.read.min-file-split-size, unless that leaves
cores without work. The share per core also gets a third more room,
because bin packing cannot fill each split.

Signed-off-by: Robert Kruszewski <github@robertk.io>
InputFileReadable copied each range in 1 MiB reads, and each stream came
from InputFile#newStream. On S3A, each stream sent a HEAD request and each
1 MiB read sent its own GET. For TPC-DS SF100 q3, Vortex made 2649 GETs of
691 KB on average, against 472 GETs of 3.7 MB for Parquet.

Read each range with one positional read. For Hadoop input files, open the
streams with FileSystem#openFile, the known length and the random read
policy, so that S3A sends no HEAD and one GET for each range.

Signed-off-by: Robert Kruszewski <github@robertk.io>
length() took the larger of the bytes in the sink and the uncompressed
Arrow size of the accepted rows. Vortex compresses well, so rolling
writers closed files far below the target size.

The Vortex writer now estimates the size of an open file: the bytes in
the sink, the compressed data that it holds, and the data that it did
not compress yet, scaled by the compression ratio that it measured.
Use that estimate. The rows of the current Arrow batch are not in the
writer yet, so scale their Arrow size by the writer's compression
ratio and add them. After close, length() is the file size.

Signed-off-by: Robert Kruszewski <github@robertk.io>
A runtime filter on a data column delays the scan until its subquery
completes. The Vortex reader drops the rows that the filter rules out,
which is worth that delay: TPC-DS at SF100 on Vortex takes 218 s with
these filters and 357 s without them. The Parquet reader keeps those
rows, and the files of most tables are not clustered by the join keys,
so the filter often drops nothing and only delays the scan.

Scans offer data columns only for tables that write Vortex files, and
only when every file of the scan is a Vortex file.
spark.sql.iceberg.runtime-filtering.data-columns.enabled still turns the
feature on or off for all tables.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…n plans

A runtime filter on a data column can hold hundreds of thousands of
values. The scan built the full description of its runtime filters on
every call to equals, hashCode and description, and Spark calls these
many times while it plans adaptively and posts plan updates. In TPC-DS
q33 at SF100 the filter on ss_addr_sk holds 364,761 values, and the
plan updates of one run grew to 214 MB.

The scan now builds the full description once, when it applies a
runtime filter, and keeps using it for equals and hashCode. Plans show
a shorter description that lists at most 10 values of each IN.

Signed-off-by: Robert Kruszewski <github@robertk.io>
…untime filtering

spark.sql.iceberg.runtime-filtering.data-columns.formats lists the file
formats whose scans offer data columns to runtime filters. The default
is vortex, as before. Listing parquet lets benchmarks measure Parquet
with data-column runtime filters.

Signed-off-by: Robert Kruszewski <github@robertk.io>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant