Repository navigation
Spark 4.2, Vortex: faster scans of Vortex tables - #48
Open
robert3005 wants to merge 28 commits into
Open
robert3005 wants to merge 28 commits into
robert3005 wants to merge 28 commits into
Conversation
Scans offer their data columns, not only the partition source columns, to Spark's runtime filters such as dynamic pruning. A join on any key can then prune an Iceberg table, which helps unpartitioned tables most. - The Iceberg session extensions install NestedDynamicPruning. The rule drops each pruning filter whose filtering plan holds pruning filters of its own, which otherwise grow the plan exponentially through self-joined common table expressions. Scans offer data columns only when the session runs the rule. - A runtime filter drops the files whose column metrics rule it out. The scan asks for the metrics of the columns it offers. - The runtime filter on data columns reaches the file readers with the residual, so they skip row groups and chunks. An IN with more than spark.sql.iceberg.runtime-filtering.max-values values reaches them as its range. - Scans of row-level operations do not offer data columns. Spark keys the groups of rows they rewrite by every filter attribute, and a null in a data column matches no key. - spark.sql.iceberg.runtime-filtering.data-columns.enabled turns the feature off. Signed-off-by: Robert Kruszewski <github@robertk.io>
Map each split boundary to the nearest natural split of the file, so scans of neighbouring splits do not both decode the chunk that holds the boundary. Signed-off-by: Robert Kruszewski <github@robertk.io>
…ex scans Spark predicates such as CONTAINS and ENDS_WITH have no Iceberg expression, so Iceberg reads every column of every row and Spark filters afterwards. The scan builder now converts them into LIKE filters for the Vortex reader, which applies them in the scan and reads the other columns only for the rows that match. Spark still evaluates the predicates after the scan. Signed-off-by: Robert Kruszewski <github@robertk.io>
Adaptive split sizing aims for one split per shuffle partition by file bytes. A scan of a few columns of a wide table then makes many tasks that each read little data, and every task adds partial aggregation and shuffle output. The adaptive size now counts the column sizes that the manifests record for the columns the scan reads, and converts the result back to file bytes, with at least one split per core. spark.sql.iceberg.read.column-aware-split-size.enabled turns this off. Signed-off-by: Robert Kruszewski <github@robertk.io>
Adaptive split sizing now makes splits of at least spark.sql.iceberg.read.min-file-split-size (64 MB by default), unless that leaves cores without work. The share per core also gets a third more room, because bin packing cannot fill each split. Before this change, two 16 MiB pieces were slightly larger than the even share per core, so a scan of store_sales ran 60 tasks in two waves on 32 cores. It now runs 30 tasks. Signed-off-by: Robert Kruszewski <github@robertk.io>
Finer split offsets let adaptive split sizing give each core a task for tables of about 1 GB. On TPC-DS SF10 with Spark 4.2 this changes the total from 104.8 s to 96.6 s (q22 15.4 s to 4.4 s, q72 7.4 s to 4.9 s). Signed-off-by: Robert Kruszewski <github@robertk.io>
Each task found the row splits of its file from the layouts of all the columns. Use only the projected columns, or the first column when the projection is empty. Signed-off-by: Robert Kruszewski <github@robertk.io>
Integer, long and string sets go to native code as one array, rather than one Java literal expression per value. Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Scans offer their data columns to Spark's runtime filters, drop files by the column metrics, and give the filter to the file readers. The Iceberg session extensions install NestedDynamicPruning, which stops pruning subqueries from nesting into each other. Spark 4.1 has no partition predicates, so the scans keep their Spark 4.1 constructors and only add the data column flag. Signed-off-by: Robert Kruszewski <github@robertk.io>
…into Vortex scans Backport of the Spark 4.2 change. A CONTAINS or ENDS_WITH predicate on a string column, which has no Iceberg expression, reaches the Vortex reader as a filter that skips rows early. Spark still evaluates the predicate after the scan. Signed-off-by: Robert Kruszewski <github@robertk.io>
…columns Backport of the Spark 4.2 change. Adaptive split sizing counts only the bytes of the columns that a scan reads, from the column sizes in the file metrics. A scan of a few columns of a wide table then gets enough splits for the cluster. Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. The share of bytes per core gets a third more room, because cuts fall on split offsets and packing cannot fill each split. Without it, a piece slightly larger than the share makes the scan take two waves of tasks. spark.sql.iceberg.read.min-file-split-size sets the smallest split. Signed-off-by: Robert Kruszewski <github@robertk.io>
A batch scan that pins its snapshot finds the names of its stats columns in the schema of that snapshot. Column-aware split sizing asked for the stats of the projected columns by their names in the table schema. After a column rename, the snapshot schema does not hold the new name, and planning failed with a NullPointerException. Spark then planned the query without runtime filters. The scan builder now gives the schema that the scan reads with to includeStats, and finds each stats column there by its field ID. The stats of a renamed column stay available. TestRuntimeFiltering.testRenamedSourceColumnTable covers this. Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 fix. A batch scan that pins its snapshot finds the names of its stats columns in the schema of that snapshot. After a column rename, planning failed with a NullPointerException, and Spark planned the query without runtime filters. The scan builder now finds each stats column by its field ID in the schema that the scan reads with. TestRuntimeFiltering.testRenamedSourceColumnTable covers this. Signed-off-by: Robert Kruszewski <github@robertk.io>
spotlessApply moves the VortexLikeFilters import into order. Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Scans offer their data columns to Spark's runtime filters, drop files by the column metrics, and give the filter to the file readers. The Iceberg session extensions install NestedDynamicPruning, which stops pruning subqueries from nesting into each other. Spark 4.0 has no SparkRuntimeFilterableScan, so the runtime filter logic goes into SparkBatchQueryScan, which implements runtime filtering there. The scan builder finds stats columns by field ID in the schema that the scan reads with, as in the Spark 4.2 fix for renamed columns. Signed-off-by: Robert Kruszewski <github@robertk.io>
…into Vortex scans Backport of the Spark 4.2 change. The scan builder turns CONTAINS and ENDS_WITH predicates on string columns, which Iceberg expressions cannot state, into filters for the Vortex reader. Spark still evaluates these predicates after the scan. Spark 4.0 has no BaseSparkScanBuilder, so the conversion goes into SparkScanBuilder.pushPredicates. Signed-off-by: Robert Kruszewski <github@robertk.io>
…columns Backport of the Spark 4.2 change. Adaptive split sizing counts only the bytes of the columns that a scan reads, from the column sizes in the file metrics. A scan of a few columns of a wide table then gets enough splits for the cluster. The scan builder finds the stats columns by field ID in the schema that the scan reads with, as in the Spark 4.2 fix for renamed columns. Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Adaptive split sizing makes splits of at least spark.sql.iceberg.read.min-file-split-size, unless that leaves cores without work. The share per core also gets a third more room, because bin packing cannot fill each split. Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Scans offer their data columns to Spark's runtime filters, drop files by the column metrics, and give the filter to the file readers. The Iceberg session extensions install NestedDynamicPruning, which stops pruning subqueries from nesting into each other. As in Spark 4.0, the runtime filter logic goes into SparkBatchQueryScan. Spark 3.5 has no variant stats columns, so the scan builder asks only for the stats of the data filter columns. Signed-off-by: Robert Kruszewski <github@robertk.io>
…into Vortex scans Backport of the Spark 4.2 change. The scan builder turns CONTAINS and ENDS_WITH predicates on string columns, which Iceberg expressions cannot state, into filters for the Vortex reader. Spark still evaluates these predicates after the scan. As in Spark 4.0, the conversion goes into SparkScanBuilder.pushPredicates. Signed-off-by: Robert Kruszewski <github@robertk.io>
…columns Backport of the Spark 4.2 change. Adaptive split sizing counts only the bytes of the columns that a scan reads, from the column sizes in the file metrics. A scan of a few columns of a wide table then gets enough splits for the cluster. The scan builder finds the stats columns by field ID in the schema that the scan reads with, as in the Spark 4.2 fix for renamed columns. Signed-off-by: Robert Kruszewski <github@robertk.io>
Backport of the Spark 4.2 change. Adaptive split sizing makes splits of at least spark.sql.iceberg.read.min-file-split-size, unless that leaves cores without work. The share per core also gets a third more room, because bin packing cannot fill each split. Signed-off-by: Robert Kruszewski <github@robertk.io>
InputFileReadable copied each range in 1 MiB reads, and each stream came from InputFile#newStream. On S3A, each stream sent a HEAD request and each 1 MiB read sent its own GET. For TPC-DS SF100 q3, Vortex made 2649 GETs of 691 KB on average, against 472 GETs of 3.7 MB for Parquet. Read each range with one positional read. For Hadoop input files, open the streams with FileSystem#openFile, the known length and the random read policy, so that S3A sends no HEAD and one GET for each range. Signed-off-by: Robert Kruszewski <github@robertk.io>
length() took the larger of the bytes in the sink and the uncompressed Arrow size of the accepted rows. Vortex compresses well, so rolling writers closed files far below the target size. The Vortex writer now estimates the size of an open file: the bytes in the sink, the compressed data that it holds, and the data that it did not compress yet, scaled by the compression ratio that it measured. Use that estimate. The rows of the current Arrow batch are not in the writer yet, so scale their Arrow size by the writer's compression ratio and add them. After close, length() is the file size. Signed-off-by: Robert Kruszewski <github@robertk.io>
A runtime filter on a data column delays the scan until its subquery completes. The Vortex reader drops the rows that the filter rules out, which is worth that delay: TPC-DS at SF100 on Vortex takes 218 s with these filters and 357 s without them. The Parquet reader keeps those rows, and the files of most tables are not clustered by the join keys, so the filter often drops nothing and only delays the scan. Scans offer data columns only for tables that write Vortex files, and only when every file of the scan is a Vortex file. spark.sql.iceberg.runtime-filtering.data-columns.enabled still turns the feature on or off for all tables. Signed-off-by: Robert Kruszewski <github@robertk.io>
…n plans A runtime filter on a data column can hold hundreds of thousands of values. The scan built the full description of its runtime filters on every call to equals, hashCode and description, and Spark calls these many times while it plans adaptively and posts plan updates. In TPC-DS q33 at SF100 the filter on ss_addr_sk holds 364,761 values, and the plan updates of one run grew to 214 MB. The scan now builds the full description once, when it applies a runtime filter, and keeps using it for equals and hashCode. Plans show a shorter description that lists at most 10 values of each IN. Signed-off-by: Robert Kruszewski <github@robertk.io>
…untime filtering spark.sql.iceberg.runtime-filtering.data-columns.formats lists the file formats whose scans offer data columns to runtime filters. The default is vortex, as before. Listing parquet lets benchmarks measure Parquet with data-column runtime filters. Signed-off-by: Robert Kruszewski <github@robertk.io>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance changes to the Spark 4.2 integration and the Vortex format.
Spark 4.2
e4630f4Prune data columns with runtime filters. Scans offer their data columns, not only the partition source columns, to Spark's runtime filters. A filter drops the files whose column metrics rule it out, and it reaches the file readers with the residual. The session extensions installNestedDynamicPruning, which drops pruning filters that hold pruning filters of their own.3e45b8cOffer data columns to runtime filtering only for Vortex. A scan offers data columns only when the table writes Vortex files and every file of the scan is a Vortex file.6062c7eAdd a setting for the formats that offer data columns.spark.sql.iceberg.runtime-filtering.data-columns.formatslists the formats. The default isvortex.cd1735dDescribe runtime filters once and shorten large IN lists in plans. The scan builds the full description when it applies a runtime filter, and plans show at most 10 values of each IN.a498d62Size adaptive splits by the bytes of the scanned columns. A scan of a few columns of a wide table no longer gets as few splits as a scan of the full rows.spark.sql.iceberg.read.column-aware-split-size.enabledturns this off.b4db8b2Fit adaptive splits to one wave of tasks. The share per core gets a third more room, because cuts fall on chunk boundaries. Addsspark.sql.iceberg.read.min-file-split-size(default 64 MB).e53bb56Find stats columns by field ID in the scan schema. After a column rename, planning failed with aNullPointerException.Vortex
54fdee5Align byte-range splits to chunk boundaries, so that no two tasks decode the same chunk.fbb6587Push string matches that Iceberg cannot state (CONTAINS,ENDS_WITH) into Vortex scans. Spark still evaluates the predicates after the scan.cc0c385Write split offsets every 16 MB.cad0a4ePlan row splits from the projected columns only.57abc8bConvert large IN sets in one native call.72b8624Read each range with one request. Streams open withopenFile, the known file length and the random read policy.994e129Use the writer's size estimate for the length of open files.length()returnsVortexWriter.estimatedSize(), plus the rows that are not in the writer yet, scaled byVortexWriter.compressionRatio().Backports
The Spark-side changes also go to Spark 4.1, 4.0 and 3.5, with one commit per version.
3e45b8c,6062c7eandcd1735dare in Spark 4.2 only.e4630f4Prune data columns with runtime filterscf8f62c0285e9e98bafb5fbb6587Push string matches into Vortex scans (Spark part)c8b13bb45e5b05b070883a498d62Size adaptive splits by the bytes of the scanned columns954f144bd6bb20467a888b4db8b2Fit adaptive splits to one wave of tasksdd06cd6ac27725b35e14ae53bb56Find stats columns by field ID in the scan schema01f62260285e9e,bd6bb2098bafb5,467a888SparkRuntimeFilterableScanorBaseSparkScanBuilder. The runtime filter logic goes intoSparkBatchQueryScan, and the string match conversion goes intoSparkScanBuilder.pushPredicates.Dependency
cad0a4e,57abc8band994e129use Vortex JNI methods that are not in a released version yet (vortex-data/vortex#11096 and vortex-data/vortex#11357). Merge this PR after a Vortex release that holds both, and bump the Vortex version.