Repository navigation
Add support for schemaMapping, Geometry/Unknown types and more expression pushdown - #38
Merged
Merged
Conversation
Vortex 0.86.1 replaces VortexWriter.create with a builder, and it can now store Arrow maps, so the Iceberg integration no longer has to skip them. - Bump dev.vortex to 0.86.1 and switch to VortexWriter.builder(...).build(). - Map Iceberg MAP to Arrow's canonical map layout (a non-nullable "entries" struct of "key"/"value") in both schema directions, and add map readers and writers to the generic and Spark row-based paths. The Spark columnar path already handled maps. - Store Iceberg FIXED as variable-width binary. Vortex rejects Arrow FixedSizeBinary unless it carries the arrow.uuid extension, so the declared length is enforced by the writer and re-imposed by the reader from the Iceberg schema. This also un-skips every test whose schema embeds SUPPORTED_PRIMITIVES (which contains fixed[7]). - Stop truncating FIXED bounds in VortexMetrics. A truncated value is not a valid fixed[N], and Parquet metrics do not truncate them either. This was unreachable while FIXED could not be written. - Drop the map/fixed assumptions from TestGenericVortex and TestGenericReadProjection. testMapOfStructsProjection stays disabled for the same reason as the other rename cases: rebinding a renamed column needs Iceberg field ids the file does not carry. The comment there now records that 0.86.1 adds a file-level metadata channel that could carry them. Vortex module tests go from 23 skipped to 11 (unknown, geometry/geography, row lineage, and the four rename cases). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Vortex drops Arrow field and schema metadata, so files carried no Iceberg field ids and readers had to bind columns by name. 0.86 adds a file-level metadata channel, which is enough to persist the ids. - Writers store the Iceberg schema JSON under the "iceberg.schema" file metadata key. - Readers load it and tag the file's Arrow fields with PARQUET:field_id (in memory only), then bind expected columns to file columns by id: the scan projects each column under its physical name, and the generic, Spark row-based and Spark columnar readers all resolve through the same VortexSchemas.FieldBinding. - Binding is by id alone when the file carries ids, so a column added under a name that used to belong to another column reads as missing instead of picking up the old column's values. Fields the file's schema does not describe - the synthetic _pos column the scan materializes - carry no id and still match by name. Files with no stored schema fall back to name-based binding, which is what they have always used. This re-enables testRename, testRenamedAddedField, testListOfStructsProjection and testMapOfStructsProjection; TestGenericReadProjection now has no skips. Also drops the FIXED support added in the previous commit: Vortex has no fixed-width binary type, so rather than storing FIXED as variable-width binary the schema conversion now refuses it with a message naming the column, and tests whose schemas embed fixed[7] are skipped again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Three of the four remaining DataTestBase skips were unimplemented integration work rather than Vortex limitations. unknown: stored as an Arrow null column, which Vortex round-trips natively. Iceberg requires unknown fields to be optional and always null, so writes go through writeNull and reads return a constant null. Parquet instead omits the column from the file; storing an empty null column keeps the writer's positional column mapping intact and is equivalent on read. row lineage: _row_id and _last_updated_sequence_number are stored data columns when the engine supplies no inheritance base, but the read projection dropped every metadata column by name, so the scan never requested them. It now drops only the two columns the reader materializes itself (_pos from row_idx, and _is_deleted), and VortexIterable drops anything the file turns out not to have. Lineage columns supplied through idToConstant are still excluded, since they are constants and are projected separately as a packed expression. geospatial: geometry and geography are WKB binary per the spec, so both map to Arrow binary. The Iceberg schema now stored in file metadata is what tells them apart from BINARY on read. Bounds are deliberately not produced: Iceberg geospatial bounds are min/max coordinate points, not byte ranges, and Comparators has no comparator for these types. Geospatial also needed the shared test harness, which could not generate or compare these values for any format: RandomUtil now emits a well-formed little-endian WKB point, RandomGenericData wraps it as a ByteBuffer like binary, and DataTestHelpers compares it as binary. These are additive cases on switches that previously threw, and every other format still leaves supportsGeospatial() false. Vortex module: 144 tests, 6 skipped (all schemas containing fixed), 0 failures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Predicate pushdown: - A transform term (bucket, truncate, ...) threw and failed the whole scan. ReadBuilder#filter is explicitly allowed to return partially filtered rows, so an untranslatable predicate is now dropped instead. - STARTS_WITH / NOT_STARTS_WITH translate to Vortex LIKE over a prefix pattern. Vortex LIKE treats % and _ as wildcards and \ as its escape character, and silently drops a backslash that does not introduce a valid escape, so all three are escaped in the prefix. NOT LIKE does not match null while Iceberg's notStartsWith does, so nulls are added back. - An IN set past the expansion limit is approximated by the inclusive range its values span instead of being dropped. NOT_IN has no such bound. - Approximated expressions are tracked so a negation over one degrades to "read everything" rather than matching fewer rows than the predicate. - IS_NAN / NOT_NAN stay untranslatable, now with the reason recorded: Vortex compares NaN equal to itself (verified against 0.86.1, where `f == f` matches NaN rows and `f != f` matches none), so nothing isolates NaN. - TIME stays untranslatable: Vortex has no time literal or time DType. TestVortexFilterPushdown is new and checks the property that actually matters: for each predicate, the rows the scan returns are a superset of the rows Iceberg's own Evaluator matches. The previous tests only checked that conversion returned a non-sentinel expression, so a filter that dropped matching rows would have passed. withNameMapping: a file with no stored Iceberg schema is exactly the case a name mapping exists for. The mapping now annotates the file's Arrow fields with ids the same way the stored schema does, so reads of files written by another producer resolve columns by id. A stored schema still wins. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Vortex now matches Parquet: an unknown column holds nothing but nulls, so it is left out of the file entirely and the reader fills it back in. Struct children are dropped the same way, and a list element or map key/value of type unknown is rejected, because those positions have no slot to drop -- both exactly where TypeToMessageType skips and rejects. Dropping columns means the Arrow root holds fewer vectors than the schema has columns, so the generic and Spark writers now map each column to its vector and skip the ones that have none, rather than indexing the root positionally. VortexMetrics indexes the written columns for the same reason: Vortex reports statistics per stored column. This also closes the Spark gap without a special case. Previously a Spark read or write of an unknown column threw from the default branch of SparkVortexReader/SparkVortexWriter, even though TypeToSparkType maps unknown to NullType. Now the column is simply absent from the file, so the existing "column not in the file" paths cover it: the writer skips it and the reader fills null through defaultReader. TestSparkVortexUnknown covers the round trip, including a struct child, on all three Spark versions. The read side still understands an Arrow null column, so a file written by another producer that does store one is read as unknown. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Vortex was absent from several shared suites that Parquet, ORC and Avro all run. Adding it found one real gap and closed three coverage holes. Enabled with no production change needed: - TestBaseTaskWriter (3 cases) and TestPositionDeltaWriters (4 cases) pass as they stand. - TestTaskEqualityDeltaWriter (21 cases) needed only its own read helper taught to read Vortex, through FormatModelRegistry like TestFileWriterFactory already does. These cover equality deltas and upserts, which the module had no coverage for despite accepting EQUALITY_DELETES. New: TestSparkVortexReader on all three Spark versions, the counterpart of TestGenericVortex that Parquet, ORC and Avro each have and Vortex did not. Spark had no type coverage on the Vortex read path at all. 55 cases; the skips are fixed (no Vortex fixed-width binary) and time, timestamp_ns and geospatial, which Iceberg's own Spark type conversion does not map. That test found a real gap. Unknown as a list element or map key/value is stored as an Arrow null column, because unlike a struct field those positions have no slot to drop -- the previous commit refused them instead, which was wrong. Structs and top-level columns still omit the field entirely. The generic reader already handled this through the Arrow type, but the Spark reader switches on the Iceberg type and threw, so it now reads unknown as a null constant. Not enabled: TestAppenderFactory, because GenericAppenderFactory is @deprecated for removal in 1.12.0 in favour of GenericFileWriterFactory, which Vortex already supports and is already covered. TestWriterMetrics, because Vortex only produces top-level column metrics, so nested bounds assertions fail; that limitation is known and a permanently red suite would not record it any better. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
The "Cannot write pos-deletes for unsupported file format: VORTEX" error I flagged earlier is not missing wiring, and wiring Vortex in would be wrong. SparkFileWriterFactory extends RegistryBasedFileWriterFactory, and newPositionDeleteWriter delegates to that registry writer -- which supports Vortex -- unless useDeprecatedPositionDeleteWriter is set. That flag is only set when a positionDeleteRowSchema is present, i.e. for position deletes that carry row data, a feature the class itself marks deprecated and slated for removal in 1.12.0. Vortex deliberately does not persist deleted rows: PositionDeleteVortexWriter writes only [file_path, pos], and TestFileWriterFactory already skips testPositionDeleteWriterWithRow for Vortex for that reason. So the format is not unsupported, the combination is, and only on a path that is going away. The error now says that instead of implying Vortex is unwired. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Several comments described the state of the tree rather than the behaviour they sit next to: comparisons to how Parquet currently handles the same case, notes about which Vortex version a behaviour was verified against, and history about what these files used to do or used not to cover. None of that stays true on its own, and none of it helps a reader understand the code. Each is rewritten to state the behaviour directly. The reasoning is kept wherever it explains a decision -- that Vortex compares NaN as equal to itself so nothing isolates NaN, that an unescaped backslash in a LIKE pattern is dropped, that unknown columns hold nothing but nulls so they are not stored -- just without anchoring it to a version number or to another format's implementation. Comments that predate this branch are left alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
|
|
robert3005
commented
Sep 14, 2026
…numerating tests
VortexArrowColumnVector bound nested struct children positionally
(StructVector#getVectorById), but Spark addresses struct children by
their ordinal in the projected schema. Once a nested projection prunes,
reorders, renames, or adds a field, the two disagree: selecting
struct.innerName from a file whose struct is {innerId, innerName} read
child 0 instead. When the Arrow types happened to differ this surfaced
as an UnsupportedOperationException from the wrong accessor; when they
matched it silently returned a sibling field's values. Vortex always
takes the columnar read path, so this was reachable for every engine
read of a pruned nested struct.
Thread the id-annotated Arrow field and the projected Iceberg type down
through the vector tree so struct children resolve by field id (falling
back to name for files without ids), mirroring what SparkVortexReader's
StructReader already did on the row path. Children absent from the file
become constant vectors carrying the field's initial default, or nulls.
Lists and maps recurse so nested structs inside them are reordered too.
Also add Vortex to the upstream suites that enumerate file formats and
now pass: TestFileFormat, TestSparkDataWrite, TestFilteredScan,
TestPartitionPruning, TestPartitionValues, TestSparkReadProjection,
TestSparkMetadataColumns, TestTimestampWithoutZone and
TestDataFrameWriterV2Coercion. Vortex reads are always vectorized, so
each Vortex case pins the suite's vectorized flag to false rather than
running the same path twice under both values.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
…PartitionEvolution Both suites enumerate file formats and pass unmodified for Vortex: 4 and 11 Vortex cases respectively. As with the other Vortex cases, the vectorized flag is pinned to false because Vortex reads are always vectorized. Two further format-enumerating suites were evaluated and deliberately left alone, because Vortex fails them for reasons in the writers rather than in the tests: - TestPositionDeletesTable writes its delete files through GenericFileWriterFactory, whose position-delete switch handles only Avro, ORC and Parquet and so never reaches the registered Vortex position-delete format model. - TestSparkWriterMetrics asserts bounds on a nested sorted column and reads nanValueCounts, neither of which the Vortex writer reports, and one of its cases needs position deletes carrying row data, which Vortex files do not persist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
SparkRowLevelOperationsTestBase enumerates file formats, and both concrete delete suites pass for Vortex: 44 cases in TestCopyOnWriteDelete and 49 in TestMergeOnReadDelete. testDeleteNonExistingRecords expects the snapshot operation to be a delete only for ORC and Parquet, and a rewrite for everything else. That split is really about whether the format reports column bounds: with them the planner proves no file matches and commits a delete of zero files, without them it falls back to rewriting. Vortex reports bounds and so commits a delete, so add it alongside ORC and Parquet and say what the branch is actually testing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Mirrors the v4.1 changes to TestIdentityPartitionData, TestMetadataTablesWithPartitionEvolution, SparkRowLevelOperationsTestBase and TestDelete. 109 Vortex cases, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Mirrors the v4.1 and v4.0 changes to TestIdentityPartitionData, TestMetadataTablesWithPartitionEvolution, SparkRowLevelOperationsTestBase and TestDelete. 109 Vortex cases, all passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
A set predicate past the expansion limit was pushed down as the inclusive range its values span. That range matches a superset of the rows the predicate does, which is sound on its own but needed machinery to keep it sound under negation: every approximate expression was tracked by identity so that a NOT over one could be dropped, and AND/OR had to propagate the marking. Large IN is now simply left to the engine, which retires the tracking set and both propagation paths along with it. Vortex will grow real support for these sets. The test that covered this asserted the range still excluded rows outside it, but built its 300 values from only two distinct literals, so the literal set was of size two and never crossed the limit it meant to exercise. It now uses distinct values and asserts what the contract actually requires: the predicate is not pushed, and no matching row is lost. Flink has no Vortex support and its tests are not part of this work, so the Flink CI workflow no longer runs automatically; it stays available via workflow_dispatch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
TestPositionDeltaWriters gained a Vortex case when the upstream writer suites were enabled for Vortex, but its parameters() is protected static and TestFlinkPositionDeltaWriters declares none of its own, so all three Flink versions inherited that case. Vortex is not on the Flink test runtime classpath and FlinkFileWriterFactory has no Vortex branch, so the inherited case could only fail: the base declares four @testtemplate methods, which is exactly the four failures Flink CI reported. Override parameters() in the Flink subclass to list only the formats that factory writes, matching TestFlinkFileWriterFactory, TestFlinkPartitioningWriters and TestFlinkRollingFileWriters, which each already do this for the same reason. This also restores the Flink CI workflow triggers. Disabling the workflow hid this failure rather than fixing it. The other shared bases that mention Vortex are fine as they stand: BaseFormatModelTests only adds the format when org.apache.iceberg.vortex.VortexFormatModels resolves, which it does not for Flink, and TestBaseTaskWriter, TestTaskEqualityDeltaWriter, TestLocalScan and TestFileFormat have no Flink subclasses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Iceberg treats a position delete file as covering a single data file when its file_path lower and upper bounds are equal, and the engine only rewrites deletes it can attribute to one data file: SparkBatchQueryScan skips anything ContentFileUtil.isFileScoped rejects. Vortex reports string bounds from its native statistics as a truncated 64 byte prefix range, so for any data file path longer than that the two bounds differed and no Vortex delete file was ever file scoped. Superseded position deletes were therefore never removed, which TestMergeOnReadUpdate caught through removedDeleteFiles coming back empty. PositionDeleteVortexWriter now tracks file_path and pos itself, and the appender prefers those exact bounds over the native ones. This is scoped to the position delete path: data files keep using Vortex's statistics, where a widened bound only costs pruning rather than correctness. The writer copies the path string because callers reuse one PositionDelete across rows. Writer supplied metrics had no effect before this: VortexFileAppender built metrics only from the native write summary, and the FieldMetrics path through VortexMetrics.buildMetrics was reached from tests alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
A column written as int and later widened to long still reads from a file that stores ints, but VortexArrowColumnVector mapped every IntVector to an IntAccessor that implements only getInt. Spark's generated code asks the vector for the projected type, so the first getLong on such a column threw UnsupportedOperationException from the accessor base class. The row reader already handled this through intsAsLongs and floatsAsDoubles; the vectorized reader, which Vortex always takes, did not. Use the projected Iceberg type the vector already carries to pick an accessor that answers the wider getter: any integer vector narrower than 64 bits can serve getLong, and a float vector can serve getDouble. The vector also has to report the promoted type, since that is what Spark picks its getters from. Decimal precision widening is the remaining legal promotion and is not covered here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
VortexValueWriter.metrics() was never read. VortexFileAppender builds a file's Metrics from Vortex's native write summary, and the FieldMetrics path through VortexMetrics.buildMetrics was reached only from tests, so GenericVortexWriter accumulated per-column counts and bounds on every written row that nothing ever asked for. Drop the interface method, the builder it fed, and the tracker machinery behind it. Position deletes were the one real consumer, and they need one specific thing rather than a general metrics channel: Iceberg reads a delete file as covering a single data file when file_path's bounds are equal, and Vortex reports string bounds as a truncated prefix range, so any real path compares unequal. PositionDeleteVortexWriter now tracks the lowest and highest path it writes and reports them through VortexExactBounds, which the appender picks up by type. That replaces the flag threaded through VortexFormatModel: only a writer that tracked something declares it. The pos column needs no such handling, since Vortex reports numeric bounds exactly. Three of the tests covering the removed builder were really covering MetricsConfig handling, which still applies to the summary path, so they move onto the appender rather than being deleted; the truncate case now also pins down that Iceberg truncation layers on top of Vortex's bounds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
robert3005
pushed a commit
that referenced
this pull request
Sep 16, 2026
Two of the features the previous commit marked missing for Vortex are in fact supported; the flags were wrong. Vortex 0.86.1, which #38 pulled in, added VortexWriter.Builder.metadata()/putMetadata(), and #38 already uses putMetadata to persist the Iceberg schema so readers can rebind columns renamed since the file was written. - FEATURE_EVOLUTION_BY_FIELD_ID: ungated, no code needed. It has worked since #38 -- TestVortexSchemaMetadata.testRenamedColumnsBindByFieldId already covers it. The flag was introduced by #36 and left stale when #38 closed the underlying gap. - FEATURE_WRITER_METADATA: ungated. The capability was there and only the Iceberg plumbing was missing -- WriteBuilderWrapper.meta() was a no-op carrying the stale comment "Vortex files carry no user-defined key/value metadata". It now collects entries and passes them to putMetadata, rejecting the reserved iceberg.schema key. New public VortexFiles.metadata(InputFile) reads them back, so VortexFormat can implement metadataValue; VortexIO stays package-private. FEATURE_WRITER_PROPERTIES stays gated: Vortex has no write properties -- the writer accepts none today, though some are planned -- so there is nothing to set and nothing to observe in the written file. FEATURE_FIXED and FEATURE_WRITER_OVERWRITE stay gated as before. iceberg-data now takes iceberg-vortex as testImplementation rather than testRuntimeOnly so VortexFormat can read file metadata directly. This is not circular: data:compileTestJava needs vortex:jar, and vortex:compileTestJava needs data:testJar. The Class.forName guard in FileFormatTestSupport.all() still keeps Vortex out of the Flink modules, which do not ship it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GxC1sZXqWV6UibNP9qaD9w
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This resolves the biggest issue of schema roundtripping by leveraging vortex file metadata.
We still need some more expressions in vortex to account for all possible pushdown but these are purely optimisations