Skip to content

Add support for schemaMapping, Geometry/Unknown types and more expression pushdown - #38

Merged
robert3005 merged 19 commits into
mainfrom
claude/jolly-einstein-mesa2k
Sep 16, 2026
Merged

robert3005 merged 19 commits into
mainfrom
claude/jolly-einstein-mesa2k

Conversation

@robert3005

Copy link
Copy Markdown
Member

This resolves the biggest issue of schema roundtripping by leveraging vortex file metadata.

We still need some more expressions in vortex to account for all possible pushdown but these are purely optimisations

Vortex 0.86.1 replaces VortexWriter.create with a builder, and it can now
store Arrow maps, so the Iceberg integration no longer has to skip them.

- Bump dev.vortex to 0.86.1 and switch to VortexWriter.builder(...).build().
- Map Iceberg MAP to Arrow's canonical map layout (a non-nullable "entries"
  struct of "key"/"value") in both schema directions, and add map readers and
  writers to the generic and Spark row-based paths. The Spark columnar path
  already handled maps.
- Store Iceberg FIXED as variable-width binary. Vortex rejects Arrow
  FixedSizeBinary unless it carries the arrow.uuid extension, so the declared
  length is enforced by the writer and re-imposed by the reader from the
  Iceberg schema. This also un-skips every test whose schema embeds
  SUPPORTED_PRIMITIVES (which contains fixed[7]).
- Stop truncating FIXED bounds in VortexMetrics. A truncated value is not a
  valid fixed[N], and Parquet metrics do not truncate them either. This was
  unreachable while FIXED could not be written.
- Drop the map/fixed assumptions from TestGenericVortex and
  TestGenericReadProjection. testMapOfStructsProjection stays disabled for the
  same reason as the other rename cases: rebinding a renamed column needs
  Iceberg field ids the file does not carry. The comment there now records
  that 0.86.1 adds a file-level metadata channel that could carry them.

Vortex module tests go from 23 skipped to 11 (unknown, geometry/geography,
row lineage, and the four rename cases).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Vortex drops Arrow field and schema metadata, so files carried no Iceberg
field ids and readers had to bind columns by name. 0.86 adds a file-level
metadata channel, which is enough to persist the ids.

- Writers store the Iceberg schema JSON under the "iceberg.schema" file
  metadata key.
- Readers load it and tag the file's Arrow fields with PARQUET:field_id (in
  memory only), then bind expected columns to file columns by id: the scan
  projects each column under its physical name, and the generic, Spark
  row-based and Spark columnar readers all resolve through the same
  VortexSchemas.FieldBinding.
- Binding is by id alone when the file carries ids, so a column added under a
  name that used to belong to another column reads as missing instead of
  picking up the old column's values. Fields the file's schema does not
  describe - the synthetic _pos column the scan materializes - carry no id and
  still match by name. Files with no stored schema fall back to name-based
  binding, which is what they have always used.

This re-enables testRename, testRenamedAddedField, testListOfStructsProjection
and testMapOfStructsProjection; TestGenericReadProjection now has no skips.

Also drops the FIXED support added in the previous commit: Vortex has no
fixed-width binary type, so rather than storing FIXED as variable-width binary
the schema conversion now refuses it with a message naming the column, and
tests whose schemas embed fixed[7] are skipped again.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Three of the four remaining DataTestBase skips were unimplemented
integration work rather than Vortex limitations.

unknown: stored as an Arrow null column, which Vortex round-trips natively.
Iceberg requires unknown fields to be optional and always null, so writes go
through writeNull and reads return a constant null. Parquet instead omits the
column from the file; storing an empty null column keeps the writer's
positional column mapping intact and is equivalent on read.

row lineage: _row_id and _last_updated_sequence_number are stored data columns
when the engine supplies no inheritance base, but the read projection dropped
every metadata column by name, so the scan never requested them. It now drops
only the two columns the reader materializes itself (_pos from row_idx, and
_is_deleted), and VortexIterable drops anything the file turns out not to
have. Lineage columns supplied through idToConstant are still excluded, since
they are constants and are projected separately as a packed expression.

geospatial: geometry and geography are WKB binary per the spec, so both map to
Arrow binary. The Iceberg schema now stored in file metadata is what tells
them apart from BINARY on read. Bounds are deliberately not produced: Iceberg
geospatial bounds are min/max coordinate points, not byte ranges, and
Comparators has no comparator for these types.

Geospatial also needed the shared test harness, which could not generate or
compare these values for any format: RandomUtil now emits a well-formed
little-endian WKB point, RandomGenericData wraps it as a ByteBuffer like
binary, and DataTestHelpers compares it as binary. These are additive cases
on switches that previously threw, and every other format still leaves
supportsGeospatial() false.

Vortex module: 144 tests, 6 skipped (all schemas containing fixed), 0 failures.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Predicate pushdown:
- A transform term (bucket, truncate, ...) threw and failed the whole scan.
  ReadBuilder#filter is explicitly allowed to return partially filtered rows,
  so an untranslatable predicate is now dropped instead.
- STARTS_WITH / NOT_STARTS_WITH translate to Vortex LIKE over a prefix
  pattern. Vortex LIKE treats % and _ as wildcards and \ as its escape
  character, and silently drops a backslash that does not introduce a valid
  escape, so all three are escaped in the prefix. NOT LIKE does not match
  null while Iceberg's notStartsWith does, so nulls are added back.
- An IN set past the expansion limit is approximated by the inclusive range
  its values span instead of being dropped. NOT_IN has no such bound.
- Approximated expressions are tracked so a negation over one degrades to
  "read everything" rather than matching fewer rows than the predicate.
- IS_NAN / NOT_NAN stay untranslatable, now with the reason recorded: Vortex
  compares NaN equal to itself (verified against 0.86.1, where `f == f`
  matches NaN rows and `f != f` matches none), so nothing isolates NaN.
- TIME stays untranslatable: Vortex has no time literal or time DType.

TestVortexFilterPushdown is new and checks the property that actually
matters: for each predicate, the rows the scan returns are a superset of the
rows Iceberg's own Evaluator matches. The previous tests only checked that
conversion returned a non-sentinel expression, so a filter that dropped
matching rows would have passed.

withNameMapping: a file with no stored Iceberg schema is exactly the case a
name mapping exists for. The mapping now annotates the file's Arrow fields
with ids the same way the stored schema does, so reads of files written by
another producer resolve columns by id. A stored schema still wins.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Vortex now matches Parquet: an unknown column holds nothing but nulls, so it
is left out of the file entirely and the reader fills it back in. Struct
children are dropped the same way, and a list element or map key/value of
type unknown is rejected, because those positions have no slot to drop --
both exactly where TypeToMessageType skips and rejects.

Dropping columns means the Arrow root holds fewer vectors than the schema has
columns, so the generic and Spark writers now map each column to its vector
and skip the ones that have none, rather than indexing the root positionally.
VortexMetrics indexes the written columns for the same reason: Vortex reports
statistics per stored column.

This also closes the Spark gap without a special case. Previously a Spark read
or write of an unknown column threw from the default branch of
SparkVortexReader/SparkVortexWriter, even though TypeToSparkType maps unknown
to NullType. Now the column is simply absent from the file, so the existing
"column not in the file" paths cover it: the writer skips it and the reader
fills null through defaultReader. TestSparkVortexUnknown covers the round
trip, including a struct child, on all three Spark versions.

The read side still understands an Arrow null column, so a file written by
another producer that does store one is read as unknown.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Vortex was absent from several shared suites that Parquet, ORC and Avro all
run. Adding it found one real gap and closed three coverage holes.

Enabled with no production change needed:
- TestBaseTaskWriter (3 cases) and TestPositionDeltaWriters (4 cases) pass as
  they stand.
- TestTaskEqualityDeltaWriter (21 cases) needed only its own read helper
  taught to read Vortex, through FormatModelRegistry like TestFileWriterFactory
  already does. These cover equality deltas and upserts, which the module had
  no coverage for despite accepting EQUALITY_DELETES.

New: TestSparkVortexReader on all three Spark versions, the counterpart of
TestGenericVortex that Parquet, ORC and Avro each have and Vortex did not.
Spark had no type coverage on the Vortex read path at all. 55 cases; the
skips are fixed (no Vortex fixed-width binary) and time, timestamp_ns and
geospatial, which Iceberg's own Spark type conversion does not map.

That test found a real gap. Unknown as a list element or map key/value is
stored as an Arrow null column, because unlike a struct field those positions
have no slot to drop -- the previous commit refused them instead, which was
wrong. Structs and top-level columns still omit the field entirely. The
generic reader already handled this through the Arrow type, but the Spark
reader switches on the Iceberg type and threw, so it now reads unknown as a
null constant.

Not enabled: TestAppenderFactory, because GenericAppenderFactory is
@deprecated for removal in 1.12.0 in favour of GenericFileWriterFactory, which
Vortex already supports and is already covered. TestWriterMetrics, because
Vortex only produces top-level column metrics, so nested bounds assertions
fail; that limitation is known and a permanently red suite would not record it
any better.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
The "Cannot write pos-deletes for unsupported file format: VORTEX" error I
flagged earlier is not missing wiring, and wiring Vortex in would be wrong.

SparkFileWriterFactory extends RegistryBasedFileWriterFactory, and
newPositionDeleteWriter delegates to that registry writer -- which supports
Vortex -- unless useDeprecatedPositionDeleteWriter is set. That flag is only
set when a positionDeleteRowSchema is present, i.e. for position deletes that
carry row data, a feature the class itself marks deprecated and slated for
removal in 1.12.0. Vortex deliberately does not persist deleted rows:
PositionDeleteVortexWriter writes only [file_path, pos], and
TestFileWriterFactory already skips testPositionDeleteWriterWithRow for Vortex
for that reason.

So the format is not unsupported, the combination is, and only on a path that
is going away. The error now says that instead of implying Vortex is unwired.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Several comments described the state of the tree rather than the behaviour
they sit next to: comparisons to how Parquet currently handles the same case,
notes about which Vortex version a behaviour was verified against, and history
about what these files used to do or used not to cover. None of that stays
true on its own, and none of it helps a reader understand the code.

Each is rewritten to state the behaviour directly. The reasoning is kept
wherever it explains a decision -- that Vortex compares NaN as equal to itself
so nothing isolates NaN, that an unescaped backslash in a LIKE pattern is
dropped, that unknown columns hold nothing but nulls so they are not stored --
just without anchoring it to a version number or to another format's
implementation.

Comments that predate this branch are left alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

…numerating tests

VortexArrowColumnVector bound nested struct children positionally
(StructVector#getVectorById), but Spark addresses struct children by
their ordinal in the projected schema. Once a nested projection prunes,
reorders, renames, or adds a field, the two disagree: selecting
struct.innerName from a file whose struct is {innerId, innerName} read
child 0 instead. When the Arrow types happened to differ this surfaced
as an UnsupportedOperationException from the wrong accessor; when they
matched it silently returned a sibling field's values. Vortex always
takes the columnar read path, so this was reachable for every engine
read of a pruned nested struct.

Thread the id-annotated Arrow field and the projected Iceberg type down
through the vector tree so struct children resolve by field id (falling
back to name for files without ids), mirroring what SparkVortexReader's
StructReader already did on the row path. Children absent from the file
become constant vectors carrying the field's initial default, or nulls.
Lists and maps recurse so nested structs inside them are reordered too.

Also add Vortex to the upstream suites that enumerate file formats and
now pass: TestFileFormat, TestSparkDataWrite, TestFilteredScan,
TestPartitionPruning, TestPartitionValues, TestSparkReadProjection,
TestSparkMetadataColumns, TestTimestampWithoutZone and
TestDataFrameWriterV2Coercion. Vortex reads are always vectorized, so
each Vortex case pins the suite's vectorized flag to false rather than
running the same path twice under both values.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
…PartitionEvolution

Both suites enumerate file formats and pass unmodified for Vortex:
4 and 11 Vortex cases respectively. As with the other Vortex cases,
the vectorized flag is pinned to false because Vortex reads are always
vectorized.

Two further format-enumerating suites were evaluated and deliberately
left alone, because Vortex fails them for reasons in the writers rather
than in the tests:

- TestPositionDeletesTable writes its delete files through
  GenericFileWriterFactory, whose position-delete switch handles only
  Avro, ORC and Parquet and so never reaches the registered Vortex
  position-delete format model.
- TestSparkWriterMetrics asserts bounds on a nested sorted column and
  reads nanValueCounts, neither of which the Vortex writer reports, and
  one of its cases needs position deletes carrying row data, which
  Vortex files do not persist.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
SparkRowLevelOperationsTestBase enumerates file formats, and both
concrete delete suites pass for Vortex: 44 cases in
TestCopyOnWriteDelete and 49 in TestMergeOnReadDelete.

testDeleteNonExistingRecords expects the snapshot operation to be a
delete only for ORC and Parquet, and a rewrite for everything else.
That split is really about whether the format reports column bounds:
with them the planner proves no file matches and commits a delete of
zero files, without them it falls back to rewriting. Vortex reports
bounds and so commits a delete, so add it alongside ORC and Parquet
and say what the branch is actually testing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Mirrors the v4.1 changes to TestIdentityPartitionData,
TestMetadataTablesWithPartitionEvolution, SparkRowLevelOperationsTestBase
and TestDelete. 109 Vortex cases, all passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
Mirrors the v4.1 and v4.0 changes to TestIdentityPartitionData,
TestMetadataTablesWithPartitionEvolution, SparkRowLevelOperationsTestBase
and TestDelete. 109 Vortex cases, all passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
A set predicate past the expansion limit was pushed down as the inclusive
range its values span. That range matches a superset of the rows the
predicate does, which is sound on its own but needed machinery to keep it
sound under negation: every approximate expression was tracked by identity
so that a NOT over one could be dropped, and AND/OR had to propagate the
marking. Large IN is now simply left to the engine, which retires the
tracking set and both propagation paths along with it. Vortex will grow
real support for these sets.

The test that covered this asserted the range still excluded rows outside
it, but built its 300 values from only two distinct literals, so the
literal set was of size two and never crossed the limit it meant to
exercise. It now uses distinct values and asserts what the contract
actually requires: the predicate is not pushed, and no matching row is
lost.

Flink has no Vortex support and its tests are not part of this work, so
the Flink CI workflow no longer runs automatically; it stays available
via workflow_dispatch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
@github-actions github-actions Bot added the INFRA label Sep 16, 2026
TestPositionDeltaWriters gained a Vortex case when the upstream writer
suites were enabled for Vortex, but its parameters() is protected static
and TestFlinkPositionDeltaWriters declares none of its own, so all three
Flink versions inherited that case. Vortex is not on the Flink test
runtime classpath and FlinkFileWriterFactory has no Vortex branch, so the
inherited case could only fail: the base declares four @testtemplate
methods, which is exactly the four failures Flink CI reported.

Override parameters() in the Flink subclass to list only the formats that
factory writes, matching TestFlinkFileWriterFactory,
TestFlinkPartitioningWriters and TestFlinkRollingFileWriters, which each
already do this for the same reason.

This also restores the Flink CI workflow triggers. Disabling the workflow
hid this failure rather than fixing it.

The other shared bases that mention Vortex are fine as they stand:
BaseFormatModelTests only adds the format when
org.apache.iceberg.vortex.VortexFormatModels resolves, which it does not
for Flink, and TestBaseTaskWriter, TestTaskEqualityDeltaWriter,
TestLocalScan and TestFileFormat have no Flink subclasses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
@github-actions github-actions Bot added FLINK and removed INFRA labels Sep 16, 2026
Iceberg treats a position delete file as covering a single data file when
its file_path lower and upper bounds are equal, and the engine only
rewrites deletes it can attribute to one data file: SparkBatchQueryScan
skips anything ContentFileUtil.isFileScoped rejects. Vortex reports string
bounds from its native statistics as a truncated 64 byte prefix range, so
for any data file path longer than that the two bounds differed and no
Vortex delete file was ever file scoped. Superseded position deletes were
therefore never removed, which TestMergeOnReadUpdate caught through
removedDeleteFiles coming back empty.

PositionDeleteVortexWriter now tracks file_path and pos itself, and the
appender prefers those exact bounds over the native ones. This is scoped
to the position delete path: data files keep using Vortex's statistics,
where a widened bound only costs pruning rather than correctness. The
writer copies the path string because callers reuse one PositionDelete
across rows.

Writer supplied metrics had no effect before this: VortexFileAppender
built metrics only from the native write summary, and the FieldMetrics
path through VortexMetrics.buildMetrics was reached from tests alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
A column written as int and later widened to long still reads from a file
that stores ints, but VortexArrowColumnVector mapped every IntVector to an
IntAccessor that implements only getInt. Spark's generated code asks the
vector for the projected type, so the first getLong on such a column threw
UnsupportedOperationException from the accessor base class. The row reader
already handled this through intsAsLongs and floatsAsDoubles; the
vectorized reader, which Vortex always takes, did not.

Use the projected Iceberg type the vector already carries to pick an
accessor that answers the wider getter: any integer vector narrower than
64 bits can serve getLong, and a float vector can serve getDouble. The
vector also has to report the promoted type, since that is what Spark
picks its getters from.

Decimal precision widening is the remaining legal promotion and is not
covered here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
VortexValueWriter.metrics() was never read. VortexFileAppender builds a
file's Metrics from Vortex's native write summary, and the FieldMetrics
path through VortexMetrics.buildMetrics was reached only from tests, so
GenericVortexWriter accumulated per-column counts and bounds on every
written row that nothing ever asked for. Drop the interface method, the
builder it fed, and the tracker machinery behind it.

Position deletes were the one real consumer, and they need one specific
thing rather than a general metrics channel: Iceberg reads a delete file
as covering a single data file when file_path's bounds are equal, and
Vortex reports string bounds as a truncated prefix range, so any real path
compares unequal. PositionDeleteVortexWriter now tracks the lowest and
highest path it writes and reports them through VortexExactBounds, which
the appender picks up by type. That replaces the flag threaded through
VortexFormatModel: only a writer that tracked something declares it. The
pos column needs no such handling, since Vortex reports numeric bounds
exactly.

Three of the tests covering the removed builder were really covering
MetricsConfig handling, which still applies to the summary path, so they
move onto the appender rather than being deleted; the truncate case now
also pins down that Iceberg truncation layers on top of Vortex's bounds.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LnkXKG8ZAzwEWeovL5LGrA
@robert3005
robert3005 merged commit b35ede8 into main Sep 16, 2026
37 of 59 checks passed
@robert3005
robert3005 deleted the claude/jolly-einstein-mesa2k branch September 16, 2026 13:24
robert3005 pushed a commit that referenced this pull request Sep 16, 2026
Two of the features the previous commit marked missing for Vortex are in
fact supported; the flags were wrong. Vortex 0.86.1, which #38 pulled in,
added VortexWriter.Builder.metadata()/putMetadata(), and #38 already uses
putMetadata to persist the Iceberg schema so readers can rebind columns
renamed since the file was written.

- FEATURE_EVOLUTION_BY_FIELD_ID: ungated, no code needed. It has worked
  since #38 -- TestVortexSchemaMetadata.testRenamedColumnsBindByFieldId
  already covers it. The flag was introduced by #36 and left stale when
  #38 closed the underlying gap.

- FEATURE_WRITER_METADATA: ungated. The capability was there and only the
  Iceberg plumbing was missing -- WriteBuilderWrapper.meta() was a no-op
  carrying the stale comment "Vortex files carry no user-defined key/value
  metadata". It now collects entries and passes them to putMetadata,
  rejecting the reserved iceberg.schema key. New public
  VortexFiles.metadata(InputFile) reads them back, so VortexFormat can
  implement metadataValue; VortexIO stays package-private.

FEATURE_WRITER_PROPERTIES stays gated: Vortex has no write properties --
the writer accepts none today, though some are planned -- so there is
nothing to set and nothing to observe in the written file. FEATURE_FIXED
and FEATURE_WRITER_OVERWRITE stay gated as before.

iceberg-data now takes iceberg-vortex as testImplementation rather than
testRuntimeOnly so VortexFormat can read file metadata directly. This is
not circular: data:compileTestJava needs vortex:jar, and
vortex:compileTestJava needs data:testJar. The Class.forName guard in
FileFormatTestSupport.all() still keeps Vortex out of the Flink modules,
which do not ship it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GxC1sZXqWV6UibNP9qaD9w
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants