Skip to content

fix(table): validate deletion-vector partitions during scans and rewrites - #2600

Open
iceberg-kitten wants to merge 16 commits into
apache:mainfrom
iceberg-kitten:fix/dv-scan-partition-match-20261115
Open

iceberg-kitten wants to merge 16 commits into
apache:mainfrom
iceberg-kitten:fix/dv-scan-partition-match-20261115

Conversation

@iceberg-kitten

@iceberg-kitten iceberg-kitten commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Validate deletion-vector partition specs and values in scan planning, incremental changelog planning, and copy-on-write deletes.
  • Match historical partition values across schema promotion, manifest rewrites, and source-column removal.

Why

Implementation

  • Widen float values exactly in either direction for retained identity fields, including when the source column was dropped.
  • Normalize integer widths when validating a deletion vector's partition. The branch incorporates fix(table): compare promoted partition values after widening #2620 from main, which already handles promoted partitions in equality and Parquet position deletes and tests int-to-long position deletes.
  • Keep field checks, exact equality, NaN normalization, and signed-zero behavior.
  • Include both partition tuples in mismatch errors.
  • Exercise matching and mismatched promoted deletion vectors in changelog planning and copy-on-write deletes, including a rewrite that preserves rows already deleted by the vector.

iceberg-kitten and others added 6 commits November 17, 2026 12:53
…on-match-20261115

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>

# Conflicts:
#	table/partitioned_fanout_writer.go
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Today a DV applies to whatever data file it names by path, so this check is overdue, and failing on a mismatch is what Java's DeleteFileIndex.validatePartitionMatch does too.

Before this merges I'd like the float to double gap in the equality-delete index closed. main has it as well, but after this PR DVs widen and equality deletes don't, so one partition matches on one path and silently misses on the other. I'd also like the integer normalization in comparablePartitionKey described and covered by a position-delete test. The rest are test questions and a nit.

Comment thread table/incremental_changelog_deletes.go Outdated
if _, hasDV := idx.dv[entry.DataFile().FilePath()]; hasDV && idx.specs != nil {
spec = idx.specs.PartitionSpecByID(int(entry.DataFile().SpecID()))
}
task, err := fileScanTaskForDataEntryWithSchema(entry, idx.pos, idx.dv, idx.eq, spec, idx.schema)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a changelog case with a promoted-float DV, plus one with a mismatched partition? The changelog test changes only pass a matching dataFilePartition(data) into the DV helpers, so nothing drives this lookup through widening or the new ErrInvalidMetadata path.

Comment thread table/partitioned_fanout_writer.go Outdated
switch value := value.(type) {
case []byte:
return binaryPartitionKey(value)
case int:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intended that this reaches beyond DV matching? As I read it, comparablePartitionKey also backs partitionsMatch for path-scoped position deletes, so an int32 delete partition against an int64 data partition, which fails on main, now matches. Inspect-partitions rows split by int width merge too. Both look like fixes, so could we mention them in the description and add a Parquet position-delete test across int to long?

Comment thread table/scanner.go
) bool {
dataFloat, dataIsFloat32 := dataValue.(float32)
dvFloat, dvIsFloat64 := dvValue.(float64)
if dataIsFloat32 && dvIsFloat64 && partitionFieldAllowsFloatWidening(partitionFieldID, spec, schema) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we extend this widening to the equality-delete index in this PR? As I read it, after float to double promotion a single-field equality delete with a float64 partition still keys by width, so against an old float32 data manifest the lookup misses and the delete is silently not applied. Path-scoped position deletes fail with ErrInvalidMetadata instead.

main behaves the same, and "float widths stay distinct" pins it. With this change, though, DVs and equality deletes disagree on the same partition, so I'd like both fixed together and that test updated.

Comment thread table/scanner.go Outdated
// -1 <= dvSeq for any known DV sequence.
func matchDVToData(dataEntry iceberg.ManifestEntry, dvIndex map[string]iceberg.ManifestEntry) []iceberg.DataFile {
dvEntry, ok := dvIndex[dataEntry.DataFile().FilePath()]
func matchDVToData(dataEntry iceberg.ManifestEntry, dvIndex map[string]iceberg.ManifestEntry) ([]iceberg.DataFile, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

matchDVToData and fileScanTaskForDataEntry are now only called from dv_scan_planning_test.go, and both pass nil, nil, so a future caller would silently get no float widening. Could the tests call the ...WithSchema variants and we drop these two?

task, err := fileScanTaskForDataEntry(entry, deletes.positional, deletes.dvs, deletes.equality)
var spec *iceberg.PartitionSpec
if _, hasDV := deletes.dvs[entry.DataFile().FilePath()]; hasDV {
spec, err = meta.GetSpecByID(int(entry.DataFile().SpecID()))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we cover this with a delete on a table that has a promoted-float DV? Scan planning is tested end to end in dv_partition_scan_test.go, but nothing runs rewriteFilesWithFilter with a widened or mismatched DV, so the GetSpecByID branch and the meta.CurrentSchema() choice here are untested.

Minh Vu added 3 commits November 18, 2026 19:42
…on-match-20261115

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>

# Conflicts:
#	table/scanner_internal_test.go
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
@iceberg-kitten iceberg-kitten changed the title fix(table): validate DV partitions during scan planning fix(table): validate deletion-vector partitions during scans and rewrites Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants