Repository navigation
fix(table): validate deletion-vector partitions during scans and rewrites - #2600
iceberg-kitten wants to merge 16 commits into
Conversation
…on-match-20261115 Signed-off-by: Minh Vu <vuhoangminh97@gmail.com> # Conflicts: # table/partitioned_fanout_writer.go
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
laskoviymishka
left a comment
There was a problem hiding this comment.
Today a DV applies to whatever data file it names by path, so this check is overdue, and failing on a mismatch is what Java's DeleteFileIndex.validatePartitionMatch does too.
Before this merges I'd like the float to double gap in the equality-delete index closed. main has it as well, but after this PR DVs widen and equality deletes don't, so one partition matches on one path and silently misses on the other. I'd also like the integer normalization in comparablePartitionKey described and covered by a position-delete test. The rest are test questions and a nit.
| if _, hasDV := idx.dv[entry.DataFile().FilePath()]; hasDV && idx.specs != nil { | ||
| spec = idx.specs.PartitionSpecByID(int(entry.DataFile().SpecID())) | ||
| } | ||
| task, err := fileScanTaskForDataEntryWithSchema(entry, idx.pos, idx.dv, idx.eq, spec, idx.schema) |
There was a problem hiding this comment.
Could we add a changelog case with a promoted-float DV, plus one with a mismatched partition? The changelog test changes only pass a matching dataFilePartition(data) into the DV helpers, so nothing drives this lookup through widening or the new ErrInvalidMetadata path.
| switch value := value.(type) { | ||
| case []byte: | ||
| return binaryPartitionKey(value) | ||
| case int: |
There was a problem hiding this comment.
Is it intended that this reaches beyond DV matching? As I read it, comparablePartitionKey also backs partitionsMatch for path-scoped position deletes, so an int32 delete partition against an int64 data partition, which fails on main, now matches. Inspect-partitions rows split by int width merge too. Both look like fixes, so could we mention them in the description and add a Parquet position-delete test across int to long?
| ) bool { | ||
| dataFloat, dataIsFloat32 := dataValue.(float32) | ||
| dvFloat, dvIsFloat64 := dvValue.(float64) | ||
| if dataIsFloat32 && dvIsFloat64 && partitionFieldAllowsFloatWidening(partitionFieldID, spec, schema) { |
There was a problem hiding this comment.
Could we extend this widening to the equality-delete index in this PR? As I read it, after float to double promotion a single-field equality delete with a float64 partition still keys by width, so against an old float32 data manifest the lookup misses and the delete is silently not applied. Path-scoped position deletes fail with ErrInvalidMetadata instead.
main behaves the same, and "float widths stay distinct" pins it. With this change, though, DVs and equality deletes disagree on the same partition, so I'd like both fixed together and that test updated.
| // -1 <= dvSeq for any known DV sequence. | ||
| func matchDVToData(dataEntry iceberg.ManifestEntry, dvIndex map[string]iceberg.ManifestEntry) []iceberg.DataFile { | ||
| dvEntry, ok := dvIndex[dataEntry.DataFile().FilePath()] | ||
| func matchDVToData(dataEntry iceberg.ManifestEntry, dvIndex map[string]iceberg.ManifestEntry) ([]iceberg.DataFile, error) { |
There was a problem hiding this comment.
matchDVToData and fileScanTaskForDataEntry are now only called from dv_scan_planning_test.go, and both pass nil, nil, so a future caller would silently get no float widening. Could the tests call the ...WithSchema variants and we drop these two?
| task, err := fileScanTaskForDataEntry(entry, deletes.positional, deletes.dvs, deletes.equality) | ||
| var spec *iceberg.PartitionSpec | ||
| if _, hasDV := deletes.dvs[entry.DataFile().FilePath()]; hasDV { | ||
| spec, err = meta.GetSpecByID(int(entry.DataFile().SpecID())) |
There was a problem hiding this comment.
Could we cover this with a delete on a table that has a promoted-float DV? Scan planning is tested end to end in dv_partition_scan_test.go, but nothing runs rewriteFilesWithFilter with a widened or mismatched DV, so the GetSpecByID branch and the meta.CurrentSchema() choice here are untested.
…on-match-20261115 Signed-off-by: Minh Vu <vuhoangminh97@gmail.com> # Conflicts: # table/scanner_internal_test.go
…on-match-20261115
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
What
Why
updateSpecSchema,Builder.build,PartitionSpec.resultType).InternalReader).Implementation
main, which already handles promoted partitions in equality and Parquet position deletes and tests int-to-long position deletes.