Repository navigation
fix(table): merge v2 position deletes into first v3 DV - #2624
iceberg-kitten wants to merge 24 commits into
Conversation
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
badalprasadsingh
left a comment
There was a problem hiding this comment.
first pass @iceberg-kitten
| for dataFilePath, deleteFiles := range oldPositionDeletes.fileScoped { | ||
| if _, replaced := seen[dataFilePath]; !replaced { | ||
| continue | ||
| } | ||
| for _, deleteFile := range deleteFiles { | ||
| updater.removeDeleteFile(deleteFile) | ||
| } |
There was a problem hiding this comment.
removing these files makes the commit "non-replayable", so if another writer commits first -> this delete now fails with "ErrCommitFailed" instead of retrying
the delete docs (transaction_delete.go ;ine275, table.go line422) name only dv removals, could you update both?
| for dataFilePath, dvFile := range existingDVs { | ||
| if _, replaced := seen[dataFilePath]; replaced { | ||
| updater.removeDeletionVector(dvFile) | ||
| } | ||
| } |
There was a problem hiding this comment.
no test covers a v3 delete that finds no new rows in a file with a dv -> It relies on the seen check here and the "formatVersion < 3" guard at line1027. without either, the dv is dropped and deleted rows come back.
please add a test
| // Seed a file's existing deletes only after a new position is found for | ||
| // it. A filter can select files from their stats even when no live row | ||
| // matches; seeding those files eagerly would turn an empty delete into a | ||
| // DV rewrite. The scan does not apply existing DVs, so skip positions | ||
| // already present in the seed before deciding that a replacement is | ||
| // needed. | ||
| hasEntries := false | ||
| for filePath, bitmap := range args.existingDVs { | ||
| pCtx, ok := partitionContextByFilePath[filePath] | ||
| if !ok { | ||
| yield(nil, fmt.Errorf("unexpected missing partition context for existing deletion vector path %s", filePath)) | ||
|
|
||
| return | ||
| } | ||
| writer.Load(filePath, bitmap, pCtx.specID, pCtx.partitionData) | ||
| hasEntries = true | ||
| } | ||
| seededPaths := make(map[string]struct{}, len(args.existingDVs)) |
There was a problem hiding this comment.
this now seeds older pos-deletes too, and only once a new position is found, but the doc of "recordWritingArgs.existingDVs" still describes only dvs, and both it and" existingDVBitmaps" have dv-only names.
better: can you rename them (say "existingDeletes") and update that doc?
nssalian
left a comment
There was a problem hiding this comment.
Thanks @iceberg-kitten for the PR.
I reverted arrow_utils.go, transaction_delete.go and table.go to the merge base and re-ran the new tests. On the merge base the upgrade cases bring deleted rows back. For example, TestV3MergeOnReadUpgradePreservesPositionDeletes/single_file returns ids [1 3] instead of [3], and TestV3MergeOnReadUpgradeMergesMultiplePositionDeleteFiles returns [1 2 4 5] instead of [4 5]. All of them pass on this head.
The handling of old deletes matches Java's DV writer. It merges every earlier position for a data file into the new DV, but it only removes delete files that are scoped to that one data file, and shared position-delete files stay live (https://github.com/apache/iceberg/blob/a6bc1f95e3fe388419fd62186667a3bd08ccd74a/core/src/main/java/org/apache/iceberg/deletes/BaseDVFileWriter.java#L125-L131). Go's scan also skips position-delete files once a DV applies to a data file (scanner.go:1626). So the tests that check scan results are, in effect, checking that the DV carries the old positions.
Worth fixing the CI.
| func TestV3MergeOnReadMixedExistingDVsKeepsUnchangedFile(t *testing.T) { | ||
| ctx := t.Context() | ||
| tbl := newPartitionedMergeOnReadTestTable(t) | ||
| arrowSchema := arrow.NewSchema([]arrow.Field{ |
There was a problem hiding this comment.
This test builds its six rows by a bit by hand: an Arrow schema for id and category, then array.TableFromJSON, then tbl.Append. TestV3MergeOnReadUpgradePreservesPartitionedPositionDeletes (line 712) repeats the same ~12 lines, and so does the existing TestV3MergeOnReadDoubleDeletePartitionedMergesDeletionVectors (line 232). The appendRows(t, tbl, json) helper in cow_rewrite_deletes_test.go:408 gets the Arrow schema from the table and does the append, so each block can be a single tbl = appendRows(t, tbl, ...) call.
| With `write.delete.mode=merge-on-read`, format v2 writes position-delete | ||
| files and format v3 writes deletion vectors (DVs). After a table is upgraded | ||
| from v2 to v3, its older position-delete files remain valid. The **first** | ||
| DV written for a data file includes all positions previously deleted from |
There was a problem hiding this comment.
nit: "includes all positions previously deleted from that file" could be read as including rows removed by equality deletes. Only position deletes are folded into the DV, and equality deletes keep applying on their own. The release note's wording, "applicable legacy position-delete files", is more precise and could be used here too.
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
badalprasadsingh
left a comment
There was a problem hiding this comment.
approving @iceberg-kitten :)
|
I re-ran my local tests at 6655a71. Two cases beyond the upgrade ones already covered here: if a peer commit lands while a delete that removes a legacy position-delete file is staged, the commit fails with I also applied #2639 (the v2 position-delete spec fix for #2606) on top. There are no conflicts and |
…os-deletes-2604 Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
What
Carry older position deletes into the first DV for a data file.
Why
Iceberg readers may skip position-delete files when a DV applies. If the older positions are not copied, rows can reappear.
Implementation
Merge old deletes per data file. Write a DV only when a new position is found. Keep a shared delete file live while another data file still needs it. Cover multiple old delete files and historical partition specs.
Fixes #2604.