Repository navigation
fix(table): write position deletes with the data file's partition spec - #2639
Conversation
badalprasadsingh
left a comment
There was a problem hiding this comment.
first pass @AynSiddiqui
also, lets add a release note under unreleased section, also mention the repair
|
|
||
| outputCh := make(chan iceberg.DataFile, 10) | ||
| writer, err := factory.getOrCreateRollingDataWriter(s.ctx, "", nil, outputCh) | ||
| writer, err := factory.getOrCreateRollingDataWriter(s.ctx, "", nil, outputCh, factory.currentSpec) |
There was a problem hiding this comment.
this test still calls "factory.writers.Load("")" (check lin628), but the map keys are now "rollingWriterKey", so "ok" is always false. the check passes even if the stream stops removing the writer
better: please use the writer's key, as your new test does:
_, ok := factory.writers.Load(writer.key)| }{ | ||
| {name: "old-spec"}, | ||
| {name: "mixed-specs", mixed: true}, | ||
| {name: "evolve-to-unpartitioned", mixed: true, unpartitioned: true}, | ||
| } { |
| currentSpec := latestMetadata.PartitionSpec() | ||
| useUnpartitionedWriter := currentSpec.IsUnpartitioned() | ||
| for _, partition := range partitionContextByFilePath { | ||
| if int(partition.specID) != currentSpec.ID() { | ||
| useUnpartitionedWriter = false | ||
|
|
||
| break | ||
| } | ||
| } |
There was a problem hiding this comment.
nit: a short comment on why would help here: the unpartitioned writer stamps the current spec on every file, so it is safe only when every target-file was written under that spec
without it, someone may simplify this back to "IsUnpartitioned()"
|
d6790c5 to
f812f76
Compare
Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>
…r-removal check Use the writer's key when asserting a failed rolling writer is removed, add a truncate[2] -> truncate[4] case where both specs share a partition path, explain the unpartitioned-writer guard, and add a release note. Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>
f812f76 to
03783cd
Compare
|
I pushed twice. The earlier push rebased onto main to clear a conflict in |
badalprasadsingh
left a comment
There was a problem hiding this comment.
approving @AynSiddiqui :)
laskoviymishka
left a comment
There was a problem hiding this comment.
Thanks @badalprasadsingh for review and @AynSiddiqui for contribution.
LGTM
Rationale for this change
After partition spec evolution,
Deleteon a table with data files from an older spec wrote position delete files using the table's current spec. The delete file's spec ID and partition did not match the data file it refers to, so readers could not apply it (and with an unpartitioned current spec, the unpartitioned path was taken regardless of the data file's spec).Fixes #2606
What changes are included in this PR?
positionDeleteRecordsToDataFilesonly uses the unpartitioned writer when the current spec is unpartitioned and every targeted data file is on the current spec; otherwise it uses the partitioned fanout writer.openFileWriter,newRollingDataWriterandgetOrCreateRollingDataWritertake the spec as a parameter instead of always usingcurrentSpec. The data-file write paths passfactory.currentSpec, so their behaviour is the same (hence the many one-line caller and test updates); the position-delete fanout writer passes the data file's spec.writePositionDeletesForFilesfor the v3 path; this PR changes the shared rolling-writer code and the v2 position-delete path, and I'll rebase once fix(table): merge v2 position deletes into first v3 DV #2624 lands.Are these changes tested?
Run locally on main
3d1d676:golangci-lint run(2.15.0; the Makefile pins 2.14.0 for install): 0 issuesgo test -count=1 ./...: passgo test -race -run 'SpecEvolution|PosDelete|PositionDelete' ./table/: passTestPositionDeleteAfterSpecEvolution(old-spec, mixed-specs, evolve-to-unpartitioned): fails on3d1d676without the change in all 3 cases and passes with it. It checks that each delete file's spec ID and partition equal its data file's.TestWriterKeysIncludeSpecAndAreRemoved.Not run: the integration suites (Spark/REST/catalog) and any interop check against Java Iceberg reading these delete files.
Are there any user-facing changes?
Bug fix: position delete files written after spec evolution now carry the correct spec ID and partition. No API changes (the changed functions are unexported).