Skip to content

fix(table): write position deletes with the data file's partition spec - #2639

Merged
laskoviymishka merged 2 commits into
apache:mainfrom
AynSiddiqui:fix/2606-pos-delete-spec
Nov 19, 2026
Merged

laskoviymishka merged 2 commits into
apache:mainfrom
AynSiddiqui:fix/2606-pos-delete-spec

Conversation

@AynSiddiqui

Copy link
Copy Markdown
Contributor

Rationale for this change

After partition spec evolution, Delete on a table with data files from an older spec wrote position delete files using the table's current spec. The delete file's spec ID and partition did not match the data file it refers to, so readers could not apply it (and with an unpartitioned current spec, the unpartitioned path was taken regardless of the data file's spec).

Fixes #2606

What changes are included in this PR?

  • positionDeleteRecordsToDataFiles only uses the unpartitioned writer when the current spec is unpartitioned and every targeted data file is on the current spec; otherwise it uses the partitioned fanout writer.
  • The position delete fanout writer passes each data file's own spec to the rolling writer.
  • openFileWriter, newRollingDataWriter and getOrCreateRollingDataWriter take the spec as a parameter instead of always using currentSpec. The data-file write paths pass factory.currentSpec, so their behaviour is the same (hence the many one-line caller and test updates); the position-delete fanout writer passes the data file's spec.
  • Rolling writers are keyed by (spec ID, partition path) so two specs with the same path don't share a writer, and removal from the map uses that key (it previously used the path only).
  • The v3 deletion vector path is not changed. fix(table): merge v2 position deletes into first v3 DV #2624 also changes writePositionDeletesForFiles for the v3 path; this PR changes the shared rolling-writer code and the v2 position-delete path, and I'll rebase once fix(table): merge v2 position deletes into first v3 DV #2624 lands.

Are these changes tested?

Run locally on main 3d1d676:

  • golangci-lint run (2.15.0; the Makefile pins 2.14.0 for install): 0 issues
  • go test -count=1 ./...: pass
  • go test -race -run 'SpecEvolution|PosDelete|PositionDelete' ./table/: pass
  • New TestPositionDeleteAfterSpecEvolution (old-spec, mixed-specs, evolve-to-unpartitioned): fails on 3d1d676 without the change in all 3 cases and passes with it. It checks that each delete file's spec ID and partition equal its data file's.
  • New TestWriterKeysIncludeSpecAndAreRemoved.

Not run: the integration suites (Spark/REST/catalog) and any interop check against Java Iceberg reading these delete files.

Are there any user-facing changes?

Bug fix: position delete files written after spec evolution now carry the correct spec ID and partition. No API changes (the changed functions are unexported).

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first pass @AynSiddiqui

also, lets add a release note under unreleased section, also mention the repair


outputCh := make(chan iceberg.DataFile, 10)
writer, err := factory.getOrCreateRollingDataWriter(s.ctx, "", nil, outputCh)
writer, err := factory.getOrCreateRollingDataWriter(s.ctx, "", nil, outputCh, factory.currentSpec)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this test still calls "factory.writers.Load("")" (check lin628), but the map keys are now "rollingWriterKey", so "ok" is always false. the check passes even if the stream stops removing the writer

better: please use the writer's key, as your new test does:

	_, ok := factory.writers.Load(writer.key)

Comment on lines +37 to +41
}{
{name: "old-spec"},
{name: "mixed-specs", mixed: true},
{name: "evolve-to-unpartitioned", mixed: true, unpartitioned: true},
} {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in every case the two specs write different partition paths, so only the unit test covers the spec id in the writer key

better: add a case where they share one, eg: "truncate2" replaced by "truncate4", both named c, with cat = "ab" rows before and after

Comment thread table/arrow_utils.go
Comment on lines +2447 to +2455
currentSpec := latestMetadata.PartitionSpec()
useUnpartitionedWriter := currentSpec.IsUnpartitioned()
for _, partition := range partitionContextByFilePath {
if int(partition.specID) != currentSpec.ID() {
useUnpartitionedWriter = false

break
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: a short comment on why would help here: the unpartitioned writer stamps the current spec on every file, so it is safe only when every target-file was written under that spec

without it, someone may simplify this back to "IsUnpartitioned()"

@AynSiddiqui

Copy link
Copy Markdown
Contributor Author
  1. The cache assertion now loads writer.key. With removal disabled, it fails.
  2. Added truncate[2](cat) -> truncate[4](cat) with cat="ab" rows on both specs. It checks each delete file against its data file, and it is the only case that fails when the spec ID is dropped from the writer key.
  3. Added the comment on the unpartitioned-writer guard.
  4. Added a note at the end of "Unreleased compatibility notes" in website/src/releases.md. On tables with old delete files, scans and re-running Delete can fail with the spec-mismatch error, and some deletes are silently missed instead; re-running those applies them. For the mismatch error, I checked that removing the offending delete files with Transaction.NewRewrite, then re-running the deletes, keeps valid deletes. I did not test Java Iceberg.

@AynSiddiqui
AynSiddiqui force-pushed the fix/2606-pos-delete-spec branch from d6790c5 to f812f76 Compare November 19, 2026 08:01
Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>
…r-removal check

Use the writer's key when asserting a failed rolling writer is removed, add a truncate[2] -> truncate[4] case where both specs share a partition path, explain the unpartitioned-writer guard, and add a release note.

Signed-off-by: Ayaan <ayaansiddiqui68@gmail.com>
@AynSiddiqui
AynSiddiqui force-pushed the fix/2606-pos-delete-spec branch from f812f76 to 03783cd Compare November 19, 2026 08:31
@AynSiddiqui

Copy link
Copy Markdown
Contributor Author

I pushed twice. The earlier push rebased onto main to clear a conflict in website/src/releases.md, where an upstream bullet (#2608) landed after my base. This latest force-push adds the Signed-off-by lines CONTRIBUTING.md asks for. The content is unchanged from d6790c5 apart from the release-note context and the sign-offs. The first commit has the same patch you reviewed (0e7667b), now with a sign-off.

@badalprasadsingh badalprasadsingh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving @AynSiddiqui :)

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @badalprasadsingh for review and @AynSiddiqui for contribution.

LGTM

@laskoviymishka
laskoviymishka merged commit 502180b into apache:main Nov 19, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Transaction: v2 merge-on-read Delete after a partition spec change writes the delete file under the new spec, breaking every scan

3 participants