Skip to content

fix(compaction): match float partitions across a float to double promotion - #2653

Open
nssalian wants to merge 1 commit into
apache:mainfrom
nssalian:compaction-eq-delete-float-promotion
Open

nssalian wants to merge 1 commit into
apache:mainfrom
nssalian:compaction-eq-delete-float-promotion

Conversation

@nssalian

Copy link
Copy Markdown
Contributor

What

Widen float32 partition values to float64 when building the spec-aware key that equality-delete cleanup uses to look up surviving data files.

Why

After an identity partition source is promoted from float to double, data files written before the promotion decode the partition value as float32 and equality deletes written after it carry float64. The survivor key printed 1000=1.1 while the delete key printed 1000=1.100000023841858, so DecideDeadEqualityDeletesWithSpecs found no surviving file and classified a delete that still applies as dead. Compaction then drops that delete and the rows it covered come back: the new TestCompactionKeepsEqualityDeleteAfterFloatPartitionSourcePromotion compacts two unrelated files and reads ids [1 2 3 4] on main instead of [2 3 4].

The opposite case was wrong too: a float32 1.1 data file and a double 1.1 delete printed the same key and matched, although they are different partitions after the promotion. That case only kept a dead delete around.

This is the same root cause as #2605 and #2610 (fixed for scan delete matching in #2620) and the partitions table fix in #2637, in a code path neither of them touches.

Implementation

  • Widening is limited to the unexported spec-aware key in SurvivorSurvey. PartMinSeq and appendPartitionTuple keep their fmt-compatible output, so the exported survey layout and its keys do not change.
  • int to long needed nothing: both widths already format the same.

Tests

  • TestCompactionKeepsEqualityDeleteAfterFloatPartitionSourcePromotion (v2 and v3): end to end through a rewrite. Fails on main.
  • Two promotion cases in TestDecideDeadEqualityDeletesWithSpecs_Predicate: a survivor written before the promotion keeps the delete alive, and a different double value no longer matches. Both fail on main.
  • commitPartitionedEqualityDeletes in partition_promotion_deletes_test.go now holds the equality-delete commit both promotion tests use.
  • A never-promoted float32 case, which fails if only the survivor side were widened, and float32(1.1) in TestAppendPartitionTupleMatchesFmt, which fails if the exported keys were widened.

@nssalian
nssalian requested a review from zeroshade as a code owner November 19, 2026 18:48
@nssalian

Copy link
Copy Markdown
Contributor Author

CC: @laskoviymishka

@laskoviymishka
laskoviymishka self-requested a review November 19, 2026 18:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant