You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
FanoutWriter and ClusteredWriter merge -0.0 and 0.0 identity partitions of float and double columns #3325
main. Verified at bb1e4a4 (2026-09-21); the code referenced below is unchanged on main today.
Describe the bug
FanoutWriter and ClusteredWriter put rows whose float/double identity partition value is -0.0 and rows whose value is 0.0 into the same data file, under whichever value arrived first. iceberg-java writes two partitions, because it compares partition keys with Float.compare / Double.compare, which keep the two apart.
ClusteredWriter keeps current_partition: Option<Struct> and closed_partitions: HashSet<Struct>
PrimitiveLiteral derives PartialEq, Eq and Hash over OrderedFloat, and OrderedFloat treats -0.0 and 0.0 as equal with the same hash. So the second value finds the first value's writer.
The data file is then recorded with one partition value while holding rows for both, so a scan that prunes on the other value drops rows. Seen through Spark on a table (id INT, d DOUBLE) partitioned by d, after writing (1, -0.0) and (2, 0.0) in one task:
use std::collections::HashMap;use iceberg::spec::{Literal,Struct};let neg = Struct::from_iter([Some(Literal::double(-0.0))]);let pos = Struct::from_iter([Some(Literal::double(0.0))]);// Passes: the two partition values are equal.assert_eq!(neg, pos);// Passes: looking up 0.0 returns the writer opened for -0.0, as in `FanoutWriter`.letmut partition_writers:HashMap<Struct,&str> = HashMap::new();
partition_writers.insert(neg,"writer for -0.0");assert_eq!(partition_writers.get(&pos),Some(&"writer for -0.0"));
The same holds for Literal::float.
Expected behavior
-0.0 and 0.0 are distinct partition values, as in iceberg-java, so the writers produce two data files with two partition values.
Possible directions: key the writers on a bit-exact form of the partition struct, or make equality and hashing of float literals bit-exact (one canonical NaN, signed zeros distinct), which is the Java Float.equals / Double.equals behavior that same_value in #3077 implements.
Willingness to contribute
None
Additional context
Found through apache/datafusion-comet#6138. Comet's native Iceberg writer uses these writers, and is adding a fall-back for float/double identity partitions until this is resolved.
Activity
fix: resolve partition field types by source id and address review
fix: fall back for Iceberg writes with float or double identity parti…
fix(spec): treat -0.0 and 0.0 as distinct float literals (apache#3327)