You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Iceberg writes: assert manifest metrics parity with iceberg-java for every data type #7639
Part of #5644. One of the conditions for enabling native Iceberg writes by default is manifest metrics parity per data type on the pinned Iceberg versions (1.5.2 is dropped, per the discussion in #5644). Readers such as Spark and Trino prune data files using these metrics, so a wrong bound or count from the native writer can make a reader skip a file that has matching rows.
CometIcebergWriteActionSuite already compares native and iceberg-java metrics through manifestMetricsParity, but only for some types and metrics:
bounds for date, timestamp, timestamp_ntz, decimal(9,2), decimal(38,10), float and binary, and value counts for boolean and bigint;
string bound truncation, metrics mode none, and sorted-column bounds under counts;
NaN counts for float and double, in separate tests.
Not covered yet: null counts across types; bounds for int, bigint, double, boolean and short strings; INT64-backed decimals (precision 10 to 18); all-null columns; and leaf columns inside structs, lists and maps.
Describe the potential solution
Parity tests per group of types, built on manifestMetricsParity. Each one writes the same rows, including nulls and NaN where they apply, through the native and the iceberg-java writer, and compares per column value_count, null_value_count, nan_value_count, lower_bound and upper_bound from readable_metrics.
Types: boolean, int, bigint, float, double, decimals backed by INT32, INT64 and fixed-length binary, date, timestamp, timestamp_ntz, string, binary, and leaf columns inside struct, list and map where iceberg-java records metrics for them.
column_sizes is left out, because the two writers lay out files differently.
The tests go in Comet's own suite, so they run under each Spark profile's pinned Iceberg version: 1.8.1 (Spark 3.5), 1.10.0 (Spark 4.0) and 1.11.0 (Spark 4.1). Each mismatch they find gets its own issue, since a fix may be on the native side, and the test for that type references it until it's fixed.
Additional context
Coverage summary is from CometIcebergWriteActionSuite on main at 983173e.
Activity
Visorgood commentedon Nov 16, 2026
take