Skip to content

Native Iceberg writes: assert manifest metrics parity with iceberg-java for every data type #7639

Open

Description

@Visorgood

What is the problem the feature request solves?

Part of #5644. One of the conditions for enabling native Iceberg writes by default is manifest metrics parity per data type on the pinned Iceberg versions (1.5.2 is dropped, per the discussion in #5644). Readers such as Spark and Trino prune data files using these metrics, so a wrong bound or count from the native writer can make a reader skip a file that has matching rows.

CometIcebergWriteActionSuite already compares native and iceberg-java metrics through manifestMetricsParity, but only for some types and metrics:

  • bounds for date, timestamp, timestamp_ntz, decimal(9,2), decimal(38,10), float and binary, and value counts for boolean and bigint;
  • string bound truncation, metrics mode none, and sorted-column bounds under counts;
  • NaN counts for float and double, in separate tests.

Not covered yet: null counts across types; bounds for int, bigint, double, boolean and short strings; INT64-backed decimals (precision 10 to 18); all-null columns; and leaf columns inside structs, lists and maps.

Describe the potential solution

Parity tests per group of types, built on manifestMetricsParity. Each one writes the same rows, including nulls and NaN where they apply, through the native and the iceberg-java writer, and compares per column value_count, null_value_count, nan_value_count, lower_bound and upper_bound from readable_metrics.

  • Types: boolean, int, bigint, float, double, decimals backed by INT32, INT64 and fixed-length binary, date, timestamp, timestamp_ntz, string, binary, and leaf columns inside struct, list and map where iceberg-java records metrics for them.
  • column_sizes is left out, because the two writers lay out files differently.
  • Types the native writer doesn't accept yet (for example UUID, feat: support native Iceberg UUID writes #7198) are out of scope.

The tests go in Comet's own suite, so they run under each Spark profile's pinned Iceberg version: 1.8.1 (Spark 3.5), 1.10.0 (Spark 4.0) and 1.11.0 (Spark 4.1). Each mismatch they find gets its own issue, since a fix may be on the native side, and the test for that type references it until it's fixed.

Additional context

Coverage summary is from CometIcebergWriteActionSuite on main at 983173e.

Willingness to contribute

I can contribute a fix for this bug independently

Activity

  1. Visorgood commented on Nov 16, 2026

    @Visorgood
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions