Skip to content

fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata - #5

Closed
c-thiel wants to merge 24 commits into
ct/fork-upstream-audit-fixesfrom
ct/fork-reader-fixes
Closed

c-thiel wants to merge 24 commits into
ct/fork-upstream-audit-fixesfrom
ct/fork-reader-fixes

Conversation

@c-thiel

@c-thiel c-thiel commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Fixes for the reader, the manifest reader and v1 table metadata. Most of them stop a read that returned wrong values without an error. A rewrite of such a file wrote the wrong values into the new file. This PR is stacked on #4. After that PR merges, this PR is retargeted to lakekeeper-compaction-base. Review commit by commit.

Fixes for Parquet timestamps read through the INT96 schema hint (INT96 files in a timestamptz column, and INT64 nanosecond files in a timestamp column) follow in #7, stacked on this PR.

Commits

  • Cherry-pick of fix(arrow): project date columns promoted to timestamp apache/iceberg-rust#3485 (open), author ump45nose: project date columns promoted to timestamp. Format v3 allows promoting date to timestamp and timestamp_ns. The reader had no rule for this promotion. When another column was read with it, the column was left out and read as null. The column is now read and cast. Issue: Date-to-timestamp promoted column reads as NULL when another column is projected apache/iceberg-rust#3404.
  • Fail the read of a column whose file type cannot be promoted. When a data file stored a column in a type that cannot be promoted to the table type, for example date read as string or long read as int, the reader filled the column with null or its initial-default, without an error. The read now fails and names the column, the file type and the table type. This also covers a type change that a catalog accepted on a format v2 table. No upstream issue or PR.
  • Fail the read when a promoted value is out of range. The reader cast promoted columns with arrow's default options, which turn a value that does not fit the target type into null. A date after 2262-04-11 promoted to timestamp_ns read as null. The spec requires such a read to fail, and it now does, naming both types. Date-to-timestamp promoted column reads as NULL when another column is projected apache/iceberg-rust#3404 names this case; there is no upstream PR.
  • Probe an equality key column that the file does not store as its initial-default. An equality delete keyed on a column that was added after the data file was written probed that column as null. A delete of the column's initial-default value removed no rows, although the reader returns that value for every row of the file. The probe now uses the identity partition value, then the initial-default of a top-level field, then null, as projection and Java's reader do. This is case B of Equality deletes ignore spec projection rules for dropped and newly added delete columns apache/iceberg-rust#3369 and affects only v3 tables. Related: perf(deletes): apply equality deletes via a hashed key set inside the parquet RowFilter apache/iceberg-rust#2961 (the hash-set probe, open).
  • Cherry-pick of fix(arrow): evaluate filters on columns missing from a data file against their identity partition value or initial-default apache/iceberg-rust#3302 (open), author Matt Butrovich: evaluate filters on columns missing from a data file against their identity partition value or initial-default. The row filter read a column that the file does not store as null, while projection fills it with the identity partition value or the initial-default. A filter such as p = 5 on a file added from a Hive table without the column p dropped every row. Each filter leaf on such a column is now decided against that value before the row filter and every pruning step. Issue: Rows are dropped when a filter references a column that is missing from the data file but has an identity partition value or initial-default apache/iceberg-rust#3298. A conflict in the test module of row_filter.rs was resolved by keeping both sides, and two type paths were shortened for a lint. The commit comes after the probe fix above, because one of its tests needs that fix.
  • Test filters on an identity partition column that the file does not store. A file that does not store its identity partition column keeps exactly the rows that each filter selects when the column is read as its partition value: =, IN, >=, IS NOT NULL, NOT, AND and OR with a stored column, and < and <= on both sides of the value, with row selection off and on. This is a separate commit so that the fix(arrow): evaluate filters on columns missing from a data file against their identity partition value or initial-default apache/iceberg-rust#3302 pick can be replaced when upstream merges it.
  • Match no row with <, <=, NOT IN and NOT STARTS WITH on a column that the file does not store. Without an identity partition value or initial-default, such a column reads as null. The row filter still kept every row for these four operators, so WHERE c < 5 returned every row written before c was added. They now keep no row, like >, >=, = and != already did, and like the same filters on a file that stores the column as all null. This matches SQL, where a comparison with null is never true. Java's generic row Evaluator sorts null first and keeps such rows for <, <=, != and NOT IN. Issue: Row filter keeps every row for < and <= on a null column that is missing from the data file apache/iceberg-rust#3355 (for < and <=); no upstream PR.
  • Adapted port of fix(scan): derive manifest schema/spec from table metadata (cf. iceberg-java specsById) apache/iceberg-rust#2683 (open), author Raghvendra Singh: take the manifest's schema and partition spec from the table metadata. The manifest reader parsed the schema from the manifest's own Avro metadata and failed when it was not a valid Iceberg schema. DuckDB 1.4 writes the Avro schema of the manifest entry there, so the fork could not read or compact tables that DuckDB 1.4 wrote. Java, PyIceberg and Spark read them. The reader now looks up the schema and spec in the table metadata by the IDs the manifest records, as Java's ManifestReader does with specsById. Manifests without recorded IDs, and reads without table metadata, still use the manifest's own keys. Issue: Manifest reader hard-depends on the manifest schema/partition-spec keys; should derive from table metadata (cf. iceberg-java specsById) apache/iceberg-rust#2682. The PR does not apply to the fork's ManifestReader, so it was adapted: the table metadata is passed with ManifestReader::with_table_metadata, which Table::manifest_reader and the scan's manifest cache use. So scans and every transaction that reads manifests through the table (append's duplicate check, rewrite, row delta, overwrite, rewrite manifests) resolve the schema the same way. The PR's load_manifest_with is left out, and ManifestFile::load_manifest still reads only the manifest (see the next commit for its rustdoc). One scan test fixture wrote a manifest that recorded schema ID 0 with a different schema; it now registers the schema it writes, as a real table does.
  • Resolve manifest schemas from the table metadata when purging a table. drop_table_data read manifests with the manifest's own header, so purging a DuckDB 1.4 table failed partway through. It now uses Table::manifest_reader. The load_manifest rustdoc now says that it reads the header and points to Table::manifest_reader.
  • Keep refs in format-version 1 table metadata. Reading v1 metadata dropped every branch and tag except a main built from current-snapshot-id, and writing v1 metadata left refs out. A catalog that builds its loadTable answer with this crate hid the tags and branches it stores. A client that expires snapshots on that answer removed the snapshots that only those refs kept. The spec allows refs in v1, and Java reads and writes it for every format version. v1 metadata now reads and writes refs as v2 and v3 do. Without refs, main still comes from current-snapshot-id. The test fixture is v1 metadata that Spark 3.5.6 with Iceberg 1.9.2 wrote after creating a tag and a branch. No upstream issue or PR.

Compatibility

  • Reads that returned null or a default for a column whose type cannot be promoted, or for a promoted value out of range, now fail with an error. A rewrite of such a file fails as well.
  • Filters on columns that a file does not store return different rows: the identity partition value or initial-default decides, and without a value <, <=, NOT IN and NOT STARTS WITH match no row.
  • Equality deletes on a column with an initial-default now remove rows of files written before the column was added (v3 only).
  • Manifests that record a schema ID and spec ID are read with the table's schema and spec when the table metadata is known.
  • v1 table metadata now always has refs when written. Older readers of this crate ignore the key.
  • New public method ManifestMetadata::parse_with. No public item is removed or changed.

Tests

Each commit builds and passes cargo test -p iceberg --all-features --lib. Each of our fixes has tests that fail without it. New tests:

c-thiel and others added 24 commits October 26, 2026 08:51
…not allow

Java writes a partition field whose name is not a valid Avro name under a sanitized name (`event.ts_day` becomes `event_x2Ets_day`, `my-col` becomes `my_x2Dcol`, `1st` becomes `_1st`), with the original name in `iceberg-field-name` and the partition field ID in `field-id`. Spark names the field of `days(event.ts)` `event.ts_day` by default. The manifest reader matched partition fields by name only, so it read every partition value of such a manifest as null, without an error. A rewrite then wrote output files and the surviving entries of the rewritten manifests with null partitions, which other engines prune away for partition filters. This is upstream apache#2536; its fix PR apache#2540 was closed as stale and no other PR is open.

The reader now matches each partition field to the writer's Avro field by field ID, then by `iceberg-field-name`, then by name, as Java projects by field ID. It gives the fields of the partition type the writer's names before it decodes the entries, so the existing match by name finds every value.

`apache-avro` rejects a header whose field names are not `[A-Za-z_][A-Za-z0-9_]*`. Java writes names with non-ASCII letters (`größe`) as they are, and iceberg-rust wrote every partition field name as it is, so the fork could not read those manifests at all, including the ones it wrote itself after one commit. On that error the reader now renames such fields in the header, keeping the original name in `iceberg-field-name`, and parses again. Field names do not change how a record is encoded. This joins the existing header fallback for repeated named type definitions; a header with both problems gets both rewrites.

Review: the field-ID-first match in `writer_partition_type` (spec/manifest/mod.rs), and the retry loop in `rewrite_rejected_header`, which replaces the earlier single fallback and keeps its warning text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fork wrote partition field names into the manifest's Avro schema as they are, for example `event.ts_day` for `days(event.ts)` or `my-col`. Such a manifest is not valid Avro: `apache-avro` refuses to parse its header, so strict third-party readers fail on it, and so did the fork itself.

Field names in Avro schemas built from Iceberg types are now made valid with the escapes of Java's `AvroSchemaUtil.sanitize` (`event_x2Ets_day`, `my_x2Dcol`, `_1st`, `a_x20b`), with the original name in `iceberg-field-name` and the field ID unchanged, as Java's `TypeToSchema` writes them. The partition values of manifest entries and of `write_data_files_to_avro` are serialized under the same names. Java keeps non-ASCII letters and digits, which Avro does not allow, so the fork escapes them as well (`größe` becomes `gr_xF6_xDFe`); Java readers find the field by its field ID, and the fork's reader by field ID or `iceberg-field-name`.

Upstream: apache#2536; PR apache#2540 did the schema half of this and was closed as stale.

Review: `with_avro_field_names` must give the names `schema_to_avro_schema` writes, because the entry values are resolved against that schema by name. The round-trip test of the previous commit for a manifest the fork writes is replaced by a test that also checks the header.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ounds

`parquet_to_data_file_builder` used a row group's min or max only when Parquet marked it exact, and skipped column chunks without statistics. arrow-rs and DataFusion truncate byte-array statistics to 64 bytes by default and mark them inexact. In a file with several row groups, a string or binary column whose extreme value is longer than 64 bytes then got bounds from the other row groups only, so the recorded bounds excluded data in the file. Java and Trino prune such files for filters that match the excluded values, and a metadata-only `DELETE` (Spark's `StrictMetricsEvaluator`) can drop the whole file although some rows do not match. Every writer that writes Parquet through arrow-rs or DataFusion and takes its metrics from this function is affected. Upstream apache#3464 reports the related case where the bounds go missing, and no upstream PR exists.

Truncated statistics still bound the values: Parquet keeps a prefix for the min and increments the truncated max. They are now used for string and binary columns. Other types cannot hold a truncated value, so a truncated `fixed` value drops the column's bounds. Deprecated byte-array min and max, which parquet-mr wrote before 1.10, compare bytes as signed and bound nothing, so they drop the column's bounds as well. A truncated string value that is not valid UTF-8, which a truncation inside a character leaves, drops the column's bounds. A column chunk without statistics, or without min and max while it may hold non-null values, also drops the column's bounds, because its values are unknown.

As in Java apache/iceberg#17557, a column with a column chunk without a null count no longer gets a null value count. The fork added only the known counts, which undercounts.

Review: `MinMaxColAggregator::update`; the rule that a chunk with a null count equal to its value count holds only nulls; and dropping bounds for truncated `fixed` and `uuid` values, for deprecated byte-array statistics and for truncated strings that are not valid UTF-8.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every snapshot commit sent `SetSnapshotRef(main, branch(None, None, None))`. `SetSnapshotRef` replaces the whole reference, so each fork commit cleared `min-snapshots-to-keep`, `max-snapshot-age-ms` and `max-ref-age-ms` on `main`, and the catalog stored the cleared reference. Snapshot expiration then fell back to the table defaults and could expire snapshots, and delete their files, that the operator had set `main` to keep. Current data was not affected. Java keeps the settings (`SnapshotRef.builderFrom(ref, id)` in `TableMetadata`).

The snapshot producer and `RewriteManifestsAction` now copy the retention of the existing `main` branch into the new reference. A retention-only change committed between the table load and the commit is still overwritten, because `RefSnapshotIdMatch` does not compare retention; Java has the same race.

Upstream: apache/iceberg-rust main has the same code and no issue or PR yet.

Review: `main_branch_retention` in transaction/snapshot.rs, and that a `main` that is a tag keeps getting an empty branch retention.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
[Upstream PR apache#3479, open on 2026-10-25.]
(cherry picked from commit db7178b)
[Upstream PR apache#3479, open on 2026-10-25.]
(cherry picked from commit 8d8e1b6)
…rning field names

Record::serialize satisfied serde's `&'static str` field-name bound with
Box::leak(k.clone().into_boxed_str()) on every field of every record,
permanently leaking one allocation per field per serialized record.
Workloads that serialize many manifest entries (large fast_append
batches, manifest rewrites during maintenance) leak O(records x fields)
heap that is never reclaimed for the lifetime of the process.

Fix: intern each distinct field name once in a process-lifetime table
and reuse the 'static reference thereafter. Total leaked memory drops
from O(records x fields) to O(distinct field names) - a handful of
schema field names, leaked once, reused forever.

The interned name is content-identical to the previously leaked copy,
so serialized output is unchanged. Regression tests pin content
equality, pointer stability, and bounded growth across 10k records.

[Upstream PR apache#2951, open on 2026-10-25.]
(cherry picked from commit da75431)
[Only datum.rs and its partition.rs test; the overwrite.rs changes of this commit are left out.]
[Upstream PR apache#3300, open on 2026-10-25.]
(cherry picked from commit 3966f6b)
…rted load

`try_start_eq_del_load` marks an equality delete file as loading with a notifier, and `insert_equality_delete` then replaced that state with a second notifier and woke only the second one when the set arrived. A task on another thread that read the state between the two calls waited on the first notifier forever. A reader that shares one delete filter across tasks on a multi-thread runtime could hang at the first load of an equality delete file.

`insert_equality_delete` now takes the notifier from `try_start_eq_del_load` and keeps the loading state it set. Upstream PR apache#2630 fixes the same race against older code and is still open.

Review: the changed `insert_equality_delete` contract (callers must start the load first); the test helper `insert_set` follows it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
try_start_pos_del_load marks a positional delete file Loading, and only
the success path marked it Loaded. If opening or parsing the file
failed, or the load was dropped because another delete file failed,
the entry stayed Loading: a retry on the same loader, a clone of the
same ArrowReader, or a concurrent load of the same file waited for a
notification that never came.

The Load action now carries a PosDelLoadGuard. finish() marks the file
loaded; dropping the guard without finishing removes the Loading marker
and wakes the waiters. A woken waiter checks the state again and loads
the file itself if it isn't loaded, instead of assuming it is.

Closes apache#3434.

[Upstream PR apache#3435, open on 2026-10-25.]
(cherry picked from commit deff41b)
Only try_start_pos_del_load inserts into positional_deletes, and only the
guard that it returns changes the entry afterwards, so the entry is always
the guard's Loading marker when it is abandoned.

[Upstream PR apache#3435, open on 2026-10-25.]
(cherry picked from commit 6532513)
Hold the Load guard, poll load_file_for_task for the same file until it
waits, then drop the guard: the waiter loads the file itself, or returns
the open error when the file is gone. Also load a positional delete file
that opens but fails to parse, then rewrite it and load it again on the
same loader, which hangs without the guard.

[Upstream PR apache#3435, open on 2026-10-25.]
(cherry picked from commit 6757eb2)
…iled

Tasks that share a delete filter load each equality delete file once; the others wait for that load. When the load failed, the spawned task that stores the set panicked on the dropped sender and left the file marked as loading. Every task that waited for, or later needed, that file then waited forever, and a reader that shares the filter across tasks hung after the first error instead of failing.

A failed equality delete load now removes the file's state and wakes the waiters, which return the existing "Missing equality delete set" error. A later task loads the file again. The spawned task and the drop path of a positional delete load also take a poisoned state lock: a panic there would leave the waiters waiting forever, and in the drop path a panic during unwinding would abort the process.

Upstream issue apache#3274 reports this hang. PR apache#3275 fixed it the same way and was closed without merging on 2026-10-20. The positional counterpart is issue apache#3434 and PR apache#3435, cherry-picked before this commit. One more test here checks that a task waiting for a positional delete load that a failing sibling cancelled loads the file itself, with all its deletes.

Review: `insert_equality_delete` and the end of `get_equality_delete_set_for_delete_file_path` in delete_filter.rs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s field ID

The fixture's manifest names partition field 1001 `tpep_pickup_datetime`, while the partition spec names it `tpep_pickup_day`. Matching by name read the value as null. The reader now matches partition fields by field ID, as Java does, and returns the stored value 1925.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Format v3 allows promoting `date` to `timestamp` and `timestamp_ns`. `type_promotion_is_valid` had no arm for these, so the date leaf was left out of the projection mask whenever another column was read with it, and the column came back as NULL. The leaf is now projected, and `RecordBatchTransformer` casts it.

[Upstream PR apache#3485, open on 2026-10-26.]
When a data file stored a column with a type the reader cannot promote to the table type, the projection left the column out, and the reader filled it with NULL or its `initial-default`. A query returned wrong values without an error, and a rewrite of the file wrote them into the new file. The read now fails and names the column, the file type and the table type. This also covers a type change the catalog accepted on a format v2 table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reader casts a promoted column with arrow's default cast options, which turn a value that does not fit the target type into NULL. A `date` after 2262-04-11 promoted to `timestamp_ns` read as NULL, and a rewrite wrote NULL into the new file. The spec requires such a value to fail the read, and it now does, naming both types.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts initial default

An equality delete keyed on a column that was added after the data file was written probed that column as null, so a delete of the column's `initial-default` value removed no rows, although projection reads every row with that value. The probe now uses the identity partition value, then the `initial-default` of a top-level field, then null, as projection and Java's reader fill the column. This is case B of apache#3369 and only affects v3 tables.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nst their identity partition value or initial-default

The reader read a top-level column that a data file does not store as null when it evaluated the scan filter, although projection fills it with the identity partition value or the field's `initial-default`. A filter such as `p = 5` on a migrated file without the partition column `p` dropped every row. `residual_for_missing_fields` now folds each leaf on such a column to `AlwaysTrue` or `AlwaysFalse` against that value, before the row filter and every pruning layer. Leaves on nested fields and on missing columns without a value keep the null handling. On this fork, equality deletes do not go through the scan predicate; their probe fills absent key columns itself.

[Upstream PR apache#3302, open on 2026-10-26.]
…oes not store

A file that does not store its identity partition column, as after a Hive migration or `add_files`, keeps exactly the rows that each filter selects when the column is read as its partition value: `=`, `IN`, `>=`, `IS NOT NULL`, `NOT`, `AND` and `OR` with a stored column, and `<` and `<=` on both sides of the value, with row selection off and on.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…H` on a column the file does not store

A column that a data file does not store, and that has no identity partition value or `initial-default`, reads as null. The row filter still kept every row of such a file for `<`, `<=`, `NOT IN` and `NOT STARTS WITH`, so `WHERE c < 5` returned every row written before `c` was added. These now keep no row, like `>`, `>=`, `=` and `!=` already did, and like the same filters on a file that stores the column as all null.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rg-java specsById)

The manifest reader parses the table schema and partition spec from the
manifest Avro file's own `schema`/`partition-spec` key-value metadata and
hard-fails if `schema` is not a valid Iceberg schema. This makes tables
written by some engines unreadable (e.g. duckdb-iceberg 1.4 serializes the
manifest_entry Avro schema there, using Avro type names like `array`/
`record`), while pyiceberg, Doris, and Spark (iceberg-java) read them fine.

The manifest's embedded schema is redundant with the authoritative table
metadata. Prefer table_metadata.{schema_by_id, partition_spec_by_id},
looked up by the manifest's recorded schema-id / partition-spec-id, over
the manifest's own keys; manifests that do not record the ids, and readers
without table metadata, stay on the self-describing path. Mirrors
iceberg-java's ManifestReader(specsById), whose reading of the schema from
file metadata is deprecated.

Closes apache#2682.

[Adapted to the fork's ManifestReader: the table metadata goes through
ManifestReader::with_table_metadata, which Table::manifest_reader and the
scan's ObjectCache use, so every transaction that reads manifests through
the table (append's duplicate check, rewrite, row delta, overwrite,
rewrite manifests) resolves the schema the same way. The PR's
load_manifest_with is left out; ManifestFile::load_manifest stays
self-describing. The dropped-date scan fixture now registers the schema
its data manifest records, as a real table would.]
[Upstream PR apache#2683, open on 2026-10-26.]

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The v1 serde struct had no `refs` field. Reading v1 metadata dropped every branch and tag except a `main` built from `current-snapshot-id`, and writing it left `refs` out. A catalog that answers loadTable with this crate therefore hid the tags and branches it stores, and a client that expires snapshots on that answer removed the snapshots only those refs kept. The spec allows `refs` in v1, and Java reads and writes it for every format version.

v1 metadata now reads and writes `refs` as v2 and v3 do. Without `refs`, `main` still comes from `current-snapshot-id`. The new fixture is v1 metadata that Spark 3.5.6 with Iceberg 1.9.2 wrote after creating a tag and a branch.

No upstream issue or PR covers this as of 2026-10-26.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…urging a table

`drop_table_data` read manifests with the manifest's own header, so purging a DuckDB 1.4 table failed partway through. It now uses `Table::manifest_reader`, as the scan planner does. The `load_manifest` rustdoc says that it reads the header and points to `Table::manifest_reader`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@JanKaul

JanKaul commented Oct 26, 2026

Copy link
Copy Markdown

Closed in favor of 8

@JanKaul JanKaul closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants