Repository navigation
fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata - #5
Closed
c-thiel wants to merge 24 commits into
Conversation
…not allow Java writes a partition field whose name is not a valid Avro name under a sanitized name (`event.ts_day` becomes `event_x2Ets_day`, `my-col` becomes `my_x2Dcol`, `1st` becomes `_1st`), with the original name in `iceberg-field-name` and the partition field ID in `field-id`. Spark names the field of `days(event.ts)` `event.ts_day` by default. The manifest reader matched partition fields by name only, so it read every partition value of such a manifest as null, without an error. A rewrite then wrote output files and the surviving entries of the rewritten manifests with null partitions, which other engines prune away for partition filters. This is upstream apache#2536; its fix PR apache#2540 was closed as stale and no other PR is open. The reader now matches each partition field to the writer's Avro field by field ID, then by `iceberg-field-name`, then by name, as Java projects by field ID. It gives the fields of the partition type the writer's names before it decodes the entries, so the existing match by name finds every value. `apache-avro` rejects a header whose field names are not `[A-Za-z_][A-Za-z0-9_]*`. Java writes names with non-ASCII letters (`größe`) as they are, and iceberg-rust wrote every partition field name as it is, so the fork could not read those manifests at all, including the ones it wrote itself after one commit. On that error the reader now renames such fields in the header, keeping the original name in `iceberg-field-name`, and parses again. Field names do not change how a record is encoded. This joins the existing header fallback for repeated named type definitions; a header with both problems gets both rewrites. Review: the field-ID-first match in `writer_partition_type` (spec/manifest/mod.rs), and the retry loop in `rewrite_rejected_header`, which replaces the earlier single fallback and keeps its warning text. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fork wrote partition field names into the manifest's Avro schema as they are, for example `event.ts_day` for `days(event.ts)` or `my-col`. Such a manifest is not valid Avro: `apache-avro` refuses to parse its header, so strict third-party readers fail on it, and so did the fork itself. Field names in Avro schemas built from Iceberg types are now made valid with the escapes of Java's `AvroSchemaUtil.sanitize` (`event_x2Ets_day`, `my_x2Dcol`, `_1st`, `a_x20b`), with the original name in `iceberg-field-name` and the field ID unchanged, as Java's `TypeToSchema` writes them. The partition values of manifest entries and of `write_data_files_to_avro` are serialized under the same names. Java keeps non-ASCII letters and digits, which Avro does not allow, so the fork escapes them as well (`größe` becomes `gr_xF6_xDFe`); Java readers find the field by its field ID, and the fork's reader by field ID or `iceberg-field-name`. Upstream: apache#2536; PR apache#2540 did the schema half of this and was closed as stale. Review: `with_avro_field_names` must give the names `schema_to_avro_schema` writes, because the entry values are resolved against that schema by name. The round-trip test of the previous commit for a manifest the fork writes is replaced by a test that also checks the header. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ounds `parquet_to_data_file_builder` used a row group's min or max only when Parquet marked it exact, and skipped column chunks without statistics. arrow-rs and DataFusion truncate byte-array statistics to 64 bytes by default and mark them inexact. In a file with several row groups, a string or binary column whose extreme value is longer than 64 bytes then got bounds from the other row groups only, so the recorded bounds excluded data in the file. Java and Trino prune such files for filters that match the excluded values, and a metadata-only `DELETE` (Spark's `StrictMetricsEvaluator`) can drop the whole file although some rows do not match. Every writer that writes Parquet through arrow-rs or DataFusion and takes its metrics from this function is affected. Upstream apache#3464 reports the related case where the bounds go missing, and no upstream PR exists. Truncated statistics still bound the values: Parquet keeps a prefix for the min and increments the truncated max. They are now used for string and binary columns. Other types cannot hold a truncated value, so a truncated `fixed` value drops the column's bounds. Deprecated byte-array min and max, which parquet-mr wrote before 1.10, compare bytes as signed and bound nothing, so they drop the column's bounds as well. A truncated string value that is not valid UTF-8, which a truncation inside a character leaves, drops the column's bounds. A column chunk without statistics, or without min and max while it may hold non-null values, also drops the column's bounds, because its values are unknown. As in Java apache/iceberg#17557, a column with a column chunk without a null count no longer gets a null value count. The fork added only the known counts, which undercounts. Review: `MinMaxColAggregator::update`; the rule that a chunk with a null count equal to its value count holds only nulls; and dropping bounds for truncated `fixed` and `uuid` values, for deprecated byte-array statistics and for truncated strings that are not valid UTF-8. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every snapshot commit sent `SetSnapshotRef(main, branch(None, None, None))`. `SetSnapshotRef` replaces the whole reference, so each fork commit cleared `min-snapshots-to-keep`, `max-snapshot-age-ms` and `max-ref-age-ms` on `main`, and the catalog stored the cleared reference. Snapshot expiration then fell back to the table defaults and could expire snapshots, and delete their files, that the operator had set `main` to keep. Current data was not affected. Java keeps the settings (`SnapshotRef.builderFrom(ref, id)` in `TableMetadata`). The snapshot producer and `RewriteManifestsAction` now copy the retention of the existing `main` branch into the new reference. A retention-only change committed between the table load and the commit is still overwritten, because `RefSnapshotIdMatch` does not compare retention; Java has the same race. Upstream: apache/iceberg-rust main has the same code and no issue or PR yet. Review: `main_branch_retention` in transaction/snapshot.rs, and that a `main` that is a tag keeps getting an empty branch retention. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
[Upstream PR apache#3479, open on 2026-10-25.] (cherry picked from commit db7178b)
[Upstream PR apache#3479, open on 2026-10-25.] (cherry picked from commit 8d8e1b6)
…rning field names Record::serialize satisfied serde's `&'static str` field-name bound with Box::leak(k.clone().into_boxed_str()) on every field of every record, permanently leaking one allocation per field per serialized record. Workloads that serialize many manifest entries (large fast_append batches, manifest rewrites during maintenance) leak O(records x fields) heap that is never reclaimed for the lifetime of the process. Fix: intern each distinct field name once in a process-lifetime table and reuse the 'static reference thereafter. Total leaked memory drops from O(records x fields) to O(distinct field names) - a handful of schema field names, leaked once, reused forever. The interned name is content-identical to the previously leaked copy, so serialized output is unchanged. Regression tests pin content equality, pointer stability, and bounded growth across 10k records. [Upstream PR apache#2951, open on 2026-10-25.] (cherry picked from commit da75431)
[Only datum.rs and its partition.rs test; the overwrite.rs changes of this commit are left out.] [Upstream PR apache#3300, open on 2026-10-25.] (cherry picked from commit 3966f6b)
…rted load `try_start_eq_del_load` marks an equality delete file as loading with a notifier, and `insert_equality_delete` then replaced that state with a second notifier and woke only the second one when the set arrived. A task on another thread that read the state between the two calls waited on the first notifier forever. A reader that shares one delete filter across tasks on a multi-thread runtime could hang at the first load of an equality delete file. `insert_equality_delete` now takes the notifier from `try_start_eq_del_load` and keeps the loading state it set. Upstream PR apache#2630 fixes the same race against older code and is still open. Review: the changed `insert_equality_delete` contract (callers must start the load first); the test helper `insert_set` follows it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
try_start_pos_del_load marks a positional delete file Loading, and only the success path marked it Loaded. If opening or parsing the file failed, or the load was dropped because another delete file failed, the entry stayed Loading: a retry on the same loader, a clone of the same ArrowReader, or a concurrent load of the same file waited for a notification that never came. The Load action now carries a PosDelLoadGuard. finish() marks the file loaded; dropping the guard without finishing removes the Loading marker and wakes the waiters. A woken waiter checks the state again and loads the file itself if it isn't loaded, instead of assuming it is. Closes apache#3434. [Upstream PR apache#3435, open on 2026-10-25.] (cherry picked from commit deff41b)
Only try_start_pos_del_load inserts into positional_deletes, and only the guard that it returns changes the entry afterwards, so the entry is always the guard's Loading marker when it is abandoned. [Upstream PR apache#3435, open on 2026-10-25.] (cherry picked from commit 6532513)
Hold the Load guard, poll load_file_for_task for the same file until it waits, then drop the guard: the waiter loads the file itself, or returns the open error when the file is gone. Also load a positional delete file that opens but fails to parse, then rewrite it and load it again on the same loader, which hangs without the guard. [Upstream PR apache#3435, open on 2026-10-25.] (cherry picked from commit 6757eb2)
…iled Tasks that share a delete filter load each equality delete file once; the others wait for that load. When the load failed, the spawned task that stores the set panicked on the dropped sender and left the file marked as loading. Every task that waited for, or later needed, that file then waited forever, and a reader that shares the filter across tasks hung after the first error instead of failing. A failed equality delete load now removes the file's state and wakes the waiters, which return the existing "Missing equality delete set" error. A later task loads the file again. The spawned task and the drop path of a positional delete load also take a poisoned state lock: a panic there would leave the waiters waiting forever, and in the drop path a panic during unwinding would abort the process. Upstream issue apache#3274 reports this hang. PR apache#3275 fixed it the same way and was closed without merging on 2026-10-20. The positional counterpart is issue apache#3434 and PR apache#3435, cherry-picked before this commit. One more test here checks that a task waiting for a positional delete load that a failing sibling cancelled loads the file itself, with all its deletes. Review: `insert_equality_delete` and the end of `get_equality_delete_set_for_delete_file_path` in delete_filter.rs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s field ID The fixture's manifest names partition field 1001 `tpep_pickup_datetime`, while the partition spec names it `tpep_pickup_day`. Matching by name read the value as null. The reader now matches partition fields by field ID, as Java does, and returns the stored value 1925. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Format v3 allows promoting `date` to `timestamp` and `timestamp_ns`. `type_promotion_is_valid` had no arm for these, so the date leaf was left out of the projection mask whenever another column was read with it, and the column came back as NULL. The leaf is now projected, and `RecordBatchTransformer` casts it. [Upstream PR apache#3485, open on 2026-10-26.]
When a data file stored a column with a type the reader cannot promote to the table type, the projection left the column out, and the reader filled it with NULL or its `initial-default`. A query returned wrong values without an error, and a rewrite of the file wrote them into the new file. The read now fails and names the column, the file type and the table type. This also covers a type change the catalog accepted on a format v2 table. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reader casts a promoted column with arrow's default cast options, which turn a value that does not fit the target type into NULL. A `date` after 2262-04-11 promoted to `timestamp_ns` read as NULL, and a rewrite wrote NULL into the new file. The spec requires such a value to fail the read, and it now does, naming both types. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts initial default An equality delete keyed on a column that was added after the data file was written probed that column as null, so a delete of the column's `initial-default` value removed no rows, although projection reads every row with that value. The probe now uses the identity partition value, then the `initial-default` of a top-level field, then null, as projection and Java's reader fill the column. This is case B of apache#3369 and only affects v3 tables. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nst their identity partition value or initial-default The reader read a top-level column that a data file does not store as null when it evaluated the scan filter, although projection fills it with the identity partition value or the field's `initial-default`. A filter such as `p = 5` on a migrated file without the partition column `p` dropped every row. `residual_for_missing_fields` now folds each leaf on such a column to `AlwaysTrue` or `AlwaysFalse` against that value, before the row filter and every pruning layer. Leaves on nested fields and on missing columns without a value keep the null handling. On this fork, equality deletes do not go through the scan predicate; their probe fills absent key columns itself. [Upstream PR apache#3302, open on 2026-10-26.]
…oes not store A file that does not store its identity partition column, as after a Hive migration or `add_files`, keeps exactly the rows that each filter selects when the column is read as its partition value: `=`, `IN`, `>=`, `IS NOT NULL`, `NOT`, `AND` and `OR` with a stored column, and `<` and `<=` on both sides of the value, with row selection off and on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…H` on a column the file does not store A column that a data file does not store, and that has no identity partition value or `initial-default`, reads as null. The row filter still kept every row of such a file for `<`, `<=`, `NOT IN` and `NOT STARTS WITH`, so `WHERE c < 5` returned every row written before `c` was added. These now keep no row, like `>`, `>=`, `=` and `!=` already did, and like the same filters on a file that stores the column as all null. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rg-java specsById)
The manifest reader parses the table schema and partition spec from the
manifest Avro file's own `schema`/`partition-spec` key-value metadata and
hard-fails if `schema` is not a valid Iceberg schema. This makes tables
written by some engines unreadable (e.g. duckdb-iceberg 1.4 serializes the
manifest_entry Avro schema there, using Avro type names like `array`/
`record`), while pyiceberg, Doris, and Spark (iceberg-java) read them fine.
The manifest's embedded schema is redundant with the authoritative table
metadata. Prefer table_metadata.{schema_by_id, partition_spec_by_id},
looked up by the manifest's recorded schema-id / partition-spec-id, over
the manifest's own keys; manifests that do not record the ids, and readers
without table metadata, stay on the self-describing path. Mirrors
iceberg-java's ManifestReader(specsById), whose reading of the schema from
file metadata is deprecated.
Closes apache#2682.
[Adapted to the fork's ManifestReader: the table metadata goes through
ManifestReader::with_table_metadata, which Table::manifest_reader and the
scan's ObjectCache use, so every transaction that reads manifests through
the table (append's duplicate check, rewrite, row delta, overwrite,
rewrite manifests) resolves the schema the same way. The PR's
load_manifest_with is left out; ManifestFile::load_manifest stays
self-describing. The dropped-date scan fixture now registers the schema
its data manifest records, as a real table would.]
[Upstream PR apache#2683, open on 2026-10-26.]
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The v1 serde struct had no `refs` field. Reading v1 metadata dropped every branch and tag except a `main` built from `current-snapshot-id`, and writing it left `refs` out. A catalog that answers loadTable with this crate therefore hid the tags and branches it stores, and a client that expires snapshots on that answer removed the snapshots only those refs kept. The spec allows `refs` in v1, and Java reads and writes it for every format version. v1 metadata now reads and writes `refs` as v2 and v3 do. Without `refs`, `main` still comes from `current-snapshot-id`. The new fixture is v1 metadata that Spark 3.5.6 with Iceberg 1.9.2 wrote after creating a tag and a branch. No upstream issue or PR covers this as of 2026-10-26. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…urging a table `drop_table_data` read manifests with the manifest's own header, so purging a DuckDB 1.4 table failed partway through. It now uses `Table::manifest_reader`, as the scan planner does. The `load_manifest` rustdoc says that it reads the header and points to `Table::manifest_reader`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
JanKaul
force-pushed
the
ct/fork-upstream-audit-fixes
branch
from
October 26, 2026 10:15
6e79695 to
8842141
Compare
|
Closed in favor of 8 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for the reader, the manifest reader and v1 table metadata. Most of them stop a read that returned wrong values without an error. A rewrite of such a file wrote the wrong values into the new file. This PR is stacked on #4. After that PR merges, this PR is retargeted to
lakekeeper-compaction-base. Review commit by commit.Fixes for Parquet timestamps read through the INT96 schema hint (INT96 files in a
timestamptzcolumn, and INT64 nanosecond files in atimestampcolumn) follow in #7, stacked on this PR.Commits
datecolumns promoted totimestamp. Format v3 allows promotingdatetotimestampandtimestamp_ns. The reader had no rule for this promotion. When another column was read with it, the column was left out and read as null. The column is now read and cast. Issue: Date-to-timestamp promoted column reads as NULL when another column is projected apache/iceberg-rust#3404.dateread asstringorlongread asint, the reader filled the column with null or itsinitial-default, without an error. The read now fails and names the column, the file type and the table type. This also covers a type change that a catalog accepted on a format v2 table. No upstream issue or PR.dateafter 2262-04-11 promoted totimestamp_nsread as null. The spec requires such a read to fail, and it now does, naming both types. Date-to-timestamp promoted column reads as NULL when another column is projected apache/iceberg-rust#3404 names this case; there is no upstream PR.initial-default. An equality delete keyed on a column that was added after the data file was written probed that column as null. A delete of the column'sinitial-defaultvalue removed no rows, although the reader returns that value for every row of the file. The probe now uses the identity partition value, then theinitial-defaultof a top-level field, then null, as projection and Java's reader do. This is case B of Equality deletes ignore spec projection rules for dropped and newly added delete columns apache/iceberg-rust#3369 and affects only v3 tables. Related: perf(deletes): apply equality deletes via a hashed key set inside the parquet RowFilter apache/iceberg-rust#2961 (the hash-set probe, open).initial-default. The row filter read a column that the file does not store as null, while projection fills it with the identity partition value or theinitial-default. A filter such asp = 5on a file added from a Hive table without the columnpdropped every row. Each filter leaf on such a column is now decided against that value before the row filter and every pruning step. Issue: Rows are dropped when a filter references a column that is missing from the data file but has an identity partition value or initial-default apache/iceberg-rust#3298. A conflict in the test module ofrow_filter.rswas resolved by keeping both sides, and two type paths were shortened for a lint. The commit comes after the probe fix above, because one of its tests needs that fix.=,IN,>=,IS NOT NULL,NOT,ANDandORwith a stored column, and<and<=on both sides of the value, with row selection off and on. This is a separate commit so that the fix(arrow): evaluate filters on columns missing from a data file against their identity partition value or initial-default apache/iceberg-rust#3302 pick can be replaced when upstream merges it.<,<=,NOT INandNOT STARTS WITHon a column that the file does not store. Without an identity partition value orinitial-default, such a column reads as null. The row filter still kept every row for these four operators, soWHERE c < 5returned every row written beforecwas added. They now keep no row, like>,>=,=and!=already did, and like the same filters on a file that stores the column as all null. This matches SQL, where a comparison with null is never true. Java's generic rowEvaluatorsorts null first and keeps such rows for<,<=,!=andNOT IN. Issue: Row filter keeps every row for<and<=on a null column that is missing from the data file apache/iceberg-rust#3355 (for<and<=); no upstream PR.ManifestReaderdoes withspecsById. Manifests without recorded IDs, and reads without table metadata, still use the manifest's own keys. Issue: Manifest reader hard-depends on the manifestschema/partition-speckeys; should derive from table metadata (cf. iceberg-javaspecsById) apache/iceberg-rust#2682. The PR does not apply to the fork'sManifestReader, so it was adapted: the table metadata is passed withManifestReader::with_table_metadata, whichTable::manifest_readerand the scan's manifest cache use. So scans and every transaction that reads manifests through the table (append's duplicate check, rewrite, row delta, overwrite, rewrite manifests) resolve the schema the same way. The PR'sload_manifest_withis left out, andManifestFile::load_manifeststill reads only the manifest (see the next commit for its rustdoc). One scan test fixture wrote a manifest that recorded schema ID 0 with a different schema; it now registers the schema it writes, as a real table does.drop_table_dataread manifests with the manifest's own header, so purging a DuckDB 1.4 table failed partway through. It now usesTable::manifest_reader. Theload_manifestrustdoc now says that it reads the header and points toTable::manifest_reader.mainbuilt fromcurrent-snapshot-id, and writing v1 metadata leftrefsout. A catalog that builds itsloadTableanswer with this crate hid the tags and branches it stores. A client that expires snapshots on that answer removed the snapshots that only those refs kept. The spec allowsrefsin v1, and Java reads and writes it for every format version. v1 metadata now reads and writesrefsas v2 and v3 do. Withoutrefs,mainstill comes fromcurrent-snapshot-id. The test fixture is v1 metadata that Spark 3.5.6 with Iceberg 1.9.2 wrote after creating a tag and a branch. No upstream issue or PR.Compatibility
initial-defaultdecides, and without a value<,<=,NOT INandNOT STARTS WITHmatch no row.initial-defaultnow remove rows of files written before the column was added (v3 only).refswhen written. Older readers of this crate ignore the key.ManifestMetadata::parse_with. No public item is removed or changed.Tests
Each commit builds and passes
cargo test -p iceberg --all-features --lib. Each of our fixes has tests that fail without it. New tests:test_read_date_promoted_to_timestamp_with_other_columns(from fix(arrow): project date columns promoted to timestamp apache/iceberg-rust#3485),test_read_column_with_unpromotable_type_fails,test_read_promoted_value_out_of_range_fails;test_eq_delete_on_column_absent_from_data_filenow expects the delete of theinitial-defaultto remove every row;test_filter_on_absent_identity_partition_column_keeps_matching_rows,test_filter_on_absent_column_without_value_matches_all_null_column;test_manifest_metadata_parse_prefers_table_metadata_over_bad_schema,test_manifest_metadata_parse_self_describes_when_ids_not_recorded,test_fast_append_duplicate_check_tolerates_foreign_schema_metadata(all from fix(scan): derive manifest schema/spec from table metadata (cf. iceberg-java specsById) apache/iceberg-rust#2683);test_table_metadata_v1_refs_round_trip,test_table_metadata_v1_without_refs_has_main_branch.