Repository navigation
fix: partition names Avro does not allow, truncated statistics, main retention, failed delete loads, and upstream cherry-picks - #4
Merged
JanKaul merged 17 commits intoOct 26, 2026
Conversation
c-thiel
force-pushed
the
ct/fork-upstream-audit-fixes
branch
from
October 26, 2026 08:04
632f433 to
6e79695
Compare
…not allow Java writes a partition field whose name is not a valid Avro name under a sanitized name (`event.ts_day` becomes `event_x2Ets_day`, `my-col` becomes `my_x2Dcol`, `1st` becomes `_1st`), with the original name in `iceberg-field-name` and the partition field ID in `field-id`. Spark names the field of `days(event.ts)` `event.ts_day` by default. The manifest reader matched partition fields by name only, so it read every partition value of such a manifest as null, without an error. A rewrite then wrote output files and the surviving entries of the rewritten manifests with null partitions, which other engines prune away for partition filters. This is upstream apache#2536; its fix PR apache#2540 was closed as stale and no other PR is open. The reader now matches each partition field to the writer's Avro field by field ID, then by `iceberg-field-name`, then by name, as Java projects by field ID. It gives the fields of the partition type the writer's names before it decodes the entries, so the existing match by name finds every value. `apache-avro` rejects a header whose field names are not `[A-Za-z_][A-Za-z0-9_]*`. Java writes names with non-ASCII letters (`größe`) as they are, and iceberg-rust wrote every partition field name as it is, so the fork could not read those manifests at all, including the ones it wrote itself after one commit. On that error the reader now renames such fields in the header, keeping the original name in `iceberg-field-name`, and parses again. Field names do not change how a record is encoded. This joins the existing header fallback for repeated named type definitions; a header with both problems gets both rewrites. Review: the field-ID-first match in `writer_partition_type` (spec/manifest/mod.rs), and the retry loop in `rewrite_rejected_header`, which replaces the earlier single fallback and keeps its warning text. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fork wrote partition field names into the manifest's Avro schema as they are, for example `event.ts_day` for `days(event.ts)` or `my-col`. Such a manifest is not valid Avro: `apache-avro` refuses to parse its header, so strict third-party readers fail on it, and so did the fork itself. Field names in Avro schemas built from Iceberg types are now made valid with the escapes of Java's `AvroSchemaUtil.sanitize` (`event_x2Ets_day`, `my_x2Dcol`, `_1st`, `a_x20b`), with the original name in `iceberg-field-name` and the field ID unchanged, as Java's `TypeToSchema` writes them. The partition values of manifest entries and of `write_data_files_to_avro` are serialized under the same names. Java keeps non-ASCII letters and digits, which Avro does not allow, so the fork escapes them as well (`größe` becomes `gr_xF6_xDFe`); Java readers find the field by its field ID, and the fork's reader by field ID or `iceberg-field-name`. Upstream: apache#2536; PR apache#2540 did the schema half of this and was closed as stale. Review: `with_avro_field_names` must give the names `schema_to_avro_schema` writes, because the entry values are resolved against that schema by name. The round-trip test of the previous commit for a manifest the fork writes is replaced by a test that also checks the header. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ounds `parquet_to_data_file_builder` used a row group's min or max only when Parquet marked it exact, and skipped column chunks without statistics. arrow-rs and DataFusion truncate byte-array statistics to 64 bytes by default and mark them inexact. In a file with several row groups, a string or binary column whose extreme value is longer than 64 bytes then got bounds from the other row groups only, so the recorded bounds excluded data in the file. Java and Trino prune such files for filters that match the excluded values, and a metadata-only `DELETE` (Spark's `StrictMetricsEvaluator`) can drop the whole file although some rows do not match. Every writer that writes Parquet through arrow-rs or DataFusion and takes its metrics from this function is affected. Upstream apache#3464 reports the related case where the bounds go missing, and no upstream PR exists. Truncated statistics still bound the values: Parquet keeps a prefix for the min and increments the truncated max. They are now used for string and binary columns. Other types cannot hold a truncated value, so a truncated `fixed` value drops the column's bounds. Deprecated byte-array min and max, which parquet-mr wrote before 1.10, compare bytes as signed and bound nothing, so they drop the column's bounds as well. A truncated string value that is not valid UTF-8, which a truncation inside a character leaves, drops the column's bounds. A column chunk without statistics, or without min and max while it may hold non-null values, also drops the column's bounds, because its values are unknown. As in Java apache/iceberg#17557, a column with a column chunk without a null count no longer gets a null value count. The fork added only the known counts, which undercounts. Review: `MinMaxColAggregator::update`; the rule that a chunk with a null count equal to its value count holds only nulls; and dropping bounds for truncated `fixed` and `uuid` values, for deprecated byte-array statistics and for truncated strings that are not valid UTF-8. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every snapshot commit sent `SetSnapshotRef(main, branch(None, None, None))`. `SetSnapshotRef` replaces the whole reference, so each fork commit cleared `min-snapshots-to-keep`, `max-snapshot-age-ms` and `max-ref-age-ms` on `main`, and the catalog stored the cleared reference. Snapshot expiration then fell back to the table defaults and could expire snapshots, and delete their files, that the operator had set `main` to keep. Current data was not affected. Java keeps the settings (`SnapshotRef.builderFrom(ref, id)` in `TableMetadata`). The snapshot producer and `RewriteManifestsAction` now copy the retention of the existing `main` branch into the new reference. A retention-only change committed between the table load and the commit is still overwritten, because `RefSnapshotIdMatch` does not compare retention; Java has the same race. Upstream: apache/iceberg-rust main has the same code and no issue or PR yet. Review: `main_branch_retention` in transaction/snapshot.rs, and that a `main` that is a tag keeps getting an empty branch retention. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
[Upstream PR apache#3479, open on 2026-10-25.] (cherry picked from commit db7178b)
[Upstream PR apache#3479, open on 2026-10-25.] (cherry picked from commit 8d8e1b6)
…rning field names Record::serialize satisfied serde's `&'static str` field-name bound with Box::leak(k.clone().into_boxed_str()) on every field of every record, permanently leaking one allocation per field per serialized record. Workloads that serialize many manifest entries (large fast_append batches, manifest rewrites during maintenance) leak O(records x fields) heap that is never reclaimed for the lifetime of the process. Fix: intern each distinct field name once in a process-lifetime table and reuse the 'static reference thereafter. Total leaked memory drops from O(records x fields) to O(distinct field names) - a handful of schema field names, leaked once, reused forever. The interned name is content-identical to the previously leaked copy, so serialized output is unchanged. Regression tests pin content equality, pointer stability, and bounded growth across 10k records. [Upstream PR apache#2951, open on 2026-10-25.] (cherry picked from commit da75431)
[Only datum.rs and its partition.rs test; the overwrite.rs changes of this commit are left out.] [Upstream PR apache#3300, open on 2026-10-25.] (cherry picked from commit 3966f6b)
…rted load `try_start_eq_del_load` marks an equality delete file as loading with a notifier, and `insert_equality_delete` then replaced that state with a second notifier and woke only the second one when the set arrived. A task on another thread that read the state between the two calls waited on the first notifier forever. A reader that shares one delete filter across tasks on a multi-thread runtime could hang at the first load of an equality delete file. `insert_equality_delete` now takes the notifier from `try_start_eq_del_load` and keeps the loading state it set. Upstream PR apache#2630 fixes the same race against older code and is still open. Review: the changed `insert_equality_delete` contract (callers must start the load first); the test helper `insert_set` follows it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
try_start_pos_del_load marks a positional delete file Loading, and only the success path marked it Loaded. If opening or parsing the file failed, or the load was dropped because another delete file failed, the entry stayed Loading: a retry on the same loader, a clone of the same ArrowReader, or a concurrent load of the same file waited for a notification that never came. The Load action now carries a PosDelLoadGuard. finish() marks the file loaded; dropping the guard without finishing removes the Loading marker and wakes the waiters. A woken waiter checks the state again and loads the file itself if it isn't loaded, instead of assuming it is. Closes apache#3434. [Upstream PR apache#3435, open on 2026-10-25.] (cherry picked from commit deff41b)
Only try_start_pos_del_load inserts into positional_deletes, and only the guard that it returns changes the entry afterwards, so the entry is always the guard's Loading marker when it is abandoned. [Upstream PR apache#3435, open on 2026-10-25.] (cherry picked from commit 6532513)
Hold the Load guard, poll load_file_for_task for the same file until it waits, then drop the guard: the waiter loads the file itself, or returns the open error when the file is gone. Also load a positional delete file that opens but fails to parse, then rewrite it and load it again on the same loader, which hangs without the guard. [Upstream PR apache#3435, open on 2026-10-25.] (cherry picked from commit 6757eb2)
…iled Tasks that share a delete filter load each equality delete file once; the others wait for that load. When the load failed, the spawned task that stores the set panicked on the dropped sender and left the file marked as loading. Every task that waited for, or later needed, that file then waited forever, and a reader that shares the filter across tasks hung after the first error instead of failing. A failed equality delete load now removes the file's state and wakes the waiters, which return the existing "Missing equality delete set" error. A later task loads the file again. The spawned task and the drop path of a positional delete load also take a poisoned state lock: a panic there would leave the waiters waiting forever, and in the drop path a panic during unwinding would abort the process. Upstream issue apache#3274 reports this hang. PR apache#3275 fixed it the same way and was closed without merging on 2026-10-20. The positional counterpart is issue apache#3434 and PR apache#3435, cherry-picked before this commit. One more test here checks that a task waiting for a positional delete load that a failing sibling cancelled loads the file itself, with all its deletes. Review: `insert_equality_delete` and the end of `get_equality_delete_set_for_delete_file_path` in delete_filter.rs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s field ID The fixture's manifest names partition field 1001 `tpep_pickup_datetime`, while the partition spec names it `tpep_pickup_day`. Matching by name read the value as null. The reader now matches partition fields by field ID, as Java does, and returns the stored value 1925. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…aits for fails A task that found another task's load of an equality delete file in progress waited for it in `build_equality_delete_sets` and failed with "Missing equality delete set" when that load failed or was cancelled, for example because a sibling delete file of the loading task failed. Only tasks that started later loaded the file again. `CachingDeleteFileLoader::build_equality_delete_sets` now loads such a file itself before building the sets, so one task's failure does not fail another. Java behaves the same way: `SparkExecutorCache.getOrLoad` uses Caffeine's `Cache.get(key, loader)`, which stores nothing when the loader throws, and each blocked caller then runs the loader itself. The wait stays after `load_deletes` rather than inside it, as positional deletes do, so that a task's buffered loads do not wait on loads queued behind another task's waiters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ly NaN arrow-rs leaves NaN out of a column chunk's min and max, so a float or double chunk whose non-null values are all NaN has no min and max but a null count below its value count. `MinMaxColAggregator::update` treated that chunk as unbounded and dropped the column's bounds for the whole file, so engines could no longer prune the file on that column. NaN is tracked in `nan_value_counts`, and bounds do not cover it. Such a chunk is now skipped and the other chunks' bounds are kept. Java's `ParquetMetrics.bounds` skips every chunk without a non-null min and max (`!stats.hasNonNullValue()`). Here the skip is limited to float and double: arrow-rs, the only writer whose footers reach this code, omits min and max for those only when the chunk holds only NaN and nulls, while other types keep the conservative drop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`compare_schemas` now rebuilds every batch whose columns lack field IDs, and the rebuild matches columns by field ID only. `BasicDeleteFileLoader` read delete files straight from Parquet without assigning IDs, so every equality column of a delete file written without `PARQUET:field_id` came out null: the file deleted rows with null keys instead of its own keys, with no error. Before, such a file was passed through when its column names, types and order matched the equality columns. `parquet_to_batch_stream` now gives a file without field IDs top-level IDs from column positions, as the data-file reader already does and as Java does for delete files (`BaseDeleteLoader` reads without a name mapping, so `ReadConf` calls `ParquetSchemaUtil.addFallbackIds`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
JanKaul
force-pushed
the
ct/fork-upstream-audit-fixes
branch
from
October 26, 2026 10:15
6e79695 to
8842141
Compare
This was referenced Oct 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes found by comparing the fork with upstream apache/iceberg-rust and Java. This PR is stacked on #3. After that PR merges, this PR is retargeted to
lakekeeper-compaction-base. Review commit by commit.Our fixes:
days(event.ts)event.ts_day. Java writes it asevent_x2Ets_daywith the original name iniceberg-field-name. The reader matched partition fields by name only and read every partition value of such a manifest as null, without an error. A rewrite then wrote those nulls back. The reader now matches by field ID, theniceberg-field-name, then name, and it reads headers whose names Avro rejects. The writer sanitizes the names as Java does, so strict Avro readers accept the fork's manifests.DELETEdropped it. Truncated string and binary statistics are now used as bounds. A truncatedfixedvalue, deprecated byte-array statistics (signed byte order, parquet-mr before 1.10), a truncated string value that is not valid UTF-8 and a column chunk without statistics drop the column's bounds. A column with a chunk without a null count gets no null count, as in Java Core, Parquet: Fix null counting for parquet that doesn't have null_count stats apache/iceberg#17557.main. Every commit replaced themainreference with one withoutmin-snapshots-to-keep,max-snapshot-age-msandmax-ref-age-ms. Snapshot expiration then fell back to the table defaults. Commits now keep the retention ofmain, as Java does.Cherry-picks of open upstream PRs. Authorship is kept, and each is marked in its message:
Record::serialize, which leaked one allocation per field per serialized record.datum.rschange and itspartition.rstest of one commit. A rewrite builds its snapshot summary withpartition_to_pathfor each removed file. When the source column of an identity partition on a decimal or uuid was dropped, the partition type is unknown and the value keeps its stored type.Datum'sDisplaythen reachedunreachable!(), so every rewrite of such a table panicked. The test rows fail without thedatum.rschange. The fork already has the overwrite action of that PR; theoverwrite.rschanges of this commit are left out.delete_filter.rs, where the fork changed the function next to the new one, was resolved by keeping both sides. Our equality delete commit adds a test for a waiter whose load a failing sibling cancelled.Compatibility
Partition field names that Avro does not allow are now written as Java writes them, for example
event_x2Ets_dayforevent.ts_day, with the original name iniceberg-field-name. Names with non-ASCII letters are escaped as well (größebecomesgr_xF6_xDFe). Java keeps those as they are, and Java readers find the field by its field ID.Fork revisions before this PR do not find the new names. They read these partition values as null, without an error, as they already read manifests that Java writes. A rewrite on such a revision then writes the nulls back. So upgrade all workers that write to the same tables together.
No public item is added, removed or changed.
Tests
Each commit builds and passes
cargo test -p iceberg --all-features --lib. Each of our fixes has tests that fail without it.