Skip to content

fix: partition names Avro does not allow, truncated statistics, main retention, failed delete loads, and upstream cherry-picks - #6

Merged
JanKaul merged 17 commits into
lakekeeper-compaction-basefrom
ct/fork-upstream-audit-fixes
Oct 26, 2026
Merged

JanKaul merged 17 commits into
lakekeeper-compaction-basefrom
ct/fork-upstream-audit-fixes

Conversation

@JanKaul

@JanKaul JanKaul commented Oct 26, 2026

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

Fixes found by comparing the fork with upstream apache/iceberg-rust and Java. #4 has the full description of each. Review commit by commit.

Fixes:

Cherry-picks of open upstream PRs, with authorship kept: apache#3479 (projection of renamed columns by field ID), apache#2951 (field-name interning in Record::serialize), the datum.rs part of apache#3300 (no panic rendering a partition value whose source column was dropped), and apache#3435 (failed positional delete loads).

Compatibility

Partition field names that Avro does not allow are now written as Java writes them, for example event_x2Ets_day for event.ts_day. Fork revisions before this PR read these partition values as null, so upgrade all workers that write to the same tables together.

No public item is added, removed or changed.

Are these changes tested?

Each fix has unit tests that fail without it. On the head commit, workspace clippy with -D warnings, cargo nextest run --all-targets --all-features --workspace against the Docker services (2583 passed, 1 skipped) and the doctests pass locally.

AI Disclosure

The review, the comparison with Java and the three new commits were done with Claude Code. Areas to check:

  • CachingDeleteFileLoader::build_equality_delete_sets reloads an equality delete file after the loading phase, not inside it as positional deletes do. Waiting inside it could deadlock tasks whose buffered loads wait on each other's queued loads.
  • Skipping a float row group without min and max assumes arrow-rs writes the file. It leaves out min and max only for NaN. Only files this crate writes reach this code (parquet_files_to_data_files is unused). Other types still drop the bounds.
  • A partition value of an unknown type with a decimal value still renders unscaled (1.23 as 123). This is not fixed here.

🤖 Generated with Claude Code

c-thiel and others added 17 commits October 26, 2026 10:47
…not allow

Java writes a partition field whose name is not a valid Avro name under a sanitized name (`event.ts_day` becomes `event_x2Ets_day`, `my-col` becomes `my_x2Dcol`, `1st` becomes `_1st`), with the original name in `iceberg-field-name` and the partition field ID in `field-id`. Spark names the field of `days(event.ts)` `event.ts_day` by default. The manifest reader matched partition fields by name only, so it read every partition value of such a manifest as null, without an error. A rewrite then wrote output files and the surviving entries of the rewritten manifests with null partitions, which other engines prune away for partition filters. This is upstream apache#2536; its fix PR apache#2540 was closed as stale and no other PR is open.

The reader now matches each partition field to the writer's Avro field by field ID, then by `iceberg-field-name`, then by name, as Java projects by field ID. It gives the fields of the partition type the writer's names before it decodes the entries, so the existing match by name finds every value.

`apache-avro` rejects a header whose field names are not `[A-Za-z_][A-Za-z0-9_]*`. Java writes names with non-ASCII letters (`größe`) as they are, and iceberg-rust wrote every partition field name as it is, so the fork could not read those manifests at all, including the ones it wrote itself after one commit. On that error the reader now renames such fields in the header, keeping the original name in `iceberg-field-name`, and parses again. Field names do not change how a record is encoded. This joins the existing header fallback for repeated named type definitions; a header with both problems gets both rewrites.

Review: the field-ID-first match in `writer_partition_type` (spec/manifest/mod.rs), and the retry loop in `rewrite_rejected_header`, which replaces the earlier single fallback and keeps its warning text.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The fork wrote partition field names into the manifest's Avro schema as they are, for example `event.ts_day` for `days(event.ts)` or `my-col`. Such a manifest is not valid Avro: `apache-avro` refuses to parse its header, so strict third-party readers fail on it, and so did the fork itself.

Field names in Avro schemas built from Iceberg types are now made valid with the escapes of Java's `AvroSchemaUtil.sanitize` (`event_x2Ets_day`, `my_x2Dcol`, `_1st`, `a_x20b`), with the original name in `iceberg-field-name` and the field ID unchanged, as Java's `TypeToSchema` writes them. The partition values of manifest entries and of `write_data_files_to_avro` are serialized under the same names. Java keeps non-ASCII letters and digits, which Avro does not allow, so the fork escapes them as well (`größe` becomes `gr_xF6_xDFe`); Java readers find the field by its field ID, and the fork's reader by field ID or `iceberg-field-name`.

Upstream: apache#2536; PR apache#2540 did the schema half of this and was closed as stale.

Review: `with_avro_field_names` must give the names `schema_to_avro_schema` writes, because the entry values are resolved against that schema by name. The round-trip test of the previous commit for a manifest the fork writes is replaced by a test that also checks the header.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ounds

`parquet_to_data_file_builder` used a row group's min or max only when Parquet marked it exact, and skipped column chunks without statistics. arrow-rs and DataFusion truncate byte-array statistics to 64 bytes by default and mark them inexact. In a file with several row groups, a string or binary column whose extreme value is longer than 64 bytes then got bounds from the other row groups only, so the recorded bounds excluded data in the file. Java and Trino prune such files for filters that match the excluded values, and a metadata-only `DELETE` (Spark's `StrictMetricsEvaluator`) can drop the whole file although some rows do not match. Every writer that writes Parquet through arrow-rs or DataFusion and takes its metrics from this function is affected. Upstream apache#3464 reports the related case where the bounds go missing, and no upstream PR exists.

Truncated statistics still bound the values: Parquet keeps a prefix for the min and increments the truncated max. They are now used for string and binary columns. Other types cannot hold a truncated value, so a truncated `fixed` value drops the column's bounds. Deprecated byte-array min and max, which parquet-mr wrote before 1.10, compare bytes as signed and bound nothing, so they drop the column's bounds as well. A truncated string value that is not valid UTF-8, which a truncation inside a character leaves, drops the column's bounds. A column chunk without statistics, or without min and max while it may hold non-null values, also drops the column's bounds, because its values are unknown.

As in Java apache/iceberg#17557, a column with a column chunk without a null count no longer gets a null value count. The fork added only the known counts, which undercounts.

Review: `MinMaxColAggregator::update`; the rule that a chunk with a null count equal to its value count holds only nulls; and dropping bounds for truncated `fixed` and `uuid` values, for deprecated byte-array statistics and for truncated strings that are not valid UTF-8.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every snapshot commit sent `SetSnapshotRef(main, branch(None, None, None))`. `SetSnapshotRef` replaces the whole reference, so each fork commit cleared `min-snapshots-to-keep`, `max-snapshot-age-ms` and `max-ref-age-ms` on `main`, and the catalog stored the cleared reference. Snapshot expiration then fell back to the table defaults and could expire snapshots, and delete their files, that the operator had set `main` to keep. Current data was not affected. Java keeps the settings (`SnapshotRef.builderFrom(ref, id)` in `TableMetadata`).

The snapshot producer and `RewriteManifestsAction` now copy the retention of the existing `main` branch into the new reference. A retention-only change committed between the table load and the commit is still overwritten, because `RefSnapshotIdMatch` does not compare retention; Java has the same race.

Upstream: apache/iceberg-rust main has the same code and no issue or PR yet.

Review: `main_branch_retention` in transaction/snapshot.rs, and that a `main` that is a tag keeps getting an empty branch retention.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
[Upstream PR apache#3479, open on 2026-10-25.]
(cherry picked from commit db7178b)
[Upstream PR apache#3479, open on 2026-10-25.]
(cherry picked from commit 8d8e1b6)
…rning field names

Record::serialize satisfied serde's `&'static str` field-name bound with
Box::leak(k.clone().into_boxed_str()) on every field of every record,
permanently leaking one allocation per field per serialized record.
Workloads that serialize many manifest entries (large fast_append
batches, manifest rewrites during maintenance) leak O(records x fields)
heap that is never reclaimed for the lifetime of the process.

Fix: intern each distinct field name once in a process-lifetime table
and reuse the 'static reference thereafter. Total leaked memory drops
from O(records x fields) to O(distinct field names) - a handful of
schema field names, leaked once, reused forever.

The interned name is content-identical to the previously leaked copy,
so serialized output is unchanged. Regression tests pin content
equality, pointer stability, and bounded growth across 10k records.

[Upstream PR apache#2951, open on 2026-10-25.]
(cherry picked from commit da75431)
[Only datum.rs and its partition.rs test; the overwrite.rs changes of this commit are left out.]
[Upstream PR apache#3300, open on 2026-10-25.]
(cherry picked from commit 3966f6b)
…rted load

`try_start_eq_del_load` marks an equality delete file as loading with a notifier, and `insert_equality_delete` then replaced that state with a second notifier and woke only the second one when the set arrived. A task on another thread that read the state between the two calls waited on the first notifier forever. A reader that shares one delete filter across tasks on a multi-thread runtime could hang at the first load of an equality delete file.

`insert_equality_delete` now takes the notifier from `try_start_eq_del_load` and keeps the loading state it set. Upstream PR apache#2630 fixes the same race against older code and is still open.

Review: the changed `insert_equality_delete` contract (callers must start the load first); the test helper `insert_set` follows it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
try_start_pos_del_load marks a positional delete file Loading, and only
the success path marked it Loaded. If opening or parsing the file
failed, or the load was dropped because another delete file failed,
the entry stayed Loading: a retry on the same loader, a clone of the
same ArrowReader, or a concurrent load of the same file waited for a
notification that never came.

The Load action now carries a PosDelLoadGuard. finish() marks the file
loaded; dropping the guard without finishing removes the Loading marker
and wakes the waiters. A woken waiter checks the state again and loads
the file itself if it isn't loaded, instead of assuming it is.

Closes apache#3434.

[Upstream PR apache#3435, open on 2026-10-25.]
(cherry picked from commit deff41b)
Only try_start_pos_del_load inserts into positional_deletes, and only the
guard that it returns changes the entry afterwards, so the entry is always
the guard's Loading marker when it is abandoned.

[Upstream PR apache#3435, open on 2026-10-25.]
(cherry picked from commit 6532513)
Hold the Load guard, poll load_file_for_task for the same file until it
waits, then drop the guard: the waiter loads the file itself, or returns
the open error when the file is gone. Also load a positional delete file
that opens but fails to parse, then rewrite it and load it again on the
same loader, which hangs without the guard.

[Upstream PR apache#3435, open on 2026-10-25.]
(cherry picked from commit 6757eb2)
…iled

Tasks that share a delete filter load each equality delete file once; the others wait for that load. When the load failed, the spawned task that stores the set panicked on the dropped sender and left the file marked as loading. Every task that waited for, or later needed, that file then waited forever, and a reader that shares the filter across tasks hung after the first error instead of failing.

A failed equality delete load now removes the file's state and wakes the waiters, which return the existing "Missing equality delete set" error. A later task loads the file again. The spawned task and the drop path of a positional delete load also take a poisoned state lock: a panic there would leave the waiters waiting forever, and in the drop path a panic during unwinding would abort the process.

Upstream issue apache#3274 reports this hang. PR apache#3275 fixed it the same way and was closed without merging on 2026-10-20. The positional counterpart is issue apache#3434 and PR apache#3435, cherry-picked before this commit. One more test here checks that a task waiting for a positional delete load that a failing sibling cancelled loads the file itself, with all its deletes.

Review: `insert_equality_delete` and the end of `get_equality_delete_set_for_delete_file_path` in delete_filter.rs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s field ID

The fixture's manifest names partition field 1001 `tpep_pickup_datetime`, while the partition spec names it `tpep_pickup_day`. Matching by name read the value as null. The reader now matches partition fields by field ID, as Java does, and returns the stored value 1925.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…aits for fails

A task that found another task's load of an equality delete file in progress waited for it in `build_equality_delete_sets` and failed with "Missing equality delete set" when that load failed or was cancelled, for example because a sibling delete file of the loading task failed. Only tasks that started later loaded the file again.

`CachingDeleteFileLoader::build_equality_delete_sets` now loads such a file itself before building the sets, so one task's failure does not fail another. Java behaves the same way: `SparkExecutorCache.getOrLoad` uses Caffeine's `Cache.get(key, loader)`, which stores nothing when the loader throws, and each blocked caller then runs the loader itself.

The wait stays after `load_deletes` rather than inside it, as positional deletes do, so that a task's buffered loads do not wait on loads queued behind another task's waiters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ly NaN

arrow-rs leaves NaN out of a column chunk's min and max, so a float or double chunk whose non-null values are all NaN has no min and max but a null count below its value count. `MinMaxColAggregator::update` treated that chunk as unbounded and dropped the column's bounds for the whole file, so engines could no longer prune the file on that column. NaN is tracked in `nan_value_counts`, and bounds do not cover it.

Such a chunk is now skipped and the other chunks' bounds are kept. Java's `ParquetMetrics.bounds` skips every chunk without a non-null min and max (`!stats.hasNonNullValue()`). Here the skip is limited to float and double: arrow-rs, the only writer whose footers reach this code, omits min and max for those only when the chunk holds only NaN and nulls, while other types keep the conservative drop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`compare_schemas` now rebuilds every batch whose columns lack field IDs, and the rebuild matches columns by field ID only. `BasicDeleteFileLoader` read delete files straight from Parquet without assigning IDs, so every equality column of a delete file written without `PARQUET:field_id` came out null: the file deleted rows with null keys instead of its own keys, with no error. Before, such a file was passed through when its column names, types and order matched the equality columns.

`parquet_to_batch_stream` now gives a file without field IDs top-level IDs from column positions, as the data-file reader already does and as Java does for delete files (`BaseDeleteLoader` reads without a name mapping, so `ReadConf` calls `ParquetSchemaUtil.addFallbackIds`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants