Repository navigation
fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata - #8
Merged
Conversation
Format v3 allows promoting `date` to `timestamp` and `timestamp_ns`. `type_promotion_is_valid` had no arm for these, so the date leaf was left out of the projection mask whenever another column was read with it, and the column came back as NULL. The leaf is now projected, and `RecordBatchTransformer` casts it. [Upstream PR apache#3485, open on 2026-10-26.]
When a data file stored a column with a type the reader cannot promote to the table type, the projection left the column out, and the reader filled it with NULL or its `initial-default`. A query returned wrong values without an error, and a rewrite of the file wrote them into the new file. The read now fails and names the column, the file type and the table type. This also covers a type change the catalog accepted on a format v2 table. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reader casts a promoted column with arrow's default cast options, which turn a value that does not fit the target type into NULL. A `date` after 2262-04-11 promoted to `timestamp_ns` read as NULL, and a rewrite wrote NULL into the new file. The spec requires such a value to fail the read, and it now does, naming both types. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts initial default An equality delete keyed on a column that was added after the data file was written probed that column as null, so a delete of the column's `initial-default` value removed no rows, although projection reads every row with that value. The probe now uses the identity partition value, then the `initial-default` of a top-level field, then null, as projection and Java's reader fill the column. This is case B of apache#3369 and only affects v3 tables. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…nst their identity partition value or initial-default The reader read a top-level column that a data file does not store as null when it evaluated the scan filter, although projection fills it with the identity partition value or the field's `initial-default`. A filter such as `p = 5` on a migrated file without the partition column `p` dropped every row. `residual_for_missing_fields` now folds each leaf on such a column to `AlwaysTrue` or `AlwaysFalse` against that value, before the row filter and every pruning layer. Leaves on nested fields and on missing columns without a value keep the null handling. On this fork, equality deletes do not go through the scan predicate; their probe fills absent key columns itself. [Upstream PR apache#3302, open on 2026-10-26.]
…oes not store A file that does not store its identity partition column, as after a Hive migration or `add_files`, keeps exactly the rows that each filter selects when the column is read as its partition value: `=`, `IN`, `>=`, `IS NOT NULL`, `NOT`, `AND` and `OR` with a stored column, and `<` and `<=` on both sides of the value, with row selection off and on. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…H` on a column the file does not store A column that a data file does not store, and that has no identity partition value or `initial-default`, reads as null. The row filter still kept every row of such a file for `<`, `<=`, `NOT IN` and `NOT STARTS WITH`, so `WHERE c < 5` returned every row written before `c` was added. These now keep no row, like `>`, `>=`, `=` and `!=` already did, and like the same filters on a file that stores the column as all null. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…rg-java specsById)
The manifest reader parses the table schema and partition spec from the
manifest Avro file's own `schema`/`partition-spec` key-value metadata and
hard-fails if `schema` is not a valid Iceberg schema. This makes tables
written by some engines unreadable (e.g. duckdb-iceberg 1.4 serializes the
manifest_entry Avro schema there, using Avro type names like `array`/
`record`), while pyiceberg, Doris, and Spark (iceberg-java) read them fine.
The manifest's embedded schema is redundant with the authoritative table
metadata. Prefer table_metadata.{schema_by_id, partition_spec_by_id},
looked up by the manifest's recorded schema-id / partition-spec-id, over
the manifest's own keys; manifests that do not record the ids, and readers
without table metadata, stay on the self-describing path. Mirrors
iceberg-java's ManifestReader(specsById), whose reading of the schema from
file metadata is deprecated.
Closes apache#2682.
[Adapted to the fork's ManifestReader: the table metadata goes through
ManifestReader::with_table_metadata, which Table::manifest_reader and the
scan's ObjectCache use, so every transaction that reads manifests through
the table (append's duplicate check, rewrite, row delta, overwrite,
rewrite manifests) resolves the schema the same way. The PR's
load_manifest_with is left out; ManifestFile::load_manifest stays
self-describing. The dropped-date scan fixture now registers the schema
its data manifest records, as a real table would.]
[Upstream PR apache#2683, open on 2026-10-26.]
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The v1 serde struct had no `refs` field. Reading v1 metadata dropped every branch and tag except a `main` built from `current-snapshot-id`, and writing it left `refs` out. A catalog that answers loadTable with this crate therefore hid the tags and branches it stores, and a client that expires snapshots on that answer removed the snapshots only those refs kept. The spec allows `refs` in v1, and Java reads and writes it for every format version. v1 metadata now reads and writes `refs` as v2 and v3 do. Without `refs`, `main` still comes from `current-snapshot-id`. The new fixture is v1 metadata that Spark 3.5.6 with Iceberg 1.9.2 wrote after creating a tag and a branch. No upstream issue or PR covers this as of 2026-10-26. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…urging a table `drop_table_data` read manifests with the manifest's own header, so purging a DuckDB 1.4 table failed partway through. It now uses `Table::manifest_reader`, as the scan planner does. The `load_manifest` rustdoc says that it reads the header and points to `Table::manifest_reader`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The row filter cast the literal down to the type the file stores. For a `date` column promoted to `timestamp`, arrow's cast truncated the timestamp literal to a day, so `d < TIMESTAMP '2020-01-01 12:00'` dropped a row of 2020-01-01 and `d = ...` kept it. For an `int` column promoted to `long`, a literal outside the `int` range cast to null and the filter matched no row. The filter now casts the column up to the table type instead. The equality-delete probe promoted each key with `Datum::to`, which cannot convert a `date` to a `timestamp`, so a delete keyed on a promoted `date` column failed the read. The probe now casts the key column to the table type with arrow, as the projection does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…te arrays as the table type The unpromotable-type check failed the read of two kinds of files that Java reads. Spark writes INT96 timestamps and Hive writes INT64 timestamps without the UTC adjustment, so a `timestamptz` column of a migrated or `add_files` table stored `timestamp` and failed. Byte arrays without the UTF8 annotation in a `string` column failed the same way. Both now read as the table type. A byte array that is not valid UTF-8 fails the read. The check covered only files with field IDs. A file read by column position cast any stored type to the table type, so a `date` column read as `string` returned date strings. The record batch transformer now applies the same rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ality deletes Filters and equality deletes on a field that a data file does not store each computed the value projection fills in, from the identity partition value and the `initial-default`, in two copies that resolved them differently. Both now use one map per task. After the residual folds the leaves on such fields to constants, the row filter now decodes only the columns the residual still references. An `AlwaysTrue` residual adds no row filter, and an `AlwaysFalse` residual reads no row group, also with row-group filtering disabled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every caller of `parse_with` is inside the crate, so it no longer grows the public API. Its tests move next to it and compare whole `ManifestMetadata` values, so a wrong partition spec or format version fails them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
lakekeeper-compaction-basenow that fix: partition names Avro does not allow, truncated statistics, main retention, failed delete loads, and upstream cherry-picks #4 has landed there (fix: partition names Avro does not allow, truncated statistics, main retention, failed delete loads, and upstream cherry-picks #6). It carries the ten commits of fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #5 unchanged and adds four commits with fixes from its review.What changes are included in this PR?
#5 has the description of its ten commits. Review commit by commit. The four new commits:
datepromoted totimestamp, the row filter cast the timestamp literal down to a day, sod < TIMESTAMP '2020-01-01 12:00'dropped a row of 2020-01-01. On anintcolumn promoted tolong, a literal outside theintrange became null and the filter matched no row. The filter now casts the column up to the table type. An equality delete keyed on a promoteddatecolumn failed the read, becauseDatum::tohas no date→timestamp conversion. The probe now casts the key column to the table type with arrow.timestamptzcolumn (they carry no UTC adjustment), and of byte arrays without the UTF8 annotation in astringcolumn. These now read as the table type. Files without field IDs, read by column position, bypassed the check and cast any type, so adatecolumn read asstringreturned date strings. The record batch transformer now applies the same rule.AlwaysFalseresidual reads no row group, also with row-group filtering off.ManifestMetadata::parse_withispub(crate). All callers are in the crate. Its tests moved next to it and compare whole values.Compatibility
Same as #5, with two differences:
ManifestMetadata::parse_withis not public. No public item is added, removed or changed.stringcolumn, read as the table type instead of failing. INT64 nanosecond files in atimestampcolumn, also listed in fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #5 as still to come, are still not handled.Are these changes tested?
Each new commit adds unit tests: filters on a promoted
dateandintcolumn, an equality delete on a promoteddatecolumn, the new type pairs with and without field IDs, invalid UTF-8 in astringcolumn, and an unpromotable type in a file without field IDs. The two filter tests fail without the fix. On each commit,cargo fmt --check,cargo clippy -p iceberg --all-targets --all-features -- -D warningsandcargo test -p iceberg --lib --all-featurespass (2135 passed, 1 ignored at the head). Integration tests and nextest against the Docker services were not run.AI Disclosure
The four new commits were written with Claude Code. Areas to check:
int→long,float→double,date→timestampand decimal precision widening, and keeps casting the literal for every other pair.timestamp↔timestamptzis allowed in both directions, on the assumption that Java reads stored micros unchanged whatever the file's UTC adjustment.arrow_type_to_typecannot convert, which keep the previous cast.🤖 Generated with Claude Code