Repository navigation
fix(arrow): read INT96 and nanosecond Parquet timestamps by value, and promoted unknown columns - #9
Merged
Conversation
…y value The INT96 schema hint matched every `Timestamp(ns)` Arrow field, not only INT96 leaves. parquet-rs accepts the same hint for an INT64 `TIMESTAMP(NANOS)` leaf and then reinterprets its nanosecond values as microseconds. A nanosecond file in a `timestamp` column therefore read 1000 times too large without an error (2024-01-01 read as year 55969), and a nanosecond UTC file in a `timestamptz` column failed every read of the file, even when the column was not projected. Such files come from pandas and pyarrow, added with `add_files`. The hint now applies only to leaves that the Parquet schema stores as INT96. A nanosecond value in a `timestamp` or `timestamptz` column reads as the same instant in microseconds; a value with sub-microsecond digits fails the read, because truncating it would change the value silently. Every other path that sees the stored nanoseconds uses the same values: - Equality deletes compare the key converted to microseconds, with the table column's UTC flag, as microsecond files do. A key with sub-microsecond digits matches no delete. - Row-group statistics, the page index and bloom filters hold nanoseconds, so they no longer prune such columns. They read microsecond literals against nanosecond values and dropped matching rows. - The INT96 hint runs only for files with INT96 leaves, and an error while building it fails the read, so no INT96 leaf can be read without it. Upstream has the same hint since apache#2301; no upstream issue or PR covers this as of 2026-10-26. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The INT96 schema hint kept the time zone of the file's Arrow type, which is none for files without an Arrow schema. An INT96 column in a `timestamptz` column was therefore read as `timestamp` and reached the table type only through the reader's `timestamp` to `timestamptz` promotion. pyarrow files that store a UTC Arrow type took the opposite promotion in `timestamp` columns. INT96 holds UTC instants, and Java reads it into either column type. The hint now takes the time zone from the Iceberg column: UTC for `timestamptz`, none for `timestamp`, regardless of the time zone the file's Arrow schema names. INT96 in a `timestamp_ns` column read without a hint, and parquet-rs wraps INT96 values outside 1677-2262 at nanosecond resolution: year 9999 read as a wrong instant without an error. Reading as microseconds would drop the nanoseconds INT96 can hold. Nanosecond columns now get the microsecond hint too, and the type check fails the read in both `timestamp_ns` and `timestamptz_ns`. Upstream has the same hint since apache#2301; no upstream issue or PR covers this as of 2026-10-26. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s NULL
Format version 3 allows promoting `unknown` to any type. The spec says writers omit `unknown` columns, but iceberg-rust's writers store them as a Parquet null column. After the reader started to check file types, such a file failed to read once the column was promoted ("stores it as unknown, which cannot be read as string"). The column now reads as NULL, its only possible value.
No upstream issue or PR covers this as of 2026-10-26.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
unknowncolumns #7, rebased ontolakekeeper-compaction-basenow that fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #8 (which replaced fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #5) has merged. Same three commits; the conflict resolutions are described below.What changes are included in this PR?
Fixes for Parquet timestamp columns that other writers produce, and for stored
unknowncolumns. #7 has the full description of each commit. Review commit by commit.TIMESTAMP(NANOS)in atimestampcolumn 1000 times too large without an error, and a nanosecond UTC file in atimestamptzcolumn failed every read. pandas and pyarrow write such files. The hint now applies only to INT96 columns, and nanosecond values read as the same instant in microseconds. A value with sub-microsecond digits fails the read. Statistics, page index and bloom filters of such a column no longer prune it.timestamptzcolumns as UTC. The hint takes the time zone from the table column instead of the file. INT96 intimestamp_nsandtimestamptz_nscolumns read wrong instants outside 1677-2262; it now fails the read.unknowncolumn promoted to another type as NULL. iceberg-rust's writers storeunknownas a null column, so the type check failed the read after a v3 promotion.Changes from #7 in the rebase
arrow/schema.rsthat the record batch transformer also uses. The nanosecond→microsecond andunknown→any rules are added there. The transformer'sTimestampNanosToMicrosarm runs before its unpromotable-type check.Datum::towith one arrow cast of the key column to the table type. fix(arrow): read INT96 and nanosecond Parquet timestamps by value, check filter columns, and read promotedunknowncolumns #7'spromote_keyis ported into that design: before the cast, the probe marks keys with sub-microsecond digits in a nanosecond column, and those rows match no delete key, as in fix(arrow): read INT96 and nanosecond Parquet timestamps by value, check filter columns, and read promotedunknowncolumns #7.timestamptz. fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #8 already readstimestampfiles intimestamptzcolumns, so INT96 in atimestamptzcolumn no longer failed on this base. The INT96 commit now makes such a column read at the table type directly instead of through that promotion. Its message is reworded accordingly. Its nanosecond-column test still fails without it.Compatibility
timestampandtimestamptzcolumns read as their values; before, they read 1000 times too large or failed. A value with sub-microsecond digits fails the read. Filters on such columns no longer use the file's statistics, page index or bloom filters.timestamptzcolumns as UTC, and intimestampcolumns regardless of the time zone in the file's Arrow schema. Intimestamp_nsandtimestamptz_nscolumns INT96 now fails the read.unknowncolumn promoted to another type reads as null instead of failing.Are these changes tested?
Each commit has the tests listed in #7, and each passes
cargo fmt --check,cargo clippy -p iceberg --all-targets --all-features -- -D warningsandcargo test -p iceberg --lib --all-features(2142 passed, 1 ignored at the head).cargo public-apiforicebergis unchanged. On the head commit, the local CI passes: taplo,cargo fmt, license and dependency-license checks, machete, typos, workspace clippy with-D warnings, the workspace build, the build without default features,make check-public-api, the doctests, andcargo nextest run --all-targets --all-features --workspaceagainst the Docker services (2612 passed, 1 skipped).AI Disclosure
The commits were written with Claude Code, and the rebase conflicts were resolved with it. Areas to check:
TimestampNanosToMicrosarm and fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #8's unpromotable-type check.timestamptznow that fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #8 covers the time-zone promotion, beyond its nanosecond-column fix.🤖 Generated with Claude Code