Skip to content

fix(arrow): read INT96 and nanosecond Parquet timestamps by value, and promoted unknown columns - #9

Merged
JanKaul merged 3 commits into
lakekeeper-compaction-basefrom
jk/fork-timestamp-fixes
Oct 26, 2026
Merged

JanKaul merged 3 commits into
lakekeeper-compaction-basefrom
jk/fork-timestamp-fixes

Conversation

@JanKaul

@JanKaul JanKaul commented Oct 26, 2026 •

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

Fixes for Parquet timestamp columns that other writers produce, and for stored unknown columns. #7 has the full description of each commit. Review commit by commit.

  • Read INT64 nanosecond timestamps in microsecond columns by value. The INT96 schema hint matched every nanosecond Arrow field, so parquet-rs read INT64 TIMESTAMP(NANOS) in a timestamp column 1000 times too large without an error, and a nanosecond UTC file in a timestamptz column failed every read. pandas and pyarrow write such files. The hint now applies only to INT96 columns, and nanosecond values read as the same instant in microseconds. A value with sub-microsecond digits fails the read. Statistics, page index and bloom filters of such a column no longer prune it.
  • Read INT96 timestamps in timestamptz columns as UTC. The hint takes the time zone from the table column instead of the file. INT96 in timestamp_ns and timestamptz_ns columns read wrong instants outside 1677-2262; it now fails the read.
  • Read a stored unknown column promoted to another type as NULL. iceberg-rust's writers store unknown as a null column, so the type check failed the read after a v3 promotion.

Changes from #7 in the rebase

Compatibility

  • INT64 nanosecond timestamps in timestamp and timestamptz columns read as their values; before, they read 1000 times too large or failed. A value with sub-microsecond digits fails the read. Filters on such columns no longer use the file's statistics, page index or bloom filters.
  • INT96 timestamps read in timestamptz columns as UTC, and in timestamp columns regardless of the time zone in the file's Arrow schema. In timestamp_ns and timestamptz_ns columns INT96 now fails the read.
  • A stored unknown column promoted to another type reads as null instead of failing.
  • No public item is added, removed or changed.

Are these changes tested?

Each commit has the tests listed in #7, and each passes cargo fmt --check, cargo clippy -p iceberg --all-targets --all-features -- -D warnings and cargo test -p iceberg --lib --all-features (2142 passed, 1 ignored at the head). cargo public-api for iceberg is unchanged. On the head commit, the local CI passes: taplo, cargo fmt, license and dependency-license checks, machete, typos, workspace clippy with -D warnings, the workspace build, the build without default features, make check-public-api, the doctests, and cargo nextest run --all-targets --all-features --workspace against the Docker services (2612 passed, 1 skipped).

AI Disclosure

The commits were written with Claude Code, and the rebase conflicts were resolved with it. Areas to check:

🤖 Generated with Claude Code

c-thiel and others added 3 commits October 26, 2026 14:34
…y value

The INT96 schema hint matched every `Timestamp(ns)` Arrow field, not only INT96 leaves. parquet-rs accepts the same hint for an INT64 `TIMESTAMP(NANOS)` leaf and then reinterprets its nanosecond values as microseconds. A nanosecond file in a `timestamp` column therefore read 1000 times too large without an error (2024-01-01 read as year 55969), and a nanosecond UTC file in a `timestamptz` column failed every read of the file, even when the column was not projected. Such files come from pandas and pyarrow, added with `add_files`.

The hint now applies only to leaves that the Parquet schema stores as INT96. A nanosecond value in a `timestamp` or `timestamptz` column reads as the same instant in microseconds; a value with sub-microsecond digits fails the read, because truncating it would change the value silently. Every other path that sees the stored nanoseconds uses the same values:

- Equality deletes compare the key converted to microseconds, with the table column's UTC flag, as microsecond files do. A key with sub-microsecond digits matches no delete.
- Row-group statistics, the page index and bloom filters hold nanoseconds, so they no longer prune such columns. They read microsecond literals against nanosecond values and dropped matching rows.
- The INT96 hint runs only for files with INT96 leaves, and an error while building it fails the read, so no INT96 leaf can be read without it.

Upstream has the same hint since apache#2301; no upstream issue or PR covers this as of 2026-10-26.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The INT96 schema hint kept the time zone of the file's Arrow type, which is none for files without an Arrow schema. An INT96 column in a `timestamptz` column was therefore read as `timestamp` and reached the table type only through the reader's `timestamp` to `timestamptz` promotion. pyarrow files that store a UTC Arrow type took the opposite promotion in `timestamp` columns.

INT96 holds UTC instants, and Java reads it into either column type. The hint now takes the time zone from the Iceberg column: UTC for `timestamptz`, none for `timestamp`, regardless of the time zone the file's Arrow schema names.

INT96 in a `timestamp_ns` column read without a hint, and parquet-rs wraps INT96 values outside 1677-2262 at nanosecond resolution: year 9999 read as a wrong instant without an error. Reading as microseconds would drop the nanoseconds INT96 can hold. Nanosecond columns now get the microsecond hint too, and the type check fails the read in both `timestamp_ns` and `timestamptz_ns`.

Upstream has the same hint since apache#2301; no upstream issue or PR covers this as of 2026-10-26.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s NULL

Format version 3 allows promoting `unknown` to any type. The spec says writers omit `unknown` columns, but iceberg-rust's writers store them as a Parquet null column. After the reader started to check file types, such a file failed to read once the column was promoted ("stores it as unknown, which cannot be read as string"). The column now reads as NULL, its only possible value.

No upstream issue or PR covers this as of 2026-10-26.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants