Skip to content

fix(arrow): read INT96 and nanosecond Parquet timestamps by value, check filter columns, and read promoted unknown columns - #7

Closed
c-thiel wants to merge 4 commits into
jk/fork-reader-fixesfrom
ct/fork-timestamp-fixes
Closed

c-thiel wants to merge 4 commits into
jk/fork-reader-fixesfrom
ct/fork-timestamp-fixes

Conversation

@c-thiel

@c-thiel c-thiel commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

Fixes for Parquet timestamp columns that other writers produce, for filters and equality deletes on columns the reader cannot read, and for stored unknown columns. Each fix stops a read that returned wrong rows or values without an error, or makes a file readable that failed. A rewrite of such a file wrote the wrong rows into the new file. This PR is stacked on #8. After #8 merges, this PR is retargeted to lakekeeper-compaction-base. Review commit by commit.

#8 already reads timestamp files in timestamptz columns and the reverse, so INT96 in timestamptz columns (Spark migrate, snapshot and add_files tables) reads with #8 alone. This PR no longer sets the INT96 time zone from the table column. Equality-delete keys now use the Arrow cast of #8, with one exact conversion for nanoseconds in microsecond columns.

Commits

  • Read INT64 nanosecond timestamps in microsecond columns by value. The schema hint that reads INT96 at microseconds matched every nanosecond Arrow field, not only INT96 columns. parquet-rs then read an INT64 TIMESTAMP(NANOS) column in a timestamp column as microseconds: values came back 1000 times too large without an error (2024-01-01 read as year 55969), and equality deletes on the column never matched, so deleted rows came back. With fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #8, the same happens in timestamptz columns for files without UTC adjustment. A nanosecond UTC file in a timestamptz column failed every read of the file. pandas and pyarrow write such files, and add_files accepts them. The hint now applies only to INT96 columns. A nanosecond value reads as the same instant in microseconds, whatever the file's UTC flag; a value with sub-microsecond digits fails the read. Equality-delete keys use the same conversion, so a key with sub-microsecond digits also fails the read when the key column is not projected; the Arrow cast truncated it. Row-group statistics, the page index and bloom filters of such a column hold nanoseconds, so they no longer prune it. The INT96 hint runs only for files with INT96 columns, and an error while building it fails the read. The hint comes from fix: incorrect Parquet INT96 timestamp values from ArrowReader apache/iceberg-rust#2301, so upstream has the same bug; no upstream issue or PR.
  • Fail filters and equality deletes on columns the reader cannot read. The reader checks the file type of each projected column, but filters and equality deletes read their columns outside the projection and skipped the check. On a column the projection refuses, they compared values, statistics and bloom filters of another type or unit: a microsecond file in a timestamp_ns column returned no row for ts = <value>. Filters and equality deletes now check their columns with the same rule, so such a read fails. No upstream issue or PR.
  • Read INT96 timestamps as microseconds in UTC, and fail in nanosecond columns. INT96 in a timestamp_ns column read without the hint, and parquet-rs wraps INT96 values outside 1677-2262 at nanosecond resolution, so year 9999 read as a wrong instant without an error. With fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #8, the same happens in timestamptz_ns columns. Every INT96 column now reads as microseconds, so the read fails in nanosecond columns. The hint also kept the unit and time zone of a stored Arrow schema: an INT96 column that pyarrow wrote from timestamp[ms] dropped its sub-millisecond digits without an error in files without field IDs, and failed with field IDs, as did any time zone other than UTC. INT96 holds UTC instants and Java ignores the stored zone, so such a column now reads as microseconds, in UTC if the stored type names a zone. No upstream issue or PR.
  • Read a stored unknown column promoted to another type as NULL. Format v3 allows promoting unknown to any type. The spec says writers omit unknown columns, but iceberg-rust's writers store them as a null column, so the type check of fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #5 failed the read after such a promotion. The column now reads as null, its only possible value, with or without field IDs in the file. No upstream issue or PR.

Compatibility

  • INT64 nanosecond timestamps in timestamp and timestamptz columns read as their values; before, they read 1000 times too large or failed. A value with sub-microsecond digits fails the read, also through an equality delete on the column. Filters on such columns no longer use the file's statistics, page index or bloom filters.
  • A filter or equality delete on a column whose file type the reader cannot read as the table type fails the read, as reading the column does.
  • INT96 timestamps read as microseconds whatever unit or time zone the file's Arrow schema names. As in Java, INT96 digits below a microsecond are dropped. In timestamp_ns and timestamptz_ns columns INT96 now fails the read.
  • No public item is added, removed or changed.

Tests

Each commit builds and passes cargo test -p iceberg --all-features --lib, and each has tests that fail without it. New tests:

  • nanosecond timestamps: test_read_int64_nanosecond_timestamps_into_microsecond_columns, test_read_sub_microsecond_timestamp_into_microsecond_column_fails, test_no_coercion_of_int64_nanosecond_leaf, test_eq_delete_on_nanosecond_column_in_microsecond_field (key projected and not, file UTC flag equal to and different from the column), test_eq_delete_on_sub_microsecond_key_fails, test_read_file_with_list_view_column, test_pruning_on_nanosecond_column_in_microsecond_field (row-group statistics, page index and bloom filters, each on its own);
  • filter columns: test_filter_and_eq_delete_on_unreadable_column_fail;
  • INT96: test_read_int96_timestamps_as_timestamptz, test_read_int96_timestamps_as_nanoseconds_fails, test_coerce_unit_and_time_zone (replaces test_coerce_timestamptz_ns_to_us; test_no_coercion_when_iceberg_is_timestamp_ns, test_no_coercion_when_iceberg_is_timestamptz_ns and test_defaults_to_us_when_iceberg_type_is_not_timestamp are removed, because the hint no longer depends on the table column);
  • unknown promotion: test_read_unknown_column_promoted_to_another_type.

c-thiel and others added 4 commits October 26, 2026 13:27
…y value

The INT96 schema hint matched every `Timestamp(ns)` Arrow field, not only INT96 leaves. parquet-rs accepts the same hint for an INT64 `TIMESTAMP(NANOS)` leaf and then reinterprets its nanosecond values as microseconds. A nanosecond file in a `timestamp` column therefore read 1000 times too large without an error (2024-01-01 read as year 55969), and a nanosecond UTC file in a `timestamptz` column failed every read of the file, even when the column was not projected. Such files come from pandas and pyarrow, added with `add_files`.

The hint now applies only to leaves that the Parquet schema stores as INT96. A nanosecond value in a `timestamp` or `timestamptz` column reads as the same instant in microseconds, whatever the file's UTC flag, as microsecond files do. A value with sub-microsecond digits fails the read, because truncating it would change the value silently. Every other path that sees the stored nanoseconds uses the same values:

- Equality deletes convert the key column with the same conversion as the projection, so they match by value. A key with sub-microsecond digits fails the read, also when the key column is not projected; the plain Arrow cast truncated it, so a delete of the truncated value removed the row.
- Row-group statistics, the page index and bloom filters hold nanoseconds, so they no longer prune such columns. They read microsecond literals against nanosecond values and dropped matching rows.
- The INT96 hint runs only for files with INT96 leaves, and an error while building it fails the read, so no INT96 leaf can be read without it.

Upstream has the same hint since apache#2301; no upstream issue or PR covers this as of 2026-10-26.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…annot read

The reader checks the file type of every projected column against the table type, but filters and equality deletes decode their columns outside the projection and skipped that check. On a column the projection refuses, they compared values, statistics and bloom filters of another type or unit: a microsecond file in a `timestamp_ns` column pruned every row group for `ts = <value>` and returned no row, and an equality delete on such a column compared wrongly cast keys. The rows came out wrong without an error whenever the column was not projected.

Filters and equality deletes now check their columns with the same rule as the projection, so such a read fails.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… nanosecond columns

INT96 in a `timestamp_ns` or `timestamptz_ns` column read without the INT96 hint, and parquet-rs wraps INT96 values outside 1677-2262 at nanosecond resolution: year 9999 read as a wrong instant without an error. Since the reader accepts `timestamp` files in `timestamptz` columns, the same happens in `timestamptz_ns` columns, which failed before. Reading as microseconds would drop the nanoseconds INT96 can hold.

The hint kept the unit and time zone of a stored Arrow schema. pyarrow stores the type it wrote from, so an INT96 column written from `timestamp[ms]` read as milliseconds: files without field IDs dropped the sub-millisecond digits without an error, and files with field IDs failed, as did any time zone other than UTC.

The hint now reads every INT96 leaf as microseconds, in UTC if the stored Arrow type names any time zone. INT96 holds UTC instants, and Java ignores the stored zone too. In a nanosecond column the type check fails the read, also for filters and equality deletes on the column.

Upstream has the same hint since apache#2301; no upstream issue or PR covers this as of 2026-10-26.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s NULL

Format version 3 allows promoting `unknown` to any type. The spec says writers omit `unknown` columns, but iceberg-rust's writers store them as a Parquet null column. After the reader started to check file types, such a file failed to read once the column was promoted ("stores it as unknown, which cannot be read as string"). The column now reads as NULL, its only possible value, with or without field IDs in the file.

No upstream issue or PR covers this as of 2026-10-26.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@c-thiel c-thiel changed the title fix(arrow): read INT96 and nanosecond Parquet timestamps by value, and promoted unknown columns fix(arrow): read INT96 and nanosecond Parquet timestamps by value, check filter columns, and read promoted unknown columns Oct 26, 2026
@c-thiel
c-thiel force-pushed the ct/fork-timestamp-fixes branch from 342de61 to ba48b98 Compare October 26, 2026 13:44
@c-thiel
c-thiel changed the base branch from ct/fork-reader-fixes to jk/fork-reader-fixes October 26, 2026 13:44
@JanKaul

JanKaul commented Oct 26, 2026

Copy link
Copy Markdown

Closed in favor of #9

@JanKaul JanKaul closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants