Repository navigation
Conversation
…y value The INT96 schema hint matched every `Timestamp(ns)` Arrow field, not only INT96 leaves. parquet-rs accepts the same hint for an INT64 `TIMESTAMP(NANOS)` leaf and then reinterprets its nanosecond values as microseconds. A nanosecond file in a `timestamp` column therefore read 1000 times too large without an error (2024-01-01 read as year 55969), and a nanosecond UTC file in a `timestamptz` column failed every read of the file, even when the column was not projected. Such files come from pandas and pyarrow, added with `add_files`. The hint now applies only to leaves that the Parquet schema stores as INT96. A nanosecond value in a `timestamp` or `timestamptz` column reads as the same instant in microseconds, whatever the file's UTC flag, as microsecond files do. A value with sub-microsecond digits fails the read, because truncating it would change the value silently. Every other path that sees the stored nanoseconds uses the same values: - Equality deletes convert the key column with the same conversion as the projection, so they match by value. A key with sub-microsecond digits fails the read, also when the key column is not projected; the plain Arrow cast truncated it, so a delete of the truncated value removed the row. - Row-group statistics, the page index and bloom filters hold nanoseconds, so they no longer prune such columns. They read microsecond literals against nanosecond values and dropped matching rows. - The INT96 hint runs only for files with INT96 leaves, and an error while building it fails the read, so no INT96 leaf can be read without it. Upstream has the same hint since apache#2301; no upstream issue or PR covers this as of 2026-10-26. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…annot read The reader checks the file type of every projected column against the table type, but filters and equality deletes decode their columns outside the projection and skipped that check. On a column the projection refuses, they compared values, statistics and bloom filters of another type or unit: a microsecond file in a `timestamp_ns` column pruned every row group for `ts = <value>` and returned no row, and an equality delete on such a column compared wrongly cast keys. The rows came out wrong without an error whenever the column was not projected. Filters and equality deletes now check their columns with the same rule as the projection, so such a read fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… nanosecond columns INT96 in a `timestamp_ns` or `timestamptz_ns` column read without the INT96 hint, and parquet-rs wraps INT96 values outside 1677-2262 at nanosecond resolution: year 9999 read as a wrong instant without an error. Since the reader accepts `timestamp` files in `timestamptz` columns, the same happens in `timestamptz_ns` columns, which failed before. Reading as microseconds would drop the nanoseconds INT96 can hold. The hint kept the unit and time zone of a stored Arrow schema. pyarrow stores the type it wrote from, so an INT96 column written from `timestamp[ms]` read as milliseconds: files without field IDs dropped the sub-millisecond digits without an error, and files with field IDs failed, as did any time zone other than UTC. The hint now reads every INT96 leaf as microseconds, in UTC if the stored Arrow type names any time zone. INT96 holds UTC instants, and Java ignores the stored zone too. In a nanosecond column the type check fails the read, also for filters and equality deletes on the column. Upstream has the same hint since apache#2301; no upstream issue or PR covers this as of 2026-10-26. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s NULL
Format version 3 allows promoting `unknown` to any type. The spec says writers omit `unknown` columns, but iceberg-rust's writers store them as a Parquet null column. After the reader started to check file types, such a file failed to read once the column was promoted ("stores it as unknown, which cannot be read as string"). The column now reads as NULL, its only possible value, with or without field IDs in the file.
No upstream issue or PR covers this as of 2026-10-26.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
unknown columnsunknown columns
c-thiel
force-pushed
the
ct/fork-timestamp-fixes
branch
from
October 26, 2026 13:44
342de61 to
ba48b98
Compare
c-thiel
changed the base branch from
ct/fork-reader-fixes
to
jk/fork-reader-fixes
October 26, 2026 13:44
|
Closed in favor of #9 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for Parquet timestamp columns that other writers produce, for filters and equality deletes on columns the reader cannot read, and for stored
unknowncolumns. Each fix stops a read that returned wrong rows or values without an error, or makes a file readable that failed. A rewrite of such a file wrote the wrong rows into the new file. This PR is stacked on #8. After #8 merges, this PR is retargeted tolakekeeper-compaction-base. Review commit by commit.#8 already reads
timestampfiles intimestamptzcolumns and the reverse, so INT96 intimestamptzcolumns (Sparkmigrate,snapshotandadd_filestables) reads with #8 alone. This PR no longer sets the INT96 time zone from the table column. Equality-delete keys now use the Arrow cast of #8, with one exact conversion for nanoseconds in microsecond columns.Commits
TIMESTAMP(NANOS)column in atimestampcolumn as microseconds: values came back 1000 times too large without an error (2024-01-01 read as year 55969), and equality deletes on the column never matched, so deleted rows came back. With fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #8, the same happens intimestamptzcolumns for files without UTC adjustment. A nanosecond UTC file in atimestamptzcolumn failed every read of the file. pandas and pyarrow write such files, andadd_filesaccepts them. The hint now applies only to INT96 columns. A nanosecond value reads as the same instant in microseconds, whatever the file's UTC flag; a value with sub-microsecond digits fails the read. Equality-delete keys use the same conversion, so a key with sub-microsecond digits also fails the read when the key column is not projected; the Arrow cast truncated it. Row-group statistics, the page index and bloom filters of such a column hold nanoseconds, so they no longer prune it. The INT96 hint runs only for files with INT96 columns, and an error while building it fails the read. The hint comes from fix: incorrect Parquet INT96 timestamp values from ArrowReader apache/iceberg-rust#2301, so upstream has the same bug; no upstream issue or PR.timestamp_nscolumn returned no row forts = <value>. Filters and equality deletes now check their columns with the same rule, so such a read fails. No upstream issue or PR.timestamp_nscolumn read without the hint, and parquet-rs wraps INT96 values outside 1677-2262 at nanosecond resolution, so year 9999 read as a wrong instant without an error. With fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #8, the same happens intimestamptz_nscolumns. Every INT96 column now reads as microseconds, so the read fails in nanosecond columns. The hint also kept the unit and time zone of a stored Arrow schema: an INT96 column that pyarrow wrote fromtimestamp[ms]dropped its sub-millisecond digits without an error in files without field IDs, and failed with field IDs, as did any time zone other than UTC. INT96 holds UTC instants and Java ignores the stored zone, so such a column now reads as microseconds, in UTC if the stored type names a zone. No upstream issue or PR.unknowncolumn promoted to another type as NULL. Format v3 allows promotingunknownto any type. The spec says writers omitunknowncolumns, but iceberg-rust's writers store them as a null column, so the type check of fix: reader errors and filters on promoted and absent columns, manifest schemas from table metadata, and refs in v1 metadata #5 failed the read after such a promotion. The column now reads as null, its only possible value, with or without field IDs in the file. No upstream issue or PR.Compatibility
timestampandtimestamptzcolumns read as their values; before, they read 1000 times too large or failed. A value with sub-microsecond digits fails the read, also through an equality delete on the column. Filters on such columns no longer use the file's statistics, page index or bloom filters.timestamp_nsandtimestamptz_nscolumns INT96 now fails the read.Tests
Each commit builds and passes
cargo test -p iceberg --all-features --lib, and each has tests that fail without it. New tests:test_read_int64_nanosecond_timestamps_into_microsecond_columns,test_read_sub_microsecond_timestamp_into_microsecond_column_fails,test_no_coercion_of_int64_nanosecond_leaf,test_eq_delete_on_nanosecond_column_in_microsecond_field(key projected and not, file UTC flag equal to and different from the column),test_eq_delete_on_sub_microsecond_key_fails,test_read_file_with_list_view_column,test_pruning_on_nanosecond_column_in_microsecond_field(row-group statistics, page index and bloom filters, each on its own);test_filter_and_eq_delete_on_unreadable_column_fail;test_read_int96_timestamps_as_timestamptz,test_read_int96_timestamps_as_nanoseconds_fails,test_coerce_unit_and_time_zone(replacestest_coerce_timestamptz_ns_to_us;test_no_coercion_when_iceberg_is_timestamp_ns,test_no_coercion_when_iceberg_is_timestamptz_nsandtest_defaults_to_us_when_iceberg_type_is_not_timestampare removed, because the hint no longer depends on the table column);unknownpromotion:test_read_unknown_column_promoted_to_another_type.