Skip to content

Fixture: Add reference table for primitive types read from Parquet - #29

Open
laskoviymishka wants to merge 2 commits into
apache:mainfrom
laskoviymishka:factory/t1-primitives
Open

laskoviymishka wants to merge 2 commits into
apache:mainfrom
laskoviymishka:factory/t1-primitives

Conversation

@laskoviymishka

Copy link
Copy Markdown
Contributor

This adds v2/primitive-types-parquet-data, a v2 table with one append of one Parquet file holding three rows (typical values, boundaries, all null) across every v1 and v2 primitive type. No existing table checks that a reader decodes boolean, float, decimal, date, time, timestamp, uuid, fixed or binary from Parquet. The trap is in the physical encodings: the three decimal columns are stored as INT32, INT64 and FIXED_LEN_BYTE_ARRAY, and timestamp and timestamptz differ only in isAdjustedToUTC, so a reader that handles one decimal encoding or ignores the zone flag returns different rows. expected/scan.json is derived from the README Input. GenerateTables.java gains input conversions for the missing types; the existing tables still generate with the same rows, schemas and summaries.

The Parquet footer was checked for the expected physical types, and the table was read raw with fastavro and pyarrow and scanned with PyIceberg 0.12.0 and HEAD (33b4ce5), Iceberg Java 1.11.0 and 1.12.0, iceberg-go HEAD (a9e2cc3), and iceberg-rust 0.11.0 and HEAD (eeb1e41), all with the local zone set to Asia/Kolkata. All match expected/scan.json. make check passes, and the table is 20.4 KB.

NaN, the infinities and -0.0 are left out, since the single-value serialization gives the first three no JSON form and a multiset comparison of decoded doubles cannot tell -0.0 from 0.0. They are open questions for the dev list, as is how to write a float whose decimal text does not round-trip through binary32.

Part of #10.

AI-assisted, reviewed by a human.

…from Parquet

Part of apache#10.

The reference tables have no column of most primitive types, so nothing checks
that a reader decodes boolean, float, decimal, date, time, timestamp, uuid,
fixed or binary from Parquet. This adds `v2/primitive-types-parquet-data`, one
column of every v1 and v2 primitive type.

The one snapshot appends `a.parquet` with three rows. Row 1 holds typical
values, with the int and long maxima. Row 2 holds the boundaries: the int and
long minima, negative decimals that use every digit of their precision, a date
and timestamps one microsecond before the epoch, `23:59:59.999999`, an empty
string and an empty binary. Row 3 is null in every optional column. The trap is
in the physical encodings. The spec maps `decimal(P,S)` to "`P <= 9`: `int32`,
`P <= 18`: `int64`, `fixed` otherwise", and the table has one column of each,
so a reader that decodes only one of them fails on the others. `timestamp` and
`timestamptz` differ only in `TIMESTAMP_MICROS` with `adjustToUtc=false` or
`true`, so a reader that ignores the flag returns one of them with the wrong
type or shifted by its local zone. The Parquet footer shows the expected
physical types: INT32, INT64 and FIXED_LEN_BYTE_ARRAY(16) for the decimals,
and isAdjustedToUTC false and true for the two timestamps.

`expected/scan.json` is derived from the README Input. Decimals are the spec's
decimal string, which carries the `decimal(38, 10)` value whose unscaled value
overflows 64 bits. `fixed` and `binary` are uppercase hex, following
CONTRIBUTING.md, while the spec's example is lowercase. A uuid is lowercase,
as the spec says. Every float and double is exactly representable, and NaN,
the infinities and -0.0 are left out, because the single-value serialization
gives no JSON form for the first three and a multiset comparison of decoded
doubles cannot tell -0.0 from 0.0.

`GenerateTables.java` gains input conversions for boolean, float, decimal,
date, time, timestamp without zone, uuid, fixed and binary. A decimal must
have exactly its type's scale, and binary and fixed values must be uppercase
hex, with a fixed value of its type's length. No commit key is added. The
existing tables generate unchanged: regenerated, their data file rows,
snapshot summaries, schemas and partition specs match the committed ones.

Checked with a raw fastavro/pyarrow reader, PyIceberg 0.12.0 and HEAD
(33b4ce5), Iceberg Java 1.11.0 and 1.12.0, iceberg-go HEAD (a9e2cc3), and
iceberg-rust 0.11.0 and HEAD (eeb1e41). All of them match
`expected/scan.json`. Every scan ran with the local zone set to Asia/Kolkata,
so a reader that applied the local zone would have shown up. The table is
20.4 KB.

Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
@laskoviymishka
laskoviymishka marked this pull request as ready for review November 16, 2026 21:23

@nssalian nssalian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran this locally, using AI: raw fastavro/pyarrow, PyIceberg 0.12.0, Java 1.12.0, iceberg-go and iceberg-rust 0.10.1 and 0.11.0 all return expected/scan.json, and the footer has the INT32/INT64/FIXED_LEN_BYTE_ARRAY decimals and both isAdjustedToUTC values. The new value() cases also leave the existing tables' Input rows unchanged. Two small wording notes inline, neither blocking. LGTM.

otherwise. A reader that decodes only one of them fails on the others. The
`timestamp` and `timestamptz` columns differ only in the `TIMESTAMP_MICROS`
annotation's `isAdjustedToUTC` flag, so a reader that ignores the flag returns
one of them with the wrong type, or shifted by its local zone.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: timestamp_col and timestamptz_col hold the same micros in the file, and the Procedure decodes each column by its type in the Iceberg schema. A reader that ignores isAdjustedToUTC but takes the type from the schema still returns the right rows, and a zone shift only shows when the check runs outside UTC. Could this sentence say the check should run in a non-UTC zone, or limit the claim to readers that take the type from the Parquet annotation?

from its Input, and a PyIceberg 0.12.0 scan returns the same rows. The rows of
`v2/primitive-types-parquet-data` were also derived from its Input, then checked
against the same read with no Iceberg library and against scans with PyIceberg,
Iceberg Java, iceberg-go and iceberg-rust.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the sentence before pins PyIceberg 0.12.0, could this one carry the versions too (PyIceberg 0.12.0, Java 1.11.0 and 1.12.0, iceberg-go a9e2cc3, iceberg-rust 0.11.0)? Right now they're only in the PR description.

@laskoviymishka

Copy link
Copy Markdown
Contributor Author

@sungwy can you plz also take a look?

Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
@sungwy

sungwy commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Thanks @laskoviymishka - on a tangent, reviewing this PR got me thinking about: #30

Could I ask for your review on it? I think it'll make contributing and maintaining reference tables more efficient in the longer run.

@sungwy sungwy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one small nit, but otherwise LGTM

for them. A -0.0 compares equal to 0.0 under most decoded-double equality, so a
multiset comparison would not tell them apart. These are open questions for the
dev list.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a dedicated Scope section now that #30 is in?

@sungwy sungwy changed the title Add reference table for primitive types read from Parquet Fixture: Add reference table for primitive types read from Parquet Nov 19, 2026
@github-actions github-actions Bot added the fixtures Surfaces, cases, reference tables, and fixture corrections label Nov 19, 2026
@slachiewicz

This comment has been minimized.

@nssalian

Copy link
Copy Markdown
Collaborator

@slachiewicz appreciate the suggestions but we should let the author of the PR push the changes. Reviewing and suggestions are most welcome but we shouldn't be trying to push to their fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fixtures Surfaces, cases, reference tables, and fixture corrections

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants