Repository navigation
Fixture: Add reference table for primitive types read from Parquet - #29
laskoviymishka wants to merge 2 commits into
Conversation
…from Parquet Part of apache#10. The reference tables have no column of most primitive types, so nothing checks that a reader decodes boolean, float, decimal, date, time, timestamp, uuid, fixed or binary from Parquet. This adds `v2/primitive-types-parquet-data`, one column of every v1 and v2 primitive type. The one snapshot appends `a.parquet` with three rows. Row 1 holds typical values, with the int and long maxima. Row 2 holds the boundaries: the int and long minima, negative decimals that use every digit of their precision, a date and timestamps one microsecond before the epoch, `23:59:59.999999`, an empty string and an empty binary. Row 3 is null in every optional column. The trap is in the physical encodings. The spec maps `decimal(P,S)` to "`P <= 9`: `int32`, `P <= 18`: `int64`, `fixed` otherwise", and the table has one column of each, so a reader that decodes only one of them fails on the others. `timestamp` and `timestamptz` differ only in `TIMESTAMP_MICROS` with `adjustToUtc=false` or `true`, so a reader that ignores the flag returns one of them with the wrong type or shifted by its local zone. The Parquet footer shows the expected physical types: INT32, INT64 and FIXED_LEN_BYTE_ARRAY(16) for the decimals, and isAdjustedToUTC false and true for the two timestamps. `expected/scan.json` is derived from the README Input. Decimals are the spec's decimal string, which carries the `decimal(38, 10)` value whose unscaled value overflows 64 bits. `fixed` and `binary` are uppercase hex, following CONTRIBUTING.md, while the spec's example is lowercase. A uuid is lowercase, as the spec says. Every float and double is exactly representable, and NaN, the infinities and -0.0 are left out, because the single-value serialization gives no JSON form for the first three and a multiset comparison of decoded doubles cannot tell -0.0 from 0.0. `GenerateTables.java` gains input conversions for boolean, float, decimal, date, time, timestamp without zone, uuid, fixed and binary. A decimal must have exactly its type's scale, and binary and fixed values must be uppercase hex, with a fixed value of its type's length. No commit key is added. The existing tables generate unchanged: regenerated, their data file rows, snapshot summaries, schemas and partition specs match the committed ones. Checked with a raw fastavro/pyarrow reader, PyIceberg 0.12.0 and HEAD (33b4ce5), Iceberg Java 1.11.0 and 1.12.0, iceberg-go HEAD (a9e2cc3), and iceberg-rust 0.11.0 and HEAD (eeb1e41). All of them match `expected/scan.json`. Every scan ran with the local zone set to Asia/Kolkata, so a reader that applied the local zone would have shown up. The table is 20.4 KB. Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
nssalian
left a comment
There was a problem hiding this comment.
Ran this locally, using AI: raw fastavro/pyarrow, PyIceberg 0.12.0, Java 1.12.0, iceberg-go and iceberg-rust 0.10.1 and 0.11.0 all return expected/scan.json, and the footer has the INT32/INT64/FIXED_LEN_BYTE_ARRAY decimals and both isAdjustedToUTC values. The new value() cases also leave the existing tables' Input rows unchanged. Two small wording notes inline, neither blocking. LGTM.
| otherwise. A reader that decodes only one of them fails on the others. The | ||
| `timestamp` and `timestamptz` columns differ only in the `TIMESTAMP_MICROS` | ||
| annotation's `isAdjustedToUTC` flag, so a reader that ignores the flag returns | ||
| one of them with the wrong type, or shifted by its local zone. |
There was a problem hiding this comment.
nit: timestamp_col and timestamptz_col hold the same micros in the file, and the Procedure decodes each column by its type in the Iceberg schema. A reader that ignores isAdjustedToUTC but takes the type from the schema still returns the right rows, and a zone shift only shows when the check runs outside UTC. Could this sentence say the check should run in a non-UTC zone, or limit the claim to readers that take the type from the Parquet annotation?
| from its Input, and a PyIceberg 0.12.0 scan returns the same rows. The rows of | ||
| `v2/primitive-types-parquet-data` were also derived from its Input, then checked | ||
| against the same read with no Iceberg library and against scans with PyIceberg, | ||
| Iceberg Java, iceberg-go and iceberg-rust. |
There was a problem hiding this comment.
nit: the sentence before pins PyIceberg 0.12.0, could this one carry the versions too (PyIceberg 0.12.0, Java 1.11.0 and 1.12.0, iceberg-go a9e2cc3, iceberg-rust 0.11.0)? Right now they're only in the PR description.
|
@sungwy can you plz also take a look? |
Signed-off-by: laskoviymishka <laskoviymishka@gmail.com>
|
Thanks @laskoviymishka - on a tangent, reviewing this PR got me thinking about: #30 Could I ask for your review on it? I think it'll make contributing and maintaining reference tables more efficient in the longer run. |
sungwy
left a comment
There was a problem hiding this comment.
Just one small nit, but otherwise LGTM
| for them. A -0.0 compares equal to 0.0 under most decoded-double equality, so a | ||
| multiset comparison would not tell them apart. These are open questions for the | ||
| dev list. | ||
|
|
There was a problem hiding this comment.
Should we add a dedicated Scope section now that #30 is in?
This comment has been minimized.
This comment has been minimized.
|
@slachiewicz appreciate the suggestions but we should let the author of the PR push the changes. Reviewing and suggestions are most welcome but we shouldn't be trying to push to their fork. |
This adds
v2/primitive-types-parquet-data, a v2 table with one append of one Parquet file holding three rows (typical values, boundaries, all null) across every v1 and v2 primitive type. No existing table checks that a reader decodes boolean, float, decimal, date, time, timestamp, uuid, fixed or binary from Parquet. The trap is in the physical encodings: the three decimal columns are stored as INT32, INT64 and FIXED_LEN_BYTE_ARRAY, andtimestampandtimestamptzdiffer only inisAdjustedToUTC, so a reader that handles one decimal encoding or ignores the zone flag returns different rows.expected/scan.jsonis derived from the README Input.GenerateTables.javagains input conversions for the missing types; the existing tables still generate with the same rows, schemas and summaries.The Parquet footer was checked for the expected physical types, and the table was read raw with fastavro and pyarrow and scanned with PyIceberg 0.12.0 and HEAD (33b4ce5), Iceberg Java 1.11.0 and 1.12.0, iceberg-go HEAD (a9e2cc3), and iceberg-rust 0.11.0 and HEAD (eeb1e41), all with the local zone set to Asia/Kolkata. All match
expected/scan.json.make checkpasses, and the table is 20.4 KB.NaN, the infinities and -0.0 are left out, since the single-value serialization gives the first three no JSON form and a multiset comparison of decoded doubles cannot tell -0.0 from 0.0. They are open questions for the dev list, as is how to write a float whose decimal text does not round-trip through binary32.
Part of #10.
AI-assisted, reviewed by a human.