Repository navigation
feat(parquet): support FINITE_VECTOR in Arrow Go - #1605
Draft
MashrafeeAryan wants to merge 5 commits into
Draft
MashrafeeAryan wants to merge 5 commits into
MashrafeeAryan wants to merge 5 commits into
Conversation
Read and save the size stored in FINITE_VECTOR. Check the list shape before linking its children so a bad schema leaves the original nodes alone.
Use the size in the Parquet label to rebuild each list. Keep missing rows missing and reject lists with the wrong number of values.
Add an opt-in setting to save the size in Parquet. Keep using normal list storage and reject missing or non-finite values inside a present vector.
Use the same schema checks whether the caller passes a finite vector type directly or by pointer. Reject a nil pointer before reading the type.
Stop the direct writer from saving a finite vector whose dictionary hides a NaN or other invalid value. Ordinary list writes still support dictionaries.
Author
|
Hi @rok, I opened this PR for the FINITE_VECTOR implementation. Whenver you have time, would you please review it? Thank you so much! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Arrow Go can already save a fixed-size list as a Parquet
LIST. However, it still needs the saved Arrow schema to recover the list size when reading. WithFINITE_VECTOR(3)in the Parquet schema, Arrow Go can read the column as a fixed-size list of length 3 even when the file has no saved Arrow schema. We are basically trying to save the size in Parquet's own schema.What changes are included in this PR?
FINITE_VECTORpqarrow.WithFiniteVectorLogicalType()so callers can write eligible fixed-size lists withFINITE_VECTOR.NaN, and infinity when writing a present vector. A whole vector can still be null if the column allows it.References: Parquet spec, Arrow C++ PR, and Parquet Java PR.
Are these changes tested?
Yes. The Parquet schema suite and focused
pqarrowtests passed on Windows.Are there any user-facing changes?
Yes. Arrow Go can now read FINITE_VECTOR() in the Parquet schema and turn it into a fixed-size list of length. To save the FINITE_VECTOR label when writing a file, users need to turn on pqarrow.WithFiniteVectorLogicalType(). This option is off by default because I believe older Parquet readers may not be able to open files with new added FINITE_VECTOR() label.
Codex and Gemini assisted with reviewing it.