Skip to content

feat(parquet): support FINITE_VECTOR in Arrow Go - #1605

Draft
MashrafeeAryan wants to merge 5 commits into
apache:mainfrom
MashrafeeAryan:parquet-finite-vector
Draft

MashrafeeAryan wants to merge 5 commits into
apache:mainfrom
MashrafeeAryan:parquet-finite-vector

Conversation

@MashrafeeAryan

Copy link
Copy Markdown

Rationale for this change

Arrow Go can already save a fixed-size list as a Parquet LIST. However, it still needs the saved Arrow schema to recover the list size when reading. With FINITE_VECTOR(3) in the Parquet schema, Arrow Go can read the column as a fixed-size list of length 3 even when the file has no saved Arrow schema. We are basically trying to save the size in Parquet's own schema.

What changes are included in this PR?

  • Read and write the FINITE_VECTOR
  • Add pqarrow.WithFiniteVectorLogicalType() so callers can write eligible fixed-size lists with FINITE_VECTOR.
  • Reject null elements, NaN, and infinity when writing a present vector. A whole vector can still be null if the column allows it.

References: Parquet spec, Arrow C++ PR, and Parquet Java PR.

Are these changes tested?

Yes. The Parquet schema suite and focused pqarrow tests passed on Windows.

Are there any user-facing changes?

Yes. Arrow Go can now read FINITE_VECTOR() in the Parquet schema and turn it into a fixed-size list of length. To save the FINITE_VECTOR label when writing a file, users need to turn on pqarrow.WithFiniteVectorLogicalType(). This option is off by default because I believe older Parquet readers may not be able to open files with new added FINITE_VECTOR() label.

Codex and Gemini assisted with reviewing it.

Read and save the size stored in FINITE_VECTOR. Check the list shape before linking its children so a bad schema leaves the original nodes alone.
Use the size in the Parquet label to rebuild each list. Keep missing rows missing and reject lists with the wrong number of values.
Add an opt-in setting to save the size in Parquet. Keep using normal list storage and reject missing or non-finite values inside a present vector.
Use the same schema checks whether the caller passes a finite vector type directly or by pointer. Reject a nil pointer before reading the type.
Stop the direct writer from saving a finite vector whose dictionary hides a NaN or other invalid value. Ordinary list writes still support dictionaries.
@MashrafeeAryan

Copy link
Copy Markdown
Author

Hi @rok, I opened this PR for the FINITE_VECTOR implementation. Whenver you have time, would you please review it? Thank you so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant