Repository navigation
GH-430: [Format] Add FINITE_VECTOR logical type with LIST storage #624
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
c10a4a3
a4bb651
9715b8e
28b762f
ce8f53f
2bdc543
daac474
6c9f73e
90fab07
ec52d25
8206462
8c42418
0596fde
8ad956d
4443ddb
70b3023
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1097,6 +1097,48 @@ optional group my_map (MAP_KEY_VALUE) { | |
| } | ||
| ``` | ||
|
|
||
| ### Finite Vectors | ||
|
|
||
| `FINITE_VECTOR` is used to annotate fixed-length ordered sequences of finite and | ||
| non-null elements. | ||
|
|
||
| `FiniteVectorType` annotation has one required parameter, `num_elements`, | ||
| which is the number of elements in each non-null vector and MUST be greater | ||
| than zero. | ||
|
|
||
| `FINITE_VECTOR` MUST always annotate the canonical 3-level structure: | ||
|
|
||
| ``` | ||
| <vector-repetition> group <name> (FINITE_VECTOR(<num_elements>)) { | ||
| repeated group list { | ||
| required <element-type> element; | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The element types supported are: | ||
|
|
||
| * unannotated `BOOLEAN`, `FLOAT` or `DOUBLE` | ||
| * `INT32` or `INT64`, either unannotated or annotated with `INTEGER` or | ||
| `DECIMAL` | ||
| * `FIXED_LEN_BYTE_ARRAY` annotated with `FLOAT16` or `DECIMAL`. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just curious if
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For completeness, it could be added, but I'd leave it out. Every supported element type is fixed width. Any decimal precision can already be stored as
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fixed width allows for some optimizations that variable width doesn't. I believe allowing variable-width types would make writers more complex as they'd have to enable or disable these optimizations at write time. |
||
|
|
||
| Every numeric element MUST be finite. For `FLOAT`, `DOUBLE`, and `FLOAT16`, | ||
| this excludes NaN, positive infinity, and negative infinity. | ||
|
|
||
| For example, a nullable vector containing 768 required `FLOAT` elements is: | ||
|
|
||
| ``` | ||
| optional group embedding (FINITE_VECTOR(768)) { | ||
| repeated group list { | ||
| required float element; | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| Every non-null vector must contain exactly `num_elements` elements. | ||
| Because `num_elements` is greater than zero, a non-null vector cannot be empty. | ||
|
|
||
| ## UNKNOWN (always null) | ||
|
|
||
| Sometimes, when discovering the schema of existing data, values are always null | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧑🍳 👌