Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions LogicalTypes.md
Original file line number Diff line number Diff line change
Expand Up @@ -1097,6 +1097,48 @@ optional group my_map (MAP_KEY_VALUE) {
}
```

### Finite Vectors

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧑‍🍳 👌


`FINITE_VECTOR` is used to annotate fixed-length ordered sequences of finite and
non-null elements.

`FiniteVectorType` annotation has one required parameter, `num_elements`,
which is the number of elements in each non-null vector and MUST be greater
than zero.

`FINITE_VECTOR` MUST always annotate the canonical 3-level structure:

```
<vector-repetition> group <name> (FINITE_VECTOR(<num_elements>)) {
repeated group list {
required <element-type> element;
}
}
```

The element types supported are:

* unannotated `BOOLEAN`, `FLOAT` or `DOUBLE`
* `INT32` or `INT64`, either unannotated or annotated with `INTEGER` or
`DECIMAL`
* `FIXED_LEN_BYTE_ARRAY` annotated with `FLOAT16` or `DECIMAL`.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious if BYTE_ARRAY annotated with DECIMAL should be supported.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For completeness, it could be added, but I'd leave it out. Every supported element type is fixed width. Any decimal precision can already be stored as FIXED_LEN_BYTE_ARRAY, so excluding BYTE_ARRAY loses nothing.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed width allows for some optimizations that variable width doesn't. I believe allowing variable-width types would make writers more complex as they'd have to enable or disable these optimizations at write time.
If it turns out there's a need for this we can revisit later too!


Every numeric element MUST be finite. For `FLOAT`, `DOUBLE`, and `FLOAT16`,
this excludes NaN, positive infinity, and negative infinity.

For example, a nullable vector containing 768 required `FLOAT` elements is:

```
optional group embedding (FINITE_VECTOR(768)) {
repeated group list {
required float element;
}
}
```

Every non-null vector must contain exactly `num_elements` elements.
Because `num_elements` is greater than zero, a non-null vector cannot be empty.

## UNKNOWN (always null)

Sometimes, when discovering the schema of existing data, values are always null
Expand Down
15 changes: 15 additions & 0 deletions src/main/thrift/parquet.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -479,6 +479,19 @@ struct GeographyType {
struct FileType {
}

/**
* Finite fixed-length vector logical type annotation
*
* Annotates the outer group of a three-level structure that is the same as
* the one used for LIST-annotated groups. Each non-null vector MUST contain
* exactly num_elements elements.
*
* See LogicalTypes.md for details.
*/
struct FiniteVectorType {
1: required i32 num_elements
}

/**
* LogicalType annotations to replace ConvertedType.
*
Expand Down Expand Up @@ -513,6 +526,7 @@ union LogicalType {
17: GeometryType GEOMETRY // no compatible ConvertedType
18: GeographyType GEOGRAPHY // no compatible ConvertedType
19: FileType FILE // no compatible ConvertedType
20: FiniteVectorType FINITE_VECTOR // use ConvertedType LIST
}

/**
Expand Down Expand Up @@ -1129,6 +1143,7 @@ union ColumnOrder {
* GEOMETRY - undefined
* GEOGRAPHY - undefined
* FILE - undefined
* FINITE_VECTOR - undefined
*
* In the absence of logical types, the sort order is determined by the physical type:
* BOOLEAN - false, true
Expand Down
Loading