diff --git a/LogicalTypes.md b/LogicalTypes.md index 559030fe..a5cf0843 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -1097,6 +1097,48 @@ optional group my_map (MAP_KEY_VALUE) { } ``` +### Finite Vectors + +`FINITE_VECTOR` is used to annotate fixed-length ordered sequences of finite and +non-null elements. + +`FiniteVectorType` annotation has one required parameter, `num_elements`, +which is the number of elements in each non-null vector and MUST be greater +than zero. + +`FINITE_VECTOR` MUST always annotate the canonical 3-level structure: + +``` + group (FINITE_VECTOR()) { + repeated group list { + required element; + } +} +``` + +The element types supported are: + +* unannotated `BOOLEAN`, `FLOAT` or `DOUBLE` +* `INT32` or `INT64`, either unannotated or annotated with `INTEGER` or + `DECIMAL` +* `FIXED_LEN_BYTE_ARRAY` annotated with `FLOAT16` or `DECIMAL`. + +Every numeric element MUST be finite. For `FLOAT`, `DOUBLE`, and `FLOAT16`, +this excludes NaN, positive infinity, and negative infinity. + +For example, a nullable vector containing 768 required `FLOAT` elements is: + +``` +optional group embedding (FINITE_VECTOR(768)) { + repeated group list { + required float element; + } +} +``` + +Every non-null vector must contain exactly `num_elements` elements. +Because `num_elements` is greater than zero, a non-null vector cannot be empty. + ## UNKNOWN (always null) Sometimes, when discovering the schema of existing data, values are always null diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index fea13fbe..c58b5631 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -479,6 +479,19 @@ struct GeographyType { struct FileType { } +/** +* Finite fixed-length vector logical type annotation +* +* Annotates the outer group of a three-level structure that is the same as +* the one used for LIST-annotated groups. Each non-null vector MUST contain +* exactly num_elements elements. +* +* See LogicalTypes.md for details. +*/ +struct FiniteVectorType { + 1: required i32 num_elements +} + /** * LogicalType annotations to replace ConvertedType. * @@ -513,6 +526,7 @@ union LogicalType { 17: GeometryType GEOMETRY // no compatible ConvertedType 18: GeographyType GEOGRAPHY // no compatible ConvertedType 19: FileType FILE // no compatible ConvertedType + 20: FiniteVectorType FINITE_VECTOR // use ConvertedType LIST } /** @@ -1129,6 +1143,7 @@ union ColumnOrder { * GEOMETRY - undefined * GEOGRAPHY - undefined * FILE - undefined + * FINITE_VECTOR - undefined * * In the absence of logical types, the sort order is determined by the physical type: * BOOLEAN - false, true