From f82e3e3779fb5cb91dbbe974c452729ee7c57470 Mon Sep 17 00:00:00 2001 From: Jan Finis Date: Mon, 2 Nov 2026 18:57:01 +0100 Subject: [PATCH] Draft commit for showing an alternative to FINITE_VECTOR FINITE_VECTOR, as proposed in https://github.com/apache/parquet-format/pull/624 has two shortcomings that this PR is trying to avoid: * This PR does not tie finiteness to the list/vector type but instead to the contained float. This way, we can have finite floats that aren't vectors. They are useful by themselves, so having the finiteness annotation at the type to which it applies makes this more composable. * This PR does not introduce FINITE_VECTOR as a new type that basically mimics LIST with a fixed length. Instead, the LIST type itself receives an optional fixed length field. This way, legacy readers can still read vectors and simply interpret them as lists. Also, by decoupling finiteness from fixed length and not calling it vector, we gain general fixed-size lists for *all* types, instead of having just some blessed types be able to live in fixed-size lists / vectors. --- LogicalTypes.md | 49 ++++++++++++++++++++++++++++++++++ src/main/thrift/parquet.thrift | 31 ++++++++++++++++++--- 2 files changed, 76 insertions(+), 4 deletions(-) diff --git a/LogicalTypes.md b/LogicalTypes.md index 559030fe..c4044b4b 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -255,6 +255,33 @@ comparison. To support compatibility with older readers, implementations of parquet-format should write `DecimalType` precision and scale into the corresponding SchemaElement field in metadata. +### FLOATING_POINT + +`FLOATING_POINT` may annotate a primitive column of physical type `FLOAT` or +`DOUBLE`. It does not change the physical representation or sort order of the +column. It provides the optional `isFinite` property. When `isFinite` is `true`, +every non-null value in the column must be finite: neither NaN nor positive or +negative infinity. Nulls, positive and negative zero, and subnormal values are +allowed. Readers may rely on this guarantee without inspecting the values. + +If `isFinite` is absent or `false`, or if the annotation is absent, no +finiteness guarantee is made; non-finite values may or may not occur. The +property applies to the annotated leaf column only, not to its containing +groups or other columns. It is a file-wide guarantee across all row groups, +including values decoded using dictionary or other encodings. + +There is no corresponding `ConvertedType`. Older readers may ignore this +annotation and read the physical `FLOAT` or `DOUBLE` values as usual. Per the +rules for unsupported logical types, a reader that does not recognize +`FLOATING_POINT` should also ignore its column order for statistics. + +`isFinite = true` does not change the requirements for writing `nan_count`, +min/max statistics, page indexes, or `ColumnOrder`. Readers unaware of this +property must continue to apply the existing statistics rules, including +treating an absent `nan_count` as no guarantee that NaNs are absent. Readers +that recognize `isFinite = true` may infer that `nan_count` and page-index +`nan_counts` are zero when those statistics are absent. + ### FLOAT16 The `FLOAT16` annotation represents half-precision floating-point numbers in the 2-byte IEEE little-endian format. @@ -263,6 +290,13 @@ Used in contexts where precision is traded off for smaller footprint and potenti The primitive type is a 2-byte `FIXED_LEN_BYTE_ARRAY`. +`FLOAT16` also has an optional `isFinite` property with the same meaning as +for `FLOATING_POINT`: if `true`, every non-null value must be finite; if absent +or `false`, no guarantee is made. This allows the constraint to coexist with +the `FLOAT16` logical type. It applies only to the annotated leaf, across all +row groups and encodings, and does not change statistics requirements. Readers +that recognize `FLOAT16` but not this new property may ignore the property. + Like `FLOAT` and `DOUBLE`, the sort order for `FLOAT16` is signed with special handling for NaNs and signed zeros. Writers should use IEEE754TotalOrder for consistent handling of these edge cases. See the `ColumnOrder` union in the @@ -895,6 +929,21 @@ the annotations, no unannotated repeated types are allowed. * The `element` field encodes the list's element type and repetition. Element repetition must be `required` or `optional`. +The `LIST` logical type may set the optional `elementCount` field. If set, it +must be non-negative, and every non-null list annotated by that `LIST` must +contain exactly `elementCount` elements, including any null elements. A value +of 0 means every non-null list is empty. Null lists remain distinct from empty +lists and are governed by the outer group's repetition. + +Writers must still encode definition and repetition levels as usual, including +the repetition levels of list elements. Readers may use `elementCount` instead +of inspecting repetition levels to determine the number of elements in each +list; repetition levels needed for other nested structures must still be +handled. When `elementCount` is absent, list lengths are determined from the +encoded levels as usual. The deprecated `ConvertedType.LIST` annotation cannot +carry `elementCount`, so writers using this field must also write the `LIST` +logical type annotation. + The following examples demonstrate two of the possible lists of string values. ``` diff --git a/src/main/thrift/parquet.thrift b/src/main/thrift/parquet.thrift index fea13fbe..4c75799c 100644 --- a/src/main/thrift/parquet.thrift +++ b/src/main/thrift/parquet.thrift @@ -313,7 +313,8 @@ struct Statistics { * Count of NaN values in the column; only present if physical type is FLOAT * or DOUBLE, or logical type is FLOAT16. * If this field is not present, readers MUST assume NaNs may be present - * (i.e. MUST assume nan_count > 0 and MAY NOT assume nan_count == 0). + * (i.e. MUST assume nan_count > 0 and MAY NOT assume nan_count == 0), + * unless the column's FLOATING_POINT or FLOAT16 logical type has isFinite = true. */ 9: optional i64 nan_count; } @@ -322,10 +323,29 @@ struct Statistics { struct StringType {} // allowed for BYTE_ARRAY, must be encoded with UTF-8 struct UUIDType {} // allowed for FIXED[16], must be encoded as raw UUID bytes struct MapType {} // see LogicalTypes.md -struct ListType {} // see LogicalTypes.md +/** See LogicalTypes.md for the LIST encoding and elementCount semantics. */ +struct ListType { + /** + * If set, every non-null list has exactly this many elements (including null + * elements). Must be non-negative; zero denotes an empty list. Repetition + * levels are still encoded, but readers need not inspect them to determine + * the length of this list. + */ + 1: optional i32 elementCount; +} struct EnumType {} // allowed for BYTE_ARRAY, must be encoded with UTF-8 struct DateType {} // allowed for INT32 -struct Float16Type {} // allowed for FIXED[2], must be encoded as raw FLOAT16 bytes (see LogicalTypes.md) +/** Allowed for FIXED[2], must be encoded as raw FLOAT16 bytes (see LogicalTypes.md). */ +struct Float16Type { + /** If true, every non-null value is finite (not NaN or +/-infinity). */ + 1: optional bool isFinite; +} + +/** Allowed for physical types FLOAT and DOUBLE (see LogicalTypes.md). */ +struct FloatingPointType { + /** If true, every non-null value is finite (not NaN or +/-infinity). */ + 1: optional bool isFinite; +} /** * Logical type to annotate a column that is always null. @@ -513,6 +533,7 @@ union LogicalType { 17: GeometryType GEOMETRY // no compatible ConvertedType 18: GeographyType GEOGRAPHY // no compatible ConvertedType 19: FileType FILE // no compatible ConvertedType + 20: FloatingPointType FLOATING_POINT // no compatible ConvertedType } /** @@ -1115,6 +1136,7 @@ union ColumnOrder { * DECIMAL - signed comparison of the represented value * DATE - signed comparison * FLOAT16 - signed comparison of the represented value (*) + * FLOATING_POINT - signed comparison of the represented value (*) * TIME_MILLIS - signed comparison * TIME_MICROS - signed comparison * TIMESTAMP_MILLIS - signed comparison @@ -1366,7 +1388,8 @@ struct ColumnIndex { * A list containing the number of NaN values for each page. Only present * for columns of physical type FLOAT or DOUBLE, or logical type FLOAT16. * If this field is not present, readers MUST assume that there might be - * NaN values in any page. + * NaN values in any page, unless the column's FLOATING_POINT or FLOAT16 + * logical type has isFinite = true. */ 8: optional list nan_counts