Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions LogicalTypes.md
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,33 @@ comparison.
To support compatibility with older readers, implementations of parquet-format should
write `DecimalType` precision and scale into the corresponding SchemaElement field in metadata.

### FLOATING_POINT

`FLOATING_POINT` may annotate a primitive column of physical type `FLOAT` or
`DOUBLE`. It does not change the physical representation or sort order of the
column. It provides the optional `isFinite` property. When `isFinite` is `true`,
every non-null value in the column must be finite: neither NaN nor positive or
negative infinity. Nulls, positive and negative zero, and subnormal values are
allowed. Readers may rely on this guarantee without inspecting the values.

If `isFinite` is absent or `false`, or if the annotation is absent, no
finiteness guarantee is made; non-finite values may or may not occur. The
property applies to the annotated leaf column only, not to its containing
groups or other columns. It is a file-wide guarantee across all row groups,
including values decoded using dictionary or other encodings.

There is no corresponding `ConvertedType`. Older readers may ignore this
annotation and read the physical `FLOAT` or `DOUBLE` values as usual. Per the
rules for unsupported logical types, a reader that does not recognize
`FLOATING_POINT` should also ignore its column order for statistics.

`isFinite = true` does not change the requirements for writing `nan_count`,
min/max statistics, page indexes, or `ColumnOrder`. Readers unaware of this
property must continue to apply the existing statistics rules, including
treating an absent `nan_count` as no guarantee that NaNs are absent. Readers
that recognize `isFinite = true` may infer that `nan_count` and page-index
`nan_counts` are zero when those statistics are absent.

### FLOAT16

The `FLOAT16` annotation represents half-precision floating-point numbers in the 2-byte IEEE little-endian format.
Expand All @@ -263,6 +290,13 @@ Used in contexts where precision is traded off for smaller footprint and potenti

The primitive type is a 2-byte `FIXED_LEN_BYTE_ARRAY`.

`FLOAT16` also has an optional `isFinite` property with the same meaning as
for `FLOATING_POINT`: if `true`, every non-null value must be finite; if absent
or `false`, no guarantee is made. This allows the constraint to coexist with
the `FLOAT16` logical type. It applies only to the annotated leaf, across all
row groups and encodings, and does not change statistics requirements. Readers
that recognize `FLOAT16` but not this new property may ignore the property.

Like `FLOAT` and `DOUBLE`, the sort order for `FLOAT16` is signed with special
handling for NaNs and signed zeros. Writers should use IEEE754TotalOrder for
consistent handling of these edge cases. See the `ColumnOrder` union in the
Expand Down Expand Up @@ -895,6 +929,21 @@ the annotations, no unannotated repeated types are allowed.
* The `element` field encodes the list's element type and repetition. Element
repetition must be `required` or `optional`.

The `LIST` logical type may set the optional `elementCount` field. If set, it
must be non-negative, and every non-null list annotated by that `LIST` must
contain exactly `elementCount` elements, including any null elements. A value
of 0 means every non-null list is empty. Null lists remain distinct from empty
lists and are governed by the outer group's repetition.

Writers must still encode definition and repetition levels as usual, including
the repetition levels of list elements. Readers may use `elementCount` instead
of inspecting repetition levels to determine the number of elements in each
list; repetition levels needed for other nested structures must still be
handled. When `elementCount` is absent, list lengths are determined from the
encoded levels as usual. The deprecated `ConvertedType.LIST` annotation cannot
carry `elementCount`, so writers using this field must also write the `LIST`
logical type annotation.

The following examples demonstrate two of the possible lists of string values.

```
Expand Down
31 changes: 27 additions & 4 deletions src/main/thrift/parquet.thrift
Original file line number Diff line number Diff line change
Expand Up @@ -313,7 +313,8 @@ struct Statistics {
* Count of NaN values in the column; only present if physical type is FLOAT
* or DOUBLE, or logical type is FLOAT16.
* If this field is not present, readers MUST assume NaNs may be present
* (i.e. MUST assume nan_count > 0 and MAY NOT assume nan_count == 0).
* (i.e. MUST assume nan_count > 0 and MAY NOT assume nan_count == 0),
* unless the column's FLOATING_POINT or FLOAT16 logical type has isFinite = true.
*/
9: optional i64 nan_count;
}
Expand All @@ -322,10 +323,29 @@ struct Statistics {
struct StringType {} // allowed for BYTE_ARRAY, must be encoded with UTF-8
struct UUIDType {} // allowed for FIXED[16], must be encoded as raw UUID bytes
struct MapType {} // see LogicalTypes.md
struct ListType {} // see LogicalTypes.md
/** See LogicalTypes.md for the LIST encoding and elementCount semantics. */
struct ListType {
/**
* If set, every non-null list has exactly this many elements (including null
* elements). Must be non-negative; zero denotes an empty list. Repetition
* levels are still encoded, but readers need not inspect them to determine
* the length of this list.
*/
1: optional i32 elementCount;
}
struct EnumType {} // allowed for BYTE_ARRAY, must be encoded with UTF-8
struct DateType {} // allowed for INT32
struct Float16Type {} // allowed for FIXED[2], must be encoded as raw FLOAT16 bytes (see LogicalTypes.md)
/** Allowed for FIXED[2], must be encoded as raw FLOAT16 bytes (see LogicalTypes.md). */
struct Float16Type {
/** If true, every non-null value is finite (not NaN or +/-infinity). */
1: optional bool isFinite;
}

/** Allowed for physical types FLOAT and DOUBLE (see LogicalTypes.md). */
struct FloatingPointType {
/** If true, every non-null value is finite (not NaN or +/-infinity). */
1: optional bool isFinite;
}

/**
* Logical type to annotate a column that is always null.
Expand Down Expand Up @@ -513,6 +533,7 @@ union LogicalType {
17: GeometryType GEOMETRY // no compatible ConvertedType
18: GeographyType GEOGRAPHY // no compatible ConvertedType
19: FileType FILE // no compatible ConvertedType
20: FloatingPointType FLOATING_POINT // no compatible ConvertedType
}

/**
Expand Down Expand Up @@ -1115,6 +1136,7 @@ union ColumnOrder {
* DECIMAL - signed comparison of the represented value
* DATE - signed comparison
* FLOAT16 - signed comparison of the represented value (*)
* FLOATING_POINT - signed comparison of the represented value (*)
* TIME_MILLIS - signed comparison
* TIME_MICROS - signed comparison
* TIMESTAMP_MILLIS - signed comparison
Expand Down Expand Up @@ -1366,7 +1388,8 @@ struct ColumnIndex {
* A list containing the number of NaN values for each page. Only present
* for columns of physical type FLOAT or DOUBLE, or logical type FLOAT16.
* If this field is not present, readers MUST assume that there might be
* NaN values in any page.
* NaN values in any page, unless the column's FLOATING_POINT or FLOAT16
* logical type has isFinite = true.
*/
8: optional list<i64> nan_counts

Expand Down
Loading