diff --git a/docs/docs/schemas.md b/docs/docs/schemas.md index 7448e858da67..1de134c33cec 100644 --- a/docs/docs/schemas.md +++ b/docs/docs/schemas.md @@ -44,6 +44,7 @@ Iceberg tables support the following types: | **`geometry(C)`** | Geospatial features with CRS parameter | Linear edge-interpolation; added in v3 | | **`geography(C, A)`** | Geospatial features with CRS and edge algorithm | Non-linear edge-interpolation; added in v3 | | **`unknown`** | Placeholder type for undetermined columns | Must be optional; added in v3 | +| **`vector(E, D)`** | D-dimensional vector of finite numeric values with element type E | Added in v4 | | **`struct<...>`** | A record with named fields of any data type | | | **`list`** | A list with elements of any data type | | | **`map`** | A map with keys and values of any data type | | diff --git a/format/expressions-spec.md b/format/expressions-spec.md index 924c2c796a66..e90710e481e5 100644 --- a/format/expressions-spec.md +++ b/format/expressions-spec.md @@ -222,7 +222,7 @@ Note that `year`, `month`, and `hour` transforms produce ordinal values and not | Parameterized function name | Description | Source types | Result type | |-----------------------------|-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------|-------------| -| `bucket(N, value)` | Hash of value, mod `N` (see [table spec details][bucket-ref]) | Any primitive except for `geometry`, `geography`, `boolean`, `float`, or `double` | `int` | +| `bucket(N, value)` | Hash of value, mod `N` (see [table spec details][bucket-ref]) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | | `truncate(W, value)` | Value truncated to width `W` (see [table spec details][truncate-ref]) | `int`, `long`, `decimal`, `string`, `binary` | Source type | [bucket-ref]: spec.md#bucket-transform-details diff --git a/format/spec.md b/format/spec.md index 58a21d4e5c3d..47e6d98db6e9 100644 --- a/format/spec.md +++ b/format/spec.md @@ -68,6 +68,7 @@ Version 4 of the Iceberg spec restructures metadata for improved performance and * New data type: `file` * Support for table [constraints](#constraints) * [Default value expressions](#default-values) for write defaults +* New data type: `vector` The full set of changes are listed in [Appendix E](#version-4). @@ -297,6 +298,7 @@ Supported primitive types are defined in the table below. Primitive types added | | **`binary`** | Arbitrary-length byte array | | | [v3](#version-3) | **`geometry(C)`** | Geospatial features from [OGC – Simple feature access][1001]. Edge-interpolation is always linear/planar. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C. If not specified, C is `OGC:CRS84`. | | | [v3](#version-3) | **`geography(C, A)`** | Geospatial features from [OGC – Simple feature access][1001]. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C and edge-interpolation algorithm A. If not specified, C is `OGC:CRS84` and A is `spherical`. | +| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of D finite numeric values with element type E | E must be a [vector element type](#vector); D must be positive | Notes: @@ -308,6 +310,34 @@ For details on how to serialize a schema to JSON, see Appendix C. [1001]: "OGC Simple feature access" +##### Vector + +A non-null `vector(E, D)` contains exactly D elements of element type E, and every element is non-null. The vector value may be null only when the vector field is optional. + +E must be one of the following vector element types: + +| Element type | Element values | Value type | +|---------------|--------------------------------------------------------------------|------------| +| **`int`** | 32-bit signed integers | `int` | +| **`long`** | 64-bit signed integers | `long` | +| **`float16`** | `float` values that are exactly representable as IEEE 754 binary16 | `float` | +| **`float`** | 32-bit IEEE 754 floating point | `float` | +| **`double`** | 64-bit IEEE 754 floating point | `double` | + +Vector element types are valid only inside a `vector`; `float16` is not a primitive type and cannot be used as the type of a field. Element values follow the rules of the value type, including JSON single-value serialization. Vector equality is element-wise, using the equality rules of the value type. Floating-point elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. + +The element type of a `vector(E, D)` may be promoted according to the following table. The dimension D and the vector field ID must not change. + +| Element type | Permitted promotion targets | +|--------------|-----------------------------| +| `int` | `long` | +| `float16` | `float`, `double` | +| `float` | `double` | + +All other element type changes are prohibited. A `vector` must not be promoted to a non-vector type. A non-vector type other than `unknown` must not be promoted to `vector`. A `list` is not a `vector(E, D)`, even if every list instance contains exactly D elements that satisfy the vector element requirements. + +After an element type promotion, readers must still read older files by converting vector elements to the new type without changing their values. The stored dimension must match the schema. + ##### CRS For `geometry` and `geography` types, the parameter C refers to the CRS (coordinate reference system), a mapping of how coordinates refer to locations on Earth. @@ -384,7 +414,7 @@ Starting in v4, a field's `write-default` may be a [value expression](expression The `initial-default` must always be a literal value. A literal is itself a value expression, so existing `write-default` values remain valid value expressions. -All columns of `unknown`, `variant`, `geometry`, `geography`, and `file` types must default to null. Non-null values for `initial-default` or `write-default` are invalid. +All columns of `unknown`, `variant`, `geometry`, `geography`, `file`, and `vector` types must default to null. Non-null values for `initial-default` or `write-default` are invalid. Default values for the fields of a struct are tracked as `initial-default` and `write-default` at the field level. Default values for fields that are nested structs must not contain default values for the struct's fields (sub-fields). Sub-field defaults are tracked in sub-field's metadata. As a result, the default stored for a nested struct may be either null or a non-null struct with no field values. The effective default value is produced by setting each fields' default in a new struct. @@ -414,6 +444,7 @@ Valid primitive type promotions are: | `date` | | `timestamp`, `timestamp_ns` | Promotion to `timestamptz` or `timestamptz_ns` is **not** allowed; values outside the promoted type's range must result in a runtime failure | | `float` | `double` | `double` | | | `decimal(P, S)` | `decimal(P', S)` if `P' > P` | `decimal(P', S)` if `P' > P` | Widen precision only | +| `vector(E, D)` | | `vector(E', D)` | v4 only; E to E' must be a permitted [vector element type promotion](#vector); D must not change | Iceberg's Avro manifest format does not store the type of lower and upper bounds, and type promotion does not rewrite existing bounds. For example, when a `float` is promoted to `double`, existing data file bounds are encoded as 4 little-endian bytes rather than 8 little-endian bytes for `double`. To correctly decode the value, the original type at the time the file was written must be inferred according to the following table: @@ -729,6 +760,8 @@ Order id `0` is reserved for the unsorted order. Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons. +Vector values of the same `vector(E, D)` type must be sorted lexicographically using the sort order of E's value type. + A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. ### Constraints @@ -767,6 +800,7 @@ The `field-ids` of a `unique` or `primary-key` constraint: * Must not reference fields of type `geometry` or `geography`, because Iceberg does not define equality for them * Must not reference fields of type `unknown`, because their values are always null * Must not reference fields of type `file` or their sub-fields +* Must not reference fields of type `vector` A table may have at most one `primary-key` constraint. The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rejects null keys even when the constraint is not enforced. The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. @@ -1111,8 +1145,8 @@ Each stats struct holds statistics for one table field. It may contain the follo | Requirement | Offset | Name | Type | Included for | Description | |-------------|--------|---------------------------|---------------------------|-----------------------------------------------|-------------| -| _optional_ | 1 | `lower_bound` | Field type or `geo_lower` | all primitives or `variant` | Lower bound stored as the field's type, or `geo_lower` for geo types | -| _optional_ | 2 | `upper_bound` | Field type or `geo_upper` | all primitives or `variant` | Upper bound stored as the field's type, or `geo_upper` for geo types | +| _optional_ | 1 | `lower_bound` | Field type or `geo_lower` | `variant` or all primitives except `vector` | Lower bound stored as the field's type, or `geo_lower` for geo types | +| _optional_ | 2 | `upper_bound` | Field type or `geo_upper` | `variant` or all primitives except `vector` | Upper bound stored as the field's type, or `geo_upper` for geo types | | _optional_ | 3 | `value_count` | `long` | all | Number of values in the column (including null and NaN values) | | _optional_ | 4 | `null_value_count` | `long` | optional fields | Number of null values in the column | | _optional_ | 5 | `nan_value_count` | `long` | `float`, `double` | Number of NaN values in the column | @@ -1878,6 +1912,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`binary`**|`bytes`|| |**`struct`**|`record`|| |**`list`**|`array`|| +|**`vector(E, D)`**|`array`|When writing, array elements must use the Avro mapping for E's value type and must not be null. The array must not have an `element-id` property. Readers and writers must enforce the vector dimension and element requirements.| |**`map`**|`array` of key-value records, or `map` when keys are strings (optional).|Array storage must use logical type name `map` and must store elements that are 2-field records. The first field is a non-null key and the second field is the value.| |**`variant`**|`record` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. |Shredding is not supported in Avro.| |**`geometry`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)| @@ -1913,6 +1948,8 @@ Values should be stored in Parquet using the types and logical type annotations Lists must use the [3-level representation](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists). +Vectors must use the [`FINITE_VECTOR` representation](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#finite-vectors), with `num_elements` set to D. + | Type | Parquet physical type | Logical type | Notes | |--------------------|----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|----------------------------------------------------------------| | **`unknown`** | None | | Omit from data files | @@ -1934,12 +1971,25 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo | **`binary`** | `binary` | | | | **`struct`** | `group` | | | | **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | +| **`vector(E, D)`** | `3-level list` | `FINITE_VECTOR(D)` | The outer group carries the field ID of the vector. When writing, the `element` field must use the Parquet type for E listed in the vector element table below. Writing and reading an element must preserve its value exactly. The ConvertedType must be `LIST`. The `list` and `element` fields must not be assigned field IDs. | | **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | | **`variant`** | `group` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. | `VARIANT` | See Parquet docs for [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [Variant shredding encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md). | | **`geometry`** | `binary` | `GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | | **`geography`** | `binary` | `GEOGRAPHY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | | **`file`** | `group` with the `file` sub-fields. Sub-fields must be assigned field IDs. | `FILE` | See Parquet docs for the [`FILE` type](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file) and [File Type](#file-type). | +Vector elements must be written to Parquet using the following types: + +| Element type | Parquet physical type | Logical type | +|---------------|---------------------------|--------------| +| **`int`** | `int32` | | +| **`long`** | `int64` | | +| **`float16`** | `fixed_len_byte_array[2]` | `FLOAT16` | +| **`float`** | `float` | | +| **`double`** | `double` | | + +When reading `vector(E, D)`, the `element` field may use the Parquet mapping for E or for an element type that can be promoted to E. Readers must convert elements to E without changing their values. The `FINITE_VECTOR` `num_elements` parameter must equal D. + When reading an `unknown` column, any corresponding column must be ignored and replaced with `null` values. ### ORC @@ -1967,6 +2017,7 @@ When reading an `unknown` column, any corresponding column must be ignored and r | **`binary`** | `binary` | | | | **`struct`** | `struct` | | | | **`list`** | `array` | | | +| **`vector(E, D)`** | Not supported | | | | **`map`** | `map` | | | | **`variant`** | `struct` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs. | `iceberg.struct-type`=`VARIANT` | Shredding is not supported in ORC. | | **`geometry`** | `binary` | `iceberg.binary-type`=`GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | @@ -2064,6 +2115,7 @@ Types are serialized according to this table: |**`fixed(L)`**|`JSON string: "fixed[]"`|`"fixed[16]"`| |**`binary`**|`JSON string: "binary"`|`"binary"`| |**`decimal(P, S)`**|`JSON string: "decimal(

, )"`|`"decimal(9, 2)"`| +|**`vector(E, D)`**|`JSON string: "vector(, )"`|`"vector(float, 768)"`| |**`struct`**|`JSON object: {`
  `"type": "struct",`
  `"fields": [ {`
    `"id": ,`
    `"name": ,`
    `"required": ,`
    `"type": ,`
    `"doc": ,`
    `"initial-default": ,`
    `"write-default": `
    `}, ...`
  `] }`|`{`
  `"type": "struct",`
  `"fields": [ {`
    `"id": 1,`
    `"name": "id",`
    `"required": true,`
    `"type": "uuid",`
    `"initial-default": "0db3e2a8-9d1d-42b9-aa7b-74ebe558dceb",`
    `"write-default": "ec5911be-b0a7-458c-8438-c9a3e53cffae"`
  `}, {`
    `"id": 2,`
    `"name": "data",`
    `"required": false,`
    `"type": {`
      `"type": "list",`
      `...`
    `}`
  `} ]`
`}`| |**`list`**|`JSON object: {`
  `"type": "list",`
  `"element-id": ,`
  `"element-required": `
  `"element": `
`}`|`{`
  `"type": "list",`
  `"element-id": 3,`
  `"element-required": true,`
  `"element": "string"`
`}`| |**`map`**|`JSON object: {`
  `"type": "map",`
  `"key-id": ,`
  `"key": ,`
  `"value-id": ,`
  `"value-required": `
  `"value": `
`}`|`{`
  `"type": "map",`
  `"key-id": 4,`
  `"key": "string",`
  `"value-id": 5,`
  `"value-required": false,`
  `"value": "double"`
`}`| @@ -2229,6 +2281,7 @@ This serialization scheme is for storing single values as individual binary valu | **`decimal(P, S)`** | Stores unscaled value as two’s-complement big-endian binary, using the minimum number of bytes for the value | | **`struct`** | Not supported | | **`list`** | Not supported | +| **`vector(E, D)`** | Not supported | | **`map`** | Not supported | | **`variant`** | Not supported | | **`geometry`** | WKB format, see [Appendix G](#appendix-g-geospatial-notes) | @@ -2269,6 +2322,7 @@ The binary single-value serialization can be used to store the lower and upper b | **`binary`** | **`JSON string`** | `"000102ff"` | Stored as a hexadecimal string | | **`struct`** | **`JSON object by field ID`** | `{"1": 1, "2": "bar"}` | Stores struct fields using the field ID as the JSON field name; field values are stored using this JSON single-value format | | **`list`** | **`JSON array of values`** | `[1, 2, 3]` | Stores a JSON array of values that are serialized using this JSON single-value format | +| **`vector(E, D)`** | **`JSON array of values`** | `[1.0, 2.0, 3.0]` | Stores exactly D values using the JSON single-value format of E's value type | | **`map`** | **`JSON object of key and value arrays`** | `{ "keys": ["a", "b"], "values": [1, 2] }` | Stores arrays of keys and values; individual keys and values are serialized using this JSON single-value format | | **`geometry`** | **`JSON string`** | `POINT (30 10)` | Stored using WKT representation, see [Appendix G](#appendix-g-geospatial-notes) | | **`geography`** | **`JSON string`** | `POINT (30 10)` | Stored using WKT representation, see [Appendix G](#appendix-g-geospatial-notes) | @@ -2281,6 +2335,8 @@ Default value expressions are added in v4. A field's `write-default` may be a [v The `file` type is added in v4. +The `vector` type is added in v4. + Relative path support is added in v4. Reading v3 or prior metadata for v4: