From 5de3a9ac7a8f6fafed6dab411a5eae9db730c7d1 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Wed, 7 Oct 2026 21:22:31 +0000 Subject: [PATCH 01/18] SPEC: Add vector type to v4 --- format/spec.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/format/spec.md b/format/spec.md index 58a21d4e5c3d..c24ba4fb2187 100644 --- a/format/spec.md +++ b/format/spec.md @@ -68,6 +68,7 @@ Version 4 of the Iceberg spec restructures metadata for improved performance and * New data type: `file` * Support for table [constraints](#constraints) * [Default value expressions](#default-values) for write defaults +* New data type: `vector` The full set of changes are listed in [Appendix E](#version-4). @@ -295,6 +296,7 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`** | Universally unique identifiers | Should use 16-byte fixed | | | **`fixed(L)`** | Fixed-length byte array of length L | | | | **`binary`** | Arbitrary-length byte array | | +| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite numeric values with element type E and dimension D | E must be `int`, `long`, `float`, or `double`; D must be positive | | [v3](#version-3) | **`geometry(C)`** | Geospatial features from [OGC – Simple feature access][1001]. Edge-interpolation is always linear/planar. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C. If not specified, C is `OGC:CRS84`. | | | [v3](#version-3) | **`geography(C, A)`** | Geospatial features from [OGC – Simple feature access][1001]. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C and edge-interpolation algorithm A. If not specified, C is `OGC:CRS84` and A is `spherical`. | @@ -308,6 +310,12 @@ For details on how to serialize a schema to JSON, see Appendix C. [1001]: "OGC Simple feature access" +##### Vector + +Every non-null `vector(E, D)` value must contain exactly D non-null elements of type E. Floating-point elements must not be `NaN`, positive infinity, or negative infinity. The vector value itself may be null when the containing field is optional. + +No type promotion to or from `vector` is defined, including changes to its element type or dimension. Lower and upper bounds are not defined for vector values. + ##### CRS For `geometry` and `geography` types, the parameter C refers to the CRS (coordinate reference system), a mapping of how coordinates refer to locations on Earth. @@ -1878,6 +1886,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`binary`**|`bytes`|| |**`struct`**|`record`|| |**`list`**|`array`|| +|**`vector(E, D)`**|`array`|Array elements must use the Avro mapping for E and must not be null. Readers and writers must enforce the vector dimension and element requirements.| |**`map`**|`array` of key-value records, or `map` when keys are strings (optional).|Array storage must use logical type name `map` and must store elements that are 2-field records. The first field is a non-null key and the second field is the value.| |**`variant`**|`record` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. |Shredding is not supported in Avro.| |**`geometry`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)| @@ -1934,6 +1943,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo | **`binary`** | `binary` | | | | **`struct`** | `group` | | | | **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | +| **`vector(E, D)`** | `3-level list` | `VECTOR(D)` | The element must be required and use the Parquet mapping for E. See [apache/parquet-format#624](https://github.com/apache/parquet-format/pull/624). | | **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | | **`variant`** | `group` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. | `VARIANT` | See Parquet docs for [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [Variant shredding encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md). | | **`geometry`** | `binary` | `GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | @@ -1967,6 +1977,7 @@ When reading an `unknown` column, any corresponding column must be ignored and r | **`binary`** | `binary` | | | | **`struct`** | `struct` | | | | **`list`** | `array` | | | +| **`vector(E, D)`** | `array` | | Array elements must use the ORC mapping for E and must not be null. Readers and writers must enforce the vector dimension and element requirements. | | **`map`** | `map` | | | | **`variant`** | `struct` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs. | `iceberg.struct-type`=`VARIANT` | Shredding is not supported in ORC. | | **`geometry`** | `binary` | `iceberg.binary-type`=`GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | @@ -2064,6 +2075,7 @@ Types are serialized according to this table: |**`fixed(L)`**|`JSON string: "fixed[]"`|`"fixed[16]"`| |**`binary`**|`JSON string: "binary"`|`"binary"`| |**`decimal(P, S)`**|`JSON string: "decimal(

, )"`|`"decimal(9, 2)"`| +|**`vector(E, D)`**|`JSON string: "vector(, )"`|`"vector(float, 768)"`| |**`struct`**|`JSON object: {`
  `"type": "struct",`
  `"fields": [ {`
    `"id": ,`
    `"name": ,`
    `"required": ,`
    `"type": ,`
    `"doc": ,`
    `"initial-default": ,`
    `"write-default": `
    `}, ...`
  `] }`|`{`
  `"type": "struct",`
  `"fields": [ {`
    `"id": 1,`
    `"name": "id",`
    `"required": true,`
    `"type": "uuid",`
    `"initial-default": "0db3e2a8-9d1d-42b9-aa7b-74ebe558dceb",`
    `"write-default": "ec5911be-b0a7-458c-8438-c9a3e53cffae"`
  `}, {`
    `"id": 2,`
    `"name": "data",`
    `"required": false,`
    `"type": {`
      `"type": "list",`
      `...`
    `}`
  `} ]`
`}`| |**`list`**|`JSON object: {`
  `"type": "list",`
  `"element-id": ,`
  `"element-required": `
  `"element": `
`}`|`{`
  `"type": "list",`
  `"element-id": 3,`
  `"element-required": true,`
  `"element": "string"`
`}`| |**`map`**|`JSON object: {`
  `"type": "map",`
  `"key-id": ,`
  `"key": ,`
  `"value-id": ,`
  `"value-required": `
  `"value": `
`}`|`{`
  `"type": "map",`
  `"key-id": 4,`
  `"key": "string",`
  `"value-id": 5,`
  `"value-required": false,`
  `"value": "double"`
`}`| @@ -2229,6 +2241,7 @@ This serialization scheme is for storing single values as individual binary valu | **`decimal(P, S)`** | Stores unscaled value as two’s-complement big-endian binary, using the minimum number of bytes for the value | | **`struct`** | Not supported | | **`list`** | Not supported | +| **`vector(E, D)`** | Not supported | | **`map`** | Not supported | | **`variant`** | Not supported | | **`geometry`** | WKB format, see [Appendix G](#appendix-g-geospatial-notes) | @@ -2269,6 +2282,7 @@ The binary single-value serialization can be used to store the lower and upper b | **`binary`** | **`JSON string`** | `"000102ff"` | Stored as a hexadecimal string | | **`struct`** | **`JSON object by field ID`** | `{"1": 1, "2": "bar"}` | Stores struct fields using the field ID as the JSON field name; field values are stored using this JSON single-value format | | **`list`** | **`JSON array of values`** | `[1, 2, 3]` | Stores a JSON array of values that are serialized using this JSON single-value format | +| **`vector(E, D)`** | **`JSON array of values`** | `[1.0, 2.0, 3.0]` | Stores exactly D values using E's JSON single-value format | | **`map`** | **`JSON object of key and value arrays`** | `{ "keys": ["a", "b"], "values": [1, 2] }` | Stores arrays of keys and values; individual keys and values are serialized using this JSON single-value format | | **`geometry`** | **`JSON string`** | `POINT (30 10)` | Stored using WKT representation, see [Appendix G](#appendix-g-geospatial-notes) | | **`geography`** | **`JSON string`** | `POINT (30 10)` | Stored using WKT representation, see [Appendix G](#appendix-g-geospatial-notes) | @@ -2281,6 +2295,8 @@ Default value expressions are added in v4. A field's `write-default` may be a [v The `file` type is added in v4. +The `vector` type is added in v4. Writing `vector` into a v3 or earlier schema is invalid. + Relative path support is added in v4. Reading v3 or prior metadata for v4: From d375444a172d31a0acacbc9316b6d4ffeb25aa66 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Wed, 14 Oct 2026 00:59:57 +0000 Subject: [PATCH 02/18] SPEC: Clarify vector semantics and exclude ORC support Apply the reviewed vector value, promotion and equality wording. Mark ORC vectors unsupported and remove redundant version text. Generated-by: Codex --- format/spec.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/format/spec.md b/format/spec.md index c24ba4fb2187..7c9a4104d1c9 100644 --- a/format/spec.md +++ b/format/spec.md @@ -312,9 +312,11 @@ For details on how to serialize a schema to JSON, see Appendix C. ##### Vector -Every non-null `vector(E, D)` value must contain exactly D non-null elements of type E. Floating-point elements must not be `NaN`, positive infinity, or negative infinity. The vector value itself may be null when the containing field is optional. +A non-null `vector(E, D)` contains exactly D elements of type E, and every element is non-null. `float` and `double` elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. The vector value may be null only when the vector field is optional. -No type promotion to or from `vector` is defined, including changes to its element type or dimension. Lower and upper bounds are not defined for vector values. +A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` of element type E is not a `vector(E, D)`, even when every value has length D. + +Equality of vector values is element-wise, using the equality rules of element type. A vector must not be an identifier field. No sort order is defined for a vector value. Writers must not write lower or upper bounds for a vector field. ##### CRS @@ -1977,7 +1979,7 @@ When reading an `unknown` column, any corresponding column must be ignored and r | **`binary`** | `binary` | | | | **`struct`** | `struct` | | | | **`list`** | `array` | | | -| **`vector(E, D)`** | `array` | | Array elements must use the ORC mapping for E and must not be null. Readers and writers must enforce the vector dimension and element requirements. | +| **`vector(E, D)`** | Not supported | | | | **`map`** | `map` | | | | **`variant`** | `struct` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs. | `iceberg.struct-type`=`VARIANT` | Shredding is not supported in ORC. | | **`geometry`** | `binary` | `iceberg.binary-type`=`GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | @@ -2295,7 +2297,7 @@ Default value expressions are added in v4. A field's `write-default` may be a [v The `file` type is added in v4. -The `vector` type is added in v4. Writing `vector` into a v3 or earlier schema is invalid. +The `vector` type is added in v4. Relative path support is added in v4. From 7cb99d0df82df85af23b9ba100a4cbc64480b639 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Wed, 14 Oct 2026 01:08:50 +0000 Subject: [PATCH 03/18] start updating per feedback Defer the vector identifier, sorting and bounds restrictions while retaining the reviewed value, promotion and equality wording. Generated-by: Codex --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 7c9a4104d1c9..6a9f015b94e3 100644 --- a/format/spec.md +++ b/format/spec.md @@ -316,7 +316,7 @@ A non-null `vector(E, D)` contains exactly D elements of type E, and every eleme A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` of element type E is not a `vector(E, D)`, even when every value has length D. -Equality of vector values is element-wise, using the equality rules of element type. A vector must not be an identifier field. No sort order is defined for a vector value. Writers must not write lower or upper bounds for a vector field. +Equality of vector values is element-wise, using the equality rules of element type. ##### CRS From e314371094f1bda0a902dba59bddee9f8dc1c47e Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Wed, 14 Oct 2026 16:34:39 +0000 Subject: [PATCH 04/18] SPEC: Exclude vector fields from bounds statistics Keep value and null counts available for vectors while excluding lower, upper, and tight bounds. Generated-by: Codex --- format/spec.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index 6a9f015b94e3..80bac15ad759 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1121,8 +1121,8 @@ Each stats struct holds statistics for one table field. It may contain the follo | Requirement | Offset | Name | Type | Included for | Description | |-------------|--------|---------------------------|---------------------------|-----------------------------------------------|-------------| -| _optional_ | 1 | `lower_bound` | Field type or `geo_lower` | all primitives or `variant` | Lower bound stored as the field's type, or `geo_lower` for geo types | -| _optional_ | 2 | `upper_bound` | Field type or `geo_upper` | all primitives or `variant` | Upper bound stored as the field's type, or `geo_upper` for geo types | +| _optional_ | 1 | `lower_bound` | Field type or `geo_lower` | `variant` or all primitives except `vector` | Lower bound stored as the field's type, or `geo_lower` for geo types | +| _optional_ | 2 | `upper_bound` | Field type or `geo_upper` | `variant` or all primitives except `vector` | Upper bound stored as the field's type, or `geo_upper` for geo types | | _optional_ | 3 | `value_count` | `long` | all | Number of values in the column (including null and NaN values) | | _optional_ | 4 | `null_value_count` | `long` | optional fields | Number of null values in the column | | _optional_ | 5 | `nan_value_count` | `long` | `float`, `double` | Number of NaN values in the column | From aa7af5d9493fc75181d59cea83215001f947eaf1 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Wed, 14 Oct 2026 20:40:11 +0000 Subject: [PATCH 05/18] SPEC: Restrict vector defaults to null Keep non-null vector defaults out of the initial proposal, consistent with variant and geospatial types. Generated-by: Codex --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 80bac15ad759..bc854c67c1fc 100644 --- a/format/spec.md +++ b/format/spec.md @@ -394,7 +394,7 @@ Starting in v4, a field's `write-default` may be a [value expression](expression The `initial-default` must always be a literal value. A literal is itself a value expression, so existing `write-default` values remain valid value expressions. -All columns of `unknown`, `variant`, `geometry`, `geography`, and `file` types must default to null. Non-null values for `initial-default` or `write-default` are invalid. +All columns of `unknown`, `variant`, `geometry`, `geography`, `file`, and `vector` types must default to null. Non-null values for `initial-default` or `write-default` are invalid. Default values for the fields of a struct are tracked as `initial-default` and `write-default` at the field level. Default values for fields that are nested structs must not contain default values for the struct's fields (sub-fields). Sub-field defaults are tracked in sub-field's metadata. As a result, the default stored for a nested struct may be either null or a non-null struct with no field values. The effective default value is produced by setting each fields' default in a new struct. From c7cb4169995a492ba046a546f2ac9783edcb1659 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Thu, 15 Oct 2026 00:40:49 +0000 Subject: [PATCH 06/18] SPEC: Define vector Parquet mapping using LIST Use the existing LIST annotation and state the vector layout and element constraints without depending on the unmerged Parquet VECTOR proposal. Generated-by: Codex --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index bc854c67c1fc..fab5747bffda 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1945,7 +1945,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo | **`binary`** | `binary` | | | | **`struct`** | `group` | | | | **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | -| **`vector(E, D)`** | `3-level list` | `VECTOR(D)` | The element must be required and use the Parquet mapping for E. See [apache/parquet-format#624](https://github.com/apache/parquet-format/pull/624). | +| **`vector(E, D)`** | `3-level list` | `LIST` | The outer group must contain a single repeated group named `list`. The `list` group must contain a single primitive field named `element`, marked `required` in the Parquet schema. The `element` field must use the same Parquet physical type as an Iceberg column of type E, following the mapping in this table. Each non-null vector must contain exactly D non-null, finite elements. | | **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | | **`variant`** | `group` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. | `VARIANT` | See Parquet docs for [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [Variant shredding encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md). | | **`geometry`** | `binary` | `GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | From 7eb991e64a3408dd0bcb27fd953884e8267fbda6 Mon Sep 17 00:00:00 2001 From: yyanyy Date: Wed, 21 Oct 2026 16:36:38 -0700 Subject: [PATCH 07/18] Update format/spec.md Co-authored-by: Steven Zhen Wu --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index fab5747bffda..3ddd647281f1 100644 --- a/format/spec.md +++ b/format/spec.md @@ -312,7 +312,7 @@ For details on how to serialize a schema to JSON, see Appendix C. ##### Vector -A non-null `vector(E, D)` contains exactly D elements of type E, and every element is non-null. `float` and `double` elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. The vector value may be null only when the vector field is optional. +A `vector(E, D)` contains exactly D elements of type E, and every element is non-null. `float` and `double` elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. The vector value may be null only when the vector field is optional. A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` of element type E is not a `vector(E, D)`, even when every value has length D. From 246d61a57b075605c526733f91e32a3dbe29df59 Mon Sep 17 00:00:00 2001 From: yyanyy Date: Wed, 21 Oct 2026 16:40:27 -0700 Subject: [PATCH 08/18] Update format/spec.md Co-authored-by: Steven Zhen Wu --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 3ddd647281f1..a4c935f27e17 100644 --- a/format/spec.md +++ b/format/spec.md @@ -314,7 +314,7 @@ For details on how to serialize a schema to JSON, see Appendix C. A `vector(E, D)` contains exactly D elements of type E, and every element is non-null. `float` and `double` elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. The vector value may be null only when the vector field is optional. -A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` of element type E is not a `vector(E, D)`, even when every value has length D. +A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` of element type E is not a `vector(E, D)`, even if every list instance contains exactly D elements. Equality of vector values is element-wise, using the equality rules of element type. From 6980844f08444747d6afa7c1c2f2e2709be7808f Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Thu, 12 Nov 2026 01:32:44 +0000 Subject: [PATCH 09/18] SPEC: Clarify vector Avro IDs and document v4 support Keep Avro vector arrays consistent with Iceberg element identity and include vector in the schema type documentation. Generated-by: Codex --- docs/docs/schemas.md | 1 + format/spec.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/docs/schemas.md b/docs/docs/schemas.md index 7448e858da67..5e22e79cab2a 100644 --- a/docs/docs/schemas.md +++ b/docs/docs/schemas.md @@ -40,6 +40,7 @@ Iceberg tables support the following types: | **`uuid`** | Universally unique identifiers | | | **`fixed(L)`** | Fixed-length byte array of length L | | | **`binary`** | Arbitrary-length byte array | | +| **`vector(E, D)`** | D-dimensional vector of finite numeric values with element type E | Added in v4 | | **`variant`** | Semi-structured data (JSON-like) | Added in v3 | | **`geometry(C)`** | Geospatial features with CRS parameter | Linear edge-interpolation; added in v3 | | **`geography(C, A)`** | Geospatial features with CRS and edge algorithm | Non-linear edge-interpolation; added in v3 | diff --git a/format/spec.md b/format/spec.md index a4c935f27e17..3f57095c903b 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1888,7 +1888,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`binary`**|`bytes`|| |**`struct`**|`record`|| |**`list`**|`array`|| -|**`vector(E, D)`**|`array`|Array elements must use the Avro mapping for E and must not be null. Readers and writers must enforce the vector dimension and element requirements.| +|**`vector(E, D)`**|`array`|Array elements must use the Avro mapping for E and must not be null. The array must not have an `element-id` property. Readers and writers must enforce the vector dimension and element requirements.| |**`map`**|`array` of key-value records, or `map` when keys are strings (optional).|Array storage must use logical type name `map` and must store elements that are 2-field records. The first field is a non-null key and the second field is the value.| |**`variant`**|`record` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. |Shredding is not supported in Avro.| |**`geometry`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)| From 61afb96d94346a806b93f4bde84cee971d9aefe3 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Thu, 12 Nov 2026 01:38:38 +0000 Subject: [PATCH 10/18] SPEC: List vector after v3 geospatial types Keep the new v4 type after existing v3 types in the schema type tables. Generated-by: Codex --- docs/docs/schemas.md | 2 +- format/spec.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/docs/schemas.md b/docs/docs/schemas.md index 5e22e79cab2a..1de134c33cec 100644 --- a/docs/docs/schemas.md +++ b/docs/docs/schemas.md @@ -40,11 +40,11 @@ Iceberg tables support the following types: | **`uuid`** | Universally unique identifiers | | | **`fixed(L)`** | Fixed-length byte array of length L | | | **`binary`** | Arbitrary-length byte array | | -| **`vector(E, D)`** | D-dimensional vector of finite numeric values with element type E | Added in v4 | | **`variant`** | Semi-structured data (JSON-like) | Added in v3 | | **`geometry(C)`** | Geospatial features with CRS parameter | Linear edge-interpolation; added in v3 | | **`geography(C, A)`** | Geospatial features with CRS and edge algorithm | Non-linear edge-interpolation; added in v3 | | **`unknown`** | Placeholder type for undetermined columns | Must be optional; added in v3 | +| **`vector(E, D)`** | D-dimensional vector of finite numeric values with element type E | Added in v4 | | **`struct<...>`** | A record with named fields of any data type | | | **`list`** | A list with elements of any data type | | | **`map`** | A map with keys and values of any data type | | diff --git a/format/spec.md b/format/spec.md index 3f57095c903b..dfa45acf830d 100644 --- a/format/spec.md +++ b/format/spec.md @@ -296,9 +296,9 @@ Supported primitive types are defined in the table below. Primitive types added | | **`uuid`** | Universally unique identifiers | Should use 16-byte fixed | | | **`fixed(L)`** | Fixed-length byte array of length L | | | | **`binary`** | Arbitrary-length byte array | | -| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite numeric values with element type E and dimension D | E must be `int`, `long`, `float`, or `double`; D must be positive | | [v3](#version-3) | **`geometry(C)`** | Geospatial features from [OGC – Simple feature access][1001]. Edge-interpolation is always linear/planar. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C. If not specified, C is `OGC:CRS84`. | | | [v3](#version-3) | **`geography(C, A)`** | Geospatial features from [OGC – Simple feature access][1001]. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C and edge-interpolation algorithm A. If not specified, C is `OGC:CRS84` and A is `spherical`. | +| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite numeric values with element type E and dimension D | E must be `int`, `long`, `float`, or `double`; D must be positive | Notes: From aaf14a58a9aacc4cdb93e54e0520b40710b3ef67 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Thu, 12 Nov 2026 01:56:54 +0000 Subject: [PATCH 11/18] SPEC: Align identity and bucket transform type allowlists Preserve identity eligibility, including vector and unknown, with explicit source types in both specifications. Match expression bucket inputs to the table specification. Generated-by: Codex --- format/expressions-spec.md | 4 ++-- format/spec.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/format/expressions-spec.md b/format/expressions-spec.md index 924c2c796a66..ff6bd83c755d 100644 --- a/format/expressions-spec.md +++ b/format/expressions-spec.md @@ -210,7 +210,7 @@ All partition transforms produce `null` for a `null` input value. | Function name | Description | Source types | Result type | |-------------------|--------------------------------------------------------------|----------------------------------------------------------------------|-------------| -| `identity(value)` | Source value, unmodified | Any primitive except for `geometry` and `geography` | Source type | +| `identity(value)` | Source value, unmodified | `unknown`, `boolean`, `int`, `long`, `float`, `double`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary`, `vector` | Source type | | `year(value)` | Extract a date or timestamp year, as years from 1970 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | | `month(value)` | Extract a date or timestamp month, as months from 1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | | `day(value)` | Extract a date or timestamp day, as days from 1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `date` | @@ -222,7 +222,7 @@ Note that `year`, `month`, and `hour` transforms produce ordinal values and not | Parameterized function name | Description | Source types | Result type | |-----------------------------|-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------|-------------| -| `bucket(N, value)` | Hash of value, mod `N` (see [table spec details][bucket-ref]) | Any primitive except for `geometry`, `geography`, `boolean`, `float`, or `double` | `int` | +| `bucket(N, value)` | Hash of value, mod `N` (see [table spec details][bucket-ref]) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | | `truncate(W, value)` | Value truncated to width `W` (see [table spec details][truncate-ref]) | `int`, `long`, `decimal`, `string`, `binary` | Source type | [bucket-ref]: spec.md#bucket-transform-details diff --git a/format/spec.md b/format/spec.md index dfa45acf830d..59eda0fb34f8 100644 --- a/format/spec.md +++ b/format/spec.md @@ -656,7 +656,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | Transform name | Description | Source types | Result type | |-------------------|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-------------| -| **`identity`** | Source value, unmodified | Any primitive except for `geometry` and `geography` | Source type | +| **`identity`** | Source value, unmodified | `unknown`, `boolean`, `int`, `long`, `float`, `double`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary`, `vector` | Source type | | **`bucket[N]`** | Hash of value, mod `N` (see below) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | | **`truncate[W]`** | Value truncated to width `W` (see below) | `int`, `long`, `decimal`, `string`, `binary` | Source type | | **`year`** | Extract a date or timestamp year, as years from 1970 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | From 27a0b4a1b51494e44290bc14ec6f668a353889fe Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Thu, 12 Nov 2026 01:58:40 +0000 Subject: [PATCH 12/18] SPEC: Restore original identity transform type rules Restore the pre-PR identity wording in both specifications while retaining the bucket allowlist correction. Generated-by: Codex --- format/expressions-spec.md | 2 +- format/spec.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/format/expressions-spec.md b/format/expressions-spec.md index ff6bd83c755d..e90710e481e5 100644 --- a/format/expressions-spec.md +++ b/format/expressions-spec.md @@ -210,7 +210,7 @@ All partition transforms produce `null` for a `null` input value. | Function name | Description | Source types | Result type | |-------------------|--------------------------------------------------------------|----------------------------------------------------------------------|-------------| -| `identity(value)` | Source value, unmodified | `unknown`, `boolean`, `int`, `long`, `float`, `double`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary`, `vector` | Source type | +| `identity(value)` | Source value, unmodified | Any primitive except for `geometry` and `geography` | Source type | | `year(value)` | Extract a date or timestamp year, as years from 1970 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | | `month(value)` | Extract a date or timestamp month, as months from 1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | | `day(value)` | Extract a date or timestamp day, as days from 1970-01-01 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `date` | diff --git a/format/spec.md b/format/spec.md index 59eda0fb34f8..dfa45acf830d 100644 --- a/format/spec.md +++ b/format/spec.md @@ -656,7 +656,7 @@ Partition field IDs must be reused if an existing partition spec contains an equ | Transform name | Description | Source types | Result type | |-------------------|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-------------| -| **`identity`** | Source value, unmodified | `unknown`, `boolean`, `int`, `long`, `float`, `double`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary`, `vector` | Source type | +| **`identity`** | Source value, unmodified | Any primitive except for `geometry` and `geography` | Source type | | **`bucket[N]`** | Hash of value, mod `N` (see below) | `int`, `long`, `decimal`, `date`, `time`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns`, `string`, `uuid`, `fixed`, `binary` | `int` | | **`truncate[W]`** | Value truncated to width `W` (see below) | `int`, `long`, `decimal`, `string`, `binary` | Source type | | **`year`** | Extract a date or timestamp year, as years from 1970 | `date`, `timestamp`, `timestamptz`, `timestamp_ns`, `timestamptz_ns` | `int` | From fc630142ea17e8cc09bcfa5e73e282c334ee1cc8 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Thu, 12 Nov 2026 22:01:07 +0000 Subject: [PATCH 13/18] SPEC: Align vector ordering, constraints, and Parquet mapping Define lexicographic ordering, exclude vectors from unique and primary-key constraints, and follow the Parquet FINITE_VECTOR definition while permitting representations that preserve Iceberg element values exactly. Generated-by: Codex --- format/spec.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index dfa45acf830d..cae07bdebd72 100644 --- a/format/spec.md +++ b/format/spec.md @@ -312,7 +312,7 @@ For details on how to serialize a schema to JSON, see Appendix C. ##### Vector -A `vector(E, D)` contains exactly D elements of type E, and every element is non-null. `float` and `double` elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. The vector value may be null only when the vector field is optional. +A non-null `vector(E, D)` contains exactly D elements of type E, and every element is non-null. `float` and `double` elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. The vector value may be null only when the vector field is optional. A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` of element type E is not a `vector(E, D)`, even if every list instance contains exactly D elements. @@ -739,6 +739,8 @@ Order id `0` is reserved for the unsorted order. Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons. +Vector values of the same `vector(E, D)` type must be sorted lexicographically using the sort order of E. + A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. ### Constraints @@ -777,6 +779,7 @@ The `field-ids` of a `unique` or `primary-key` constraint: * Must not reference fields of type `geometry` or `geography`, because Iceberg does not define equality for them * Must not reference fields of type `unknown`, because their values are always null * Must not reference fields of type `file` or their sub-fields +* Must not reference fields of type `vector` A table may have at most one `primary-key` constraint. The fields of a `primary-key` constraint must be `required` and must not be nested in an optional struct, so that the schema rejects null keys even when the constraint is not enforced. The fields of a `unique` constraint may be optional or nested in optional structs, because a row in which any field is null is not compared. @@ -1924,6 +1927,8 @@ Values should be stored in Parquet using the types and logical type annotations Lists must use the [3-level representation](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#lists). +Vectors must use the [`FINITE_VECTOR` representation](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#finite-vectors), with `num_elements` set to D. + | Type | Parquet physical type | Logical type | Notes | |--------------------|----------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|----------------------------------------------------------------| | **`unknown`** | None | | Omit from data files | @@ -1945,7 +1950,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo | **`binary`** | `binary` | | | | **`struct`** | `group` | | | | **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | -| **`vector(E, D)`** | `3-level list` | `LIST` | The outer group must contain a single repeated group named `list`. The `list` group must contain a single primitive field named `element`, marked `required` in the Parquet schema. The `element` field must use the same Parquet physical type as an Iceberg column of type E, following the mapping in this table. Each non-null vector must contain exactly D non-null, finite elements. | +| **`vector(E, D)`** | `3-level list` | `FINITE_VECTOR(D)` | The outer group carries the field ID of the vector. The `element` field must represent values of the Iceberg element type E using a Parquet representation permitted by `FINITE_VECTOR`. Writing and reading an element must preserve its value exactly. The ConvertedType must be `LIST`. The `list` and `element` fields must not be assigned field IDs. | | **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | | **`variant`** | `group` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. | `VARIANT` | See Parquet docs for [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [Variant shredding encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md). | | **`geometry`** | `binary` | `GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | From bd621c9136724b4a808db774f30454683c69a043 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Tue, 17 Nov 2026 21:30:45 +0000 Subject: [PATCH 14/18] SPEC: Define vector element types and float16 storage Separate vector element types from scalar Iceberg types and specify their value types and Parquet mappings. Use value types for Avro, JSON, equality, and sorting. Generated-by: Codex --- format/spec.md | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/format/spec.md b/format/spec.md index cae07bdebd72..9f30af9d4bbf 100644 --- a/format/spec.md +++ b/format/spec.md @@ -298,7 +298,7 @@ Supported primitive types are defined in the table below. Primitive types added | | **`binary`** | Arbitrary-length byte array | | | [v3](#version-3) | **`geometry(C)`** | Geospatial features from [OGC – Simple feature access][1001]. Edge-interpolation is always linear/planar. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C. If not specified, C is `OGC:CRS84`. | | | [v3](#version-3) | **`geography(C, A)`** | Geospatial features from [OGC – Simple feature access][1001]. See [Appendix G](#appendix-g-geospatial-notes). Parameterized by CRS C and edge-interpolation algorithm A. If not specified, C is `OGC:CRS84` and A is `spherical`. | -| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of finite numeric values with element type E and dimension D | E must be `int`, `long`, `float`, or `double`; D must be positive | +| [v4](#version-4) | **`vector(E, D)`** | Fixed-length collection of D finite numeric values with element type E | E must be a [vector element type](#vector); D must be positive | Notes: @@ -312,11 +312,23 @@ For details on how to serialize a schema to JSON, see Appendix C. ##### Vector -A non-null `vector(E, D)` contains exactly D elements of type E, and every element is non-null. `float` and `double` elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. The vector value may be null only when the vector field is optional. +A non-null `vector(E, D)` contains exactly D elements of element type E, and every element is non-null. The vector value may be null only when the vector field is optional. -A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` of element type E is not a `vector(E, D)`, even if every list instance contains exactly D elements. +E must be one of the following vector element types: -Equality of vector values is element-wise, using the equality rules of element type. +| Element type | Element values | Value type | +|---------------|--------------------------------------------------------------------|------------| +| **`int`** | 32-bit signed integers | `int` | +| **`long`** | 64-bit signed integers | `long` | +| **`float16`** | `float` values that are exactly representable as IEEE 754 binary16 | `float` | +| **`float`** | 32-bit IEEE 754 floating point | `float` | +| **`double`** | 64-bit IEEE 754 floating point | `double` | + +Vector element types are valid only inside a `vector`; `float16` is not a primitive type and cannot be used as the type of a field. Element values follow the rules of the value type, including equality and JSON single-value serialization. Floating-point elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. + +A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` is not a `vector(E, D)`, even if every list instance contains exactly D elements that satisfy the vector element requirements. + +Equality of vector values is element-wise, using the equality rules of the value type. ##### CRS @@ -739,7 +751,7 @@ Order id `0` is reserved for the unsorted order. Sorting floating-point numbers should produce the following behavior: `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < `Infinity` < `NaN`. This aligns with the implementation of Java floating-point types comparisons. -Vector values of the same `vector(E, D)` type must be sorted lexicographically using the sort order of E. +Vector values of the same `vector(E, D)` type must be sorted lexicographically using the sort order of E's value type. A data or delete file is associated with a sort order by the sort order's id within [a manifest](#manifests). Therefore, the table must declare all the sort orders for lookup. A table could also be configured with a default sort order id, indicating how the new data should be sorted by default. Writers should use this default sort order to sort the data on write, but are not required to if the default order is prohibitively expensive, as it would be for streaming writes. @@ -1891,7 +1903,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`binary`**|`bytes`|| |**`struct`**|`record`|| |**`list`**|`array`|| -|**`vector(E, D)`**|`array`|Array elements must use the Avro mapping for E and must not be null. The array must not have an `element-id` property. Readers and writers must enforce the vector dimension and element requirements.| +|**`vector(E, D)`**|`array`|Array elements must use the Avro mapping for E's value type and must not be null. The array must not have an `element-id` property. Readers and writers must enforce the vector dimension and element requirements.| |**`map`**|`array` of key-value records, or `map` when keys are strings (optional).|Array storage must use logical type name `map` and must store elements that are 2-field records. The first field is a non-null key and the second field is the value.| |**`variant`**|`record` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. |Shredding is not supported in Avro.| |**`geometry`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)| @@ -1950,13 +1962,23 @@ Vectors must use the [`FINITE_VECTOR` representation](https://github.com/apache/ | **`binary`** | `binary` | | | | **`struct`** | `group` | | | | **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | -| **`vector(E, D)`** | `3-level list` | `FINITE_VECTOR(D)` | The outer group carries the field ID of the vector. The `element` field must represent values of the Iceberg element type E using a Parquet representation permitted by `FINITE_VECTOR`. Writing and reading an element must preserve its value exactly. The ConvertedType must be `LIST`. The `list` and `element` fields must not be assigned field IDs. | +| **`vector(E, D)`** | `3-level list` | `FINITE_VECTOR(D)` | The outer group carries the field ID of the vector. The `element` field must use the Parquet type for E listed in the vector element table below. Writing and reading an element must preserve its value exactly. The ConvertedType must be `LIST`. The `list` and `element` fields must not be assigned field IDs. | | **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | | **`variant`** | `group` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. | `VARIANT` | See Parquet docs for [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [Variant shredding encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md). | | **`geometry`** | `binary` | `GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | | **`geography`** | `binary` | `GEOGRAPHY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | | **`file`** | `group` with the `file` sub-fields. Sub-fields must be assigned field IDs. | `FILE` | See Parquet docs for the [`FILE` type](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file) and [File Type](#file-type). | +Vector elements are stored in Parquet using the following types: + +| Element type | Parquet physical type | Logical type | +|---------------|---------------------------|--------------| +| **`int`** | `int32` | | +| **`long`** | `int64` | | +| **`float16`** | `fixed_len_byte_array[2]` | `FLOAT16` | +| **`float`** | `float` | | +| **`double`** | `double` | | + When reading an `unknown` column, any corresponding column must be ignored and replaced with `null` values. ### ORC @@ -2289,7 +2311,7 @@ The binary single-value serialization can be used to store the lower and upper b | **`binary`** | **`JSON string`** | `"000102ff"` | Stored as a hexadecimal string | | **`struct`** | **`JSON object by field ID`** | `{"1": 1, "2": "bar"}` | Stores struct fields using the field ID as the JSON field name; field values are stored using this JSON single-value format | | **`list`** | **`JSON array of values`** | `[1, 2, 3]` | Stores a JSON array of values that are serialized using this JSON single-value format | -| **`vector(E, D)`** | **`JSON array of values`** | `[1.0, 2.0, 3.0]` | Stores exactly D values using E's JSON single-value format | +| **`vector(E, D)`** | **`JSON array of values`** | `[1.0, 2.0, 3.0]` | Stores exactly D values using the JSON single-value format of E's value type | | **`map`** | **`JSON object of key and value arrays`** | `{ "keys": ["a", "b"], "values": [1, 2] }` | Stores arrays of keys and values; individual keys and values are serialized using this JSON single-value format | | **`geometry`** | **`JSON string`** | `POINT (30 10)` | Stored using WKT representation, see [Appendix G](#appendix-g-geospatial-notes) | | **`geography`** | **`JSON string`** | `POINT (30 10)` | Stored using WKT representation, see [Appendix G](#appendix-g-geospatial-notes) | From d6e35943f9cf4eaa2d1d1e45c16462ecdac478dd Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Tue, 17 Nov 2026 23:22:54 +0000 Subject: [PATCH 15/18] SPEC: Allow lossless vector element type promotion Define the permitted element widenings, including float16, while preserving vector dimensions and field IDs. Require exact reads of older element encodings and explicit compatibility for type-dependent vector artifacts. Generated-by: Codex --- format/spec.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/format/spec.md b/format/spec.md index 9f30af9d4bbf..e8822434decd 100644 --- a/format/spec.md +++ b/format/spec.md @@ -326,7 +326,19 @@ E must be one of the following vector element types: Vector element types are valid only inside a `vector`; `float16` is not a primitive type and cannot be used as the type of a field. Element values follow the rules of the value type, including equality and JSON single-value serialization. Floating-point elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. -A `vector` must not be promoted to another type. No type other than `unknown` may be promoted to `vector`. The element type and dimension must not change. A `list` is not a `vector(E, D)`, even if every list instance contains exactly D elements that satisfy the vector element requirements. +The element type of a `vector(E, D)` may be promoted according to the following table. The dimension D and the vector field ID must not change. + +| Element type | Permitted promotion targets | +|--------------|-----------------------------| +| `int` | `long` | +| `float16` | `float`, `double` | +| `float` | `double` | + +All other element type changes are prohibited. A `vector` must not be promoted to a non-vector type. A non-vector type other than `unknown` must not be promoted to `vector`. A `list` is not a `vector(E, D)`, even if every list instance contains exactly D elements that satisfy the vector element requirements. + +Readers must accept vectors written with the requested element type or an element type that can be promoted to it, preserving each element's value exactly. The stored and requested dimensions must match. + +After a vector element type promotion, readers must ignore type-dependent vector-derived artifacts computed for an earlier element type, such as statistics or index blobs, unless the artifact format explicitly defines compatibility with the promoted type. Equality of vector values is element-wise, using the equality rules of the value type. @@ -436,6 +448,7 @@ Valid primitive type promotions are: | `date` | | `timestamp`, `timestamp_ns` | Promotion to `timestamptz` or `timestamptz_ns` is **not** allowed; values outside the promoted type's range must result in a runtime failure | | `float` | `double` | `double` | | | `decimal(P, S)` | `decimal(P', S)` if `P' > P` | `decimal(P', S)` if `P' > P` | Widen precision only | +| `vector(E, D)` | | `vector(E', D)` | v4 only; E to E' must be a permitted [vector element type promotion](#vector); D must not change | Iceberg's Avro manifest format does not store the type of lower and upper bounds, and type promotion does not rewrite existing bounds. For example, when a `float` is promoted to `double`, existing data file bounds are encoded as 4 little-endian bytes rather than 8 little-endian bytes for `double`. To correctly decode the value, the original type at the time the file was written must be inferred according to the following table: @@ -1903,7 +1916,7 @@ Maps with non-string keys must use an array representation with the `map` logica |**`binary`**|`bytes`|| |**`struct`**|`record`|| |**`list`**|`array`|| -|**`vector(E, D)`**|`array`|Array elements must use the Avro mapping for E's value type and must not be null. The array must not have an `element-id` property. Readers and writers must enforce the vector dimension and element requirements.| +|**`vector(E, D)`**|`array`|When writing, array elements must use the Avro mapping for E's value type and must not be null. The array must not have an `element-id` property. Readers and writers must enforce the vector dimension and element requirements.| |**`map`**|`array` of key-value records, or `map` when keys are strings (optional).|Array storage must use logical type name `map` and must store elements that are 2-field records. The first field is a non-null key and the second field is the value.| |**`variant`**|`record` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. |Shredding is not supported in Avro.| |**`geometry`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)| @@ -1962,14 +1975,14 @@ Vectors must use the [`FINITE_VECTOR` representation](https://github.com/apache/ | **`binary`** | `binary` | | | | **`struct`** | `group` | | | | **`list`** | `3-level list` | `LIST` | See Parquet docs for 3-level representation. | -| **`vector(E, D)`** | `3-level list` | `FINITE_VECTOR(D)` | The outer group carries the field ID of the vector. The `element` field must use the Parquet type for E listed in the vector element table below. Writing and reading an element must preserve its value exactly. The ConvertedType must be `LIST`. The `list` and `element` fields must not be assigned field IDs. | +| **`vector(E, D)`** | `3-level list` | `FINITE_VECTOR(D)` | The outer group carries the field ID of the vector. When writing, the `element` field must use the Parquet type for E listed in the vector element table below. Writing and reading an element must preserve its value exactly. The ConvertedType must be `LIST`. The `list` and `element` fields must not be assigned field IDs. | | **`map`** | `3-level map` | `MAP` | See Parquet docs for 3-level representation. | | **`variant`** | `group` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. | `VARIANT` | See Parquet docs for [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [Variant shredding encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md). | | **`geometry`** | `binary` | `GEOMETRY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | | **`geography`** | `binary` | `GEOGRAPHY` | WKB format, see [Appendix G](#appendix-g-geospatial-notes). | | **`file`** | `group` with the `file` sub-fields. Sub-fields must be assigned field IDs. | `FILE` | See Parquet docs for the [`FILE` type](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file) and [File Type](#file-type). | -Vector elements are stored in Parquet using the following types: +Vector elements must be written to Parquet using the following types: | Element type | Parquet physical type | Logical type | |---------------|---------------------------|--------------| @@ -1979,6 +1992,8 @@ Vector elements are stored in Parquet using the following types: | **`float`** | `float` | | | **`double`** | `double` | | +When reading `vector(E, D)`, the `element` field may use the Parquet mapping for E or for an element type that can be promoted to E. Readers must convert elements to E without changing their values. The `FINITE_VECTOR` `num_elements` parameter must equal D. + When reading an `unknown` column, any corresponding column must be ignored and replaced with `null` values. ### ORC From 222f167b48946b05382dec50f5742ddba18e53f2 Mon Sep 17 00:00:00 2001 From: Yan Yan Date: Wed, 18 Nov 2026 01:48:04 +0000 Subject: [PATCH 16/18] SPEC: Simplify vector promotion and equality wording Remove the vector artifact compatibility policy, explain how readers handle older files after promotion, and define vector equality once. Generated-by: Codex --- format/spec.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/format/spec.md b/format/spec.md index e8822434decd..d617a7ff8f00 100644 --- a/format/spec.md +++ b/format/spec.md @@ -324,7 +324,7 @@ E must be one of the following vector element types: | **`float`** | 32-bit IEEE 754 floating point | `float` | | **`double`** | 64-bit IEEE 754 floating point | `double` | -Vector element types are valid only inside a `vector`; `float16` is not a primitive type and cannot be used as the type of a field. Element values follow the rules of the value type, including equality and JSON single-value serialization. Floating-point elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. +Vector element types are valid only inside a `vector`; `float16` is not a primitive type and cannot be used as the type of a field. Element values follow the rules of the value type, including JSON single-value serialization. Vector equality is element-wise, using the equality rules of the value type. Floating-point elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. The element type of a `vector(E, D)` may be promoted according to the following table. The dimension D and the vector field ID must not change. @@ -336,11 +336,7 @@ The element type of a `vector(E, D)` may be promoted according to the following All other element type changes are prohibited. A `vector` must not be promoted to a non-vector type. A non-vector type other than `unknown` must not be promoted to `vector`. A `list` is not a `vector(E, D)`, even if every list instance contains exactly D elements that satisfy the vector element requirements. -Readers must accept vectors written with the requested element type or an element type that can be promoted to it, preserving each element's value exactly. The stored and requested dimensions must match. - -After a vector element type promotion, readers must ignore type-dependent vector-derived artifacts computed for an earlier element type, such as statistics or index blobs, unless the artifact format explicitly defines compatibility with the promoted type. - -Equality of vector values is element-wise, using the equality rules of the value type. +After an element type promotion, readers must still read older files by converting vector elements to the new type without changing their values. The stored dimension must match the schema. ##### CRS From 15c555e9832472cae0f8da7b3f7a11102265d8d4 Mon Sep 17 00:00:00 2001 From: pvary Date: Wed, 18 Nov 2026 16:41:19 +0100 Subject: [PATCH 17/18] Just table indentation changes --- format/spec.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/format/spec.md b/format/spec.md index d617a7ff8f00..a3fdfbdc9ccb 100644 --- a/format/spec.md +++ b/format/spec.md @@ -320,9 +320,9 @@ E must be one of the following vector element types: |---------------|--------------------------------------------------------------------|------------| | **`int`** | 32-bit signed integers | `int` | | **`long`** | 64-bit signed integers | `long` | -| **`float16`** | `float` values that are exactly representable as IEEE 754 binary16 | `float` | -| **`float`** | 32-bit IEEE 754 floating point | `float` | -| **`double`** | 64-bit IEEE 754 floating point | `double` | +| **`float16`** | `float` values that are exactly representable as IEEE 754 binary16 | `float` | +| **`float`** | 32-bit IEEE 754 floating point | `float` | +| **`double`** | 64-bit IEEE 754 floating point | `double` | Vector element types are valid only inside a `vector`; `float16` is not a primitive type and cannot be used as the type of a field. Element values follow the rules of the value type, including JSON single-value serialization. Vector equality is element-wise, using the equality rules of the value type. Floating-point elements must be finite: `NaN`, `+Infinity`, and `-Infinity` are invalid. @@ -331,7 +331,7 @@ The element type of a `vector(E, D)` may be promoted according to the following | Element type | Permitted promotion targets | |--------------|-----------------------------| | `int` | `long` | -| `float16` | `float`, `double` | +| `float16` | `float`, `double` | | `float` | `double` | All other element type changes are prohibited. A `vector` must not be promoted to a non-vector type. A non-vector type other than `unknown` must not be promoted to `vector`. A `list` is not a `vector(E, D)`, even if every list instance contains exactly D elements that satisfy the vector element requirements. From 57981ff5c6b8def7905e99cdf7c9203ee50074d5 Mon Sep 17 00:00:00 2001 From: pvary Date: Wed, 18 Nov 2026 16:47:32 +0100 Subject: [PATCH 18/18] Another table layout fix --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index a3fdfbdc9ccb..47e6d98db6e9 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1984,7 +1984,7 @@ Vector elements must be written to Parquet using the following types: |---------------|---------------------------|--------------| | **`int`** | `int32` | | | **`long`** | `int64` | | -| **`float16`** | `fixed_len_byte_array[2]` | `FLOAT16` | +| **`float16`** | `fixed_len_byte_array[2]` | `FLOAT16` | | **`float`** | `float` | | | **`double`** | `double` | |