Skip to content

Vector Type Support #18297

Open
Open

Description

@PFischbeck

Proposed Change

Add a new primitive type for numeric vector data, i.e., a fixed-size list of numeric values (e.g., floats or integers) to be used in the context of vector embeddings.

For Parquet handling, the new Iceberg type should be based on Parquet spec PR that introduces a new Parquet logical type for vectors.

Mailing list discussion thread: https://lists.apache.org/thread/4qj3ogom42lt4kf2jzz0y5yfk35z82pb

Proposal document

https://docs.google.com/document/d/1zA8PskNDFKXXJpzWQr25CFoX_aIZHBNHbkxk0t3PteE/edit?usp=sharing

Specifications

  • Table
  • View
  • REST
  • Puffin
  • Encryption
  • Other

Activity

  1. added
    proposalIceberg Improvement Proposal (spec/major changes/etc)
    on Sep 28, 2026
  2. pvary commented on Sep 28, 2026

    @pvary
    Contributor

    Shall we start a recurring sync on this topic?

  3. added this to the Iceberg V4 Spec milestone on Sep 28, 2026
  4. stevenzwu commented on Sep 29, 2026

    @stevenzwu
    Contributor

    Shall we start a recurring sync on this topic?

    good idea. we will reuse/extend the file type sync to also cover the vector type.

  5. raghavyadav01 commented on Oct 23, 2026

    @raghavyadav01

    One note on the open metrics question. Making this a primitive has a side effect that is easy to miss:

    • MetricsConfig decides eligibility with type.isPrimitiveType() || type.isVariantType() (MetricsConfig.java:163), so a new primitive is metrics-eligible by default, for the first 100 columns.
    • The default mode is truncate(16), and truncation applies only to string and binary. Everything else is stored as-is.
    • So a float[768] column would write a full 768-float lower bound and upper bound into every manifest entry, roughly 6 KB per data file per vector column, in the metadata that scan planning reads.
    • The bounds would not buy much either. Lexicographic min/max over coordinates cannot prune a similarity search, which is the predicate these columns exist for.

    Might be worth the spec saying vector columns default to none rather than inheriting the default mode. If useful vector statistics turn up later, Puffin looks like the right home for them, next to the sketch work in #18516.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    proposalIceberg Improvement Proposal (spec/major changes/etc)

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions