You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a new primitive type for numeric vector data, i.e., a fixed-size list of numeric values (e.g., floats or integers) to be used in the context of vector embeddings.
For Parquet handling, the new Iceberg type should be based on Parquet spec PR that introduces a new Parquet logical type for vectors.
One note on the open metrics question. Making this a primitive has a side effect that is easy to miss:
MetricsConfig decides eligibility with type.isPrimitiveType() || type.isVariantType() (MetricsConfig.java:163), so a new primitive is metrics-eligible by default, for the first 100 columns.
The default mode is truncate(16), and truncation applies only to string and binary. Everything else is stored as-is.
So a float[768] column would write a full 768-float lower bound and upper bound into every manifest entry, roughly 6 KB per data file per vector column, in the metadata that scan planning reads.
The bounds would not buy much either. Lexicographic min/max over coordinates cannot prune a similarity search, which is the predicate these columns exist for.
Might be worth the spec saying vector columns default to none rather than inheriting the default mode. If useful vector statistics turn up later, Puffin looks like the right home for them, next to the sketch work in #18516.
Activity
pvary commentedon Sep 28, 2026
Shall we start a recurring sync on this topic?
stevenzwu commentedon Sep 29, 2026
good idea. we will reuse/extend the file type sync to also cover the vector type.
raghavyadav01 commentedon Oct 23, 2026
One note on the open metrics question. Making this a primitive has a side effect that is easy to miss:
MetricsConfigdecides eligibility withtype.isPrimitiveType() || type.isVariantType()(MetricsConfig.java:163), so a new primitive is metrics-eligible by default, for the first 100 columns.truncate(16), and truncation applies only to string and binary. Everything else is stored as-is.float[768]column would write a full 768-float lower bound and upper bound into every manifest entry, roughly 6 KB per data file per vector column, in the metadata that scan planning reads.Might be worth the spec saying vector columns default to
nonerather than inheriting the default mode. If useful vector statistics turn up later, Puffin looks like the right home for them, next to the sketch work in #18516.