Add RFC for Vector Search

[gggrace14]

This RFC proposes the syntax and SPI to natively support Approximate Nearest Neighbor (ANN) vector search within Presto.

[NivinCS]

Thanks, @gggrace14 , for the write-up

The proposal currently does not describe the lifecycle management of vector indexes with respect to underlying table mutations. In a lakehouse environment such as Iceberg, operations like appends, partition backfills, deletes, file rewrites (compaction), and snapshot expiration can make the index stale or physically invalid, especially if the index stores file-level row references.

Could we clarify how index validity is expected to be maintained across such operations? Specifically:

1. How is index staleness detected?
2. Are indexes expected to be snapshot-bound?
3. How should index cleanup or rebuild be handled after file rewrites or snapshot expiry?
4. Is automatic invalidation or maintenance of index metadata in scope?

This would help ensure correctness and avoid cases where ANN search may return outdated or invalid results.

[NivinCS]

After creating the vector index metadata without building it for any partitions, the expected query behavior is not entirely clear. For example, if a user runs:

SELECT *
FROM candidates_table
ORDER BY embedding <-> query_vector
LIMIT 10;

when the index is defined but not yet populated, should the engine fall back to a full scan, ignore the index, emit a warning, or fail the query? It would be helpful to clarify the intended behavior in this scenario to ensure predictable query execution.

[gggrace14]

It is by current design for the RFC to leave it to individual connector defining the behavior, when the index only has schema but has no data.

For example, at meta internal the current strategy is to have a condition, and build the index on the fly if the condition is met, and fall back to full scan otherwise. And we put this strategy into a ConnectorPlanOptimizer subclass.

I think it's too early to define such behavior or any behavior in SPI, and let every connector respect this behavior. A better approach is to let individual connector contribute the implementation first, and we extract the common behavior into SPI afterwards.

[NivinCS]

It would be beneficial to keep the PRIMARY KEY / ROW_ID specification optional rather than mandatory for vector index creation.

From the Iceberg V3 perspective, tables with Row Level Lineage enabled can provide a stable row-level identifier via $row_id. In such cases, the identifier column can be made optional, allowing the connector to automatically fall back to $row_id when a user-defined id is not specified.

This would avoid requiring users to explicitly provide a unique identifier for tables where lineage-based row identity is already available.

[gggrace14]

Hi @NivinCS , sure, make sense. I will change rfc to make the id field optional.

[NivinCS]

It may be beneficial to make the partition specification in UPDATE VECTOR INDEX (e.g., FOR PARTITION (...)) optional rather than mandatory.

Instead of requiring users to manually specify partition ranges for index updates, the connector could automatically detect changed partitions and perform incremental index refresh, similar to materialized view incremental refresh.

This would reduce the need for manual orchestration and improve usability for incremental index maintenance.
cc : @tdcmeehan

[gggrace14]

By design UPDATE VECTOR INDEX vector_index WHERE <filter> should be the Presto interface for incremental index refresh, similar to the REFRESH MATERIALIZED VIEW ... WHERE <filter>. I think the auto detection of partition change to the base table should be out of the scope of Presto. For example, we could possibly have a pub-sub service out of Presto that auto detects based on the base table metadata. Once the change to base table partition is detected, any auto mechanism could assemble the and call UPDATE VECTOR INDEX to instruct Presto to refresh the index increment.

[NivinCS]

It will be beneficial to include a reusable toolkit/library as part of the proposal to handle internal query rewrites for vector search operations.

In many cases, executing ANN search would otherwise require users to explicitly join the base table with the corresponding vector index using a row identifier (e.g., user-defined id or $row_id). This can expose index implementation details and make query syntax more complex, requiring users to understand how embeddings and index data are related internally.

Providing a reusable component that transparently rewrites vector search predicates (e.g., ANN_SEARCH(...)) into the appropriate join between the base table and vector index would enable a simpler and more seamless user experience. This would allow users to express ANN queries without manually specifying join logic, while the underlying connector leverages the index automatically.

Such a library could be integrated by connectors (e.g., Iceberg) to perform these rewrites behind the scenes, avoiding duplication of rewrite logic across connectors and keeping index implementation details abstracted from end users.
cc : @tdcmeehan

[aditi-pandit]

Thanks @gggrace14 for this writeup. Overall I'm in favor of this proposal.

I'm also curious since FAISS functions were recently exposed in C++.... Are you planning to do any follow up work with the library ?

[aditi-pandit]

Does the optimizer know about the Index as such ? The use of the index should be abstracted by VECTOR_SEARCH function, no ?

[gggrace14]

Whether the candidates table has an index or not would be maintained by metadata, which is accessible to optimizer. If the index is available, the optimizer could pick it up and rewrite the VECTOR_SEARCH query.

[aditi-pandit]

Are we missing the SPI for these ? Can you elaborate ?

[aditi-pandit]

Have you considered implementing VECTOR_SEARCH with a TABLE function ? It is easier to express the input tables, partitioning and parameters with it.

[gggrace14]

Hi @aditi-pandit , yes, we do plan to implement VECTOR_SEARCH as a TABLE function, on top of the table function framework.

Yea, also thought to follow up. In the prior TSC meeting, you mentioned that you have some experience or resources that you could share with us to implement the VECTOR_SEARCH function as a table function. Would you kindly provide more details? Thank you!

[aditi-pandit]

Can you give more details about VectorSearchNode and VectorSearchOptions. Will VectorSearchNode be a PlanNode ?

[gggrace14]

Sorry, Aditi, I think this paragraph is outdated. I'm removing this paragraph.

[bibith4]

It would be helpful to clarify the behavior when base table partitions are dropped, since index partitions are mapped to them. Should corresponding index partitions be automatically cleaned up or require manual action?

Additionally, it may be beneficial to define lifecycle DDL such as DROP VECTOR INDEX for operational usability.

[gggrace14]

Yea, I'm adding the DDL DROP VECTOR INDEX. A good suggestion.

I'm changing this RFC to add a getVectorSearchIndexStatus() method to the SPI class MetadataManager and ConnectorMetadata. Then the search path could decide whether to consume the index partitions given the index status.

Regarding the behavior when the base table partitions are dropped, I think we should leave to users to decide whether to drop the index partitions or not. In some case, the index building is quite expensive, and the user might not want to drop the index partitions until the index quality does not meet certain criteria. We could make suggestion in the RFC. At meta, currently we drop the index partitions. I'm also adding this clarification to the RFC.

[bibith4]

It would be helpful to clarify query behavior during index build or refresh. For example, can queries read partially built partitions?

[gggrace14]

We can add a getVectorSearchIndexStatus() method to the SPI class MetadataManager and ConnectorMetadata. It's better to leave it to the individual ConnectorPlanOptimizer to decide whether to pick up the partially built index partitions or now.

[Dilli-Babu-Godari]

what do the parameters nlist and nb represent?

1. How do these parameters affect indexing performance, recall, and query latency?
2. What guidelines should users follow to choose optimal values for different dataset sizes and vector dimensions?
3. Is there any automatic tuning mechanism available, or are users expected to benchmark and manually determine the best configuration?

[gggrace14]

Hi @Dilli-Babu-Godari , generally index_options is particular to individual index_type. And each connector is responsible for checking if the index_type is supported and if the index_options is valid for the index_type. (I'm adding this detail to this RFC)

Specifically,
1. How do these parameters affect indexing performance, recall, and query latency?

nlist is one option for any ivf_* index, which is the # of centroids. The typical trade-off around the # of centroids also hold here. Larger nlist means smaller cluster size. If the data distribution itself is good, it could largely reduce the search space, and thus reduce the amount of records loaded to each node as well as memory pressure. If the data distribution is not good, then we need to increase nprobe.

nb is one option for the index RabitQ. Higher nb provides better accuracy at the cost of additional storage, as in https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#if-maximum-compression-is-required-then-rabitq.

2. What guidelines should users follow to choose optimal values for different dataset sizes and vector dimensions?
For nlist, we can use the formula here, https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#if-below-1m-vectors-ivfk.
For nb, more bits means better accuracy but at the cost of memory.

3. Is there any automatic tuning mechanism available, or are users expected to benchmark and manually determine the best configuration?
At the moment, we derive default values in code for the options, if possible. This index_options parameter is what the user can use to customize the options. Automatic tuning is next next step, which needs the index quality feedback loop.

[gggrace14]

Thanks, @gggrace14 , for the write-up

The proposal currently does not describe the lifecycle management of vector indexes with respect to underlying table mutations. In a lakehouse environment such as Iceberg, operations like appends, partition backfills, deletes, file rewrites (compaction), and snapshot expiration can make the index stale or physically invalid, especially if the index stores file-level row references.

Could we clarify how index validity is expected to be maintained across such operations? Specifically:

1. How is index staleness detected?
2. Are indexes expected to be snapshot-bound?
3. How should index cleanup or rebuild be handled after file rewrites or snapshot expiry?
4. Is automatic invalidation or maintenance of index metadata in scope?

This would help ensure correctness and avoid cases where ANN search may return outdated or invalid results.

Hi @NivinCS , in general it is designed to define the minimum required parts to be SPI and leave individual connectors with the max flexibility to define its own behavior.

Specifically with respect to how index validity is maintained, we could leave it to individual ConnectorPlanOptimizer to detect if the index partition is stale and return reasonable result when the ConnectorPlanOptimizer plans/rewrites the vector search query.

For SPI, thinking through your questions, I think what could be added here is a getVectorSearchIndexStatus() method in the SPI class MetadataManager and ConnectorMetadata, similar to how materialized view detects staleness. Let me know if there are something else in your mind that need to be added to SPI.

I could describe below what we will put into the HiveConnector to maintain the index partition validity at Meta, by answering your above questions. Meta has an internal HiveConnector extension. I think you can implement the Iceberg behaviors in the counterpart classes of IcebergConnector. Again I think these should belong to the connectors but not SPI.

1. How is index staleness detected?
By looking at the metadata of index partition on HiveMetastore. As index partition is just a Hive table partition, if the index partition is stale, we'll drop the partition from the HiveMetastore. We can also put a dirty mark on the metadata of the index partition.

We'll have a pub-sub service that subscribes to the metadata of the candidate table (base table). If the corresponding base table partition is updated/overwritten/deleted, it will trigger the dropping of the index partition.

Then in the search path, we have a VectorSearchRewritePlanOptimizer that implements a ConnectorPlanOptimizer. In the VectorSearchRewritePlanOptimizer, we compare the available partitions of the index vs the base table in the query range. If an index partition is available, we'll use it to get the kNN of the partition. Otherwise, we will build the index on the fly. Another option for an unavailable index partition is to fall back to the exact search. And then we'll aggregate across all partitions in the query range and compute the kNN from all kNNs from each partion.

2. Are indexes expected to be snapshot-bound?
I might not have enough context for Iceberg. However, I think you can decide the reasonable behavior in IcebergConnector.

3. How should index cleanup or rebuild be handled after file rewrites or snapshot expiry?
As mentioned above, we will have a pub-sub service. Once it detects the update or deletion event of the base table from HiveMetastore, it will trigger at least two actions in sequence, 1) drop the corresponding index partition; 2) call UPDATE VECTOR INDEX vector_index WHERE <filter>, where corresponds to the dropped partition.

4. Is automatic invalidation or maintenance of index metadata in scope?
No, the automatic invalidation is out of the scope of Presto as well as this RFC. The index validity is stored as the index metadata. And when and how to update the index metadata is out of the scope of Presto. Presto only provides the SQL as interface to create and update the index.

[jja725]

Hi @gggrace14 , thanks for the proposal and great to see you again in the community! This definitely would help presto involve in the AI/ML workloads. I recently worked on lance connector and and would be happy to jump in and help with the vector search integration!

[jja725]

UPDATE VECTOR INDEX: This looks like a DML UPDATE but is actually a rebuild/refresh operation. REFRESH VECTOR INDEX or REBUILD VECTOR INDEX would be clearer and avoid confusion with
standard SQL UPDATE semantics.

[gggrace14]

Hi @jja725 , I'm open with REFRESH VECTOR INDEX or REBUILD VECTOR INDEX, if no one else objects to. I could change this RFC to use REFRESH VECTOR INDEX.

[jja725]

REFRESH VECTOR INDEX make sense.

[jja725]

should be CreateVectorIndex?

[gggrace14]

You're right. Correcting the type now.

[jja725]

This is confusing — does the WHERE clause here replace the original filter, extend it, or is it ignored? Why we need this filter if it's unpartitioned

[gggrace14]

I think the confusing part is that UPDATE VECTOR INDEX sounds like an update to the index schema. Here the semantics is actually to refresh the index partition ds = '2026-01-04'. I can change the UPDATE VECTOR INDEX to REFRESH VECTOR INDEX, same to the reply to your previous comment. By design, we don't support mutating the index schema, or replace vector index.

[jja725]

https://docs.cloud.google.com/bigquery/docs/vector-index#pre-filters_and_post-filters bigquery actually have pre-filter seems like

[gggrace14]

Thanks @gggrace14 for this writeup. Overall I'm in favor of this proposal.

I'm also curious since FAISS functions were recently exposed in C++.... Are you planning to do any follow up work with the library ?

Hi @aditi-pandit , thank you for reviewing this RFC in the TSC meeting and here. Yes, at meta so far we've accumulated implementation for the HiveConnector on top of the FAISS library, for both the search path and create vector index path. And we have plan to move some implementation and, more importantly, the algorithm to open source as common lib or toolkit. The challenge is we're building a sophisticated two-level hierarchical search algorithm in order to scale to >1 billion internal dataset. We need to figure out a clever way to extract components that are universally reusable for the presto community. We've also written a paper describing this approach. I think the paper description is the best way to share the algorithm first, once accepted :)