PS-11264: Vector index support in Data Dictionary by percona-mhansson · Pull Request #6000 · percona/percona-server

percona-mhansson · 2026-06-09T14:22:54Z

Vector indexes have the type (algorithm) SE_SPECIFIC, and we add a column option in the data dictionary saying vector_index=1; which gets picked up by dedicated code in the data dictionary and the handler part of InnoDB.

In the SQL layer, the vector index is very much a thing; there is an HA_KEY_ALG_VECTOR, an HA_VECTOR and a KEYTYPE_VECTOR.

Extra SQL is added to display the type of a vector index as VECTOR rather than SE_SPECIFIC.

I you try to open a table containing a vector index in a trunk Percona server, you get a failed assertion in InnoDB when a client connects. However, if you drop the index, you can connect just fine.

Copilot

Pull request overview

Adds end-to-end server and InnoDB/DD plumbing to recognize “vector indexes” as a first-class SQL concept while storing them as SE_SPECIFIC in the DD, plus adjusts SHOW/INFORMATION_SCHEMA output to display VECTOR.

Changes:

Introduces new SQL-layer key/index types for vector indexes (KEYTYPE_VECTOR, HA_VECTOR, HA_KEY_ALG_VECTOR) and a handler capability flag (HA_CAN_VECTOR).
Propagates vector-index identification into InnoDB dictionary/index structures and skips B-tree specific logic where vector indexes don’t have trees (similar to FTS handling).
Updates SHOW/IS output paths to surface vector index type as VECTOR instead of SE_SPECIFIC.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
storage/temptable/src/table.cc	Treats vector index algorithm as unsupported in temptable index creation (abort path).
storage/temptable/src/handler.cc	Ensures vector index algorithm reports no index flags in temptable handler.
storage/innobase/include/dict0mem.ic	Initializes new `dict_index_t` vector marker during struct fill.
storage/innobase/include/dict0mem.h	Adds `is_vector_index` and helper accessor; adjusts index-type bit width constant.
storage/innobase/include/dict0dict.ic	Adds `dict_index_is_vector()` helper.
storage/innobase/handler/ha_innodb.cc	Sets vector capability; propagates vector flag into InnoDB index objects; treats vector like FTS/spatial in several code paths.
storage/innobase/dict/dict0dict.cc	Adds vector “internal index build” path and excludes vector indexes from some operations.
storage/innobase/dict/dict0dd.cc	Excludes vector indexes from length/prefix logic and sets vector marker when building dict indexes from TABLE metadata.
storage/innobase/dict/dict0crea.cc	Skips index-tree creation for vector indexes (like FTS).
storage/innobase/btr/btr0btr.cc	Skips B-tree validation for vector indexes (like FTS/online DDL).
sql/sql_table.cc	Adds KEYTYPE_VECTOR handling, vector index constraints, algorithm mapping, and related DDL validation.
sql/sql_show.cc	Displays “VECTOR KEY” and reports `INDEX_TYPE=VECTOR` in SHOW INDEX output.
sql/key_spec.h	Adds `KEYTYPE_VECTOR`.
sql/handler.h	Adds `HA_CAN_VECTOR` handler capability flag.
sql/field.cc	Allows VECTOR type to participate in key-part eligibility checks.
sql/dd/info_schema/show.cc	Attempts to map `SE_SPECIFIC` to `VECTOR` for IS/SHOW KEYS query output.
sql/dd/impl/types/column_impl.cc	Allows `vector_index` as a valid DD column option key.
sql/dd/dd_table.cc	Persists `vector_index` column option and maps vector algorithm/type into DD fields.
sql/dd_table_share.cc	Detects vector indexes when filling TABLE_SHARE key metadata from DD.
sql/create_field.cc	Allows VECTOR type through key-length calculation paths (removes prior assert).
share/messages_to_clients.txt	Adds client-visible error messages for vector-index constraints.
include/my_base.h	Adds `HA_KEY_ALG_VECTOR` and `HA_VECTOR` key flag.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  index->allow_duplicates = false;
  index->nulls_equal = false;
  index->disable_ahi = false;
+  index->is_vector_index = false;
  index->last_ins_cur = nullptr;
  index->last_sel_cur = nullptr;


+  [[nodiscard]] bool is_vector() const {
+    ut_ad(magic_n == DICT_INDEX_MAGIC_N);
+
+    return (is_vector_index);
+  }


+  if (sql_field->sql_type == MYSQL_TYPE_VECTOR &&
+      false /* !(key_info->flags & HA_VECTOR) */) {
    my_error(ER_NON_SCALAR_USED_AS_KEY, MYF(0), column->get_field_name());
    return true;
  }


+    // VECTOR indexes are only allowed on VECTOR columns.
+    if (sql_field->sql_type != MYSQL_TYPE_VECTOR) {
+      my_error(ER_UNKNOWN_ERROR, MYF(0));
+      return true;
+    }


+      if (!(file->ha_table_flags() & HA_CAN_VECTOR)) {
+        my_error(ER_UNKNOWN_ERROR, MYF(0));
+        return true;
+      }


+    if (primary_info.actual_key_parts > 1) {
+      my_error(ER_UNKNOWN_ERROR, MYF(0));
+      return true;
+    }


+  Item *sub_part_item =
+      new (thd->mem_root) Item_field(pos, NullS, NullS, alias_sub_part.str);
+  if (sub_part_item == nullptr) return nullptr;
+
+  Item *one_item = new (thd->mem_root)
+      Item_string(STRING_WITH_LEN("1"), system_charset_info);
+  if (one_item == nullptr) return nullptr;
+
+  Item *is_single_sub_part =
+      new (thd->mem_root) Item_func_eq(pos, sub_part_item, one_item);
+  if (is_single_sub_part == nullptr) return nullptr;


 constexpr uint32_t DICT_MULTI_VALUE = 512;

 /** number of bits used for SYS_INDEXES.TYPE */
-constexpr uint32_t DICT_IT_BITS = 10;
+constexpr uint32_t DICT_IT_BITS = 11;
 /** @} */


satya-bodapati

[AI-review] Review of the DD/metadata layer. Context: this commit is the base of the ps-11300-hnsw-populate work (30 commits downstream), so several items below were found the hard way there. 7 inline suggestions attached; three items that can't be anchored to diff lines:

1. ha_innobase::index_flags() must return 0 for vector keys (function not touched by this PR). The stub index has no B-tree, but index_flags() still advertises HA_READ_*/HA_KEYREAD_ONLY, so find_shortest_key picks it for COUNT(*) / index-only scans — which return 0 rows. Fix verified downstream (mirror of the FULLTEXT early-out at the top of the function):

  /* The vector index is a stub at the B-tree level: it can never serve
  ordered reads, ranges or index-only scans. Without this,
  find_shortest_key picks it for COUNT(*) and returns no rows. */
  if (table_share->key_info[key].algorithm == HA_KEY_ALG_VECTOR) {
    return (0);
  }

Side effect: SHOW INDEXES shows Collation NULL for vector keys — semantically right.

2. vector_index=1 is set on every VECTOR column, indexed or not. fill_dd_columns_from_create_fields() stamps the option on any MYSQL_TYPE_VECTOR column, so a table with an unindexed vector column also gets Percona-specific SDI, and dd_is_vector_index() then classifies any single-column SE_SPECIFIC index on such a column as a vector index. Suggest making the marker an option on the dd::Index (where the WITH(...) params will live anyway), or setting the column option only when the column is under a KEYTYPE_VECTOR key.

3. Nits. (a) The SHOW KEYS rewrite (type='SE_SPECIFIC' AND Sub_part='1' ⇒ VECTOR, string compare) is a heuristic — fine while nothing else emits SE_SPECIFIC, but deserves a comment saying so. (b) The handler.h ha_fast_update/upsert and field.cc include-comment hunks are unrelated reformatting — consider dropping to keep the diff reviewable.

satya-bodapati · 2026-07-22T12:04:10Z

+  if (sql_field->sql_type == MYSQL_TYPE_VECTOR &&
+      false /* !(key_info->flags & HA_VECTOR) */) {
    my_error(ER_NON_SCALAR_USED_AS_KEY, MYF(0), column->get_field_name());
    return true;
  }


[AI-review] With the guard stubbed to false, ALTER TABLE t ADD INDEX (v(10)) and ADD UNIQUE (v) succeed — a real B-tree/UNIQUE gets committed on vector data, and SHOW CREATE TABLE then prints a second VECTOR KEY with no TYPE clause (not parser-round-trippable; a follow-up ADD INDEX fails with a misleading ER_ONLY_SINGLE_VECTOR_INDEX_ALLOWED). key is in scope here:

Suggested change

if (sql_field->sql_type == MYSQL_TYPE_VECTOR &&

false /* !(key_info->flags & HA_VECTOR) */) {

my_error(ER_NON_SCALAR_USED_AS_KEY, MYF(0), column->get_field_name());

return true;

}

// A VECTOR column can only be used in a VECTOR key.

if (sql_field->sql_type == MYSQL_TYPE_VECTOR &&

key->type != KEYTYPE_VECTOR) {

my_error(ER_NON_SCALAR_USED_AS_KEY, MYF(0), column->get_field_name());

return true;

}

satya-bodapati · 2026-07-22T12:04:10Z

+ER_VECTOR_INDEX_NEEDS_PK
+  eng "Vector index can only be created in tables with a BIGINT UNSIGNED primary key."
+
+ER_ONLY_SINGLE_VECTOR_INDEX_ALLOWED
+  eng "A table can have at most one vector index."


[AI-review] Two more errors here would retire the three ER_UNKNOWN_ERROR placeholders (suggestions on each site follow):

Suggested change

ER_VECTOR_INDEX_NEEDS_PK

eng "Vector index can only be created in tables with a BIGINT UNSIGNED primary key."

ER_ONLY_SINGLE_VECTOR_INDEX_ALLOWED

eng "A table can have at most one vector index."

ER_VECTOR_INDEX_NEEDS_PK

eng "Vector index can only be created in tables with a BIGINT UNSIGNED primary key."

ER_ONLY_SINGLE_VECTOR_INDEX_ALLOWED

eng "A table can have at most one vector index."

ER_TABLE_CANT_HANDLE_VECTOR

eng "The used table type doesn't support VECTOR indexes."

ER_VECTOR_INDEX_REQUIRES_VECTOR_COLUMN

eng "A vector index requires a column of type VECTOR."

satya-bodapati · 2026-07-22T12:04:34Z

[AI-review] Suggested PR-description sections — @percona-mhansson feel free to paste/edit:

Summary of changes

Makes the vector index a first-class index kind in the SQL layer and a recognized stub in InnoDB — metadata only. No storage, population, or search; those come in follow-up PRs. Pattern throughout: treat a vector index like an FTS index (no B-tree, excluded from stats/validation/FK use).

High-level design

SQL layer: new HA_KEY_ALG_VECTOR, key flag HA_VECTOR (1<<30), KEYTYPE_VECTOR. Engines opt in via table flag HA_CAN_VECTOR (InnoDB only).
DD: no schema/SDI change. The index is persisted with existing enums (IA_SE_SPECIFIC + IT_MULTIPLE); a vector_index=1 column option is the marker. Identity is re-derived at open (dd_table_share.cc, dd_is_vector_index() in InnoDB).
InnoDB: dict_index_t::is_vector_index + dict_index_is_vector(); vector indexes get no index tree (dict_create_index_tree_in_mem), are skipped by btr validation, row-size checks, stats/cardinality (fixed 1.0), FK parent/supporting key lookup; index_read() refuses.
Downgrade: DD is readable by upstream, but upstream InnoDB asserts at open on an SE_SPECIFIC index — drop the vector index before downgrade.

DD changes

Item	Change
SDI / DD schema	none
`dd::Index`	reuses `algorithm=SE_SPECIFIC`, `type=MULTIPLE`
`dd::Column.options`	new whitelisted key `vector_index` (the marker)
Open path	marker + shape (single visible element) => `HA_VECTOR` key

User interface changes

CREATE TABLE ... VECTOR KEY (col) accepted (InnoDB only).
SHOW CREATE TABLE prints VECTOR KEY; no prefix-length suffix.
SHOW INDEX / I_S.STATISTICS: Index_type = VECTOR.
New errors: ER_VECTOR_INDEX_NEEDS_PK, ER_ONLY_SINGLE_VECTOR_INDEX_ALLOWED.
Disallowed: >1 key part; >1 vector index per table; missing or non-BIGINT UNSIGNED single-column PK; engines without HA_CAN_VECTOR; vector index as FK parent/supporting key; explicit USING algorithm.

- Job-level repository guard on both jobs so forks with Actions enabled stop inheriting the hourly cron, which fails without the canonical repo's secrets and OIDC trust and emails the fork owner every hour.

Vector indexes have the type (algorithm) SE_SPECIFIC, and we add a column option in the data dictionary saying `vector_index=1;` which gets picked up by dedicated code in the data dictionary and the handler part of InnoDB. In the SQL layer, the vector index is very much a thing; there is an `HA_KEY_ALG_VECTOR`, an `HA_VECTOR` and a `KEYTYPE_VECTOR`. Extra SQL is added to display the type of a vector index as VECTOR rather than SE_SPECIFIC.

percona-mhansson changed the base branch from trunk to vector-mvp June 9, 2026 14:25

percona-mhansson mentioned this pull request Jun 9, 2026

PS-11203 Vector Index Syntax #5987

Open

percona-mhansson force-pushed the vector-mvp-dd branch 2 times, most recently from af4cd72 to fee791a Compare June 10, 2026 14:24

percona-mhansson force-pushed the vector-mvp-dd branch from fee791a to 89e8d21 Compare June 23, 2026 13:04

percona-mhansson force-pushed the vector-mvp-dd branch 2 times, most recently from cd4a559 to 82b0023 Compare July 22, 2026 09:57

satya-bodapati requested a review from Copilot July 22, 2026 10:58

Copilot started reviewing on behalf of satya-bodapati July 22, 2026 10:59 View session

Copilot AI reviewed Jul 22, 2026

View reviewed changes

satya-bodapati reviewed Jul 22, 2026

View reviewed changes

Comment thread sql/handler.h

Comment thread sql/sql_table.cc

satya-bodapati reviewed Jul 22, 2026

View reviewed changes

percona-mhansson force-pushed the vector-mvp-dd branch from 82b0023 to 017dc49 Compare July 23, 2026 14:32

nogueiraanderson and others added 2 commits July 27, 2026 12:35

PS-11078 [trunk]: Skip orphan-sweep runs on forks (percona#6089)

898bf7d

- Job-level repository guard on both jobs so forks with Actions enabled stop inheriting the hourly cron, which fails without the canonical repo's secrets and OIDC trust and emails the fork owner every hour.

percona-mhansson force-pushed the vector-mvp-dd branch from 017dc49 to 97fb978 Compare July 28, 2026 10:07

Navigation Menu

Uh oh!

PS-11264: Vector index support in Data Dictionary - #6000

PS-11264: Vector index support in Data Dictionary#6000
percona-mhansson wants to merge 2 commits into
percona:vector-mvpfrom
percona-mhansson:vector-mvp-dd

percona-mhansson commented Jun 9, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

satya-bodapati left a comment

Uh oh!

satya-bodapati Jul 22, 2026

Uh oh!

percona-mhansson Jul 23, 2026

Uh oh!

satya-bodapati Jul 22, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

satya-bodapati commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

percona-mhansson commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

satya-bodapati left a comment

Choose a reason for hiding this comment

Uh oh!

satya-bodapati Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

percona-mhansson Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

satya-bodapati Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

satya-bodapati commented Jul 22, 2026

Summary of changes

High-level design

DD changes

User interface changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

percona-mhansson commented Jun 9, 2026 •

edited

Loading