From 39a218468215b76ede435cb7ceecff134a777afd Mon Sep 17 00:00:00 2001 From: Gabriel Date: Wed, 30 Sep 2026 23:23:28 +0800 Subject: [PATCH] [docs] Explain Lance array predicate pushdown and scalar index verification --- .../lakehouse/catalogs/lance-catalog.mdx | 46 ++++++++++++++++++- .../lakehouse/catalogs/lance-catalog.mdx | 46 ++++++++++++++++++- 2 files changed, 88 insertions(+), 4 deletions(-) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx index 66939e915ecd0..9bb9a6c12425e 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx @@ -544,8 +544,9 @@ Doris 会将语义兼容的谓词转换为 Substrait 表达式，并交给 Lance | `utf8`、`large_utf8` | 支持 | | `date32(day)` | 支持 | | 无时区 `timestamp(s/ms/us)` | 支持 | +| `list` | 内置 `array_contains`，标签为非 `NULL` 字符串字面量；详见[数组标签谓词](#数组标签谓词) | -其他可读取类型，例如 `float16`、`decimal256`、Binary、`date64`、Time、纳秒 Timestamp、带时区 Timestamp 和复杂类型，当前保留为 Doris 侧谓词。 +其他可读取类型，例如 `float16`、`decimal256`、Binary、`date64`、Time、纳秒 Timestamp、带时区 Timestamp，以及上述数组成员判断以外的复杂类型，当前保留为 Doris 侧谓词。 ### 支持下推的操作符 @@ -558,13 +559,14 @@ Doris 会将语义兼容的谓词转换为 Substrait 表达式，并交给 Lance | 布尔列、`NOT` 布尔列 | 直接引用布尔列 | | `LIKE`、`NOT LIKE` | 直接引用 `utf8` 或 `large_utf8` 列，模式为字符串字面量，且不包含反斜杠转义、显式 `ESCAPE` 子句或 NUL 字符 | | `starts_with`、`ends_with` | 使用内置函数，第一个参数直接引用 `utf8` 或 `large_utf8` 列，第二个参数为字符串字面量 | +| `array_contains(labels, 'red')` | 直接引用 `list` 列，标签为非 `NULL` 字符串字面量 | | `AND` | 顶层 Conjunct 可以分别下推，不能下推的部分保留在 Doris | | `OR` | 两个分支都能完整转换时下推 | | `NOT` | 操作数能完整转换时下推 | 以下形式通常不会下推： -- 除上述内置字符串函数之外的函数，或列上的算术表达式。名称为 `like`、`starts_with` 或 `ends_with` 的用户自定义函数不会按同名内置函数下推。 +- 除上述内置字符串函数和数组成员判断函数之外的函数，或列上的算术表达式。名称为 `like`、`starts_with`、`ends_with` 或 `array_contains` 的用户自定义函数不会按同名内置函数下推。 - 模式包含 NUL 字符的字符串谓词。使用反斜杠转义或显式 `ESCAPE` 子句的 `LIKE`、`NOT LIKE` 也保留在 Doris。 - `IN` 列表为空或包含 `NULL`。 - `OR` 或 `NOT` 中只有部分表达式可转换。 @@ -581,6 +583,46 @@ WHERE active AND country LIKE 'C%'; ``` +### 数组标签谓词 + +Doris 4.1 的数组标签下推更新支持对直接引用的 Lance `list` 列执行 `array_contains`，标签必须为非 `NULL` 字符串字面量。较早的构建版本可能仍将这些条件保留在 Doris，请先通过 `EXPLAIN` 确认。`NULL` 数组仍返回 `NULL`，空数组不匹配，重复标签不会导致结果行重复。 + +对于包含 `id` 列和 `list` 类型 `labels` 列的 Lance 表： + +```sql +EXPLAIN +SELECT id +FROM lance_catalog.default.items +WHERE array_contains(labels, 'red') + AND array_contains(labels, 'blue'); +``` + +该条件匹配同时包含两个标签的数组，不要求标签顺序或相邻位置。将 `AND` 改为 `OR` 表示匹配任一标签。`NOT (array_contains(labels, 'red'))` 也可以转换，并保留 SQL 的 `NULL` 语义。这些示例适用于普通表扫描，不改变搜索 TVF 的 `filter` 参数与外层 `WHERE` 条件的区别。 + +以下条件仍保留在 Doris 求值： + +- `array_contains(labels, NULL)`：Doris 可以匹配数组中的 `NULL` 元素，Lance 成员判断函数对 `NULL` 标签的语义不同。 +- `array_contains_all(labels, ['red', 'blue'])`：Doris 判断有序、连续的子序列，不能替换为 Lance 的集合包含函数或两个独立的成员判断条件。 +- 动态标签、数组列上的表达式或类型转换，以及其他数组元素或容器类型，包括 `large_list`、定长列表和 `list`。 + +### 谓词下推与标量索引使用 + +谓词下推表示由 Lance 求值，不保证使用索引。普通扫描中，Doris 可以为每个扫描任务指定一个物理 BTree、Bitmap 或 LabelList 索引段。对于同列的 `IN`、`AND`、`OR`、`NOT` 组合，Lance 支持相应完整表达式时可以使用选中的索引。`OR` 的两个分支都必须提供安全的候选集，`NOT` 不能对仅求值一部分的子树取反。不支持的索引表达式会回退为分配的 Fragment 范围内的扫描。 + +每个任务选择**一个逻辑标量索引**。同时包含标签、类别和数值条件的查询，并不表示 Doris 会对三个索引求交集。下推的完整条件会在扫描 LIMIT 生效前再次检查，索引未覆盖的 Fragment 仍会读取。没有可用索引时，兼容的谓词也可以下推，由 Lance 扫描求值。 + +应结合以下信息验证： + +| 验证信息 | 含义 | +|---|---| +| `EXPLAIN` 中的 `lancePushdownPredicate` | 已发送给 Lance 的条件；剩余的 `predicates:` 条件由 Doris 求值。 | +| `EXPLAIN` 中的 `lanceScalarIndexScan=SEGMENT` 和 `lanceGroupingIndex` | 已规划标量索引段扫描，并显示选中的逻辑索引；不能单独证明运行时使用了索引。 | +| Profile 中的 `LanceScalarIndexSegmentsSearched` | 已埋点的标量索引段路径实际执行的搜索次数。 | +| Profile 中的 `LanceScalarIndexCandidateRows` | 该路径报告的候选行数，不一定等于最终结果行数。部分 `NOT` 表达式等补集掩码可能不报告候选集基数。 | +| Profile 中的 `LanceScalarIndexSegmentFallbacks` | 从规划的标量索引段路径回退的任务数。 | + +开启 `enable_profile` 后查看这些计数器。衡量优化效果时，应保持 SQL、投影列、数据集版本和缓存状态一致。这些计数器描述普通标量索引段扫描，不能单独用于判断 ANN 或全文搜索预过滤内部是否使用了标量索引。 + ### Runtime Filter 下推 普通 Lance 表参与 Join 时，Doris 可以将 Join 构建端生成的部分 Runtime Filter 转换为 Lance SQL 条件，并在 Lance 读取数据时提前过滤。这可以减少返回给 Doris 的数据量，尤其适合大表与过滤结果较小的表进行 Join 的场景。 diff --git a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx index d7eb0961850ee..c5769a8593e18 100644 --- a/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx +++ b/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx @@ -544,8 +544,9 @@ Doris converts semantically compatible predicates into Substrait expressions and | `utf8`, `large_utf8` | Supported | | `date32(day)` | Supported | | Timezone-naive `timestamp(s/ms/us)` | Supported | +| `list` | Built-in `array_contains` with a non-`NULL` string literal; see [Array Label Predicates](#array-label-predicates) | -Predicates on other readable types, including `float16`, `decimal256`, Binary, `date64`, Time, nanosecond Timestamp, timezone-aware Timestamp, and complex types, currently remain in Doris. +Predicates on other readable types, including `float16`, `decimal256`, Binary, `date64`, Time, nanosecond Timestamp, timezone-aware Timestamp, and complex types other than the array membership case above, currently remain in Doris. ### Operators Supported for Pushdown @@ -558,13 +559,14 @@ Predicates on other readable types, including `float16`, `decimal256`, Binary, ` | Boolean column, `NOT` Boolean column | Direct Boolean column reference | | `LIKE`, `NOT LIKE` | Direct `utf8` or `large_utf8` column and a string literal pattern without backslash escapes, an explicit `ESCAPE` clause, or an embedded NUL character | | `starts_with`, `ends_with` | Built-in function with a direct `utf8` or `large_utf8` column and a string literal | +| `array_contains(labels, 'red')` | Direct `list` column and a non-`NULL` string literal | | `AND` | Top-level conjuncts can be pushed down independently, with unsupported conjuncts retained in Doris | | `OR` | Both branches must be fully convertible | | `NOT` | The operand must be fully convertible | The following forms are generally not pushed down: -- Functions other than the supported built-in string functions, or arithmetic expressions applied to a column. A user-defined function named `like`, `starts_with`, or `ends_with` is not treated as the corresponding built-in function. +- Functions other than the supported built-in string and array membership functions, or arithmetic expressions applied to a column. A user-defined function named `like`, `starts_with`, `ends_with`, or `array_contains` is not treated as the corresponding built-in function. - String predicates whose pattern contains an embedded NUL character. `LIKE` and `NOT LIKE` patterns that use a backslash escape or an explicit `ESCAPE` clause also remain in Doris. - An empty `IN` list or an `IN` list containing `NULL`. - An `OR` or `NOT` expression in which only part of the expression can be converted. @@ -581,6 +583,46 @@ WHERE active AND country LIKE 'C%'; ``` +### Array Label Predicates + +The array-label pushdown update for Doris 4.1 supports `array_contains` on a direct Lance `list` column with a non-`NULL` string literal. Older builds may keep these conditions in Doris; check `EXPLAIN` before relying on this optimization. A `NULL` array still produces `NULL`, an empty array does not match, and duplicate labels do not duplicate result rows. + +For a Lance table with an `id` column and a `labels` column of type `list`: + +```sql +EXPLAIN +SELECT id +FROM lance_catalog.default.items +WHERE array_contains(labels, 'red') + AND array_contains(labels, 'blue'); +``` + +This matches arrays containing both labels, regardless of their order or adjacency. Replacing `AND` with `OR` matches either label. `NOT (array_contains(labels, 'red'))` is also convertible and preserves SQL `NULL` semantics. These examples apply to ordinary table scans; they do not change the distinction between a search TVF's `filter` parameter and an outer `WHERE` condition. + +The following remain Doris residual predicates: + +- `array_contains(labels, NULL)`: Doris can match a `NULL` element; Lance's membership function has different `NULL`-needle semantics. +- `array_contains_all(labels, ['red', 'blue'])`: Doris tests an ordered, contiguous subsequence. It cannot be replaced with Lance's set-containment function or two independent membership predicates. +- Dynamic needles, expressions or casts around the array column, and other array element/container types, including `large_list`, fixed-size lists, and `list`. + +### Predicate Pushdown and Scalar Index Use + +Predicate pushdown means Lance evaluates a condition; it does not guarantee index use. For ordinary scans, Doris can assign a physical BTree, Bitmap, or LabelList segment to each scan task. Same-column `IN`, `AND`, `OR`, and `NOT` combinations can use the selected index when Lance supports the complete expression. `OR` requires safe candidates from both branches; `NOT` cannot negate an incompletely evaluated subtree. Unsupported index expressions fall back to a scan within the assigned fragment domain. + +Each task selects **one logical scalar index**. A query combining label, category, and numeric predicates does not imply that Doris intersects all three indexes. The complete pushed condition is rechecked before applying the scan limit, and fragments outside index coverage are still read. Without a usable index, compatible predicates can still be pushed down and evaluated by a Lance scan. + +Use the following evidence together: + +| Evidence | Meaning | +|---|---| +| `lancePushdownPredicate` in `EXPLAIN` | Conditions sent to Lance. Remaining `predicates:` conditions are evaluated by Doris. | +| `lanceScalarIndexScan=SEGMENT` and `lanceGroupingIndex` in `EXPLAIN` | A scalar segment scan was planned and identifies the selected logical index. This alone does not prove runtime index use. | +| `LanceScalarIndexSegmentsSearched` in Profile | Actual searches on the instrumented scalar segment path. | +| `LanceScalarIndexCandidateRows` in Profile | Candidate rows reported by that path, not necessarily final result rows. Complement masks, such as some `NOT` expressions, may not report a candidate cardinality. | +| `LanceScalarIndexSegmentFallbacks` in Profile | Tasks that fell back from the planned scalar segment path. | + +Enable `enable_profile` to inspect these counters. Compare the same SQL, projection, dataset version, and cache state when measuring an optimization. These counters describe ordinary scalar segment scans and must not be used alone to infer scalar-index use inside ANN or full-text prefiltering. + ### Runtime Filter Pushdown When a regular Lance table participates in a join, Doris can convert some Runtime Filters produced by the join build side into Lance SQL conditions and apply them while Lance reads the data. This can reduce the amount of data returned to Doris, especially when a large table is joined with a highly selective build-side table.